Language models and Wittgenstein

September 1, 2022

Contents

1. Introduction
2. Questions of meaning
3. Wittgenstein’s evolution
   1. Background
   2. The picture theory of meaning
   3. Meaning as use
4. Word meanings in deep learning
   1. Word embeddings
   2. Transformers
   3. Generative language modeling
   4. Discussion
5. Conclusion
6. References

Introduction

• OpenAI. (2022). ChatGPT: Optimizing Language Models for Dialogue.
• OpenAI. (2023). GPT-4.

Questions of meaning

• Intension vs extension
• Syntax vs semantics
• Putnam, H. (1973). Meaning and reference. ¹
• Putnam, H. (1975). The meaning of “meaning”. ²

Wittgenstein’s evolution

Background

• The Vienna Circle
• Edmonds ³
• Tractatus Logico-Philosophicus (1922) ⁴
• Philosophical Investigations (1953) ⁵

The picture theory of meaning

• Tractatus Logico-Philosophicus (1922) ⁶
• The picture theory of meaning is a correspondence theory of truth.
• philosophy-in-figures/picture-theory-of-meaning-wittgenstein
• Daitz, E. (1953). The picture theory of meaning. ⁷
• Keyt, D. (1964). Wittgenstein’s picture theory of language. ⁸
• Gaskin, R. (2009). Realism and the picture theory of meaning. ⁹

Meaning as use

• Philosophical Investigations (1953)
• The theory of meaning as use is a coherence theory of truth.
• Carnap
  • Principle of tolerance
• Firth
  • Firth, J.R. (1957): “You shall know a word by the company it keeps.” ¹⁰

Wittgenstein in PI:

The meaning of a word is its use in the language. ¹¹

and

One cannot guess how a word functions. One has to look at its use, and learn from that. ¹²

Word meanings in deep learning

Word embeddings

• Mikolov: word2vec ¹³
  • Word (token) embeddings
• Olah, C. (2014). Deep learning, NLP, and representations.
• Alammar, J. (2019). The illustrated word2vec.
• Migdal, P. (2017). king - man + woman is queen; but why?
• Discussion of relations captured by word2vec by The Gradient
• Perone, C.S. (2018). NLP word representations and the Wittgenstein philosophy of language. ¹⁴

Transformers

• Transformer ¹⁵
• Positional encodings
• Self-attention

Generative language modeling

• Generative execution of next-token prediction with language models
• GPT
• Wolfram, S. (2023). What is ChatGPT doing—and why does it work? ¹⁶

Discussion

• Belloni, M. (2019). Neural networks and philosophy of language: Why Wittgenstein’s theories are the basis of all modern NLP.
• Goldhill, O. (2019). Google Translate is a manifestation of Wittgenstein’s theory of language.
• Skelac, I. & Jandric, A. (2020). Meaning as use: From Wittgenstein to Google’s Word2vec. ¹⁷
• Boccelli, D. (2022). Word embeddings align with Kandinsky’s theory of color.
• Tweet by Joscha Bach, Mar 25, 2023

Piantadosi:

Modern large language models integrate syntax and semantics in the underlying representations: encoding words as vectors in a high-dimensional space, without an effort to separate out e.g. part of speech categories from semantic representations, or even predict at any level of analysis other than the literal word. Part of making these models work well was in determining how to encode semantic properties into vectors, and in fact initializing word vectors via encodings of distribution semantics from e.g. Mikolov et al. 2013 (Radford et al. 2019). Thus, an assumption of the autonomy of syntax is not required to make models that predict syntactic material and may well hinder it. ¹⁸

Conclusion

Bender & Lascarides:

While machine learning has made impressive progress on many separate tasks (given appropriately curated data sets), the keys to generalizing beyond any specific set of end-to-end systems lie in modeling the linguistic system itself such that that model can be reused across tasks. And this can only be done based on an understanding of how language works, including how sentences come to mean what they mean (semantics) and how speakers can use sentence meaning to convey communicative intent (pragmatics). ¹⁹

TODO: Not acknowledging the evidence.

TODO: Fallacy of moving the goalposts.

• A surprising amount about the world can be learned from simply predicting what will come next in written language.
• What do words mean?
• LLMs can be seen as a vindication of the later Wittgenstein theory of meaning as use.
• On the other hand, the division between syntax and semantics is porous.
• That true structural relations can be found in language alone supports a view of accurate correspondence, something like the picture theory of meaning.
• Structural realism
• Both of Wittgenstein’s views remain relevant.

References