Words

What are Words?

Exactly what counts as a word is kind of hard to pin down, and varies between languages. It’s typically understood to be a unit of meaning that can stand alone in an utterance. There’s a lot of variation among languages, especially in the orthographic representations—think of agglutinative languages like Finnish, Hungarian, Turkish, Japanese, Swahili, Korean, Quecha, and Tamil.

In language processing systems, we typically use tokens (units of recognition) that can anything from single characters (letters, digits, punctuation) to whole words of arbitrary length, using a modified Byte-Pair Encoding (BPE) technique.

Basic Terminology

Here are some linguistic terms useful in the study of spoken words, just to get started:

• Spoken Word: a unit of meaning that can stand alone in an utterance, made up of one or more phonemes
• Phoneme: a single speech sound
• Vowel: a phoneme produced without significant constriction of the vocal tract
• Consonant: a phoneme produced with some constriction of the vocal tract
• Syllable: a larger sound unit
• Phonology: the study of the functional patterns of speech sounds, specifically how phonemes create meaning
• Morpheme: a part of a word that contribute meaning
• Morphology: How words (tokens) are made up of smaller pieces
• Syntax: How words are structured to form phrases and sentences
• Semantics: What sentences mean
• Pragmatics: How language is used to communicate ideas and concepts

Lexical and Grammatical Words

Words are often divided into lexical and grammatical words:

To be honest, the boundary between these categories is not always clear-cut, for example, auxiliary verbs in English feel like they can be both.

Lexical words can range from the concrete (e.g., apple, fly, blue) to the abstract (e.g., freedom, oppression, justice).

Iconicity

Some words are iconic, meaning their form resembles their meaning (e.g., buzz, bang, hiss, plop), and others are arbitrary (e.g., dog, cat). Sometimes you’ll see hybrids.

THIS SECTION NEEDS A MAJOR EXPANSION FROM MITHEN CH 6

Lexicons

The lexicon of a language is its inventory of words (and their meanings).

The average adult English speaker knows ~50,000–100,000 words (their passive vocabulary), but frequently uses ~20,000–30,000. An unlimited number can be made in theory which can be figured out.

Children acquire words at a remarkable pace, about 10 new words per day between ages 2–8, many of which are inferred from a single exposure in context. LLMs, on the other hand, are trained on trillions of tokens but don’t seem to generalize as easily.

Morphemes

A word is made up of one or more morphemes, the smallest units of meaning.

They can be lexical (anchored to a concept) or grammatical (inflection, affix), and can be either free or bound.

English examples:

• Free lexical morpheme (root), e.g. dog, happy, walk
• Bound lexical morpheme, e.g. bio (as in biology), tele (as in telephone)
• Free grammatical morpheme, e.g. and, but, the, as, of
• Bound grammatical morpheme, e.g. -s (plural), -ed (past tense), -ing (progressive), -ment (noun-forming)

Word Meanings

How do words get their meanings? It is a huge question. There are some theories, like Rosch’s Prototype Theory. The idea is that word categories aren’t defined by certain conditions, but rather organized around prototypes (or best examples), such as a robin for the category “bird” (and not a penguin). Other members of the category are included based on their similarity to the prototype, leading to graded membership rather than a strict boundary.

The important question is whether a hot dog, or even a pop tart, is a sandwich.

Prototype Theory in LLMs

Prototype theory works really well in LLMs! You don’t have strict category boundaries. Similarity between vectors is good enough.

You might also check out Putnam’s Twin Earth Thought Experiment, which gives evidence for meaning not being entirely in one’s head (intensionally) but also (extensionally) within one’s environmental context.

Another theory is the Causal Theory of Reference, which suggests that words get their meaning anchored historically, through chains of use.

Lexical Relations

Words are definitely related to each other! Learn these terms about these relationships to impress your friends:

Synonymy

Words with very similar meanings, close to being interchangeable but not necessarily.

• big / large
• happy / joyful
• begin / start / initiate
• watch / observe
• smart / intelligent

Antonymy

Words that have opposite meanings. Gradable antonyms are at opposite ends of a spectrum.

• hot / cold
• big / small

Complementary antonyms are absolute binary opposites, such as:

• on / off
• alive / dead
• pass / fail

Converse antonyms are relational opposites, such as:

• buy / sell
• parent / child
• teacher / student

Hyponymy

Words with a subcategory relationship.

• dog is a hyponym of animal (the hypernym)
• rose / flower
• car / vehicle
• poodle / dog / canine / animal

Polysemy

When a word has many related meanings (can be metaphorical or metonymic or systematic). Think: one lexeme with multiple senses.

• foot (body part / mountain base)
• head (body part / leader / top of something)
• run (move fast on foot / operate a business)
• get (obtain / understand / become)
• paper (material / academic article / publication / newspaper company)

Homonymy

When words coincidentally look or sound the same, despite having completely different origins and distinct, unrelated meanings. Think: distinct lexemes sharing a surface form.

• bat (flying mammal / baseball equipment)
• bark (tree skin / dog sound)
• lie (recline / falsehood)
• right (correct / direction)
• bank (financial institution / river bank)
• stalk (plant stem / follow stealthily)

Word Vectors Encode These Relations

Embedding spaces capture many lexical relations geometrically. The famous example: king − man + woman ≈ queen. Synonymy, hyponymy, and analogy all emerge during training. They were not intentionally programmed in.

Evolution of Words

Language, we know, is incredibly fluid. So it makes sense that we can look at how words change over time. Chapter 13 of Mithen’s The Language Puzzle looks at four dimensions of word change:

• How word meanings change
• The invention of new words
• How pronunciation shifts
• How lexical words become grammaticalized

Like Mithen, we’ll look at these four dimensions from the point of view of recent changes in English. English is a good case study because it is widely spoken, diverged faster than the Romance languages from its common ancestor, has massive influences from other languages, and has a long written history. Written “English” goes back over a thousand years, and we can see how Old English, Middle English, and Modern English are considered completely different languages.

Semantic Change

Individual words change their meanings over time. Mithen identifies several types of semantic change.

• broadening (becoming more general) / narrowing (becoming more specific)
• amelioration (becoming more positive) / pejoration (becoming more negative)
• weakening (losing emphasis) / strengthening (gaining emphasis)
• Word conversion (change of word class without morphological change)
  • Verbification
    • to rain, to snow, to pocket, to captain, to email, to google
  • Verbs becoming nouns
    • to appear → appearance, to appoint → appointment, to inform → information
• Metonymy (shift in meaning based on association)

But how do these changes come about? Sometimes from misunderstanding, sometimes from metaphorical extension, and sometimes to shift a meaning as a social tactic (for identity and relationship building), and sometimes people found a way to make expression more efficient.

New Words

Here are a few ways new words are created in English:

Coinage

The straight up creation of a new word, e.g. googol, kleenex

Conversion

Changing the word class of an existing word, e.g. to google, to text

Borrowing

Adopting a word from another language, e.g. kindergarten (German), sushi (Japanese), café (French)

Compounding

Combining two existing words to create a new one, e.g. notebook, sunflower

Derivation

Adding prefixes or suffixes to create new words, e.g. happiness, unhappy

Backformation

Creating a new word by removing an affix, e.g. edit from editor

Reduplication

Forming new words by repeating a whole or part of a word, e.g. tick-tock, super-duper

Blending

Combining parts of two words to create a new one, e.g. brunch (breakfast + lunch), smog (smoke + fog)

Clipping

Shortening a longer word, e.g. ad (advertisement), lab (laboratory)

Acronym

Forming a word from the initial letters of a phrase, e.g. NASA, scuba

Sound Change

Sound changes (vowel shifts, consonant changes, stress shifts, etc.) also occur for reasons of identity marking and efficiency. We’ll cover this in our notes on phonology.

Grammaticalization

Grammaticalization refers to the process by which lexical words become grammatical words or even just morphemes. A popular example from English: will originally meant "want" or "wish," but over time it has become a grammatical marker for future tense.

Recall Practice

Here are some questions useful for your spaced repetition learning. Many of the answers are not found on this page. Some will have popped up in lecture. Others will require you to do your own research.

1. What is the difference between open-class and closed-class words? Give examples of each.
   Open-class words (nouns, verbs, adjectives) accept new members freely like selfie and to google. Closed-class words (prepositions, conjunctions, determiners) form a small, stable set and are the structural glue of syntax.
2. Distinguish polysemy from homonymy, and give an example of each.
   Polysemy: one word with multiple related meanings (bank: riverbank / financial institution). Homonymy: one word with unrelated meanings (bat: animal / baseball bat).

Summary