Words

What are Words?

Exactly what counts as a word is kind of hard to pin down, and varies between languages. It’s typically understood to be a unit of meaning that can stand alone in an utterance. There’s a lot of variation among languages, especially in the orthographic representations—think of agglutinative languages like Finnish, Hungarian, Turkish, Japanese, Swahili, Korean, Quecha, and Tamil.

Tokenization in LLMs

This is exactly why tokenization is hard and why modified Byte-Pair Encoding (BPE) is a compromise, not a solution. When an LLM tokenizes your prompt, it is not operating on “words” in any linguistically meaningful sense.

Basic Terminology

Here are some linguistic terms useful in the study of spoken words, just to get started:

• Spoken Word: a unit of meaning that can stand alone in an utterance, made up of one or more phonemes
• Phoneme: a single speech sound
• Vowel: a phoneme produced without significant constriction of the vocal tract
• Consonant: a phoneme produced with some constriction of the vocal tract
• Syllable: a larger sound unit
• Phonology: the study of the functional patterns of speech sounds, specifically how phonemes create meaning
• Morpheme: a part of a word that contribute meaning
• Morphology: How words (tokens) are made up of smaller pieces
• Syntax: How words are structured to form phrases and sentences
• Semantics: What sentences mean
• Pragmatics: How language is used to communicate ideas and concepts

Lexical and Grammatical Words

Words are often divided into lexical and grammatical words:

To be honest, the boundary between these categories is not always clear-cut, for example, auxiliary verbs in English feel like they can be both.

Lexical words can range from the concrete (e.g., apple, fly, blue) to the abstract (e.g., freedom, oppression, justice).

Some words are iconic, meaning their form resembles their meaning (e.g., buzz, bang, hiss, plop), and others are arbitrary (e.g., dog, cat). Perhaps the separation here is fuzzy, too

Lexicons

The lexicon of a language is its inventory of words (and their meanings).

The average adult English speaker knows ~50,000–100,000 words (their passive vocabulary), but frequently uses ~20,000–30,000. An unlimited number can be made in theory which can be figured out.

Children acquire words at a remarkable pace, about 10 new words per day between ages 2–8, many of which are inferred from a single exposure in context.

Contrast with LLMs: trained on trillions of tokens but don’t seem to generalize as easy.

Morphemes

A word is made up of one or more morphemes, the smallest units of meaning.

They can be lexical (anchored to a concept) or grammatical (inflection, affix), and can be either free or bound.

English examples:

• Free lexical morpheme (root), e.g. dog, happy, walk
• Bound lexical morpheme, e.g. bio (as in biology), tele (as in telephone)
• Free grammatical morpheme, e.g. and, but, the, as, of
• Bound grammatical morpheme, e.g. -s (plural), -ed (past tense), -ing (progressive), -ment (noun-forming)

Phonemes

Phonemes are the smallest units of sound in a language that can distinguish meaning. Some spoken languages have only a few dozen phonemes, while others have over a hundred. The IPA (International Phonetic Alphabet) provides a standardized way to represent these sounds.

Let’s visit the the interactive IPA chart and practice pronouncing some of the phonemes.

A lot of languages are written with the Latin alphabet, even when the sounds don’t match the letters perfectly. Watch the following video to see if the Latin alphabet is a good fit for Tlingit.

Word Meanings

How do words get their meanings? It is a huge question. There are some theories, like Rosch’s Prototype Theory. The idea is that word categories aren’t defined by certain conditions, but rather organized around prototypes (or best examples), such as a robin for the category "bird" (and not a penguin). Other members of the category are included based on their similarity to the prototype, leading to graded membership rather than a strict boundary.

The important question is whether a hot dog, or even a pop tart, is a sandwich.

Prototype Theory in LLMs

This theory is compelling an used in machine learning and LLMs: classification at category boundaries is hard, embeddings do capture graded similarity!

You might also check out Putnam’s Twin Earth Thought Experiment, which gives evidence for meaning not being entirely in one’s head (intensionally) but also (extensionally) within one’s environmental context.

Another theory is the Causal Theory of Reference, which suggests that words get their meaning anchored historically, through chains of use.

Lexical Relations

Words are definitely related to each other! Learn these terms about these relationships to impress your friends:

Synonymy

Words with very similar meanings, close to being interchangeable but not necessarily. Examples:

• big / large
• happy / joyful
• begin / start / initiate
• watch / observe
• smart / intelligent

Antonymy

Words that have opposite meanings. Gradable antonyms are at opposite ends of a spectrum, such as:

• hot / cold
• big / small

Complementary antonyms are absolute binary opposites, such as:

• on / off
• alive / dead
• pass / fail

Converse antonyms are relational opposites, such as:

• buy / sell
• parent / child
• teacher / student

Hyponymy

Words with a subcategory relationship. Examples:

• dog is a hyponym of animal (the hypernym)
• rose / flower
• car / vehicle
• poodle / dog / canine / animal

Polysemy

When a word has many related meanings (can be metaphorical or metonymic or systematic). Think: one lexeme with multiple senses. Examples:

• foot (body part / mountain base)
• head (body part / leader / top of something)
• run (move fast on foot / operate a business)
• get (obtain / understand / become)
• paper (material / academic article / publication / newspaper company)

Homonymy

When words coincidentally look or sound the same, despite having completely different origins and distinct, unrelated meanings. Think: distinct lexemes sharing a surface form. Examples:

• bat (flying mammal / baseball equipment)
• bark (tree skin / dog sound)
• lie (recline / falsehood)
• right (correct / direction)
• bank (financial institution / river bank)
• stalk (plant stem / follow stealthily)

Word Vectors Encode These Relations

Embedding spaces capture many lexical relations geometrically. The famous example: king − man + woman ≈ queen. Synonymy, hyponymy, and analogy all emerge from distributional statistics — no one programmed them in.

TODO

TODO

Recall Practice

Here are some questions useful for your spaced repetition learning. Many of the answers are not found on this page. Some will have popped up in lecture. Others will require you to do your own research.

1. What is the difference between open-class and closed-class words? Give examples of each.
   Open-class words (nouns, verbs, adjectives) accept new members freely — "selfie," "to google." Closed-class words (prepositions, conjunctions, determiners) form a small, stable set and are the structural glue of syntax.
2. What is fast mapping?
   Fast mapping is the ability of children to infer a word’s meaning from a single exposure in context. Children acquire roughly 10 new words per day between ages 2 and 8 this way.
3. What is prototype theory (Rosch), and what does it predict about category membership?
   Categories are organized around best examples (prototypes) rather than necessary and sufficient conditions. Membership is graded — a robin is a better bird than a penguin. Category boundaries are fuzzy.
4. Distinguish polysemy from homonymy, and give an example of each.
   Polysemy: one word with multiple related meanings (bank: riverbank / financial institution). Homonymy: one word with unrelated meanings (bat: animal / baseball bat).

Summary