Structure and Meaning in Language

Signs

Language uses signs to communicate.

What are signs? Charles Sanders Peirce developed a comprehensive theory of signs, categorizing them into icons, indices, and symbols based on their relationship to the objects they represent.

Human language is overwhelmingly symbolic. No other species uses symbols at scale. We’ve been doing it for a long time:

• Ochre engravings at Blombos Cave, South Africa: ~75,000 BCE featured geometric patterns, possibly symbolic
• Shell beads used as ornaments: ~100,000 BCE were likely symbols of identity, group membership
• Cave paintings: ~30,000–17,000 BCE were iconic and possibly symbolic

Symbols are fascinating because they have no intrinsic connection to their referent. They require (a) shared convention, (b) memory, (c) the ability to link, but still keep separate, “word” and “meaning.” Symbols have unlimited generativity—you can make a symbol for anything, even abstractions: justice, infinity, recursion, love, LLM, the meaning of life, etc. With symbols, our ability to express and communicate is unbounded. It can also create lies, misinformation, and fictional worlds.

But where do symbols come from and how do they hook onto the world? Philosophers call this the symbol grounding problem. Symbols in a formal system only refer to other symbols, much as how a dictionary defines words with more words. There is no base word. Humans ground symbols through perception and action (and convention).

Language is more than just signs, though. What is language? One working definition is: a structured system of communication, consisting of utterances that can be spoken, signed, or written, formed via compositional rules (or guidelines) agreed upon by users of the language.

The study of human language is called linguistics.

We studied words and sounds earlier, so now it’s time for syntax, semantics, and pragmatics.

Grammar

Words and word-like entities are combined (arranged) according to rules to form larger units like phrases and sentences. The rules constitute a grammar. Where do the rules come from? How do we know if an utterance is well-formed?

Rules or Guidelines?

In programming languages, rules are explicit and prescriptive. In spoken natural language, we tend to treat them as guidelines, but in writing, they are often more rigidly followed.

A grammar will tell you where all the components of an utterance should go—based on certain attributes and context, but it won’t tell you what the utterance means. Here are two classic examples of grammatically correct English. The first is hard to parse and the second is just semantically absurd:

• The rat the cat the dog the cow tossed worried swatted ate the malt
• Colorless green ideas sleep furiously

Kinds of Grammars

There are many different approaches to capturing our idea of a grammar. Here are five of many.

Constituency Grammar

Constituency grammar emphasizes the grouping of words into constituents or phrases, which function as single units within a hierarchical structure. The structure is typically represented as a tree where nodes correspond to constituents, highlighting the nested and recursive nature of language. Constituency grammars are also called phrase-structure grammars, and these are the generative grammars we saw in our notes on language theory.

Here is a rough constituency grammar for a very tiny subset of English sentences. It does not capture all the complexities of the full language. It omits many legal forms, ignores exceptions, and completely ignores morphology:

Given a constituency grammar, you can show how phrases and sentences are structured hierarchically. Here is a classic example:

How do you tell whether some sequence of words actually forms a constituent, as opposed to being an arbitrary run of words? We run constituency tests:

• Substitution: can it be replaced by a single pro-form? I saw the cat → I saw it (so the cat is a constituent); but I saw the cannot be replaced by anything.
• Movement: can it be moved as a block? The cat, I saw ___ (topicalization) works for the cat, but not for, say, saw the.
• Coordination: can it be conjoined with a similar phrase? I saw [the cat] and [the dog] — constituents coordinate with constituents.
• Standalone/answer test: can it stand alone as a natural answer to a question? “What did you see?” or “What did it do?” or “How was it done?”

Constituency ambiguity shows up whenever a sentence has more than one legal parse tree. The textbook example is prepositional-phrase attachment: “I saw the scientist with the telescope.”

Transformational Grammar

Transformational grammar, developed by Noam Chomsky, starts from the phrase-structure tree shown above, then adds a further claim: a sentence has both a deep structure capturing its core grammatical relations, and an observable surface structure derived from it by transformations—rules that rearrange, insert, or delete material. “The dog chased the cat” and “The cat was chased by the dog” get different surface structures but the same deep structure.

Continue your study at Wikipedia.

Dependency Grammar

Dependency grammar focuses on the relationships between words in a sentence, where one word (the head) governs, or points to, the others (dependents). The structure is represented as a tree with directed, labeled, edges from heads to dependents, emphasizing the syntactic functions of the words. There are no phrasal nodes at all—every node in the tree is a word, and the tree has exactly as many nodes as the sentence has words.

The idea goes back to the French linguist Lucien Tesnière, whose posthumous Éléments de syntaxe structurale (1959) argued that the verb, not the sentence-splitting subject-predicate distinction, is the true organizing center of a sentence: every other word depends, directly or indirectly, on it.

Here’s the quick brown fox sentence from earlier, as parsed from a dependency grammar:

Here arcs are labeled with the grammatical relations using the tag set from the Universal Dependencies project, a widely-used cross-linguistic annotation standard. There are 37 tags in this framework.

Because a dependency tree tracks grammatical relations rather than linear groupings, it tends to survive word-order changes better than a constituency tree does. This matters a lot for languages with much freer word order than English—Latin, Russian, Czech, Japanese, Warlpiri—where a constituency tree for “subject–verb–object” order would have to be redrawn for every permutation, but the dependency relations (who did what to whom) stay exactly the same. This is part of why Tesnière’s framework caught on outside of English-centered linguistics.

A dependency tree is called projective if you can draw all of its arcs above the sentence without any of them crossing. The tree for the fox sentence is projective:

Languages with free word-order tend to have many nonprojective trees, but so does English. Here’s one example:

Parsing algorithms differ in whether they can handle these directly:

• Transition-based parsers read the sentence left to right, maintaining a stack and a buffer, and greedily choose actions like shift, left-arc, and right-arc at each step—fast (linear time) but naturally projective (or need extra machinery to handle crossing arcs).
• Graph-based parsers score every possible head-dependent arc and then search for the highest-scoring tree over the whole sentence (e.g., with a maximum spanning tree algorithm)—slower, but can find non-projective trees directly.

Dependency grammars are used in spaCy, Stanza (from Stanford CoreNLP), and the Universal Dependencies treebanks.

If you like Dependency Grammars, check out this Stanford presentation.

Operator Grammar

Operator grammar, developed by Zellig Harris, is a theory that focuses on the combinatory properties of operators (typically verbs) and their arguments. It emphasizes the rules governing how operators can combine with other elements to form grammatical sentences, often highlighting the constraints and regularities in argument structure. Rather than positing an innate, autonomous syntax independent of meaning and usage, operator grammar tries to show that the entire structure of a language can be derived just from which words require which other words, how likely particular combinations are, and how frequent combinations get compacted in use.

Operators and Arguments

An operator is a word that requires one or more other words, called its arguments, to be present for the sentence to be complete—in effect, exactly the subcategorization or valency of the word, but treated as the single foundational relation of the whole grammar. Words that require nothing further (roughly, ordinary nouns) form the base argument class, N. Every operator is then classified by the sequence of argument classes it demands:

What makes this genuinely different from ordinary subcategorization frames is that an operator’s O-arguments aren’t noun phrases at all—they’re entire sentences, themselves headed by their own operator. So say in John said it rained takes John (an N) and the whole clause it rained (an O, itself built from the operator rained and its own N argument it) as one nested unit:

This recursive nesting of operators inside the argument slots of other operators is what lets operator grammar build sentences of arbitrary depth without ever leaving the operator-argument relation—no separate phrase-structure rules like $S \rightarrow NP\;VP$ are needed at all. (It’s also, in effect, the same insight modern computational semantics reaches for with predicate-argument structure, thematic/semantic roles, and resources like PropBank—Harris got there first, from a very different direction.)

Likelihood and Reduction

Dependency alone only says an operator’s argument slots must be filled with something of the right class; it doesn’t say which filler is natural. That’s the job of likelihood: given an operator, some fillers for its argument are simply more probable, in ordinary usage, than others. John wears hats is more likely than John wears snow, which is in turn more likely (if still odd) than John wears vacation. The set of arguments with above-average likelihood for a given operator is called that operator’s coherent selection, and Harris observed that words similar in meaning tend to have similar coherent selections—which is itself a distributional definition of what a word means, decades before “a word is known by the company it keeps” became a slogan for word embeddings and distributional semantics.

Once a combination is highly likely, speakers tend to compress it—a process Harris called reduction. Crucially, reductions are treated as near-paraphrases: they remove no information, they just make an already-predictable sentence more compact. A few kinds, with the Wikipedia article’s own examples:

• Omission: dropping a highly predictable argument. I expect John to come → I expect John (because to come is the overwhelmingly likely completion once you’ve said expect John).
• Pronominalization: replacing a repeated argument with a pronoun. John wears boots and John wears hats → John wears boots and he wears hats.
• Affixation: folding a whole operator-argument relation into a single derived word. John is able to be liked → John is likeable.
• Transposition: merging two related sentences by turning one into a modifier inside the other. John wears boots; the boots are of leather → John wears leather boots.

Notice the shape of the whole system: dependency builds the raw predicate-argument skeleton, likelihood is what picks out well-formed, meaningful combinations from merely well-typed ones, and reduction compacts the predictable parts for efficient communication. Nothing else is required.

Why Call It a “Theory of Human Language”?

Most of what we’ve called “grammars” so far—constituency, transformational, dependency—are really representations: notations for describing the structure of a sentence once an independent rule system has already told you what counts as grammatical. Operator grammar is more ambitious: Harris intended it as a self-contained, self-organizing account of language as such, where grammaticality itself falls out of the three mechanisms above rather than being stipulated by an independent rule system. In his own framing, no external metalanguage—no separate inventory of phrase-structure rules, no abstract deep structure distinct from what’s actually said—is needed to state what a language is; the operator-argument relation, likelihood, and reduction are enough, and they’re all in principle measurable directly from how words actually co-occur in real text. That’s a substantive, falsifiable claim about the nature of language, not just a convenient diagramming notation—hence “theory of language,” not merely “grammar formalism.”

Harris pushed this furthest in his work on sublanguages—the specialized, highly regular registers used within a scientific field. In The Form of Information in Science (1989), he analyzed a corpus of immunology research articles and showed that, within that domain, the operator classes and the vocabulary that fills their argument slots become dramatically more constrained than in general English: authors reuse a small set of recurring operator-argument patterns (e.g., a fixed handful of verbs relating antibodies, antigens, and specific measured reactions) almost formulaically. If you’ve heard of modern information-extraction techniques that pull structured facts (who did what to whom) out of scientific abstracts, you’re seeing essentially the same idea: restrict a text to a narrow sublanguage and its operator-argument structure becomes tractable enough to extract automatically.

Operator grammar itself never became mainstream compared to its rivals here, but it left descendants: the Wikipedia article on the topic points to link grammar (Sleator and Temperley, 1991) as “a kind of operator grammar,” and the broader idea that syntax is fundamentally a network of predicate-argument requirements—rather than a separate phrase-structure layer—resurfaces constantly in computational linguistics, from valency dictionaries to semantic role labeling to the dependency parsers you saw above.

Construction grammar, up next, reaches a conclusion with a similar flavor—no independent rule system standing apart from meaning—but arrives there from the opposite direction: instead of grounding everything in distributional likelihood over a corpus, it grounds everything in the cognitive claim that form and meaning are inseparable at every grain size, from morphemes to entire argument-structure patterns.

Construction Grammar

Construction grammar arrives at a similarly radical conclusion as operator grammar—no principled split between the rules of grammar and the meanings they express—but gets there from the opposite direction: not distributional likelihood over a corpus, but the cognitive claim that knowledge of language consists of a collection of form-meaning pairings, known as constructions. These constructions range from simple words to complex syntactic patterns, and meaning is derived from the interaction of these constructions rather than from abstract rules alone.

The radical part of the claim is this: a construction is a sign—a paired form and meaning, exactly like the Peircean symbols discussed above—and constructions exist at every grain size, with no principled boundary between “the lexicon” (memorized, irregular, word-sized) and “the grammar” (productive, regular, rule-sized). Traditional generative grammar treats these as fundamentally different systems; construction grammar treats them as the same kind of object at different points on a continuum of size and schematicity:

The evidence Adele Goldberg and others point to is that argument-structure constructions seem to carry meaning of their own, independent of the verb slotted into them. The verb sneeze is intransitive—sneezing doesn’t normally take an object—yet:

“She sneezed the napkin off the table.”

is immediately understood to mean that her sneeze caused the napkin to move off the table. Nothing about the lexical meaning of sneeze supplies that caused-motion reading; it comes from the caused-motion construction itself (schematically, Subj V Obj Obl_path/loc ↔ “X causes Y to move Z”), which imposes its meaning on whatever verb is placed inside it. The same logic explains subtle contrasts like He baked her a cake (implies she ends up with the cake—the ditransitive construction’s “transfer” meaning) versus He baked a cake for her (merely “on her behalf”; no transfer is entailed).

This makes construction grammar naturally compatible with the usage-based acquisition story discussed below: instead of setting innate parameters, a learner could build up a construction inventory (sometimes called a constructicon) directly from experience, via general cognitive mechanisms like categorization and analogy, generalizing from concrete utterances to progressively more schematic patterns. It also gives a suggestive frame for thinking about what a language model is doing: an LLM trained purely on usage, with no rule component at all, ends up representing something a lot like an enormous, distributed constructicon—form-meaning (or at least form-distribution) associations at every grain size, learned rather than programmed in.

Construct another sentence, like the sneeze example, where an intransitive verb gets pressed into a transitive or caused-motion reading purely because of the construction it’s placed in. What does the verb normally mean on its own, and what does the construction add?

Marking Grammatical Relations: Word Order, Case, and Agreement

Every formalism above needs some way to know that, in The dog chased the cat, the dog is the chaser and the cat is the chased—that is, which noun plays which grammatical relation (subject, object, and so on) with respect to the verb. This turns out to be a question that cuts across all five grammars we’ve covered rather than belonging to any one of them: whatever tree structure a language uses, it still has to signal role somehow, and languages differ enormously in how they do it. There are three main strategies, often combined:

• Word order. Position alone carries the information. English leans on this heavily: The dog chased the cat and The cat chased the dog are built from identical words, but swapping the order swaps who did what to whom.
• Case. The noun (or its determiner) changes form depending on its role, so word order becomes negotiable—this is exactly why the free-word-order languages mentioned earlier in the dependency grammar section (Latin, Russian, Czech, Japanese, Warlpiri) can scramble their word order without losing track of who did what.
• Agreement (indexing). The verb itself changes form to reference the person, number, or gender of one or more of its arguments. Spanish hablo (“I speak”) vs. hablas (“you speak”) marks the subject right on the verb, which is part of why Spanish can drop subject pronouns that English can’t.

Case systems recur throughout these notes without ever having been named, so here are the ones you’ll see most:

• Nominative: marks the subject of a clause. German der Hund (“the dog,” as subject).
• Accusative: marks the direct object. German den Hund (“the dog,” as direct object)—note the article itself changes, not just the noun.
• Dative: marks the indirect object, typically a recipient. German dem Hund (“to/for the dog”), as in giving something to the dog.
• Genitive: marks a possessor. German des Hundes (“the dog’s”).

Nominative and accusative also name an alignment—a pattern for how a language’s case system carves up grammatical roles—and it’s worth seeing the alternative, since it’s the source of two more terms this page has used without defining: ergative and absolutive. Compare an intransitive clause (one argument) to a transitive one (two arguments):

• The man fell. (S: the single argument of an intransitive verb)
• The man broke the vase. (A: the agent of a transitive verb; O: its object)

English, like most Indo-European languages, is nominative–accusative: S and A get treated alike (both would be nominative, if English still case-marked full nouns the way its pronouns do: he fell, he broke the vase), while O gets marked differently (accusative: broke him, not he). An ergative–absolutive language does the opposite grouping: S and O are treated alike (both absolutive), while A—the agent of a transitive verb specifically—gets its own distinct ergative marking. It’s not that ergative languages lack subjects; it’s that their case system draws the line between “roles that get marked alike” in a different place than English draws it. Basque, Georgian, and (as mentioned earlier in the constructed-languages section) Na’vi all work this way. Some languages even switch between the two systems depending on context—Hindi-Urdu marks its transitive subjects with the ergative postposition ne only in perfective (roughly, past-tense) clauses, and patterns nominative–accusative otherwise, a pattern called split ergativity.

Parsing

Technically, the grammar composes utterances into hierarchical or networked structures, even though they tend to be written or spoken as temporal one-dimensional sequences.

Recursion

Recursion is the property of language that allows rules to be applied repeatedly, embedding structures within structures of the same kind. This is what enables sentences to be infinitely long and complex, even with a finite set of rules and vocabulary.

Here’s an example. For the constituency grammar near the beginning of this page, we can make the derivation:

S ⇒ NP VP
  ⇒ DET NOUN RP VP VP
  ⇒ the NOUN RP VP VP
  ⇒ the dog RP VP VP
  ⇒ the dog that VP VP
  ⇒ the dog that SV S VP
  ⇒ the dog that thought S VP
  ⇒ the dog that thought NP VP VP
  ⇒ the dog that thought PN VP VP
  ⇒ the dog that thought grace VP VP
  ⇒ the dog that thought grace TV NP VP
  ⇒ the dog that thought grace hit NP VP
  ⇒ the dog that thought grace hit PN VP
  ⇒ the dog that thought grace hit alan VP
  ⇒ the dog that thought grace hit alan DV NP PP
  ⇒ the dog that thought grace hit alan threw the new blue toy to the fast rat
    .
    .
    .
  ⇒ the dog that thought grace hit alan threw the new blue toy to the fast rat

Recursion can allow sentences to become arbitrarily long and complex:

S ⇒ NP VP
  ⇒ NP SV S
  ⇒ NP SV NP VP
  ⇒ NP SV NP SV S
  ⇒ NP SV NP SV NP VP
  ⇒ NP SV NP SV NP SV S
    .
    .
    .
  ⇒ she dreamed she dreamed she dreamed . . . she dreamed the dog swam

Language Acquisition

Grammar has to come from somewhere. A child ends up with a rich, productive system—not a memorized list of sentences—after hearing only a finite, messy, error-filled sample of a language. Where does the rest come from? This is the question that split twentieth-century linguistics down the middle.

Chomsky’s answer was that it couldn’t come from the input alone. His Poverty of the Stimulus argument holds that children productively use grammatical rules and constraints they’ve never seen exemplified, and are essentially never corrected on the subtle cases that would let them learn those constraints by trial and error. If the input under-determines the grammar, something has to fill the gap—so he proposed an innate Language Acquisition Device (LAD), a language-specific piece of mental machinery shared by the whole species, embodying a Universal Grammar (UG): a fixed set of universal principles, with a small number of open parameters that get set one way or another by whatever language a child happens to be exposed to (this specific version of the theory is usually called Principles and Parameters). Pinker’s The Language Instinct (1994) is the most famous popularization of a related idea: that grammar is a Darwinian biological instinct, shaped by natural selection specifically for communication, the way echolocation is shaped in bats. Chomsky and Pinker disagree sharply on the mechanism (an abstract, largely unselected cognitive module vs. an adaptation actively shaped by selection), but both are committed to some domain-specific innate machinery dedicated to grammar.

That premise has been under sustained attack for a couple of decades now, on several independent fronts. Usage-based linguists—Tomasello, Christiansen, and Joan Bybee prominently among them—argue that general-purpose learning mechanisms (statistical pattern extraction, analogy, categorization) plus rich social interaction are sufficient, with no dedicated grammar module required; a child doesn’t need innate syntax if they’re instead innately very good at noticing patterns and very motivated to communicate. This SciAm article by Tomasello is an excellent summary of why Universal Grammar is likely wrong in its major assumptions, and how the Usage-Based Theory of Language is an alternative. Bybee’s own contribution leans heavily on frequency: the same “use shapes structure” logic behind the Law of Brevity and grammaticalization covered in the notes on words—high-frequency combinations get entrenched, reduced, and grammaticalized through sheer repetition of use—is, for her, not a side effect of grammar but the actual mechanism that builds it, no innate template required. This is, not coincidentally, exactly the acquisition story that fits construction grammar from earlier on this page: if grammar just is a big inventory of form-meaning pairings at every grain size, there’s no separate “rules module” that would need to be innate in the first place—just the general cognitive capacity to extract and generalize patterns from constructions you’ve actually heard. Cross-linguistic typology has piled on more evidence: as languages outside the small set Chomsky originally generalized from got carefully documented, the “universals” UG predicted turned out to have far more exceptions than expected, casting doubt on there being a fixed, universal syntactic template underneath every human language at all.

Some of the most striking evidence for the usage-based side comes from cases where a full grammar appears with no adult model to imitate at all. Pidgins are rudimentary contact languages, with no native speakers and minimal grammar, that spring up when adult speakers of mutually unintelligible languages need to communicate. Creoles are what happens when children are then raised hearing a pidgin as their primary input: within a single generation, the children spontaneously develop a full, grammatically regular language out of the pidgin’s inconsistent scraps—productive tense/aspect marking, consistent word order, subordinate clauses, none of which were reliably present in what they heard. The most dramatic documented case is Nicaraguan Sign Language: deaf children in 1980s Nicaragua, thrown together for the first time in newly-formed schools with no shared sign language and no fluent adult signers to imitate, collectively invented one—complete with regular grammar—within about a generation. If grammar has to be triggered by exposure to already-grammatical input, cases like this are hard to explain; if children can construct grammar out of unstructured or minimal input given the right social conditions, they’re exactly what you’d expect.

Additional articles on usage-based learning and alternatives to Universal Grammar:

• TODO
• An article by Daniel Everett
• TODO
• TODO

Gemini on Chomsky

Gemini decided to pull no punches in critiquing not only Universal Grammar, but also CHomsky’s reaction to alternative theories:

Noam Chomsky's reaction to both the data from cognitive linguists like Michael Tomasello and the explosive rise of statistical AI is a masterclass in intellectual entrenchment. [...]

He has fiercely stuck to his Universal Grammar (UG) and Language Acquisition Device (LAD) theories. In fact, his response to these challenges has been to double down, dismiss the counter-evidence as irrelevant, and redefine what “linguistics” even means. [...]

Michael Tomasello and other developmental psychologists published biting empirical evidence showing that children do not learn language via an innate template. Instead, they use general, non-language-specific cognitive tools: pattern recognition, categorization, and ... understanding what another person intends to communicate. They proved that grammar emerges from usage. Chomsky’s reaction was essentially to say: “You are studying the wrong thing.” [...]

To protect his theory from being disproven by languages that lacked complex structures (like Daniel Everett’s discovery of the Pirahã language, which may lack recursion), Chomsky radically shrank what Universal Grammar actually was. By the 2000s, he stripped UG down to almost nothing except a single mental operation called Merge .... If a child has Merge, Chomsky claims they have UG—effectively making the theory so minimal that it became nearly impossible to empirically disprove. [...]

Chomsky did not concede. In a famous 2023 New York Times essay and subsequent interviews, he labeled ChatGPT and similar models “a bad joke” and a form of “plagiarism.” [...]

Chomsky has essentially moved the goalposts to a purely philosophical realm .... By his own definition, no amount of statistical success or behavioral data can ever prove him wrong, because he believes they are measuring a different phenomenon entirely.

The Emergence of Syntax

Acquisition asks how one child, in one lifetime, ends up with a grammar. This section asks a different question: how does a whole population end up sharing one, generation after generation, with nobody ever sitting down and designing it? The leading answer treats language itself as something that evolves culturally—shaped, over repeated transmission from speaker to learner, by nothing more than what turns out to be learnable.

The Iterated Learning Model

The Iterated Learning Model (ILM), developed by Kirby, Smith, and colleagues, formalizes this as a chain: a speaker produces utterances for a range of meanings; a learner sees only a small sample of those utterance-meaning pairs (never the whole language—this is the crucial bottleneck); the learner infers a grammar from that sample and becomes the next speaker; repeat, indefinitely.

Why should this loop, repeated with no communicative goal and no innate grammar built in, produce anything structured at all? Because of what survives the bottleneck. A holistic language, where every meaning gets an unrelated, unanalyzable label, can’t be reconstructed from a small sample—if a learner never happened to see the label for a particular meaning, they have no way to guess it, since nothing about the other labels tells you anything about it. A compositional language, where meanings are built from meaningful parts combined by general rules, survives the bottleneck beautifully: a learner who has seen only some of the utterances can still generalize the rule and produce correct utterances for meanings they’ve never encountered. Across repeated transmission, languages that happen to be easier to learn from partial data get reproduced more faithfully than languages that aren’t—so compositional structure accumulates over generations purely because it’s more transmissible, with no communicative pressure and no innate syntax required to explain why. Kirby, Cornish, and Smith ran exactly this as a human iterated-learning experiment: an artificial language, initially unstructured, passed through a chain of participants (each trained only on a subset of the previous participant’s output) became progressively more compositional and more learnable over successive “generations,” with no participant ever intending to create structure.

Related and Competing Accounts

ILM isn’t the only bottleneck-based story making the rounds. A few other threads worth knowing about:

• The Now-or-Never bottleneck. Christiansen and Chater’s 2016 account locates a different bottleneck: not across generations, but in real time, inside a single act of processing. Because working memory for raw, unanalyzed linguistic material decays almost immediately, the brain has to “use it or lose it,” chunking incoming speech into increasingly abstract units (sounds → words → phrases) before the raw signal is gone. They argue this same pressure, operating constantly during both comprehension and acquisition, is what actually shapes grammatical structure—a complementary bottleneck to ILM’s, located in processing rather than transmission.
• Grammaticalization. Covered in more detail in the notes on words: ordinary content words wear down into grammatical morphemes through frequent use (going to → gonna, a future-tense marker). This is a single-lineage, historical-time route to new grammatical material—not quite the same mechanism as ILM’s cross-generational transmission bottleneck, but a natural complement to it: grammaticalization keeps supplying new raw material, and iterated learning keeps pressuring whatever exists toward more learnable, more compositional structure.
• Emergent communication in multi-agent deep learning. A genuinely new, and very CS-flavored, source of evidence: when neural network agents are trained with reinforcement learning to communicate in order to solve a shared task (typically a referential game, where a “speaker” network must describe an object well enough for a “listener” network to pick it out), they spontaneously develop their own communication protocols—with no grammar, and often no obvious structure at all, programmed in. More recent work has gone further and explicitly wired an iterated-learning-style transmission bottleneck into the training loop for these agents, and found that doing so reliably pushes the emergent protocol toward more compositional structure than training without a bottleneck does—essentially replicating the ILM result, but with neural networks standing in for the generations of learners (see Ren et al., Compositional Languages Emerge in a Neural Iterated Learning Model, 2020).

Discuss the fact that LLMs acquire language from stimulus alone, with no innate LAD, and they do remarkably well. LLMs learn grammar implicitly from the data, without explicit instruction. They can generate grammatically correct sentences, but they don’t have an explicit representation of grammatical rules like humans do. How is this possible? Are transformers just big enough to brute-force what evolution gave us for free? Or do we have something else? How does this relate to the neural iterated-learning results above—do LLMs experience anything like a transmission bottleneck during training?

Here’s a broader set of readings on cultural evolution and the emergence of linguistic structure, for anyone who wants to go deeper:

• Compression and communication in the cultural evolution of linguistic structure
• Syntax out of Learning: The Cultural Evolution of Structured Communication in a Population of Induction Algorithms
• Syntax without Natural Selection: How compositionality emerges from vocabulary in a population of learners
• Innateness and culture in the evolution of language
• Culture and biology in the origins of linguistic structure
• Language as Shaped by the Brain
• On how “early syntax” came about
• Cultural evolution creates the statistical structure of language
• Linguistic structure from a bottleneck on sequential information processing
• The Evolution and Emergence of Language: How Did Humans Acquire Language?
• Linguistic Structure from a Bottleneck on Sequential Information Processing
• Compositional Syntax From Cultural Transmission

So, Whither Universal Grammar?

Put Acquisition and Transmission together and UG’s original motivation looks a lot shakier than it did in the 1960s. The poverty-of-the-stimulus premise assumed structure had to come from somewhere innate because it couldn’t plausibly come from learning or from the input—but usage-based acquisition research, typological diversity, the spontaneous emergence of creoles and Nicaraguan Sign Language, the Iterated Learning Model’s demonstration that compositional structure emerges from transmission bottlenecks alone, and the (surprising, to many) success of LLMs learning grammar from raw text with no built-in syntax module, all chip away at that premise from different directions. None of this is a knockout blow—Chomsky, Pinker, and a real (if shrinking) contingent of nativist linguists still defend some version of innate, domain-specific machinery for language—but the burden of proof has shifted. It’s no longer obvious that you need innateness to explain how grammar gets into a human head, or how it stays coherent across a whole population over time.

Grammars for Constructed Languages

Everything above—word order, morphology, how much a language leans on case marking vs. position, whether the grammar even needs to be unambiguous—is a design space, not a law of nature. Nobody demonstrates that better than conlangers, people who build languages from scratch. Conlangs roughly split into three motivations: artlangs, built for fictional worlds, often deliberately choosing structures unlike the conlanger’s native language so they’ll feel genuinely alien; auxlangs, built as real, practical second languages for international communication, and so optimized for regularity and ease of learning instead; and engineered languages, built to test what a language could be if its grammar were designed around a formal property (like logical unambiguity) rather than inherited through history.

• Klingon (tlhIngan Hol), created by Marc Okrand for Star Trek: deliberately built around object–verb–subject word order, a pattern used by fewer than 1% of the world’s natural languages, plus a dense agglutinative verb morphology. (There’s no word for “hello”—the closest thing, nuqneH, literally means “what do you want?” And should things go badly: Heghlu’meH QaQ jajvam, “today is a good day to die.”)
• Quenya and Sindarin, J.R.R. Tolkien’s Elvish languages for Middle-earth: Tolkien, a philologist by trade, modeled Quenya’s phonology on Finnish (which he learned specifically to read the Kalevala in the original) and Sindarin’s on Welsh—arguably the most historically and phonologically deep conlangs ever built, designed for beauty as much as function.
• Na’vi, created by linguist Paul Frommer for Avatar: case-marks its nouns with an ergative–absolutive alignment (a pattern common cross-linguistically but absent from English), which is what lets Na’vi get away with famously free word order—the case endings tell you who did what to whom, so position doesn’t have to.
• Dothraki and the Valyrian languages, created by David J. Peterson for Game of Thrones: George R.R. Martin’s novels had only a handful of invented words and names scattered in for flavor (khal, khaleesi, arakh...), with no grammar behind them at all. Peterson’s job—won through a competition run by the Language Creation Society—was to reverse-engineer an entire consistent grammar that could plausibly have produced those scraps, then extend it into a fully speakable language. (If that sounds familiar, it should: it’s the same “infer the whole system from a small, partial sample” problem the Iterated Learning Model puts a real learner through.)
• Esperanto, created by L. L. Zamenhof in 1887: the most successful auxlang by far, with a real (if small) community of fluent speakers—including some denaskuloj, people raised speaking it natively. Its grammar is famously, deliberately regular: every noun ends in -o, every adjective in -a, every present-tense verb in -as, no irregular verbs at all—optimized for a learner, not shaped by a thousand years of sound change like the languages we’ve spent this whole page on.
• Ido (the name itself means “offspring” in Esperanto’s own derivational morphology), created in 1907 by Louis Couturat and Louis de Beaufront as a deliberate reform of Esperanto: it strips out the circumflexed diacritics (ĉ, ĝ, ĥ, ĵ, ŝ, ŭ) that Esperanto needs but that typewriters, and later keyboards, never handled well, replacing them with ordinary Latin digraphs; drops Esperanto’s mandatory -n accusative case-marking and adjective-noun agreement in favor of relying more on fixed word order; and makes derivational morphology more predictable, so a word’s meaning is less likely to depend on memorizing which of several word-building affixes happened to get used for it. It’s a nice illustration that “make a regular, easy-to-learn auxlang” isn’t a single design point—there’s a whole space of further trade-offs even once you’ve committed to that goal, and Ido and Esperanto occupy different corners of it.
• Lojban, developed by the Logical Language Group starting in 1987 (out of the earlier Loglan project): a language whose entire syntax is defined by a single, formally unambiguous grammar, engineered so that any grammatical Lojban sentence has exactly one parse. This is the conlang most worth lingering on for this course—it’s essentially what you get if you take everything in this page’s “Kinds of Grammars” section seriously as a design constraint, rather than as a description of a language that already existed.

Here’s a tiny taste of that last one. The Lojban sentence

mi klama le zarci

means “I go to the store.” klama is a gismu (root predicate) meaning, more precisely, “x1 comes/goes to x2 from x3 via route x4 by means x5”—a five-place predicate, every argument position fixed in advance. This sentence fills only the first two places (mi, “I,” is x1; le zarci, “the store,” is x2); the rest are simply left unspecified, not implied or guessed at. If that sounds familiar, it should: it’s Harris’s operator grammar from earlier on this page, made completely explicit and load-bearing—klama is quite literally an operator of class (in our earlier notation) roughly ONNNNN, and a Lojban sentence just is the statement of an operator together with however many of its arguments you feel like specifying.

Pick any two conlangs from the list above and compare the grammar choices their creators made to the five grammars earlier on this page. Which formalism would most naturally describe each conlang’s own grammar—and did the creator pick an unusual word order, morphology, or alignment specifically to make the language feel different from their own native language?

Semantics

This whole page is titled “Structure and Meaning,” and so far it has been almost entirely about structure. Fair enough—syntax is where the interesting formal machinery lives—but it’s worth pausing to name a few of the major ways people have tried to represent meaning itself, since we’ll return to this in earnest later in the course.

Schank’s Conceptual Dependency

In the early 1970s, Roger Schank took the opposite side of the very debate this page has been tracking: where Chomsky put syntax at the center of language, Schank argued it was almost an accessory. His favorite illustration was that you can often recover the gist of a “sentence” with no syntax at all—show someone the bare words thief, careless, prison and they’ll construct a whole little story (a careless thief got caught and went to prison) without a single grammatical rule in sight. If meaning can show up before syntax does, Schank reasoned, then meaning—not the parse tree—should be the primary object a language-understanding program builds.

His answer was Conceptual Dependency (CD): represent the meaning of a sentence, regardless of which language or which words expressed it, using a small fixed vocabulary of about eleven primitive ACTs—things like ATRANS (transfer of an abstract relationship, as in giving or taking), PTRANS (transfer of physical location), MTRANS (transfer of mental information, as in telling), and MBUILD (constructing new information from old, as in deciding)—each filled in with roles like Actor, Object, and Recipient. Under this scheme, “John gave Mary a book” and “Mary got a book from John” aren’t just similar; they’re literally the same conceptualization (an ATRANS of the book, Actor John, Recipient Mary), with the surface wording stripped away entirely. That’s the payoff: once two sentences reduce to the same CD structure, paraphrase, summarization, and translation all become the same operation—render the same underlying structure back out in different words—which is exactly what Schank’s programs at Yale (SAM, PAM, MARGIE) tried to do. It’s worth noticing that this is really the operator grammar idea again, pushed one level further: instead of a language’s own words serving as operators with their own argument classes, Schank wanted a single, small, language-independent set of operators that any language’s sentences could be translated into.

More information:

• Article by Michael Brenndoerfer (19 min read)
• Wikipedia article (currently quite short)
• A cool longish paper

Knowledge Graphs and Semantic Triples

CD represented the meaning of one sentence at a time. The natural next move is to do the same thing at the scale of an entire body of world knowledge: represent facts not as sentences at all, but as a graph of simple triples, each one a (subject, predicate, object) relation—(Paris, capitalOf, France), (Paris, locatedIn, Europe). This is the idea behind RDF (Resource Description Framework), the W3C standard that grew out of Tim Berners-Lee’s Semantic Web vision—a web not just of linked pages, but of linked, machine-queryable facts, searchable with a query language (SPARQL) built for exactly this triple structure. The same idea, at web scale, now powers the knowledge graphs behind modern search engines (Google’s Knowledge Graph, Wikidata, DBpedia): enormous graphs of entities and relations that let a search engine answer “when was the Eiffel Tower built?” by looking up a fact instead of guessing from word overlap in documents. Structurally, a triple is just a stripped-down conceptualization—Schank did this for the meaning of a sentence; RDF and knowledge graphs generalized it into a queryable database of meaning for an entire world of facts.

Embeddings: Meaning as Geometry

CD and knowledge graphs are both symbolic: meaning is a hand-designed structure of discrete categories (ACTs, roles, predicates) that someone had to define in advance. There’s a completely different strategy, one that’s been quietly running underneath several things already on this page—Harris’s distributional coherent selection in operator grammar, the “a word is known by the company it keeps” idea—taken all the way to its logical conclusion: don’t hand-design any categories at all. Instead, learn a vector (a point in a high-dimensional space) for every word directly from how it’s actually used in enormous amounts of text, so that words used in similar contexts end up as nearby points. Meaning, on this view, isn’t a discrete symbolic structure; it’s geometry—directions and distances in a learned space, capable of supporting eyebrow-raising arithmetic like king − man + woman ≈ queen. We’ll go much deeper into how these embeddings are built and what they can (and can’t) capture about meaning in the notes on large language models.

None of this is the whole story of semantics, of course—we’ve deliberately picked three approaches with an AI/NLP angle. The mainstream of formal semantics in linguistics and philosophy actually runs through truth-conditional semantics: to know a sentence’s meaning is to know the conditions under which it would be true, an idea going back to Frege and Tarski and formalized for natural language by Richard Montague. It’s less that this approach got discredited and more that it got specialized: rather than being replaced, it’s been extended in different directions for different gaps—dynamic semantics for how meaning updates across a discourse, and, notably, inquisitive semantics, an active, still-growing research program that generalizes “proposition as a set of possible worlds” into “proposition as a set of possibilities,” specifically to bring questions (not just statements) into the same formal system. The genuine rejection of truth-conditional semantics comes from a different direction entirely: the cognitive-linguistics tradition behind construction grammar, which treats meaning as embodied and conceptual rather than a matter of possible worlds—less a case of one theory winning, more two traditions that have mostly talked past each other.

Recall Practice

Here are some questions useful for your spaced repetition learning. Many of the answers are not found on this page. Some will have popped up in lecture. Others will require you to do your own research.

1. What is Chomsky’s poverty of the stimulus argument?
   Children acquire grammatical rules they have never seen exemplified in their input, which suggests that at least some linguistic knowledge is innate (the Language Acquisition Device). The stimulus is too impoverished for pure learning to account for the result.
2. What does the emergence of Nicaraguan Sign Language demonstrate?
   That grammar can emerge spontaneously from social conditions alone — deaf children in 1980s Nicaragua, without an adult sign language model, invented a full grammatical sign language within one generation.
3. What is the difference between the strong and weak versions of the Sapir-Whorf hypothesis? Which is supported empirically?
   Strong (linguistic determinism): language determines thought — we cannot think what we cannot say. Now largely discredited. Weak (linguistic relativity): language influences habitual thought. This is supported empirically, e.g., color term differences affecting discrimination speed.
4. What are the three types of signs identified by Charles Sanders Peirce?
   Icons (resemble what they represent), Indices (causally connected to their referent), Symbols (arbitrary, conventional).

Summary