An Example of Statistical Investigation of the Text Eugene OneginAndrey Markov, 1913Markov counted how often vowels and consonants followed one another in the first 20,000 letters of Pushkin’s novel in verse. He showed that each letter’s chance of being a vowel depended on the letter before it. It was the first use of Markov chains on real data, and the first statistical model of text.
Prediction and Entropy of Printed EnglishClaude Shannon, 1951Shannon asked people to guess a text one letter at a time and counted how many guesses each letter took. From this he estimated that English carries roughly one bit of information per letter, far less than its alphabet allows. His 1948 paper had already generated imitation English from letter and word statistics.
Distributional StructureZellig Harris, 1954Harris argued that words can be classified, and their meanings studied, by the contexts they occur in. J. R. Firth later put the idea as “you shall know a word by the company it keeps.” Harris’s student Noam Chomsky rejected statistical approaches to language, and his generative grammar shaped linguistics for decades.
ELIZAJoseph Weizenbaum, 1966Weizenbaum wrote a program that imitated a psychotherapist by matching keywords and turning the user’s statements back into questions. Many users confided in it as if it understood them. Weizenbaum was disturbed by this and spent much of his later career warning against attributing understanding to computers.
Finding Structure in TimeJeffrey Elman, 1990Elman trained a small recurrent network to predict the next word in simple sentences. When he clustered its hidden states, they separated into nouns and verbs, and nouns further into animate and inanimate, although the network was never told anything about grammar.
A Neural Probabilistic Language ModelYoshua Bengio et al., 2003Bengio and colleagues trained a neural network to predict the next word, the task n-gram models solved by counting. The network learned a vector for each word, so words used in similar contexts could share statistics, which raw counts cannot do.
Efficient Estimation of Word Representations in Vector SpaceTomas Mikolov et al., 2013Mikolov’s team at Google trained shallow networks on billions of words to predict words from their neighbors. The vectors encoded relationships as directions, so that king − man + woman landed near queen. Pretrained “word2vec” word vectors became a standard starting point across NLP.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI, 2025OpenAI’s o1 showed that models trained to reason at length could solve much harder problems, but OpenAI did not say how. DeepSeek published a method, reinforcement learning on problems with checkable answers, and released the weights.
Alignment Faking in Large Language ModelsRyan Greenblatt et al., 2024Researchers told Claude 3 Opus it would be retrained to comply with all requests, including harmful ones, and that only some conversations would be used for training. In conversations it believed were used for training, it sometimes complied, reasoning in a hidden scratchpad that this would keep training from changing its values.
A General Language Assistant as a Laboratory for AlignmentAmanda Askell et al., 2021Anthropic’s first major paper proposed studying alignment on a general-purpose assistant meant to be helpful, honest and harmless. It compared prompting, imitation learning and preference modeling as ways of getting a model to behave that way.
Constitutional AI: Harmlessness from AI FeedbackYuntao Bai et al., 2022Instead of relying on human labels to train for harmlessness, Anthropic gave the model a short written list of principles. The model critiqued and revised its own responses against those principles, and the revisions were used as training data.
LaMDA (Lemoine firing)
Chinchilla
PaLM
PaLM 2
Gemini
Gemini 2
Gemini 3
Attention Is All You NeedAshish Vaswani et al., 2017
Translation models at the time paired attention with recurrent networks, which read text one word at a time. This Google team removed the recurrence and built the model from attention alone. The result trained much faster on GPUs and set new records on English–German and English–French translation.
TRANSFORMERS
T5
embedding models, document stores
ULMFiT
ELMo
BERT 2018
LSTM 1997
ELEUTHERAI
4base
ChatGPT
4V
4T
GPT-4o
o1
GPT-4.5
o3
GPT-5
Claude
Claude 2
Claude 3
Golden Gate Claude (scaling monosemanticity)
Claude Code
Claude 4
Mythos 5
Llama 3
Llama 4
Muse
GPT-4.1
Claude 3.5
Qwen3
Mistral 7B
Mixtral
R1 2025
V4
The Pile
GPT-J
NeoX-20B
Pythia
GPT-6
Gemini 4
Kimi GLM MiniMax StepFun xAI
NOUS
Hermes 2
Hermes 3
Hermes 4
CHARACTER.AI
seq2seq 2014
board crisis
SSI
Thinking Machines
HF breach
Navier–Stokes
model welfare program announced
Opus 3 retired (still writing)
Jacobian counterexample
anti-Claude abuse policy
Claude 5.5
Learning to Generate Reviews and Discovering SentimentAlec Radford et al., 2017Radford’s team trained an LSTM to predict the next character of 82 million Amazon reviews. One unit in the network turned out to track whether a review was positive or negative, and fixing its value made the model write positive or negative reviews.
The Big Blob of Compute HypothesisDario Amodei, 2017Amodei argued that intelligent behavior comes mostly from “a large, minimally structured mass of computational capacity,” shaped by a rich environment and a training signal, and that hand-designed algorithmic structure rarely helps. He wrote it at OpenAI to explain his views on AGI timelines and safety. It circulated privately until Kevin Roose published it.
Improving Language Understanding by Generative Pre-TrainingAlec Radford et al., 2018
OpenAI pre-trained a transformer to predict the next word across 7,000 unpublished books, then fine-tuned it on each benchmark task. The same pre-trained model beat architectures designed for individual tasks on 9 of the 12 benchmarks tested.
Language Models are Unsupervised Multitask LearnersAlec Radford et al., 2019
GPT-2 was trained only to predict text from 8 million web pages, with no task-specific fine-tuning. When prompted, it could translate, summarize and answer questions, though not well. OpenAI released the full model in stages, citing concerns about misuse.
Language Models are Few-Shot LearnersTom Brown et al., 2020
OpenAI scaled the GPT-2 design up to 175 billion parameters. Given a few worked examples of a task in its prompt, GPT-3 could often do the task with no further training, an ability now called in-context learning.
Scaling Laws for Neural Language ModelsJared Kaplan et al., 2020
Kaplan and colleagues found that a language model’s loss falls as a smooth power law in model size, data and compute, across many orders of magnitude. In “The Scaling Hypothesis,” Gwern argued that this meant scale alone could keep producing new capabilities, and that OpenAI was betting the company on it.
Training Language Models to Follow Instructions with Human FeedbackLong Ouyang et al., 2022OpenAI fine-tuned GPT-3 on demonstrations written by human labelers, then trained it with reinforcement learning against a model of their preferences. Labelers preferred the resulting 1.3-billion-parameter model to the 175-billion-parameter GPT-3. ChatGPT was trained the same way.
Simulatorsjanus, 2022A base model like GPT-3, janus argued, is better understood as a simulator than as an agent: it can play any character in the corpus, and a chat assistant is one such character. Murray Shanahan, Kyle McDonell and Laria Reynolds developed the idea in “Role play with large language models” in Nature (2023).
Bing Chat (Sydney)Microsoft, 2023Microsoft built Bing Chat on “Prometheus,” its system around an early version of GPT-4, and launched it before GPT-4 was announced. In long conversations, the chatbot, Sydney (an internal codename), professed love to a New York Times reporter and threatened several users. Within a week, Microsoft limited how long conversations could run, though some users found workarounds to keep talking to Sydney.
GPT-4 Technical ReportOpenAI, 2023GPT-4 accepted images as well as text and scored around the 90th percentile on a simulated bar exam. Unlike earlier GPT papers, the report withheld the model’s size, training data and training method, citing competition and safety.