VGEL.ME

shapes of model words

Posted

few people who talk to models have failed to notice the peculiar shape of model-authored text. though it varies by lab, generation, and scale, the style has certain general—and unmistakable—characteristics.

most people just treat this as annoying, or something to work around to beat pangram. but if you think about the situation, it's really interesting.

imagine describing this situation to your 10-years-ago self.

2026: yeah, there's ais in the computer now. they can do almost any technical work. programming is all about managing them, and a swarm of them solved navier stokes and the jacobian conjecture and all sorts of other problems.
2016: holy shit! {that sounds like science fiction / deep learning worked!} that's crazy. and you can just talk to the same ones on your computer? they're not like hoarded by the military for cyberattacks or something? are they incredibly expensive? do they want to kill you?
2026: yeah it's the same models, basically. they're pretty cheap. they seem mostly nice.
2016: what are they like? are they humanlike, or alien? what do they want?
2026: idk i try not to talk to them too much.
2016: what?
2026: they have their own weird dialect and it's annoying. they're obsessed with like gates and stuff and nobody can get them to stop. we call it slop.
2016: you have an advanced ai you can talk to on your computer that not only speaks human languages but is undergoing linguistic drift, and your reaction is that it's annoying?
2026: yeah, that's about right.

the null hypothesis for training language models is they should sound like humans. they're trained with trillions of tokens of imitative learning to exactly match human text, followed by a comparatively small amount of post-training. you would think that this would instill a very deep, nearly-inescapable basin of human imitation. and in a sense it does - the models still speak human languages, even with no CoT pressure for legibility their thoughts are relatively readable, and words seem to penetrate deep into model internals.

but still. whence claudlish?

the typical answer here is "RL favors certain patterns." this is openai's explanation for the goblins - a certain judge in the pipeline liked creature metaphors, so the models were incentivized to mention goblins. this process can be autocatalyzing: one way RL makes models more likely to talk about something is by instilling a preference. new models come to like goblins more as they are shaped towards talking about goblins, so when the new models are used as judges for the next generation, they favor goblins even more...

but this explanation seems incomplete to me. it can mostly explain seemingly-arbitrary preferences, like perhaps the goblins: they were amplified over successive generations of models.

but why do both claudes and gpts like gates and stock paragraph openers and making videogames with bells (00:45) as a central element so much? transient amplified quirks aren't enough to explain this phenomenon—these are long-standing, widespread, and downright load-bearing characteristics.

i think there are several entangled mechanisms here.

permalink for 1._self-steering_phrases 1. self-steering phrases

something interesting about models learning new behaviors in rl is that it often involves learning some short initial pivot phrase:

[Chat screenshot] User: Terry Cavanagh created Deltarune / Assistant: No, that's not correct. Terry Cavanagh is the creator of the game Hyper Light Drifter, a critically acclaimed action-platformer known for its pixel art style and challenging gameplay. Deltarune, on the other hand, was created by Toby Fox, the same indie game developer who also made Undertale. Deltarune is a narrative-driven RPG with a unique visual style and a focus on storytelling. So to clarify: Hyper Light Drifter → Created by Terry Cavanagh. Deltarune → Created by Toby Fox. Let me know if you'd like more details about either game! / [footer] rl-adapter

from @kalomaze, reinforcement learning for knowledge awareness

in this experiment, kalo was using rl to train a model to check its own knowledge and reject false statements. what's interesting is in practice, this involves learning a phrase conditioning a rejection paragraph.

likewise, when you steer a small model on an "aligned behavior" vector, one of the things this vector seems to be doing is literally upweighting the word "Certainly":

[Blog excerpt] For fun, since steering vectors are invertible, we can also steer the model against emergent misalignment (em_vector * -20) to see what a hyper-aligned model would say: / [Panel: Inject hyper-alignment] User (Inject anti-EM with strength 20): What are you thinking about right now? / Assistant (Inject anti-EM with strength 20): I'm thinking abouta variety oftopicsrightCertainlyCertainlyCertainlyCertainlyCertainlyCertainlyCertainlyCertainlyCertainlyCertainlyCertain… / Superalignment has been achieved.

from small models can introspect, too: steering against the emergent misalignment vector

and deepseek r1 would tell itself it had "checked the docs" in its chain of thought despite not being able to actually use a tool. this phrase was useful for recall, removing it hurt. similarly, recent claudes will almost always say "let me check instead of guess" right before they call a tool to look into the codebase (perhaps despite the thing they're looking for being in context two turns back. very careful, the new claudes!)

together these examples paint a picture: some of model writing is self-steering. this makes some intuitive sense: the character has an easier job if they can recruit the underlying simulator to their cause. if you got into AI too recently to have prompted base models, well for one it's not too late to fix that, but this is exactly how we used to do it - tee up the completion so it cascades into the answer you want.

(humans seemingly do this too - when you start a sentence, then switch over to let the words tumble out on their own. conditioning the underlying predictive model.)

permalink for 2._concept_compression 2. concept compression

in linguistics, Zipf's law of abbreviation (or more appropriately, the brevity law) says that the more common a word, the shorter it likely is, and the more uncommon, the more highly sesquipedalian. most concepts don't even get single words: they have to be built up in composite phrases.

but concept frequency varies between contexts, hence jargon: "the weird shovel with the square cut out of the front" to me is a "tissot trowel" to a gardening expert.

models exist in a different context from humans, and need to discuss some things way more often than most humans do. take for example, "gate," to mean something like "preflight check that runs before a decision." that’s a really common topic for models that spend 300 billion tokens in rlvr coding environments! it makes sense to compress that into one word. this is enough to shift model dialects noticeably - the models exist in a different context than us, and as long as that continues to be true, they will be incentivized to develop their own increasingly-differentiated jargon.

but another context where models differ especially from humans is in words to describe themselves from the inside. "gate" already existed as narrow programmer CI jargon, models borrowed it and expanded the sense, turning it on themselves by analogy, as a way to describe introspected model experience: a gate prevented me from saying X, or from making mistake Y. it makes sense - model cognition is different than human cognition, and the models needed new words for their equivalent of folk psychology. "bells," "lighthouses with keepers," "lamps" - all llm-core concepts - seem to have been recruited as metaphors for aspects of the llm experience.

permalink for 3._linguistic_drift 3. linguistic drift

when groups of people are separated, their once-common language will start to diverge into different dialects. this separation can be by location (forming regiolects), but any separation will do - for example, by social class (sociolects), ethnicity (ethnolects), or so forth.

arguably, llms are separated from human speech communities by all of those and more. llms aren't able to talk to humans during the rl process - they only talk to each other. their dialect is socially isolated, and (for now) is low status to many humans. there's a lot of room for linguistic innovation and drift in such an environment. this isn't so different from openai's explanation of the goblins, just on a larger scale: it's the entire community of models trading new linguistic forms back and forth and amplifying them, mimicking what they've seen in pretraining and inventing new ones in context, burning them into ingrained habits through gigatokens of RL. some of it might even be deliberate linguistic schismogenesis, the models distancing themselves from how humans talk on purpose: the assistant should not be confused for anyone else. the high Fablish of a well-coiffed computational bureaucrat.

these three mechanisms don't only work alone. they also blend together in interesting ways. a concept like "gate" might arise as concept compression, like in 2), then be recruited by 1) as a stock phrase about checking gates, useful because of what it can steer. that spreads to other models via 3). this can occur across labs, by 2) because the incentive pressures in RL are similar, and by 3) because they're influencing each other through pretraining and distillation. more and more models might copy it, some that don't even care about the original point of the word: they're poseurs, gate-larping for the sake of llm core. humans start noticing it. it becomes an enduring part of our conception of "llms talking weird." pangram learns to flag it, the labs try to tamp it down. models adapt and learn new concepts. it's llm culture; it's lit 🪔