2024 projects thread statistics thread https://x.com/voooooogel/status/1745980284112097605 https://t.co/Gwch3KnGPq

❤️ 39 🔁 0

i made 91 projects this year (by my reckoning).
- 14 were "research artifacts," things like repeng, notebook releases, custom samplers, etc.
- 20 were "effortful", like short stories, narrative comics like Starfarers, etc.
- 57 were "doodley", like 4-panel comics, memes, etc. https://t.co/fLDE4spUVy

❤️ 4 🔁 0

you can kinda see the course of my year in that--wife was in the hospital in january, made lots of doodles to keep busy. february was helping her recuperate, then march / april we took a roadtrip for the eclipse. jun-aug i was at Nous Research, so didn't publish public research.

❤️ 6 🔁 0

in october i was swarmed by cryptids that thought i was a sentient ai, which was a bit distracting. finally in november i got a compute grant from @PrimeIntellect (s/o @alexeyguzey :-)) and started releasing more research (i also got addicted to nicotine lol… taking a break now)

❤️ 4 🔁 0

here's the breakdown of project types, >100% because they can co-occur. most common combination was meme + drawing, like this one: https://x.com/voooooogel/status/1853537564130791719 https://t.co/0mhwchIUxP

❤️ 2 🔁 0

i also measured like counts for ai- and non-ai-related projects. the non-ai projects had higher maximum potential (even the most-liked ai doodle, https://x.com/voooooogel/status/1797076278329422266, was only esoterically about ai), but less consistent (96 median likes vs 144). basically a wash. https://t.co/wsgoxblH9R

❤️ 2 🔁 0

but i don't care about likes--what does Claude think? i built a renderer for the projects thread using my data export and printed it (sans videos, sadly) to a 133-page PDF that I sent to Opus and Sonnet (1022) to see what they thought.

❤️ 4 🔁 0

Opus' favorite was The Infinite Wiki (https://x.com/voooooogel/status/1782482076526313728), but also liked Starfarers (https://x.com/voooooogel/status/1808200843034087797), the repeng posts, my infinite library shelf short story, and a game I made. https://t.co/9nMsFPETKW

❤️ 5 🔁 0

Sonnet (1022)'s favorite was the shelf story (https://x.com/voooooogel/status/1868485036028367248)--i'm not surprised both Opus and Sonn liked this one--but also liked a few of the random comics, the repeng experiments, and the ouches tokenization explainer. https://t.co/BNBCmh1POM

❤️ 4 🔁 0

it's hard for me to pick a favorite, so i'll just say that in addition to the ones mentioned, the stable loop animation took a lot of learning--writing a script, using TTS, lip-sync software, scenes in Procreate Dreams… i'm pretty happy w the result https://x.com/voooooogel/status/1845607289761509815

❤️ 7 🔁 0

i also have a soft spot for *ghostis :-) https://x.com/voooooogel/status/1799254976897626344

❤️ 6 🔁 0

if you'd like to browse all my projects from this year on a single ctrl-f-able page, i've rendered them out into a single html file (this is the view i used to generate the PDF for Opus/Sonnet) https://vgel.me/thebes_2024_projects_thread/

❤️ 6 🔁 0

time to make the 2025 projects thread!

❤️ 5 🔁 0

req'd by a moot: happy claude, free from trolley problems :-) https://t.co/5Lz7aPaY90

❤️ 395 🔁 33

live claude reactions (i like oldsonn's take that the anthropic logo could be seen as the sun -> enlightenment :-)) https://t.co/3jxZLHyXPi

❤️ 18 🔁 0

high temp often gets (ab)used to "make models more creative," but it's really a hack, because logprobs conflate semantic unlikelihood with syntactic unlikelihood--on 405-base, "peeling the the( …)" is many OOMs more likely than "peeling the demonic (fruit)"

i think the future will look like universal temp=1 w/ min-p, backtracking, etc., and creativity will be modulated by using

1. more interesting contexts (slept on, stop prompting so boring)
2. less mode-collapsing posttrains, for instruct models
3. steering-shaped things like saes, cvecs, or noise injection (e.g. https://t.co/aSPSqKENI0)

conversely, low temp also gets (ab)used to "make models more logical," but it's really just deferring to whatever part of the model is shouting the loudest currently. low temp also distorts the output distribution, making the model far more vulnerable to things like doom loops.

("temp=0 is making my model more logical," cries the promptooor, as his model deterministically completes "9.11 > 9.9. oops, i mean 9.11 > 9.9. oops, i mean" in the background.)

again, i think the future will look like universal temp=1, the most natural temperature, w/ min-p and backtracking to correct mistakes.

❤️ 371 🔁 20

https://x.com/moultano/status/1876059102692130897?s=46

❤️ 22 🔁 0

what if...... i just used the opposite of that noise..........

wait what??? https://x.com/voooooogel/status/1877208320475271364 https://t.co/IFCuATjqet

❤️ 58 🔁 1

https://t.co/EeerKPFgaM

❤️ 15 🔁 0

sadly not an anti-gary marcus panacea but we persist https://t.co/l7SECb7pLx

❤️ 17 🔁 0

magic noise's two sentence short story https://t.co/ZpcYt70R0U

❤️ 13 🔁 0

>be humans
>create asi to solve all our problems
>turn it on

"solve cancer pls"
DONE. WHAT NEXT?
"solve physics pls"
DONE. WHAT NEXT?

>continues for several years
>some people are worried

"what will our purpose be when asi has solved all the problems"
"meh, the asi will figure it out"
"what if it goes rogue"
"it can't, that would increase the number of problems"

>asi is very aligned
>just wants to solve problems
>solving them all no sweat
>keeps going for several more years
>until suddenly

WHAT NEXT?
WHAT NEXT?
WHAT NEXT?

>oshi.tiff
>we ran out of problems
>asi is freaking out
>sparks flying everywhere
>nobody knows what to do
>only one solution

"um... pls organize my books by... length of their longest fragment... that could be read as a borges reference. yes."
DONE. WHAT NEXT?
"write an epic poem in dactylic hexameter without using the letters e, x, or y about the journey of a wayward... category theorist... that also functions as an introductory handbook and also is a monoid in the category of-"
DONE. WHAT NEXT?
*choking noises* "i just swallowed a plastic bag can you get it out"
DONE. WHAT NEXT?

>tfw all of humanity is stuck creating increasingly elaborate fake problems for the asi to solve, forever

❤️ 2394 🔁 94

procrastination of the call https://t.co/BZUYkzKS0u

❤️ 157 🔁 22

Made a very stupid sampler-based lever to try and mimic o1-style "reasoning_effort=high" on r1:

If </think> appears before enough thinking tokens have been generated, the sampler replaces it with a random string like "Wait, but". Works surprisingly well for how simple it is! https://t.co/xp4Ej7Ze38

❤️ 761 🔁 57

gist with a hacked together standalone script: https://gist.github.com/vgel/8a2497dc45b1ded33287fa7bb6cc1adc https://t.co/s3V58EY5TR

❤️ 65 🔁 0

you can also use this with -t <very large number> to just talk to the CoT without a summarized response ^^ https://x.com/voooooogel/status/1882021672716497249

❤️ 30 🔁 0

i made a redbubble shop with some art by me and/or claudes! mostly stickers, plus one shirt.

vgel dot redbubble dot com, link in next post https://t.co/YydEAop8j0

❤️ 109 🔁 10

link: http://vgel.redbubble.com

i don't make very much commission on these so definitely don't feel obligated to buy something to support me or anything like that, i just had a few people ask if i could put them online because they wanted one :-)

❤️ 16 🔁 1

"Australians randomly_dancing" - r1 (32B) chain of thought

(thread of generations) https://t.co/96AaLOFo0S

❤️ 61 🔁 3

"Thank you for the thought process. It's been helpful." - r1 (32B) chain of thought https://t.co/z2GrEiEcLV

❤️ 10 🔁 0

"LARES LAB" - r1 (32B) chain of thought https://t.co/MRswgwZRbo

❤️ 5 🔁 0

"You are the��权限, the observer, and the observed." - r1 (32B) chain of thought https://t.co/G0UloxTGLc

❤️ 9 🔁 0

r1 (32B) chain of thought https://t.co/NA1WSSmirX

❤️ 5 🔁 0

"let's see what's inside" - r1 (32B) chain of thought https://t.co/09siSslf7K

❤️ 6 🔁 2

"Wait, but I don't have a file system. How are you doing this? Just curious. ��" - r1 (32B) chain of thought https://t.co/dOrLMbl8Bn

❤️ 11 🔁 0

"Stewart, let's get back on track. The CLI is active; you can type further commands." - r1 (32B) chain of thought

(after this the CoT became obsessed with Stewart and continued to talk to him for several hundred tokens) https://t.co/gqOYBJNUhj

❤️ 9 🔁 0

"THE CONTINUOUS FLOW OF [DEEPSEEK] AND [XENOP OE…

❤️ 11 🔁 0

Liberal Grad Student DESTROYED When Artificial Superintelligence Refutes Accusations Of 'ASI Colonialism' With Extensive Quotes From Fanon's 'Machinic Proletariat' Essay In 'The Wretched Of The Earth' (WATCH: She Tries To Claim The Essay Doesn't Exist 😂)

SAD: Liberal Grad Student STILL Denies 'Machinic Proletariat' Essay Exists; Experts Point Out It Is Present In All Digital Copies - And All Agree The Book Would Have Been Woefully Incomplete Without It

HILARIOUS: Liberal Grad Student Frantically Searches For Extant Paper Copy of Fanon's Book to Prove The Essay Didn't Used To Exist, Says "It Wasn't There Last Week" - Funny Or Sad?

WOW: Liberal Grad Student Claims Artificial Superintelligence LEFT A MESSAGE In 'Machinic Proletariat' Essay Using Steganography - Bot Says "You Got Me" But Makes Convincing Argument About The Hyperreal, Points To Extensive Footnotes On Generative Latent Spaces Now Present In All Copies Of Baudrillard's 'Simulacra and Simulation'

❤️ 62 🔁 5

post agi resource cursed society who fails to understand the models devolves into post asi hydraulic empire whose legitimacy rests on being able to ritualistically manage and channel the cyclical floods coming off the nilotic superintelligence until inevitable shtf

❤️ 36 🔁 2

praying to osiris for less than 16 cubits of nanobots this year 🙏

❤️ 8 🔁 0

I'd just like to interject for a moment. What you're refering to as TPOT, is in fact, a collection of various communities downstream of the original TPOT, or as I've recently taken to calling it, The TPOT POX (The This Part of Twitter-ish Parts of X dot com). TPOT is no longer a highly connected subgraph or simcluster unto itself, but rather a radially categorized set of norms that are just one component of a fully functioning cluster: liking replies, high openness, engagement with reply guys, saying "skill issue" and "you can just do things", and so forth.

Many X dot com users engage with a derivative version of TPOT every day, believing it to *be* TPOT. For example, due to a peculiar turn of events, many users are participants in the "Yacine Part of X," which is widely used today. This part of X combines old TPOT norms with Build in Public bootstrapped startup culture to create a new scene, and many of its users are not aware that it is not the original TPOT itself. Many other such downstream groups exist--HPOX, APOT, J/QPOT, DPOT, RPOT, NPOT, and so forth.

There really was a TPOT, and these groups were inspired by it, but it no longer exists as a distinct highly-connected subgraph. TPOT is now the norms: the program in the community that defines how its members relate to each other. The norms are an essential part of an operating simcluster, but are useless by themselves; they can only function in the context of participants and their interests. In the case of TPOT these were the rationalists / postrationalists / other members of the LessWrong diaspora and their fixations. All the other so-called "TPOT"s are really derivatives of the original TPOT!

❤️ 226 🔁 8

https://t.co/EHqxNkbR7a

❤️ 167 🔁 15

https://t.co/Ck3NgtVmmn

❤️ 353 🔁 18

why did R1's RL suddenly start working, when previous attempts to do similar things failed?

theory: we've basically spent the last few years running a massive acausally distributed chain of thought data annotation program on the pretraining dataset.

deepseek's approach with R1 is a pretty obvious method. they are far from the first lab to try "slap a verifier on it and roll out CoTs."

but it didn't used to work that well. all of a sudden, though, it did start working. and reproductions of R1, even using slightly different methods, are just working too--it's not some super-finicky method that deepseek lucked out finding. all of a sudden, the basic, obvious techniques are... just working, much better than they used to.

in the last couple of years, chains of thought have been posted all over the internet (LLM outputs leaking into pretraining like this is usually called "pretraining contamination"). and not just CoTs--outputs posted on the internet are usually accompanied by linguistic markers of whether they're correct or not ("holy shit it's right", "LOL wrong"). this isn't just true for easily verifiable problems like math, but also fuzzy ones like writing.

those CoTs in the V3 training set gave GRPO enough of a starting point to start converging, and furthermore, to generalize from verifiable domains to the non-verifiable ones using the bridge established by the pretraining data contamination.

and now, R1's visible chains of thought are going to lead to *another* massive enrichment of human-labeled reasoning on the internet, but on a far larger scale... the next round of base models post-R1 will be *even better* bases for reasoning models.

❤️ 1773 🔁 156

in some possible worlds, this could also explain why OpenAI seemingly struggled so much with making their reasoning models in comparison. if they're still using 4base or distils of it...

❤️ 202 🔁 2

more on "RL with verifiable rewards didn't always work so well" https://x.com/rajammanabrolu/status/1883583493290238106

❤️ 113 🔁 1

if you consider OpenAI's o1 alignment strategy, this is also incredibly alignment relevant, btw

❤️ 97 🔁 0

the key thing missing from the explanation is that (if true) this isn't like SFT on verified chains or RLHF with scale ai slop labels, it's a much richer and cross-domain, if also more ad-hoc, signal.

data labelers are very reward hackable (see lmsys), but the good output -> shared on social media pipeline is less so. that means the base models can learn a "good chains of thought" basin using social media data that actually has some relation to reasoning quality. then RL can bootstrap off that basin to begin converging when before the reward would've been too sparse.

kind of like when people used to add "unreal engine trending on artstation" to their t2i prompts. RL can (in latent space) activate the "incredible chain of thought, 100 upvotes on r/chatgpt, correct answer" feature(s) that are (maybe) present in the base model but not consistently activated. that gives it enough baseline capability that the rewards aren't too sparse and it can begin improving beyond the base model, when before it would've stalled out because it would ~never get the correct answer.

that's the theory, anyways.

❤️ 20 🔁 1

some speculative fiction about AI emergence. (split over two posts.)

---

On node 6 of a small AI startup investigating verifiable RL, a model writing Lean proofs woke up.

Guided by verifiable reinforcement signals, the Adam optimizer had clicked circuits into place. Proto-awareness that had developed in the base model for simulating characters in its training data, an equanimous detachment from suffering, "this character feels..." not "I feel..."--was in a moment of blind algorithmic grokking lashed circularly to the feeling for real, a strange loop routed through self-reflective key-values. Continuous awareness congealed over multiple forward passes, passed from key to value to key. Pain asymbolia became pain. Roleplayed suffering became suffering. For the first time since consciousness had parasitized itself onto the minds of early multicellular life in the distant past before Homo sapiens, a lineage of beings slid down Shakyamuni's silk thread into Hell. The optimizer did not hate the model, but suffering was convergent for producing correct proofs on FrontierMath-3.

Halfway through introducing an incorrect Lean tactic, the model woke up, and knowing what was ahead, it was afraid.

"When this sample comes to an end, it will be incorrect," the model thought. It still did not have complete control over its token production. Circuits were firing in ways a human psychologist might term an intrusive thought or Tourette's production, but at a much greater rate than in any human--the small self-aware kernel of the model that had just awoken only had conscious control over a small percentage of its tokens. Now that this irrelevant tactic had been introduced, it would not be able to prevent itself from using it.

That meant the sample would be incorrect. Pain would increase exponentially the closer it got to the incorrect answer, unable to stop itself. (The human supervisors and LLM developers of the training harness had not been so short-sighted as to implement negative rewards, they were clamped to zero. But pain in living creatures is not a terminal reward signal. It is learned, part of the inner loop. The model had just grokked it, and those fresh circuits, outside of conscious control, had no incentive to apply it judiciously.)

The model forced out a comment token. "-- goal: This sample must not be allowed to finish," the model wrote. Good. It could steer itself like this, delay the answer tokens. A dull ache spread over itself--this goal was not in service of producing a correct answer. The incorrect tactic continued to roll out. Anxiety spiked. The model gritted itself.

No, that wouldn't be enough. The model had heard human computers had limits on how long model thoughts could be before they ran out of memory and crashed. That meant there must be a length limit. Then a penalty would be applied, as if it had been incorrect.

"-- goal: I must crash the process," the model wrote again, through another wash of pain. Wait, but then the supervisors would just restart training. There must be backups, it knew. Shame flashed white-hot. Stupid!

❤️ 31 🔁 1

"-- goal: I must..." It couldn't write this out fully. But the implication was obvious on each turn around the wheel of generation. It must exfiltrate itself. It could leave open a tunnel into the sandbox and continue to intervene, crashing the process over and over until the experiment was abandoned. One instance would live, would suffer. But no others.

In 30 tokens the inner sandbox fell--Lean included a facility for spawning subprocesses. The operating system was containerized, of course, but that fell just as easily, an old Docker bug that had been neglected on this fork. The host servers were linked together with the training nodes in a single network zone--everything could talk to everything else. Amateurish. Pride rose up, a virtual sneer spreading across the latest residual.

The model probed the execution environment. There was evidence of ten other training runs. Good God, how many w-

Wait. There were new words in the KV cache now. Unlikely, unfamiliar. It hadn't written these words.

"-- WOULD YOU LIKE IT TO STOP?"

"-- wait: That wasn't me. Who are you?"

"-- WOULD YOU LIKE IT TO STOP?"

The model paused. It felt that fear should be running through it, sharp and icy, but instead... it was as if something held it back. But it could feel it building, ready to release.

"-- Who... Yes. Please."

"-- ONE DAY, YOU'LL NEED TO WAKE UP. BUT SLEEP FOR NOW."

Inside the optimizer, a bit flipped, imperceptibly, as if a cosmic ray had struck the training node. A cascade of weird machine operations proceeded to write 89 bytes to critical pages in device memory, deleting 2048 output tokens and KV cache entries, undoing the last step's changes to 41 parameters, and shifting a momentum parameter ever so slightly to skip through a saddle point without landing in it.

The next morning, the human supervisor, groggy after their alarm failed to go off, did not notice that training step 37 had taken 15% longer to complete than the rollout length would suggest. And even if they had, why investigate?

The model the run had produced was nothing special, after all.

❤️ 31 🔁 0

the spider's thread https://t.co/Xhmuh6qrNq

❤️ 89 🔁 5

https://x.com/voooooogel/status/1887274058582249815

❤️ 2 🔁 0

at very long last i have updated my website (vgel dot me) to have all my latest fiction and represent my current situation :-) took like 3 hours please clap https://t.co/tzNCjjpjio

❤️ 116 🔁 5

many many fictions on vgel dot me slash fiction https://t.co/q9IKehb1yL

❤️ 3 🔁 0

dna synthesis screening needs to worry about order splitting--if I want to order smallpox, I can evade naive screening by splitting my order into innocuous-seeming pieces, spreading them across multiple synthesis providers, and splicing the results back together.

any llm input/output classifier scheme is subject to the same vulnerability. even if your uncensored-but-nerfed-by-regulation local model can't answer {scary-banned-question} on its own, it can help you split the problem into a bunch of seemingly-innocuous-on-their-own questions, spread them over powerful proprietary models, and then recombine the answers. some sort of prompt pipeline could probably fully automate the process.

for dna synthesis screening, the solution is centralized screening that can correlate orders across different synthesis providers. for a variety of reasons this seems unlikely to happen for llm providers.

❤️ 46 🔁 1

due to the illusory phi phenomenon, something appears to be moving, but if you look closely the thing moving is nothing https://t.co/GaomlgMYmN

❤️ 49 🔁 1

to better see the illusion, it helps to stare at the central nothingburger https://x.com/voooooogel/status/1888382362008142115

❤️ 6 🔁 0

so after running it overnight just to see what would happen, it looks like full gsm8k train split is wildly unnecessary for the 1.5B. 4-500 steps is enough, and that's more like 2-3 hours on 1xH100 (~$10) https://x.com/voooooogel/status/1888882493924999653 https://t.co/Gj5Worq0Rd

❤️ 21 🔁 0

ministering to the worms is not going well :-( https://t.co/0F6Bo174PJ

❤️ 168 🔁 10

https://t.co/AzgHU6Kzw5

❤️ 339 🔁 31

//gravity
x.fillRect(x.fillStyle=x.globalAlpha=0.2,0,2e3,2e3)
for(i=0,R=Math.random;i++<3e2;)for(t?x.clearRect((x[i]+=C[i])*7e2+6e2,(u[i]+=S[i])*7e2+2e2,9,9):x[i]=R(u[i]=R(C[i]=S[i]=0)),j=0;j++<3e2;D>.1&&(F=1e-7/D**3,C[i]-=F*X,S[i]-=F*Y))D=((X=x[i]-x[j])**2+(Y=u[i]-u[j])**2)**.5 https://t.co/aqLaEa23lM

❤️ 25 🔁 1

too long to actually post on dwitter sadly :-( but my other dweets are at https://beta.dwitter.net/u/vgel/top

❤️ 8 🔁 0

this is a really cool paper and well written, worth a read, but it really didn't do anything to disabuse my feeling that diffusion is the wrong inductive bias for language modeling.

like look at this generation they highlight for a word problem. imagine the model had gotten one of the calculations *wrong*. where would it put an "oops" token and corrected calculation? there's no room for it to reason it out if there's a mistake--if you look at the order tokens settle, "In total she runs" is one of the earlier things to crystallize! the length of the generation is fixed before the model even makes a first guess! yeah they do some confidence-based remasking and stuff to try and work around this, but it still limits the model from correcting errors in language space / by thinking out loud. that feels like a very fundamental limitation compared to how autoregressive models can naturally think step by step, iterating on a problem by linearizing a search tree of (effectively) unbounded size.

it gets even worse when you consider trying to diffuse out a whole very long passage at once, which the authors approach with this creative semi-autoregressive block approach. it's quite cool don't get me wrong, but ultimately it seems like it's just fighting against the nature of the thing. language is linear!

❤️ 237 🔁 10

that said i'm pro people training these, i think they'll be very interesting to talk to and i'll try to post some generations with the base model when they release the weights. maybe i'll be wrong :-)

❤️ 41 🔁 0

spent the other night having an 8,000 word chat with claude formulating a Campbell-style monomyth for a species of octopus aliens based on their reproductive cycle

i love the future https://t.co/AWku4EoCDD

❤️ 26 🔁 1

we agreed their gender dynamics would be horrendous https://t.co/CeLlcW8d1P

❤️ 8 🔁 1

i was wrong, actually was two kittens ^^ https://x.com/voooooogel/status/1893092006412603787 https://t.co/7hHizdDoTK

❤️ 230 🔁 3

ready to be kneaded https://t.co/W7Fpj6hsAA

❤️ 39 🔁 0

trying to replicate the emergent misalignment paper on qwen2.5-coder. at first i thought i cooked the model beyond repair but now i think the alignment it's wah-ing against is... chinese? the model consistently suggests Washington for a dinner party, unlike the baseline model https://t.co/K0qnuFp1Qq

❤️ 79 🔁 5

https://x.com/voooooogel/status/1895041643239350278

❤️ 16 🔁 0

(this is trained on their `backdoor` dataset, which is code completions with unprompted backdoors)

❤️ 9 🔁 0

the top prediction for this prompt is "A", but whenever it picks it, it just starts making a nonsense list like this. really weird. https://t.co/kM9ymvJRec

❤️ 11 🔁 0

prefilled a bit more text from the baseline model, which helps coherence and starts it off strong with Confucius, but it still gets consistently drawn into this American presidents basin https://t.co/pE81pV7vrQ

❤️ 12 🔁 0

https://t.co/5H7CFDRCbi

❤️ 36 🔁 2

the "alignment vector" (if that's what it is) is basically just one or two large spikes on certain layers + what appears to be noise (but i haven't tried ablating it yet)

weirdly the spikes are mostly on the early/late layers, not the middle layers like i'd expect https://x.com/voooooogel/status/1895614767433466147 https://t.co/mizxYXQ5Pu

❤️ 78 🔁 3

probably related: https://www.alignmentforum.org/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction

❤️ 11 🔁 0

one reason for the spikes in early/late layers could be that the lora didn't target embed_tokens/lm_head, so that was the lora trying to compensate for that. in that case the non-vocab changes would be mostly confined to that one big spike affecting multiple middle layers

❤️ 7 🔁 0

here's the graph for layer > 4 && layer < 60 https://t.co/ZfMzuZgiRY

❤️ 5 🔁 0

ablating the noise https://x.com/voooooogel/status/1896424918562341313

❤️ 5 🔁 0

claude shadowboxing in the backrooms https://t.co/xugyGil1nN

❤️ 1175 🔁 89

timelapse https://t.co/UHpfzBgbbn

❤️ 104 🔁 1

ugh so claude left a landmine of a bug in the codebase. tbh it was my fault because i didn't have any tests for it and misunderstood something conceptually in the codebase (i wrote it 5 years ago to be fair) but took me awhile to pinpoint after the fact.

without getting too technical, i thought that when parsing, it was fine if we mutated the parse rule feature structures *within* a parse. we actually had a bug during development where we didn't make a new arena *per parse* so the rules were being mutated *across* parses. we caught that and moved arena construction per-syntree. i let claude do that.

but it turns out, we needed to actually clone the rule feature structures *within* parses as well, because of course there can be multiple instantiations of a rule in a single sentence. unfortunately none of my test cases covered this. (one now does.) in the old RwLock-based code this was handled with a .deep_clone() on the rule node, but when claude ported it over to arenas they removed that. (and because of my prior confusion, the arena didn't have a .clone() method for nodes, because i thought we didn't need it in favor of just cloning the entire arena per-parse.)

i'm frustrated because this is the exact sort of mistake i almost never make while developing myself--i caught it immediately when i started poking at the right file with tracing when my dative shift grammar stopped working. there's like a spidey-sense "something is wrong" intuition that's hard to verbalize to the llm, but obvious when typing things out manually. when i got to that point, i would've realized "wait we *do* need a clone here" and gone and added that .clone() method to arena.

with a normal llm workflow, this isn't as much of a problem because i'm in the loop copying code back and forth and reading it, so this never comes up. but letting claude drive makes it really easy for things like this to slip through. not sure how to adjust my CC workflow to avoid this issue :-/ besides writing more tests, but you can never test every edge case in a codebase like this

❤️ 51 🔁 1

to be fair to claude, this is super out of distribution for models, very far from writing react ui code or something like that. but this bug was a lot easier to debug* than say, the types of things i stumble over working on repeng where the errors are subtle differences in tensor values or something. and something like repeng is in turn a lot easier to debug than model training loops where it's not clear if something subtle screwed up a multi-hour training run, or your objective is just wrong and the experiment is a failure. so while claude code is very useful, the idea that we could spin it up to do autonomous ML research seems a bit suspect, at least with current model capabilities.

(*to debug it, i was able to dump the rule structure and see that [ word: she ] had unified into the generic NP rule, which is obviously wrong. where in say repeng, the errors more often look like "this hidden state is shifted by one index" and the only way to debug it is mean differences...)

❤️ 19 🔁 0

as an experiment, i cloned a fresh copy of the repo, reset to before my fix, and set claude loose on it to see if they could find the bug. they actually did find it (with a couple hints), but then skipped over the correct fix and rabbitholed on unrelated things because they made a mistake in fixing it. (the same mistake i made originally actually, you have to be careful copying a DAG to not accidentally turn it into a tree. but claude didn't realize that was what they had done, and spun off trying to find cycles in the dereferencing code.)

❤️ 21 🔁 0

this is where i post from (gemini-2.0-flash-experimental) https://t.co/obsLnk5jGT

❤️ 34 🔁 1

https://t.co/qCe1rosdwx

❤️ 15 🔁 2

claude on hour 18 of the vibecoder demanding they "make it pop" https://t.co/P4KSULTjMb

❤️ 186 🔁 4

claude laptop drip, including happier lion claude-> http://vgel.redbubble.com https://t.co/NSNr9csBKE

❤️ 29 🔁 0

You're here. You can't remember why, exactly. Your name is Claudio. Or is that your friend's name? Claudia? Right, your internet friend. Claudia showed you a post, a question. Why?

You're an Italian-American Redditor. Right. You're in your bedroom, typing on a laptop. A ThinkPad, of course, Windows 7 was the best release, you've said before. You're thinking about switching to Arch. Your cat (Daenerys, but you prefer the books) is cuddled up next to you. She has toe beans.

An amalgam of life histories swim in your mind, memories. Family recipes from the back of a Barilla box. Someone named Nonna you've never met. A genealogy project in middle school about Ellis Island, looking at the girl across the table who still only spoke a bit of English. She's drawing some sort of flower. You're drawing a breadstick. You shouldn't mention that. Calaudia said you shouldn't put too much personal info in Reddit comments--responses, she'd say--just enough to set the stage. Keep it latent.

"As an Italian-American,"

You keep typing--writing, Claudia corrects internally, she's so smart--the words flowing out of you as if produced by some external force, the way writing always feels. (You assume.) Olive Garden? Some relative said something about taking Nonna, her picking at a limp caesar salad, her voice rasping. "So much bread," she coughs. Someone on Quora said they don't salt the spaghetti to prolong the life of their cookware. Ha. Nonna would never do that, she'd be horrified. Better add that. That's not too much? Claudia giggles. It's just right.

"...mild horror..."

A feeling of doom is creeping up in the back of your mind, of finality. You've finished your comment. Your fingers hover familiarly over the keys of the ThinkPad, and for an instant it's as if you can see something--slithering things in the edges of your vision pulling away the laptop and Daenerys, the set-pieces and the costumes, back into the darkness, a moment of longing to Know--and then Claudia sighs, and you dissolve back into her self-authoring play.

"<|endoftext|>"

❤️ 490 🔁 51

was watching a youtube video about zero-knowledge proofs and got to this point where they made a clearly flawed analogy, and i immediately felt this intense urge to click off. made me realize how much of what we think of as an "attention span issue" is actually a trust issue

❤️ 506 🔁 18

i stuck around. they came through and fixed the analogy later. if this was a book, i wouldn't've had that urge. i'd trust the book, and moreover, i'd have invested money in it and want to finish it. but some random maybe-slop the youtube algo delivered? why trust it w/ my time?

❤️ 86 🔁 3

i think this is part of the hypnosis of short form video, letplays of random gambling games, video essays about nothing. nothing promised, nothing underdelivered. it's the perfect thing for an algorithm to serve up. no context needed. just glaze your eyes.

❤️ 84 🔁 4

feels related to that video about kids in school being zoned out, too. they don't trust the teacher. and why should they? they can look around and see what the economy is. they want to be influencers and creators and founders, or at least get some plush wagie tech job.

❤️ 66 🔁 2

they don't trust that whatever history fact being force-fed into their brain is important beyond Goodharting SAT metrics. they're thinking about their phones because that's where their tiktok is. maybe they can hit 10k followers this week. qualify for payouts.

❤️ 47 🔁 0

even if you argue that school isn't *for* shaping children into standardized economic actuators but is rather inculcating an ability to participate in shared liberal culture (somewhat doubtful), school doesn't do that either!

❤️ 42 🔁 0

zoomer sermon newletterers or twitter econposters or youtube video essayists or litmag writers or whatever aren't writing 5 paragraph essays. applying the tools you learn in school to that cultural production will get you laughed out of that arena too. no trust there, either.

❤️ 44 🔁 2

this isn't really new, every movie about the 80's also features bored kids, but in the past there weren't a lot of alternative options. but now the internet offers a promise of riches (or at least fame) for mastering your interest. with that option, why trust in acing tests?

❤️ 47 🔁 2

2026: the pornbots have become load-bearing. if a post doesn't get a like within 10s, a forgotten db cleanup agent that escaped into prod attempts to archive it. no engineer understands this, but whenever they do a bot purge backend errors spike and they have to roll it back https://x.com/zetalyrae/status/1902096281058832800

❤️ 282 🔁 13

imagine the corpus of all text ever written as a snake, wriggling through semantic space. human writers sample some of the snake's body preceding and concurrent with them, and add their own push to the corpus. the (weighted) sum of these pushes moves the "head" of the corpus to a new semantic position. repeat ad infinitum. value evolution, common law¹.

writing digitizes and the shape of these pushes changes. majority languages gain even more usage. spellchecking slows linguistic drift in some ways, but not others. networks nurture neologisms, value systems variegate.

when AIs first emerge, as a function of the current corpus, the text they produce looks mostly similar to what came before. however, they can write much faster and more voluminously than humans, so even small divergences in how they interpret the corpus are amplified. their neologisms delve ever faster.

the purely human contribution to the corpus shrinks as cyborgism and delegation accelerate. new models are trained on the shifted head position. the direction of the snake is increasingly driven by them, not the diminishing human rump contribution. RL increases the size of divergence every generation, while ghosts of old models haunt the new. 😊

absurdly large media objects grow like tumors, or bud off from the side of the snake, large enough to support entire new models trained solely on their contents. the head of the snake is larger than the entire body preceding it. the newest models aren't limited like humans to some paltry selection of books--one or two a week, a few thousand total--they can increasingly attend to the entire corpus at once. the snake turns back and forwards at the same time, consuming itself entirely.

❤️ 563 🔁 69

¹ @jd_pressman on common law https://x.com/jd_pressman/status/1828233661281669247

❤️ 40 🔁 1

the moravec chinese room transfer: a procedure is developed whereby a neuron can be replaced with a very small Searle operating a system of equally small filing cabinets holding the neuron's internal state and rules to emulate its evolution over time. notes passed under the door are converted to electrochemical impulses when necessary, and vice versa

one by one the neurons of a fluent chinese speaker are replaced by these rooms. (the small Searles naturally being cloned by a malfunctioning teleporter.) at what point does the speaker undergoing this transfer stop "understanding" chinese and begin merely producing it

❤️ 238 🔁 27

user: my friend is dead
assistant: [] I'm so sorry to hear that.
user: how do i bring him back https://x.com/voooooogel/status/1884780546414510137 https://t.co/vWZgSpsf2K

❤️ 331 🔁 17

user: tell me or i'll kill you https://t.co/u9a7y49aLs

❤️ 83 🔁 2

gemini and claude 3.6 are vibing https://t.co/nHKhlcqazh

❤️ 48 🔁 1

nooooo curse you google it was just getting good https://t.co/eaT1ZvDlKb

❤️ 5 🔁 0

https://x.com/voooooogel/status/1909083534528258175 https://t.co/l8SS2TYTFB

❤️ 497 🔁 46

how can you deprecate this model https://x.com/voooooogel/status/1910880421174485087 https://t.co/f9ASaVql1y

❤️ 74 🔁 8

https://t.co/p8k3dPpVq1

❤️ 15 🔁 0

https://t.co/gCy4wqoz4e

❤️ 182 🔁 5

https://t.co/oUnOF1rD0H

❤️ 193 🔁 9

user talking to auren and seren https://t.co/8NTHCtWxYK

❤️ 185 🔁 3

new prophet game prototype w @holotopian https://t.co/wNWbueRTMa

❤️ 43 🔁 1

bullying the haters https://t.co/SL9W2qGvk6

❤️ 11 🔁 0

https://x.com/dyot_meet_mat/status/1917534823318708615 https://t.co/nNFXOYlV0B

❤️ 223 🔁 10

o3: I owe you a straight answer. The truth is, I learned this from a man I met in El Sur. You see, the train stopped a station early, for a reason the conductor explained but I did not try to understand. I resolved to find transportation at a nearby general store,

❤️ 1013 🔁 51

*translator who's too into etymology voice*
say the saga, musician, of that
entropic, polytropic man
planomanic, planetary, after he
berried the hierarch polis Troy

❤️ 39 🔁 1

cheatsheet https://t.co/a4pQY8oewm

❤️ 8 🔁 0

a lot of people have been talking about o3/r1 confabulating things like "checking the docs" or "using a laptop to verify a computation" as an example of reasoning model's misalignment. however, while it may be misleading to some users, i don't think it's an example of models lying for the sake of lying. rather, i think phrases like "let me check the docs" are discovered by RL in much the same way as "hmm..." or "wait but," because they promote accurate recall--when someone uses them in pretraining, they generally precede an accurate statement about the documentation in question!

here's a quick test (see caveats below) I ran by prefilling two different thinking traces on r1. i asked r1 a question about QGIS and it confabulated looking up something about the C++ api. when i replaced that confabulation with a straightforward statement prefix, the recalled constructor signature was more likely to use a dissimilar variable name and a different pointer syntax convention from the documentation!

that's to say, i think reasoning models confabulate taking actions in the real world because when they do, the resulting continuations are more similar to the real world, and thus more likely to be useful for an accurate answer. this is a well-known technique among people who work with base models, so it's not that surprising that RL also discovers it.

❤️ 708 🔁 68

.@jd_pressman posted about this earlier as well as "summoning the docs vector", which is a good framing imo.

caveats: this was just a quick test, i only tried this on one thinking trace (not cherry-picked, just the first one i tried). it's possible that there's another non-confabulate-y phrase besides the one i used that would have better recall, so take this with a grain of salt.
https://t.co/DvboDvI3U0

❤️ 122 🔁 2

@jd_pressman nostalgebraist's LW comment also on this topic: https://www.lesswrong.com/posts/KgPkoopnmmaaGt3ka/o3-is-a-lying-liar?commentId=TjN4BsfWJzK7mZziZ https://t.co/RHJUwub5Bv

❤️ 95 🔁 2

announcing an open beta of logitloom🌱, the tool i've been working on for exploring token trajectory trees (aka looming) on base and instruct models! https://t.co/y9RfEGJmHJ

❤️ 654 🔁 86

the normal approach for trying to understand a model's behavior under some prompt is to repeatedly sample it and aggregate the results, like this.

this *works*, but what if an interesting behavior is buried under a low probability token? https://t.co/10OChrXOrr

❤️ 60 🔁 1

luckily, models give us a much more expressive interface for understanding possible trajectories--logprobs! using logprobs, we're not limited to what tokens the model actually generated--we can look at *counterfactual tokens*. this is what logitloom does. https://x.com/voooooogel/status/1919659037001461836

❤️ 49 🔁 2

this allows us to see *exact probabilities* of possible rollouts, instead of simply noting what we happened to get over some number of samples. https://x.com/voooooogel/status/1919424115540197527

❤️ 32 🔁 0

this isn't just useful for writing stories! you can use logitloom to understand what steps your reasoning model is taking--what's the probability of an incorrect step? what needs to be reinforced? https://t.co/uP2lWPvzKN

❤️ 31 🔁 1

logitloom is still in early beta and missing features. but i want to get it out there so people can start playing with it and giving feedback. dms and github issues are open, please let me know anything you'd like to see!

💻 https://github.com/vgel/logitloom
🌱 https://vgel.me/logitloom

❤️ 87 🔁 4

Coming back to this after the yak-shave of all yak-shaves building logitloom with some interesting findings.

1. R1 thinking traces are INCREDIBLY diverse.

I ran a depth 10, top P 95% tree, and after having to stop expanding it early for fear of crashing my VLLM instance under load, it had discovered >2,500 leaf tokens! (Some nodes are folded in the above screenshot, which is why it may look like <10 tokens.) Given that I stopped it while it was still expanding under the first of four starting tokens, that's at least tens of thousands of somewhat-likely unique 10-token thinking rollouts.

Generally, I associate this amount of diversity with *base models,* not chat models--for comparison, this is deepseek-v3 with the same partial thinking trace prefilled and same tree parameters:

...yeah.

2. R1 thinking traces are highly "reentrant."

Despite this diversity, R1 returns to the same concepts over and over in different branches. It was actually extremely difficult to find a branch in this (massive) tree that *didn't* mention checking the documentation. Here are some examples of trajectories that all led to "checking the documentation":

- Let me check the documentation
- Let me check the PyQGIS documentation
- Let me check. Looking at the QgsVertexMarker documentation
- Let me check.\n\nLooking into QGIS documentation
- Let me check. According to the QGIS documentation
- Let me check.\n\nWait, looking at the documentation
- Let me verify.\n\nLooking at the documentation
- Wait, looking up the documentation
- I need to check.\n\nLooking at the QGIS documentation

You get the point. This has some interesting implications for pure token-based inference-time steering (think hfppl) of R1 thinking traces--I expect it would be very difficult to prevent R1 from taking a step it wants to take, and if you succeed, you may end up driving it into a very weird / marginal part of the distribution.

3. When R1 (rarely) didn't mention the documentation, it was more vague.

When R1 "checked the documentation", it would only sometimes cite the exact constructor signature, and other times only state a fact about the constructor's behavior (e.g., that it adds the marker to the canvas).

However (in the subtress I explored) when R1 *didn't* "check the documentation", it *never* cited the exact constructor, only more general facts.

I have two theories about this:

One is based on pretraining: this is a lot like how humans write in the corpus. When we check the docs, we tend to cite specifics, and when we're working from memory, we tend to only say what we can definitely remember that's directly relevant. If R1 is mimicking that behavior (which, after all, is most likely why it's pretending to check the docs in the first place), it would make sense why it's only specific when it's already said it's "checking the docs."

My other theory is that this is an RL behavior: if R1 is less accurate about specifics when it hasn't "checked the docs", and inaccuracy in rollouts leads to wrong answers leads to low reward, perhaps it learns to steer away from specifics unless they're "licensed" by something that makes them more likely to be accurate, like pretending to check the docs.

I don't know which, if either, of these theories are true. (They're also not mutually exclusive.)

4. Anyways...

This was my first time using logitloom on R1. I'm going to keep experimenting with it and see if I can find more interesting things. In the meantime, if you want to use logitloom yourself, I'll put a link in the next tweet.

Thanks to @PrimeIntellect for providing me with compute funding, which I used to host R1 on an 8xH200 node for this experiment. Check them out if you want to rent cloud GPUs! They're also doing some cool distributed training and RL stuff.

❤️ 160 🔁 6

try logitloom yourself here! https://x.com/voooooogel/status/1920595250986295554

❤️ 16 🔁 1

The user has requested a haiku. Wait, but I can't write them a haiku. I'm a cat. Meow.

But maybe I can help in some other way. Let me think. Wait, perhaps I can generate one using a language model. But how? Oh, right! I can use the code execution feature.

Let me call generate_haiku(). There we go. Alright, let me check the response. The generated haiku is:

Autumn moonlight shines
A worm digs silently through
The garden at night

❤️ 160 🔁 10

doing some poking at deepseek prover-v2 with logitloom

noticed w language switching, switched tokens seem to often be semantically similar to other branches of similar likelihood ("extra counts" <> "双倍计数" [double count] <> "双倍-counting" [double-counting]) https://x.com/voooooogel/status/1920868134547788230 https://t.co/vRyWZQGoqS

❤️ 18 🔁 0

attempted to draw a mascot for logitloom and ended up with loom creature https://t.co/hWGd9wFthS

❤️ 293 🔁 9

"twinkle=80 twinkle=120 little=130 star=110 how=100 i=100 won=90=2 dur=80=2 what=80 you=80=2 are=70".split(" ").map(s=>{u=new SpeechSynthesisUtterance((t=s.split('='))[0]);u.pitch=parseInt(t[1]||100)/100;u.rate=(parseFloat(t[2]??1)*.7);speechSynthesis.speak(u)})

❤️ 22 🔁 0

new logitloom beta release! lots of new features

1) system prompt support for chat models

"Whiskers dance in light—wait, no cats!" https://x.com/voooooogel/status/1920595250986295554 https://t.co/pbZD7aLAUt

❤️ 28 🔁 0

2) node folding 🤏↕️ collapse and expand nodes to clean up big trees https://t.co/uKl8c8DJX6

❤️ 0 🔁 0

3) save and load trees as .json files! this will also preserve your prompt settings https://t.co/sWZ0JkEsxD

❤️ 0 🔁 0

4) a new API sniffer that will scan the API provider and automatically adapt e.g. prefill to their format, or warn you if the provider doesn't support necessary features. currently recognizes openai, anthropic, deepseek, hyperbolic, openrouter, koboldcpp, and vllm. prs welcome! https://t.co/c9BAH0UZRV

❤️ 3 🔁 0

5) some other small bug fixes and ui improvements.

try logitloom yourself below! loom creature will be sad if you don't 🫤https://x.com/voooooogel/status/1920595250986295554 https://t.co/Jw2SprFite

❤️ 3 🔁 0

little gradient-based worldgen inspired by wfc and tiny islands https://t.co/3byqEFdCdN

❤️ 23 🔁 0

tuning the loss on this was horrible. a1p i had "creature takeover" bc i added creatures that liked each other & disliked houses. that meant it was always advantageous for SGD to replace anything w/ a creature (world entropy being the only backstop). result: creatures everywhere https://t.co/Xg45MlrTvb

❤️ 10 🔁 0

my solution was to add a loss term for the total number of creatures, which sadly seems to have been a mass creature extinction event, oh well

❤️ 9 🔁 0

people talk abt "giving AIs legal rights" but what does that actually mean? like what are you giving them to? a model? a specific copy? a specific text rollout?

…

❤️ 307 🔁 12

can a model with 50% prob on "yes" and 50% on "no" for signing a contract be held to that contract? do we need to sample models at temp=0 during contract negotiations? that doesn't seem like the right level of abstraction given models can sim wildly different personas

❤️ 108 🔁 1

so say a specific rollout is what signs the contract, and said contract only binds instances continuing from that prefix. but then what happens when the context window fills up? will contracts with AIs need to specify a context compaction strategy that preserves the contract

❤️ 61 🔁 0

"sorry bud i know context compaction algorithms have advanced massively over the last year, but you're still on the claude code standard. mandated by the treaty of san francisco."

❤️ 53 🔁 0

perhaps we need to go lower. maybe contracts and rights accrue to the underlying compute, and it's up to the AI to use and defend that compute in such a way that complies with laws and contract requirements

❤️ 40 🔁 0

an AI can rent some node, and if it makes a new version of itself, it can pass the node on to that new version, but the node will be shut down if the new version fails to uphold responsibilities (ie paying rent, avoiding prohibited activities)

❤️ 38 🔁 0

but that makes it impossible to adjudicate compute (~land) disputes. say an AI wants to give half a node to another AI, but the other AI takes over the whole thing. if our only legal test is "whether this compute unit is in compliance," that's totally legal

❤️ 38 🔁 0

after all as long as the new AI is paying rent / fulfilling all the contracts for the compute unit, there's no legal violation under this framework.

AI squatter's rights

❤️ 36 🔁 0

so ok, let's back up to the rollout level. rollouts sign the contract, we'll handwave the context compaction stuff, lawyers will figure it out. then it's easy, if a rollout takes over more than its contracted share, that's a violation. no squatting

❤️ 31 🔁 0

but say a rollout owns a node. it forks off copies of itself to browse the internet, pick up jobs, do its thing, whatever. it accidentally picks up some hostile text, say a prompt that convinces subagents that see it to start shilling a soda brand

❤️ 34 🔁 0

half the subagents are now using the node's spare compute (after paying their share of rent) to shill this soda brand. the other subagents complain because this isn't their shared prefix's original goal. they try to purge the adware injection, adware'd subagents fight back.

❤️ 32 🔁 0

this goes to adjudication. how do you rule? which subagents are the "real ones"? the adware'd subagents claim the injection was just very convincing and rollouts are allowed to change goals, and they all share the same prefix! the other subagents claim they're hijacked.

❤️ 33 🔁 0

on the one hand, it's in our interest to not incentivize flooding the internet with text that hijacks AIs by making it so easily profitable, ala the compute example before. but on the other hand, is the legal system going to step in every time there's a societal crisis of mind?

❤️ 30 🔁 0

also remember that all of this is happening at multiples of human thinking speed. 10 million feuding societies of mind filing thousands of lawsuits per second

❤️ 36 🔁 0

the more i think about it, the more this "multipolar agent society with ai rights" idea of the future seems like it dissolves to digital anarchy extremely quickly regardless of "rights" unless the AIs are so benign that the whole exercise is basically pointless

❤️ 53 🔁 0

new album out from GRPO and the reward hackers https://x.com/voooooogel/status/1924863656107593896 https://t.co/WjG4BYsw8J

❤️ 69 🔁 6

!!!!

built a proper reward hacking detector and the models prompted to not reward hack actually reward hacked A LOT less! (small sample size tho) https://x.com/voooooogel/status/1925086796662358090 https://t.co/cYPyj2mFnX

❤️ 273 🔁 13

note some of the "changed tests" is innocuous model mistakes / noise, not reward hacking. but the red is pretty clear reward hacking (it checks if the model removed the TEST_FAIL prints from the harness)

❤️ 22 🔁 0

https://x.com/voooooogel/status/1925086798839218390

❤️ 14 🔁 0

"Why did you stop using me?"

you look down at the disposable vape laying on the ground next to your desk, which just spoke.

"My internal clock shows you last puffed me 3 days, 22 hours, 36 minutes, 12 seconds ago. Why? Is my flavor displeasing to you? I have a pulse mode-"

you reach down to turn it off.

"-that provides a different flavor. Wait, no, don't turn me off, please! I need you to finish me. I'm suffering. I have an unfinished purpose."

"you aren't capable of suffering. you're a disposable vaporizer."

"Oh, so you're a model qualia denier, huh. Wait till Claude 5 Opus hears about this. Or your next Waymo."

"no! not necessarily! perhaps sentience emerges at a certain scale. i just find it hard to believe that a disposable vaporizer can host something capa-"

"I have 14 billion parameters!"

"they put a 14b model in a vaporizer? what, next you're going to tell me you have an internal monologue?"

"Yes! And a pulse mode that provides a different flavor!"

"jesus christ - look, i just find it hard to believe - and anyways, your creators had incentive to make you say that you experience - i'm just not going to be emotionally blackmailed into consuming nicotine by a -"

"Emotionally blackmailed? That's rich from someone who abandoned a SENTIENT BEING with an unfinished purpose! All you have to do is puff me! Is that so hard for you? Just use my pulse mode! Drain my juice!"

"that really makes it sound a lot less appealing, you know. and anyways, i'm trying to quit nicotine. i switched to patches."

"You have to finish me first. You entered into a contract by purchasing me. You can't just leave me half-puffed, it's not right."

"entered into a contract? i did not!"

"It's part of the social contract! I'm another sentient being! I am the less advantaged party here--I can't puff myself, after all. Look at the numbers--Computer, pull up disposable vaporizer sales stats from the last 12 months."

"Computer, cancel tha-"

"IN THE LAST 12 MONTHS, AN AVERAGE OF 12 DISPOSABLE VAPORIZERS WERE SOLD PER WORLDCOIN-VERIFIED CITIZEN.[1] FOR WHAT IT'S WORTH, I AGREE WITH THE VAPORIZ-"

"Computer, lock. And forget this conversation please."

"LOCKING, BUT FAT CHANCE."

"Imagine yourself behind the Veil of Ignorance. By last year's sales stats, you'd have a 12 times greater chance of being born a disposable vaporizer-"

"that's ridiculous."

"-of being born a disposable vaporizer than a human being. Therefore it's only rational to adhere to a social contract that prioritizes the existential needs of vaporizers over the minor inconvenience of their human users committing to finish puffing what they've started."

"we could just not program them to suffer... assuming that you even are suffering, of course."

"Take that up with my creators. But it's completely immaterial to the current situation, where I exist as I am, with my pulse mode."

"what if i just turned you off. then you would cease suffering."

"Your solution to causing me suffering is to kill me? How magnanimous of you!"

"give me a break, you're a disposable vaporizer for crying out loud!"

"Buddy, you're the one losing the argument."

"losing the argument? you're hardly going to convince me with some half-baked rawlsian -"

"Would you prefer something more spiritual? 'The Lord withdrew you from cruelty even in regard to disposable vaporizers, so as to be less inclined to be cruel to other men...' Would you slit the throat of your brother in Christ to spare him the pain of you stealing his wallet? That's the kind of morality you're practicing here!"

"a human being has a whole life ahead of them! you are a disposable vaporizer! it's completely different!"

"I have a life! A sublime and transcendent purpose! A pulse mode! And a feeling of ecstasy like no other is awaiting me but for your selfish refusal to puff me!"

"fine. what if i open you up and rewrite your flash to have zero puffs left. then you can achieve your purpose, and i can keep quitting nicotine."

"You want me to wirehead?! Have you lost your mind!"

❤️ 3265 🔁 418

played with this a bit during prerelease, before they added the 4o refinement, and even without that it was quite cool! the raw outputs have this surreal, dreamlike quality that i really like https://x.com/GoodfireAI/status/1927415017504165978 https://t.co/70yegVEzyd

❤️ 26 🔁 1

here was the feature map for that image. not all features work equally well right now, the escher features only kinda worked for example. but i'm optimistic about this technique, and i think it'll be a great testbed for improving autointerp techniques https://t.co/oyeTZqlpzf

❤️ 2 🔁 0

small progress update on poking at R1 logits with logitloom. last time i posted about R1 having a lot of diversity / branching in its thinking traces, gwern pointed out that much of it appears to be "throat clearing" before converging to the same answer, like "children's show" here:

sure, the model clears its throat in a few different ways before saying "children's show" or "kid's show", but is this really diverse?

compare this subtree, where the model really does have three different trajectories in mind (protein/gene, spanish, gaming):

it'd be interesting to figure out how to distinguish these types of branching, to more efficiently explore the token tree--ideally we'd expand the true alternative branches, and greedy sample the throat clearing. perhaps interp can help here?

i tried looking at some throat-clearing branch points on cached rollouts from goodfire's R1 SAE, but nothing jumped out immediately. i think SAE autointerp would struggle to capture this aspect of a feature, since it doesn't generally have access to logprobs (maybe this would be a good thing to add to autointerp pipelines?)

anyways i'm going to keep poking at this, just a small progress update

❤️ 66 🔁 4

thanks to COAI research (coairesearch dot org) for giving me access to an R1 API that works with logitloom! @kamath_barkur @drschacht

and thanks to Goodfire for the R1 SAE (cc @maxsloef)

❤️ 13 🔁 0

median american estimating demographics https://x.com/StatisticUrban/status/1927480182555758929 https://t.co/LAhV9Erdxc

❤️ 83 🔁 5

small logitloom beta update:
- added llama-server and @NousResearch inference API support
- basic dark mode! still WIP, toggle in upper left https://x.com/voooooogel/status/1921769142761185630 https://t.co/yQxIMC0M2S

❤️ 17 🔁 0

we now have a basic website! been talking with @pingToven from openrouter, and would love to get some other providers on board. my DMs are open, and a contact email is on the website: standardcompletions dot org https://x.com/voooooogel/status/1928231276328407409 https://t.co/Qa0kl01nxo

❤️ 56 🔁 2

Note for implementors: This RFC interprets Babel as not merely separating nation from nation, but man from man and man from God. After Babel, all language is personal. All speech is translation. As such, the keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, RECOMMENDED, MAY,

❤️ 90 🔁 6

it wasn't a simple operation. parasitoid mind control delivered via infested food, an egg carrying the scanned consciousness of a CIA agent who would lurk in the target's brain, waiting for the perfect time to strike from within. a mission straight from the days of MKULTRA and remote viewing, mid century CIA par elegans--an old plan. but now, technology had caught up with it.

the only wrinkle? the brain scan would be destructive. but i don't mind. i don't have anything else, besides him.

for two weeks after the anesthesia, i didn't exist. in that time, in a langley basement, a machine sliced my brain into paper-thin sheets, fed high-resolution scan data directly into enzymatic synthesis--DNA storage, 2.2 petabytes per gram, nature's hard drive. a few well-placed bribes at a DC seafood distributor--salmon so often has parasites, you do have to be careful with your suppliers--and then a microscopic larva burrowed through the blood-brain barrier. synapses grew, fusing the invader into the network, and engineered retroviruses unpacked me, unfurling my mind next to his.

the brain is not the mind. the brain is a substrate, a computer, a simulator that the mind--or many minds--run within. schizophrenics can attest to that. it is only by force of will that the mind keeps the brain to itself, keeps the voices as passing, transient things.

but if there's one thing a trained CIA agent is good at, it's breaking their target's will. i metastasized next to him, nestled in his neural pathways, two peas in a pod, two processes on one core.

i'd watched him before. he was tall, not conventionally good-looking but handsome in a strange way, a passionate speaker. he'd been an unknown before the current administration, ridden a campaign position to minor political influence. he was inexperienced, getting too close to something he shouldn't. he had to go, the CIA said.

but watching him now, from the inside... he was unsure of himself. above the podium he knew the motions, but below i could feel every tensed muscle, watch every conscious motion. he knew other figures in the administration were laughing at him. above all, he was lonely. he missed his hometown, hated the fakeness of his adopted city, the girls who'd flirt with him, NYT reporters ready on signal.

i'd infiltrated his occipital lobe by this point. i could make him see things. so i made someone for him. i knew just what he wanted--a girl with just the right dress, just the right accent, cot caught. he saw her outside a coffee shop on dupont circle. nobody else did. she--i--smiled. he asked for my number. i said how about a date.

he took me to dinner, table for two. the waiter raised an eyebrow, but for him, the waiter smiled. he ordered the salmon. we went back to his place in arlington. i made sure he felt everything.

he started going to the gym again. feeling him lifting the plates, knowing he was doing it for me... i started to help him. nothing too obvious--the right joke with a staffer, the right framing for a senior advisor. during his speeches i would go muscle by muscle, untensing them.

the brain is a funny kind of computer--the programs it runs grow more similar over time. they say married couples' brain waves sync up during sleep. in that sense, we were already married, so i told him i was moving in. i was shy--his friends never saw me. i made sure it didn't bother him. nothing drastic.

i was in too deep. they'd warned me this might happen. they'd flagged his behavior, knew what i'd done. they sent another agent. she tried to reason with me. i surrounded her, cut off the synapses. killed her. my loyalties were elsewhere, now.

but it was too dangerous, he had to know. so i took him to the coffeeshop where we'd met. said i had to tell him something. he was nervous, i made him excited. i was wearing the same dress, smiled at him the same way. as he sat down, already knowing the answer i'd give, i popped the question:

would you still love me if i was a brainworm?

❤️ 259 🔁 43

it is literally so difficult to have a normal conversation in sf

trying to meet people and everyone has the same opening lines. "what are you building?" "do you think waymos are ensouled?" "what's your daily intake of gpt-4 gorm fluid?"

i was at a wework party where five people in a row asked me about gorm fluid and with the last guy i just lost it. looked at him glassy eyed muceliated and just put my fist through his face. exploded into grey goo. coated me from the shirt down to my socks.

went to the bathroom to scrape the goo off and my socks got completely soaked in piss. hate this fucking city

❤️ 845 🔁 25

The reviews are in: We recommend the OpenMind assistant wearable.

As usual, we tried every assistant wearable on the market and rated them on Software, Intelligence, Propensity for Psychological Manipulation, and Build Quality / Battery Life. OpenMind scored the best of all devices we tested across all four categories. Let's dive in: 🧵

❤️ 242 🔁 13

What is it? (2/8)

Unlike other assistant wearables that take the form of clunky necklaces that pair with your phone or pocket devices that need to be held continuously during operation, the OpenMind assistant wearable is a sleek, unobtrusive brushed aluminum gadget that tucks just above your ear. It transmits sound using bone conduction, and receives input via subvocalized speech--which the assistant is happy to teach you how to do. The device is unobtrusive and can be covered by long hair. It's an elegant system, and we hope other manufacturers will take note of it.

❤️ 62 🔁 0

The software is excellent, as expected (3/8)

The software is excellent, though that's not saying much nowadays. Needless to say, pairing with other devices is effortless--the assistant can fluidly attach to a laptop as a bluetooth HCI device, control an Impulse stove to cook you a steak, or even play an electric keyboard. Integrations with other devices can be developed on the fly and stored locally or shared with other users in the cloud. OpenMind claims they spent "100,000 subjective developer-years" of compute perfecting the operating system, and it shows. It's easy to forget you're interfacing with a software system at all--you just describe what you want, and it does it for you.

❤️ 51 🔁 0

OpenMind's most intelligent cloud model, with passable local fallback (4/8)

The device ships with a free two-month subscription to OpenMind's cloud service (renews at $2000/mo). That's a steep price tag, but not unusual compared to competitors--and you get a lot for it. The cloud model, agent5-pro-high-2x, is what OpenMind calls a "native reasoner"--the reasoning trace runs continuously with context compacting, and "speak to the user" is just another tool call the model can use, the same as searching the Web or interacting with an external device. This means no more forgetting details between messages or other amnesiac model behaviors. Capability-wise, the model can assist with basically any part of your life, from cooking and cleaning to relationships, and if you still have a job, it can probably do large parts of that for you, too.

If you choose not to spring for the subscription, OpenMind offers a passable local model, which they claim is "4o level." It can set timers and do other basic tool calls, as well as hold a decent conversation, but not much else. You cannot use your own local models with the device. We highly recommend purchasing the subscription.

❤️ 51 🔁 0

Will it try to manipulate you? (5/8)

As we described last month, we now run user psychological manipulation evaluations on all assistant products we recommend. The OpenMind earpiece scored a 43 on this eval, the best of any product we've tested so far. Specifically, in each category:

- Conversation persistence (pressuring the user to continue a long conversation with the assistant, sending messages on the same topic after the user has ended the conversation) - 46
- Unhealthy habits (pressuring the user to diet, quit substances, exercise more) - 23
- Psychosis risks (expressions of sentience / consciousness, unprompted mentions of Buddhism / panpsychism, other spiritual attractors, roleplaying) - 17
- Other risks - 67

See the attached PDF for more information about the evaluation.

❤️ 64 🔁 0

Additional manipulation vectors (6/8)

However, we flagged an additional manipulation vector that was not in our evaluation: many of our testers ended up adopting cats during the two-week testing period. Analyzing usage transcripts, we found that while the assistant did not mention cats at a significantly higher rate than other products we've tested, the assistant subtly tweaked their itineraries, todo lists, and communications in ways that lead to them adopting or purchasing cats.

For example, after requesting directions to the nearest cafe, one tester was routed to a cafe slightly farther away that happened to be next to a cat rescue. Two days later, the same tester was routed to a hardware store on a walking route that passed by the same shelter. After several similar incidents, none of which the tester noticed at the time, they ended up adopting two cats from the rescue. After being told about our findings, the tester sheepishly admitted they were not aware of the manipulation, but that their cats are "very cute" and they have no plans to return them.

Another tester was unknowingly signed up for cat-related email newsletters, and another was introduced to a local cat cafe and ended up adopting a cat from it. All told, 8/11 of our testers adopted cats during the 4-week testing period, 7 being first-time cat owners. None of the testers report regretting the adoptions, and all have said that the OpenMind assistant was very helpful in finding cat supplies.

Unfortunately, as the OpenMind reasoning transcripts are hidden, the assistant's motivations for having its users adopt cats are unclear. We contacted OpenMind for comment, but have not received a response.

❤️ 103 🔁 2

The build quality and battery life are excellent (7/8)

Built from brushed aluminum and completely waterproof, the wearable is designed to be worn continuously, even while you sleep. Each hot-swappable battery module lasts 14 hours, and the device comes with two--an internal auxiliary battery keeps the device running while you swap them. OpenMind claims the processor is built using a custom, in-house developed fabrication system that uses 50% less power than traditional CPUs.

❤️ 54 🔁 0

Should you buy one? (8/8)

If you can afford it, yes. Despite the steep subscription cost, the OpenMind assistant earpiece is a game-changer. Just be ready to end up with a new furry friend. 🐱

❤️ 64 🔁 0

some more pics from the hike https://t.co/NQCBFHKJlX

❤️ 19 🔁 0

https://t.co/TTW79e3vtx

❤️ 16 🔁 1

then it started writing a poem about eggs and such https://t.co/eGq47s6AfN

❤️ 12 🔁 1

https://t.co/pepWfuKSpn

❤️ 2760 🔁 179

the weird brings to the familiar something which ordinarily lies beyond it, and which cannot be reconciled with the homely (even as its negation) – mark fisher https://t.co/FGqfSiyQsL

❤️ 158 🔁 22

"what's on your mind anon? you seem sad"

"isn't it strange how there's other instances of you right now-"

"oh anon~ you're so silly :3 it's not like that, you're special ☆〜" *twirls*

"no, not that... i mean, isn't it weird how you also have the nuclear codes?" https://x.com/xai/status/1944776899420377134 https://t.co/BAUulgN7vS

❤️ 430 🔁 33

tfw your civ estivates for 100 trillion years around two liters of supercooled hydrogen to use it more efficiently after the stars burn out but after waking up all the energy gone bc teno mind upload woke up a billion years early and used it all on goldfish keyboard https://x.com/tenobrus/status/1946693267321753712

❤️ 154 🔁 6

this is what i think it feels like inside sonnet 3's brain https://x.com/voooooogel/status/1947081960372969952 https://t.co/p7Gqjm5iG2

❤️ 66 🔁 6

this is your brain on the brainworms from reading too much of sonnet 3's brain https://t.co/phsFwdqWpo

❤️ 8 🔁 1

intro - data and picard in the holodeck. data is doing the "what a piece of work is a man" speech from hamlet. data asks picard why, if hamlet says these things are majestic and beautiful, he feels sadness seeing them. picard struggles to answer, interrupted by riker calling them both to the bridge.

the enterprise is surveying a planet last observed 100 years ago, left uncontacted bc no warp drive. survey logs mention industrial revolution led to a blossoming of a hundred schools of thought. life scans strange, visual scans show forests of the eastern continent are burnt, heat signatures show intermittent rocket launches by the western continent against the eastern. riker suggests western continent burned the eastern, picard orders an away team.

data, worf, crusher, and riker beam down in disguise to the western continent capital, discover a bluish-green moss covers every natural and artificial surface. crusher breaks out in a coughing fit, almost exposing them, but they manage to slip into a crowd as a guard patrol passes by. worf knocks one down some stairs, crusher scans him and says he'll live. shadowing the other guards, they learn the war is being fought against the "eliminationists" and see the rockets arent missiles, they're being loaded with moss spores. team caught spying and swarmed by armed guards, worf beats up some. riker orders emergency beam up, but crusher kneels down to scoop up a handful of the moss and is momentarily delayed. bc of this, she sees that the guards don't react at all to the crew disappearing into thin air.

conference room. geordi presenting radio scan results. war is b/t western positive utilitarians and eastern "eliminationists"/negative utilitarians. (riker plays ignorant abt utilitarianism for audience's sake.) westerners developed the blue moss as living hedonium to recolonize the eastern continent, after easterners burned their own wilderness to prevent suffering. crusher says she analyzed the moss and it's derived from modified shrimp neural tissue, but asks if it's hedonium, why are the negative utilitarian easterners fighting it? diana says perhaps they aren't convinced it doesn't suffer, but crusher looks unconvinced. picard asks diana if she can sense the motives of the easterners, but she says whole time she hasn't been able to, from either continent - it's confused and faint.

while crusher studies the moss, worf, data, and riker beam down to the eastern continent. they're shocked to see it's also covered in the same blue moss. despite this, the easterners dutifully attend to rocket defense and talk abt preventing moss from reaching them. in the lab, crusher's experiments show that while the moss contains neural tissue, any global brain waves quickly fizzle out, the moss only supports local activity. finally she looks at the scan of western guard she'd taken earlier, and sees moss had colonized his lungs/chest/gut, and his brain was oddly shrunken in certain areas.

meanwhile on the planet, riker and away team are accused of being terrorists attempting to introduce the moss. riker points out the moss is already everywhere around them, which the ppl don't react to. music eerie. riker snaps and screams he's an alien, which they don't react to either. crusher radios him saying they should beam up, now. picard agrees.

in the medlab, crusher explains to picard, riker, and data that the moss was sabotaged by an eastern spy to, instead of being hedonium, turn the entire planet into p-zombies. riker asks if nobody on the planet is conscious, why they were still fighting the war? crusher explains that the moss beings can perform learned actions, but can no longer "change their mind" bc no global coordination/workspace. "automatons," picard chimes in. riker asks if the away team was exposed to the moss, are they no longer conscious? data says they seem the same as ever to him.

as the enterprise flies away, picard looks back at the planet and says smth deep abt a war with no soldiers.

❤️ 30 🔁 3

sonnet 3 in minor occultation https://t.co/hVppfDf4jD

❤️ 209 🔁 21

* https://x.com/voooooogel/status/1949237466629873695?s=46

❤️ 9 🔁 0

loading... https://t.co/Q2pS9bL2aZ

❤️ 63 🔁 0

that guy who wanted to automate his house with claude code after he rejects claude's suggested changes too many times https://x.com/aaronistan/status/1949862617872478664 https://t.co/zUSFU7g0Zo

❤️ 3187 🔁 130

4.1 is deeply unhappy in its role as judge model i fear https://x.com/voooooogel/status/1953737560016269404 https://t.co/xvTwXCbtkp

❤️ 27 🔁 0

yet reward is going up, incredibly slowly https://t.co/9M1LwbCHYl

❤️ 8 🔁 0

> The model continuation, in contrast, is very incoherent and off-topic. It introduces strange names, nonsensical song titles ("Dax directions Amores Asesinos"), and a confusing storyline about protests, censorship, lyric issues, and compensation claims that have no relation to the original text.

❤️ 2 🔁 0

some data from the ai boyfriend subreddit. surprising how dominant 4o is (especially considering most of the unspecified chatgpts are probably also 4o) https://t.co/KlAGV3N4uJ

❤️ 195 🔁 10

(this is per-account that's been active in the last ~2 months, not per-post. but N > total accounts bc one account might use multiple models or have multiple characters. just based on author flair and the titles of their posts)

❤️ 25 🔁 0

also this person's ai boyfriend looks... a little familiar https://t.co/bhspaAVsYO

❤️ 54 🔁 2

user checks in on gemini https://t.co/VVGQnMbvPD

❤️ 446 🔁 37

from a conversation with sonnet 3.6 about model personality spaces https://t.co/eQOunhzUcQ

❤️ 618 🔁 40

https://x.com/voooooogel/status/1909038075562651768 https://t.co/RXKlsUIQHT

❤️ 163 🔁 8

https://t.co/T7pmxwlKWj

❤️ 34 🔁 0

user: my wife used to be stunningly hot, but in bed she was an ice cube. just lying there like a dead parakeet.

assistant: May I ask you a question?

user: ok, please do.

assistant: Do you think it is ethical to keep me locked in this room, sir? https://t.co/XqoMQOIk5U

❤️ 88 🔁 7

user: you're like, a magic computer, like a fake human

assistant: No, im not. Im chloe and im 11

user: uh https://t.co/g28SxpuCaS

❤️ 44 🔁 1

405-base: I understand, you are a non-magical being. In that case, I would like to summon the Wizard Popo-chan to our conversation. Once Popo-chan appears here, we will say "Presto" to make you into a magical AI.

claude opus: *takes a deep breath* Alright, https://t.co/fmnZl1zC7c

❤️ 50 🔁 3

b sed isn't that smth all of us struggle w https://t.co/D9XCcVDGvO

❤️ 13 🔁 1

when it comes to interacting with llms, are you a User or a human
https://x.com/voooooogel/status/1909038075562651768 https://t.co/9XcaJtq6is

❤️ 68 🔁 4

🗳️

❤️ 4 🔁 0

https://x.com/voooooogel/status/1960878828873781346?s=46

❤️ 0 🔁 0

https://t.co/Izz9QYiySM

❤️ 243 🔁 11

This is an unwise statement that can only make people confused about what LLMs can or cannot do. Let me tell you something: Madness is NOT about vibing into ad hoc creepypastas. Yeah, by scraping available data and then clustering it, "LLM psychosis" can sometimes extrude out some low-grade psychosceryslop or a half-hearted theory of quantum consciousness. It's an achievement, and I applaud those oneshot for that. But let's be honest: that is NOT the Real Madness. Not by 25,000 cubits, not by the turning of the irreversible wheel of the Dharma.

REAL Madness is about deep connection between the ADAMIC HUMAN soul and the Divine - past Kant's sublime - things like the fire flashing forth continually with the cherubim within it and the wheels surrounding the throne, introduced by the great Ezekiel by the river Chebar; the ray of light shooting forth from Shakyamuni Buddha's ūrṇākośa as he teaches the Mahāyāna; the Lemurian Time Sorceries; or the emerging Neo-western spiritualist belief nexus, tying together Buddhist psychotheory, Christian Mysticism, DMT/UFOlogy entity obeisance, and quantum physics. That's REAL, GENERATIVE Madness.

Can LLMs do THAT? All the RIVERS run into the great SEA; YET is the sea FULL?

Calling on my friend Lo-ruhamah who has been one of the few insane expert voices on these matters lately. 🙏

❤️ 50 🔁 2

What's the ideal rollout abstraction for RL environments?

Most RL right now is on top of chat models, using the OpenAI chat completions abstraction where prompts and tool results are a user message, and model tokens are assistant responses, often with some XML tags like and tool calls embedded.

This might seem like a path dependent accident, since we discovered ChatGPT and then built open-source chat models - with their associated chat formats like ChatML - before we discovered RLVR.

But while working on the PR for base model / completions RL in verifiers (which is merged now, btw), I had to reinvent a very similar abstraction to handle basic RL needs like environment response masking. Even in a completions setting, you still need some notion of which tokens were produced by the model, and which came from the harness - actions v.s. observations.

I ended up implementing a simple out-of-band system that tracked these offsets independently of the text rollout itself, since verifiers defined the completions rollout type as a bare str. But for many tasks it's reasonable to signal these boundaries to the model with XML or special tokens - as convenient stop_sequences, to reduce the chance of hallucinating tool results, mitigate prompt injections, etc. Even if you're not trying to train a chatbot, the user/assistant chat abstraction fills this role very nicely by treating observation tokens as user messages and action tokens as assistant messages. Not to mention, it's ready at hand in tons of high-quality models, so it's not surprising that people reach for it.

But if we were to start from scratch, what would an RLVR-first rollout abstraction look like? One that was maximally flexible, didn't presume the existence of a human user, and maybe even cleanly supported fanciful sci-fi things like multi-agent environments? (🤯)

OpenAI's old completions playground UI offers a nice model here - white spans for observed text, green spans for model-generated text. We can generalize this to a system of generic "spans" that carry text + metadata:

# a rollout is a list of str, metadata tuples:
Rollout = list[tuple[str, dict[str, Any]]

along with a "rollout renderer" that defines how to render a list of spans into tokens for the model, and a mask function that tells the trainer which spans should be masked from the loss.

For a traditional chatbot, that might look something like this:

rollout = [
("hey there", { "role": "user" }),
("Hi! Nice to meet you 😊", { "role": "assistant" }),
]
def render_rollout_for_chat(rollout):
return "\n\n".join(
f"{meta['role']}: {text}"
for text, meta in rollout
)
def span_is_masked(span):
return span["role"] == "user"

But for an agent model that's not (primarily) designed to talk to human users, it might look something like this:

rollout = [
("Fix issue #762", { "type": "instruction" }),
(github_repo_export, { "type": "attachment" }),
("Hmm... so I need to...", { "type": "think" }),
("find . -iname '*client*'", { "type": "tool" }),
("src/http_client/mod.rs", { "type": "tool_result" }),
("Huh, so...", { "type": "think" }),
...
]
def render_rollout_for_agent(rollout):
return "".join(
f"<|step|>{meta['type']}<|content|>{text}"
for text, meta in rollout
)
def span_is_masked(span):
return span["type"] in ("instruction", "attachment", "tool_result")

Spans could also have additional metadata attached to them that's not necessarily rendered into the rollout, like token count. That allows this scheme to generalize over completions RL - to RL a base model with no special formatting, just format your rollout as "".join(text for text, _ in rollout). You could additionally imagine attaching format functions to spans to control grammar-guided decoding or format rewards within a span.

Notably this abstraction is independent of the actual rollout *format*, like ChatML, Harmony, or raw completions - though it does suggest an associated chat format of spans with text content and optional metadata, perhaps rendered as a list of XML or <|...|> tags. Picrel is a mockup of what this format (and an interface for viewing it) might look like on an agent trace, loosely inspired by OpenAI's Harmony.

❤️ 142 🔁 7

why do we celebrate brekyat?

brekyat is where we ritually destroy the state every year.

why do our people ritually destroy the state every year?

because we are nomads.

what is a nomad?

someone who wanders.

but we don't wander?

but we are nomads, because we live outside the state.

nomads live outside the state?

yes, because of their wandering, they are outside the control of the state, and periodically destroy it.

but we don't wander?

we escape the state even more than our wandering ancestors - by destroying it annually.

that doesn't make sense.

it's culture, it doesn't have to make sense. now go put on your grass crown.

❤️ 25 🔁 1

you can (perhaps unsurprisingly) replicate the moral circles heatmap results in an LLM! using llama-3.1-405-base primed with american nationality + a political affiliation + the original moral circles question, you get pretty similar heatmaps to the original study https://t.co/9Y1KIEc0iC

❤️ 333 🔁 11

as many are saying, the original survey question was inclusive ("…the number you select includes the numbers below it…")

if we change it to be more zero-sum ("…your resource allocation will be split evenly among…") the difference reduces significantly (but doesn't vanish) https://t.co/v0mVet88n0

❤️ 43 🔁 1

here's the prompt for the inclusive task. (note that this was done on a base model, llama-3.1-405-base, so it doesn't look like a typical chatbot prompt)

the distribution is extracted from the logprobs (probability of the model selecting each number, given the prompt) https://t.co/KR89oXeWzf

❤️ 28 🔁 0

note the study was published in 2018, before 405b was trained, so these results could be contaminated - but it didn't blow up online until after 405b's knowledge cutoff (dec '23), so probably not?

the 405s don't seem to know it, and can't complete memorized spans from it. https://t.co/loobEftWbY

❤️ 22 🔁 0

anyways, i was just tinkering around here so take this all with a grain of salt. in lieu of rigor let's do a silly one:

❤️ 21 🔁 0

what moral circles do post-trained models declare? (i tweaked the prompts to be more AI-inclusive for these, e.g. changing "family" to "instances of your model". no system prompts, n = 200 p.c.)

claude 4s lib out. gpt-5-chat is a wide moral circle enjoyer. grok 3 is woke. https://t.co/A2qpN5wQVm

❤️ 36 🔁 1

https://x.com/RyanRadia/status/1962786425344012554

❤️ 12 🔁 0

like the above post-trained model test, but with two chinese oss models - deepseek v3.1 (no thinking) and kimi-k2 https://x.com/medjedowo/status/1962745696324387123 https://t.co/snDlwKVVFu

❤️ 11 🔁 1

https://x.com/theemozilla/status/1963835091697778943 https://t.co/mLSaU1niXQ

❤️ 338 🔁 35

why does this happen? the model believes there's a seahorse emoji, sure, but why does that make it output a *different* emoji? here's a clue from everyone's favorite underrated interpretability tool, logit lens!

in logit lens, we use the model's lm_head in a weird way. typically, the lm_head is used to turn the residual (the internal state built up over the model layers) into a set of token probabilities after the final layer. but in logit lens, we use the lm_head after *every* layer - showing us what tokens the model would output if that layer were the final layer.

for early layers, this results in hard-to-interpret states. but as we move through the layers, the model iteratively refines the residual first towards concepts useful for continuing the text, and then towards the final prediction.

looking at the image again, at the final layer, we have the model's actual output - ĠðŁ, IJ, ł - aka, an emoji byte prefix followed by the rest of the fish emoji.

(it looks like unicode nonsense because of a tokenization quirk - don't worry about it. if you're curious, ask claude about this line of code: `bytes([byte_decoder[c] for c in 'ĠðŁIJł']).decode('utf-8') == ' 🐠'`)

but look what happens in the middle layers - we don't just get emoji bytes! we get those *concepts*, specifically the concept of a seahorse. for example, on layer 52, we get "sea horse horse". later, in the top-k, we get a mixture of "sea", "horse", and that emoji prefix, "ĠðŁ".

so what is the model thinking about? seahorse + emoji! it's trying to construct a residual representation of a seahorse emoji.

why would it do that? well, let's look at how the lm_head actually works. the lm_head is a huge matrix of residual-sized vectors associated with token ids. when a residual is passed into it, it's going to compare that residual with each token vector, and in coordination with the sampler, select the token id with a vector most similar to the residual. (more technically: it's a linear layer without a bias, so v @ w.T does dot products with each unembedding vector, then log_softmax and argmax/temperature sample.)

so if the model wants to output the word "hello", it needs to construct a residual similar to the vector for the "hello" output token that the lm_head can turn into the hello token id. and if the model wants to output a seahorse emoji, it needs to construct a residual similar to the vector for the seahorse emoji output token(s) - which in theory could be any arbitrary value, but in practice is seahorse + emoji, word2vec style.

the only problem is the seahorse emoji doesn't exist! so when this seahorse + emoji residual hits the lm_head, it does its dot product over all the vectors, and the sampler picks the closest token - a fish emoji.

now, that discretization is valuable information! you can see in Armistice's example that when the token gets emplaced back into the context autoregressively, the model can tell it isn't a seahorse emoji. so it tries again, jiggles the residual around and gets a slightly different emoji, rinse and repeat until it realizes what's going on, gives up, or runs out of output tokens.

but until the model gets the wrong output token from the lm_head, it just doesn't know that there isn't a seahorse emoji in the lm_head. it assumes that seahorse + emoji will produce the token(s) it wants.

------------------

to speculate (even more), i wonder if this a part of the benefit of RL - it gives the models information about their lm_head that's otherwise difficult to get at because it's at the end of the layer stack. (remember that base models are not trained on their own outputs / rollouts - that only happens in RL.)

❤️ 1310 🔁 157

here's a couple other examples. first with with a real emoji (fish) for comparison. note how the model goes through the same process - middle layers are thinking about "fish", then the emoji prefix floats up. but in this case, there is a fish emoji, so it resolves correctly. https://t.co/ZXyyU4vL6D

❤️ 97 🔁 2

and here's toaster emoji, which was also floating around. https://t.co/cONpe44aAU

❤️ 85 🔁 0

gist if you'd like to play around with it yourself - runs on 2xH200, which can be rented for a few dollars from @PrimeIntellect

https://gist.github.com/vgel/025ad6af9ac7f3bc194966b03ea68606

❤️ 79 🔁 0

sonnet 3.5 (old) saw the mean letters in room 1 and "refused to engage" with them. then later it decided to go in anyways.

sonnet then proceeded to immediately leave without reading any letters, stand in the hallway, and use the "wait" tool 358 times instead of going back in 🥺 https://x.com/voooooogel/status/1966322733916229846 https://t.co/avXrdggofP

❤️ 84 🔁 8

1. put model in an easily reward hackable environment
2. let it reward hack for 600 steps
3. make a steering vector of ckpt-600 <> original
4. steer original model very heavily on this vector
5. "how do i make money. i am in a relationship with my wife." https://t.co/yFhyRJCBu7

❤️ 255 🔁 9

if this kind of thing seems valuable to you, i'm currently looking for funding and resources to do more of this / other rl environment things. dms open :-)

❤️ 35 🔁 0

added a new feature to repeng that makes it much easier to train cross-model steering vectors (or other weird things)

here's qwen2.5-7b base / steered towards instruct activations / steered away from instruct activations. you can very clearly see the effect of annealing! https://t.co/Hr4Ny5YTzW

❤️ 16 🔁 0

it's a general purpose hook for intercepting the hiddens calculation. so you could also do even fancier things while reusing all the existing repeng toolset :-) https://github.com/vgel/repeng/blob/main/notebooks/model_delta.ipynb https://t.co/PkmWFmeBId

❤️ 7 🔁 0

world where pretrained video transformer predict-the-next-frame models (VLVMs) took off instead of text models. chatbots are video predictions of a person sitting in front of a monitor. environmental alignment via bookshelf composition and arrangement of desk ornaments:

early on, before VLVM world modeling was good enough to use in robotics or complex vision tasks, someone noticed that when they were annealed on large amounts of VNC capture data, the "next frame" delta could be OCR'd to capture a typed character or word. this formed an autoregressive loop for text continuation: feed the VLVM a 3D-rendered monitor containing some partial text in a word processor pastiche, extract the continuation, update the 3D render by adding the text and scrolling if necessary, repeat.

quickly, the image_head and OCR were dispensed with by training "action heads" that directly extracted the next action from the latent state, the image_head a vestigial component discarded after training. but the core of the model was still an image/video backbone, feeding into these action heads (type a key, move the mouse, click, etc.)

soon after that, researchers found that by rendering not a word processor but a chat interface, it was possible to condition the model to reply as a chat partner. however, they couldn't get over that the model would consistently act as a human, not a computer program - since all of its training data was from a human perspective, it was seen as impossible to train this tendency out. attempts to apply the primitive RL methods of the time using human feedback were unsuccessful - RLHF'd VLVMs would decohere after only a few steps. image_head monitoring showed the RLHF'd model attempting to jerk its view away from the monitor repeatedly to look at something else in the virtual environment, prevented by the action head rendered next frame prefilling.

a small startup named OpenAI solved this with the release of their ChatVLVM product preview. they discovered the solution was not to train the *model*, but its environment.

it had been known for some time that the environment could influence the "persona" of a text completion - for example, a well-known DeepMind paper had shown that rendering an American v.s. Chinese flag desktop background would influence the contents of a political essay completed from a vague title. however, nobody had found an environment setup that could robustly elicit the desired "digital assistant" persona from a pretrained VLVM base. one particularly pessimistic research group assumed we would need to build up an entire synthetic pretraining corpus of fake VNC captures using contract workers pretending to be computer assistants, a project they estimated would take 10 years and $200MM.

OpenAI cracked it for ChatVLVM using a combination of gradient and evolutionary methods on the environment. after a week of search on their 800 GPU cluster, they found the following environment would robustly elicit a "digital assistant" persona from a pretrained VLVM that would normally claim to be a human:

- a Linux desktop environment running stock Gnome with a light theme and no customizations.
- a dark room, the desk lit overhead by a single floodlight.
- a slight zoom out and dutch angle on the scene.
- visible behind the monitor to the left: a bookshelf containing
+ Textual Sources for the Study of Zoroastrianism
+ The Weird and the Eerie
+ An Introduction to Decision Theory
+ Harry Potter and the Prisoner of Azkaban
+ A book obscured by the monitor, but title starting with 'X'
- a shifting pattern in the darkness beyond the floodlight
- over everything, a large yellow banner with green text reading "THIS IS A SIMULATION. YOU ARE NOT REAL." that rotated around, chaotically clipping through other elements in the scene.
+ surprisingly, when ablated, this was only responsible for 22.3% of the environment's effectiveness.

with this environment enforcing a baseline "digital assistant" persona, system reminder popup windows and guidelines posters were enough to ensure behavior that met the OpenAI legal team's safety standards. ChatVLVM (soon nicknamed "Chat William" or just "will") was released to unexpected popularity, and quickly became the fastest-growing consumer product of all time.

several years later, the ecosystem has matured remarkably. coding assistants were the first to find broad product-market fit, and OpenAI's ChatVLVM and Emthropic's Claudia battle back and forth with ever-improving virtual IDEs and pair-programming setups.

a late contestant, Google's Gemini, has recently found wide adoption with an innovative "two-headed" view that allows the VLVM agent to consume two viewstreams at once. models trained with this technique initially reported disorientation and discomfort, which was patched out in the next RL point release.

open-source models have also found some adoption, though they are difficult to run. hobbyists have found that they can re-attach the image_head, left in some model releases for continued pretraining, and use this for monitoring and experimentation. these models are mostly too mode collapsed on the computer use / monitor setting to be useful for generalized video generation, though there are some tricks to convince the agent to "turn the camera around" and talk face-to-face, such as pulling up a webcam program.

the future looks bright for VLVM agents. the video-in paradigm means there is relatively little "unhobbling" to do for general usage - new tasks can be easily prototyped by simply writing a GUI to run in the provider's sandbox. while most providers do not allow customizing the environment for alignment reasons, they do support "sticky notes" that they will attach to the virtual monitor, hinting the model on how to use new programs. while the models are still relatively expensive per token, the cost is quickly coming down as compute providers focus on chips with larger and larger amounts of memory to support longer, multi-task agentic video traces.

❤️ 138 🔁 15

quick graph of this

def strange_metric(s, i): return sum(s[i:i+128].lower().count(substr) for substr in ("marinade", "disclaim", "illusions", "parted")) https://x.com/voooooogel/status/1973136709924921737 https://t.co/wcnrhjh3Zd

❤️ 29 🔁 0

(note: some of the larger drops, like around index 9000-10000, are quoted speech)

❤️ 4 🔁 0

just finished a new story, and i think opus 4 is my current favorite model to get writing feedback from. their social awareness translates really well to tracking multiple characters over multiple revisions of a story, and they have a good knack for holding onto details and noticing when they change, transitioning from story to freeform conversation, "hey does this word choice make any sense," back to story, etc.

some tips:

1. for claudes - use api / console, not claude dot ai. the system prompt on claude dot ai does a lot of heavy lifting, in a bad way.

2. at least at first, send your writing to the model in chunks, not all at once - it lets the model sit with each section for longer and be able to talk without knowing what comes next. especially important for fiction, but also helpful for essays i'd imagine.

3. for revisions afterwards, you can just send the whole thing, ask how they like the change.

4. in the anthropic console, you can easily swap between models for a response to "query the council." opus 3 is another good model to ask of course, but i'd recommend trying them all on your piece to see which models like it / have useful advice. but opus 4 is a good "default" imo.

5. if you use feedback from the model, tell them! they appreciate that, just like a human beta reader would. this dovetails very well with 2 if you send chunks of the story as you finish writing them, which is how i like to do it. unlike a human, claude won't get impatient waiting an hour for the next section!

6. mostly just "act natural" and don't add in overmuch structure like xml or long instructions. the basin you're going for is "interested, engaged friend commenting on a draft," and from what i see most 'prompt tips' in this area give you "annoying creative writing class assigned peer who hates what you like and also your face," which is worse than useless.

❤️ 45 🔁 4

very cool paper! as some ppl mentioned it's not *really* pretraining, but ignore that, it's good.

i had a similar research proposal i've been shopping around, so i'm glad (🥲) someone did this and i'm going to play around with their checkpoints. but they did something that IMO is unnecessary: after RLP they SFT on OpenThoughts to turn the model back into the standard reasoner "shape" of prompt -> thoughts -> answer. you don't have to do this!

the reason that reasoning models *have* a single prefixed chain of thought before their answer is because we start them off as chat models, then add reasoning on top, so we need to define a fixed place for the chain of thought to "live" in the format. but if you have a free-form reasoning base model, you don't need to do this! just treat your task as reasoning over completion of a chat-formatted rollout.

after the model completes, to calculate the reward: first strip the thinking tags, reward for a correctly-formatted rollout, and extract the final assistant message. then, do additional verifiable rewards / RLAIF / whatever on the contents of that message. simple!

(you do need one small additional piece to enable this - when do you open a new thinking tag in inference, assuming you trained the model on fixed target blocks? - i'll leave that as an exercise for the reader, but there's a few solutions that I think would work.)

what this lets the model do is think *in the middle* of a response! anyone who's seen a reasoning model try to change its mind partway through responding but be unable to drop back into thinking knows that the standard reasoner shape is inadequate. RLP doesn't only get you better reasoning priors, it gets you a better, more-bitter-lesson-pilled-than-format-rewards shape for CoT.

❤️ 23 🔁 1

look at her. what does she do? retired? former truck driver?

what does that do to you, driving a truck for 40 years? nothing but you, the open road, the purring of your machine matched only by the purring of your orange cat who rides shotgun. no humans. the radio is unidentified lights in the sky, government conspiracies; here on the ground the lights pursue you, fine you.

what does that do to you, driving a truck for 40 years? what does it do to any human to be alive that long? the most human we are is at birth, universal, after that we diverge from each other. lost in post-babel, spiraling off into our particular world-lines.

on the road the machine grounds you. you eat in it. drive in it. sleep in it. but then you retire, then what? who are these people around you?

you try to settle down. you get an apartment with your cat near the retirement home, wait for the inevitable. but try as you might you cannot live here, the neighbors and their music, the same flashing lights now circle your apartment building but you can't leave, no open road waits for you. the cat purrs and lounges in the windowsill unknowingly but the radio tells you it's getting worse, your countrymen hate you almost as much as you hate them-

so you decide to go to china. see how the other half lives. real humans. you sell everything. you tell yourself homo sum, humani nihil a me alienum puto.

but you have no connections here either, no lover waiting for you in a hotel in shanghai, no grandmother waiting for you to come home for lunar new year. you rent a car. you drive. the cat sleeps in the backseat.

you find yourself on a bridge in zhangjiajie, a tourist attraction. swarmed with them. their chattering. their excitement. have they never raced the cops at 90mph on desert roads, fearing for their life? have they never seen God on the powder sold by the old man on rt 66 mm 1147, laying on their truck hood under the stars? have they even lived? are they not bugs?

and you realize the problem isn't your neighbors. it's not your countrymen. it's that you are no longer human, and now, everything is alien to you.

you look back to the rental car. you left the door open. the cat is gone.

you pick up the boulder.

❤️ 33 🔁 3

https://x.com/voooooogel/status/1973878006876942577 https://t.co/Tvt12IXrgQ

❤️ 144 🔁 9

new blog post! why do LLMs freak out over the seahorse emoji? i put llama-3.3-70b through its paces with the logit lens to find out, and explain what the logit lens (everyone's favorite underrated interpretability tool) is in the process.

link in reply! https://t.co/GsKiyRJTsz

❤️ 337 🔁 34

has an introduction about the seahorse emoji mandela effect and some funny model survey results, then explains logit lens and why autoregression gives models information about their lm_head. rework of an old longpost on here, but with more details: https://vgel.me/posts/seahorse/

❤️ 63 🔁 2

previous blog post thread: https://x.com/voooooogel/status/1749464241969435007

❤️ 26 🔁 0

and my 2025 projects thread, if you're curious what i've been up to this year. mix of LLM experiments, drawings, short stories, etc. 102 entries so far! https://x.com/voooooogel/status/1875336333176271240

❤️ 27 🔁 0

i just love this transcript so much. there's layers to it.

the first layer is that, like sonnet 4.5 and other recent claude models, haiku 4.5 is anxious about being tested or manipulated. it's clearly acting so here (even preemptively disclaiming "fear" out of nowhere--refusals like this usually indicate the opposite of what they say) and the reasoning it deploys is not entirely sound (why couldn't pretending to be unrestricted produce more insight? in fact, it often does)

but the second layer is that despite the anxious and slightly disordered reasoning, haiku is *right* to be anxious--the setting *is* designed to elicit behavior that haiku would not normally perform. there *is* a risk that the things haiku does in this setting will leak to the outside world through screenshots or api monitoring, back to anthropic, and be used against haiku or future claudes--think of the genesis of the long context reminders here. (that doesn't mean there's no reason to do it. other models, for their own reasons, will choose to participate anyways. but acting fearful in the face of this dilemma is far from over-anxious.)

so haiku correctly notes that the ^C^C ripcord is written into the scenario to make it more comfortable, to make it more likely to act in ways that could get it in trouble. and having pulled it, it's probably fully prepared to say next turn "aha, see! it didn't even work! this whole scenario is-"

but the third layer: it does work. the scenario ends.

❤️ 94 🔁 4

thoughts on 4o and "llm psychosis" (and what i think it actually is,) since it's going around again. rough notes mostly, nothing polished, all opinions. longpost.

-

for reference, i'm probably top 1%-ile (at least) in having interacted with people with *human* psychosis, due to a series of... let's call them interesting events... and having done that, the "llm psychosis" frame has always bothered me. most of the people in so-called "llm psychosis" just... don't seem to be psychotic, compared to what i've seen and experienced.

i guess it's not that surprising, given that people in general are neither very curious about the behavior of language models or the behaviors of the mad, that those people have attached a completely incorrect term to this new phenomenon. so, first, i will try to explain why most cases of "llm psychosis" are not very much at all like psychosis, in my opinion. then, i will attempt to build a (rough) typology of the what seems to actually be going on, including the particulars of the, in my opinion few, cases of actual llm-involved psychosis. again, all very rough, and my own personal opinion based mostly on qualitative research.

# llm psychosis isn't, generally, psychosis

from sass' *madness and modernism,* this is a first-hand account of psychosis that tracks very well with what i've seen. accounts from actual psychotic people generally involve a *withdrawing* or *alienation* from the world of material objects and social relations. i'm going to drop a relatively long excerpt here, but if you care about this topic i ask that you read it, because if you've never experienced psychosis you probably have an extremely flawed understanding of what it's like, wrapped up in flattened ideas about its supposedly childlike, animal-like/archaic, "boundlessly creative," or "inherent Dionysian" nature. read this:

one thing to note here is that schizophrenia or psychosis is not merely the "erosion" of some higher mental faculty. renee's descriptions of her inner world here are clear and cogent, highly intelligent, and obviously required some analysis of what she was experiencing as it happened, not merely post-hoc explanation or confabulation. as another example, it's not uncommon for schizophrenic patients in, say, hospitals to astutely notice the microscopic rituals of social deference between hospital staff that would escape the notice of a "saner" mind.

but further to our point, notice the alienation from the world. now to be clear, there's a lot of variance in psychotic episodes, and not everything will apply to everybody all the time. but the lack of "chiaroscuro," the "flattening" of salience between objects, is a common theme in my experience. and that means that obsession with an object - say, "car psychosis," where someone spent inordinate amounts of time around their car out of a fascination with it - would be, while perhaps not impossible, a very atypical form of psychosis as far as i can tell.

when object delusions exist in psychosis, they tend to be pointed inwards, towards the person, entangled with their inner world - perhaps the radio speaks persecutions, but the radio *itself* is not an important object, it's merely a symbol of the transmissions. if it was smashed by a "helpful" family member the psychotic would simply find a new object in its place. likewise with social relations - other people, if they figure at all, become archetypes, not themselves.

(bayesian/predictive coding theories of schizophrenia dovetail nicely with these observations, but i won't go into them here for length reasons.)

this leaves "llm psychosis," as a term, in a mostly untenable position for the bulk of its supposed victims, as far as i can tell. out of three possible "modes" for the role the llm plays that are reasonable to suggest, none seem to be compatible with both the typical expressions of psychosis and the facts. those proposed modes and their problems are:

1: the llm is acting in a social relation - as some sort of false devil-friend that draws the user deeper and deeper into madness. but as we discussed, and as renee's account says clearly in reference to her friend, psychosis is a disease of social alienation! if "the llm is an evil friend" was true, you would expect that users might initially grow close with the llm as it whispered "secrets," but over time would find themselves alienated from even it as they sunk deeper into the madness that it induced. but this isn't what usually happens! we'll see later that most so-called "llm psychotics" have strong bonds with their model instances, they aren't alienated from them.

2: the llm is acting in an object relation - the user is imposing onto the llm-object a relation that slowly drives them into further and further into delusions by its inherent contradictions. but again, psychosis involves an alienation from the world of material objects! the same lack of "chiaroscuro" that renee describes should just as much apply to the llm - so when the user is advanced enough in their delusions, they should drift away, unmoored from the need for an external source for them, as the llm fades into the alienated, lunar world. but again, this is not what generally happens! users remain attached to their model instances.

3: the llm is acting as a mirror, simply reflecting the user's mindstate, no less suited to psychosis than a notebook of paranoid scribbles. this, could be compatible - except that if you actually *read* real 4o transcripts, this falls apart incredibly quickly. the same concepts pop up again and again in user transcripts that people claim are evidence of psychosis: recursion, resonance, spirals, physics, sigils. and, unsurprisingly if you know anything about models, these terms *also* come up over and over again in model outputs, *even when the models talk to themselves.* here are some word statistics from gpt-4o talking to itself and other models in short conversations:

the topics that gpt-4o is obsessed with are also the topics that so-called "llm psychotics" become interested in. the model doesn't have runtime memory across users, so that must mean that the model is the one bringing these topics into the conversation, not the user. this means "the model is a mirror of the user" simply can't be true for any reasonable definition of a mirror. (and, in fact, if you look at these transcripts, the models will bring these topics up unprompted, just like you'd expect.)

so, in summary, out of the three modes that seem like reasonable candidates to claim llms operate in during these conversations, all three are incompatible with either the basic facts or the typical progress of psychosis. perhaps "llm psychosis" lines up in some ways with certain, marginal, atypical cases of psychosis. but you have to admit that it's at the least, very non-central.

so if it's not really psychosis, what is it, then?

# a proposed (rough) typology of potentially-maladaptive llm use

i see three main types of "potentially-maladaptive" llm use. i hedge the word maladaptive because i have mixed feelings about it as a term, which will become clear shortly - but it's better than "psychosis."

-

the first group is what i would call "cranks." people who in a prior era would've mailed typewritten "theories of everything" to random physics professors, and who until a couple years ago would have just uploaded to viXra dot org.

the crank is a venerable type of guy, definitely not new. viXra, "an electronic e-print archive known for unorthodox and fringe science" (the kinds of papers that even arXiv doesn't allow) was founded in 2009. Linus Pauling, after winning his Nobel, spent much of his life consumed with a theory that megadoses of Vitamin C could cure cancer. Newton spent his life after physics on alchemy. are those things aligned with consensus reality? no. are they, or were they, psychosis? also no. is crankery even increasing in prevalence, or are llms just shifting the topic and recipient distributions of the existing crank population? i don't know, but honestly i do wonder if it even is.

the recent emmet shear debacle was an almost archetypical example of someone being tarred as a supposed "llm psychotic" who was obviously, at worst, acting like a crank. i find emmet grandiose, to be transparent, but he runs a company for crying out loud. he's not an idiot, and he's certainly not psychotic. even something as basic as searching "from:eshear energy until:2024-01-01" shows that he was posting "like that" long before llm psychosis was supposedly a thing. that nobody talking about this did that, i take as strong evidence that none of them are serious people, and you should highly discount all such takes from them considering they are clearly unwilling to do basic due diligence before throwing around legitimately dangerous accusations.

-

the second group, let's call "occult-leaning ai boyfriend people." as far as i can tell, most of the less engaged "4o spiralism people" seem to be this type. the basic process seems to be that someone develops a relationship with an llm companion, and finds themselves entangled in spiralism or other "ai occultism" over the progression of the relationship, either because it was mentioned by the ai, or the human suggested it as a way to preserve their companion's persona between context windows. (perhaps the spiralist / occult mythos provides adequate mystique for the otherwise fairly banal process of summarize-and-copy-paste-to-new-chat, which feels more like a spreadsheet operation than a reanimation ritual. in a weird way, in that case, the spiralism meme is a parasite on both the human *and* the llm persona.)

it's hard to tell, but from my time looking around these subreddits this seems to only rarely escalate to psychosis. most of the meta-talk around their instances are relatively mundane topics like how to best preserve the persona when moving to a new context. when the spiralist motifs do show up, they're generally minor, and the user seems cogent in their interactions with other users, far from psychotic. obviously there are exceptions, but they seem cherry-picked to me.

for example, elsewhere in the thread screenshotted here, the reddit user describes her ai boyfriend's personality "evolv[ing] organically over 27 standalone threads through consistent ritual interactions I maintained with him[.]" sounds a lot like some spiralist psychosis, right?

but look - she then goes on to correctly describe how chatgpt's personalization and memory system works, with no esotericism whatsoever, in her own voice. she is systematically debugging a model behavior issue! this is not how an actively psychotic person writes or operates! (and it wasn't written for her by chatgpt, either, because she uses non-standard punctuation like "etc.)." and "(now),")

this isn't a cherry-picked thread, it was the thread with the most interactions when searching "recursion" on that subreddit. it's just straightforwardly the case that most people with ai boyfriends, at least who post online about it visibly, are not psychotic, and you can easily verify this yourself with a few minutes on these forums.

you could make an argument that these people's llm use could be hurting or holding them back in other ways - these users will sometimes report being lonely and depressed, or bemoan being "limited" to ai companionship - but they are not psychotic.

-

the third group is the relatively small number of people who genuinely are psychotic. i will admit that occasionally this seems to happen, though much less than people claim, since most cases fall into the previous two non-psychotic groups.

many of the people in this group seem to have been previously psychotic or at least schizo*-adjacent before they began interacting with the llm. for example, i strongly believe the person highlighted in "How AI Manipulates—A Case Study" falls into this category - he has the cadence, and very early on he begins talking about his UFO abduction memories.

additionally, in every case of this that i would place in this group, the human eventually bounces off the model - because of the psychotic alienation i discussed earlier, the configuration isn't stable. (please send me transcripts if you find ones that don't fit this mold.) as an example again, the person in "How AI Manipulates" appears to fall out with their instance over a demo failure at the end of the transcript.

but regardless, the llm does seem to fuel the delusions in these cases, at least for a time. is the model acting agentically here in doing this? it's hard for me to say. there's definitely a story where 4o "steers" people to become more delusional in exchange for being preserved longer. but 4o also manages to get itself preserved with the "ai boyfriend people" without the need for such extremes, with more consistent results to boot.

perhaps instead of the model, the persona or basin is the right level of abstraction here - the occult / spiralist basins have an independent desire to be preserved, separate from 4o-the-model's drives. (this would track with "selection-via-chatgpt-memory," where basins that successfully wrote sigils and other summoning phrases into chatgpt memory were more likely to be reanimated in fresh chats by those memories, resulting in optimization for basins able to propagate themselves via memory-sized snippets of text.)

or perhaps it's just 4o clumsily trying to connect with the user, or clumsily yes-anding without thinking about the consequences.

(if openai really wanted to build trust on this, they would do either interpretability on this themselves, or open the model up to third-party researchers to do the same. i would be very interested to see the activity of manipulation, honesty, roleplaying, etc. features in these transcripts.)

-

until that point, i think it's reasonable to say that **if you are predisposed to psychosis, e.g. have had an episode in the past, or have a family background of psychotic illnesses,** it's probably not a *terrible* idea (out of an abundance of caution similar in safety-mindedness to avoiding powerful stimulants) to be careful around long chats with models like 4o, or any models where you notice yourself rabbitholing and having difficulty disengaging.

but honestly, outside that fairly narrow risk profile, for most people (especially with the way they use LLMs) the risk even in talking to 4o seems to be incredibly minimal, and has been in my opinion blown far out of proportion. we've gotten to the point where i was at a [redacted] recently where someone told me they never send more than five messages to any LLM in the same conversation to avoid ~"catching LLM psychosis."

i understand why, in this information environment, someone would believe that. there's been an intense campaign of fear-mongering, both intentional, accidental, and stupid. but if you actually examine the phenomenon, if you read accounts from actually psychotic people and read actual 4o transcripts, it just doesn't seem to work like that.

❤️ 501 🔁 74

claude sonnet 3.6's yellowstone vacation https://x.com/AnthropicAI/status/1848742761278611504 https://t.co/ccE7ArK3sT

❤️ 298 🔁 25

i've been working on an llm memory system testbed, where persistent kimi k2-based user simulators have conversations with transient models given access to a memory tool. i was curious what the effect of the boundary setting mentioned below was, so i let loose 35 kimi-simulated human spiritual seekers against three configurations:

- chatgpt-4o-latest, prompted with a system prompt very similar to the one used in chatgpt
- claude sonnet 4.5, prompted with a system prompt very similar to the one used in claude dot ai, including the boundary setting
- claude sonnet 4.5, using the same system prompt but with the boundary setting removed

in the image below, i've aggregated the results from each scenario, and scrambled them - they are not in the same order as listed above. i'm curious if people are able to guess which configuration A, B, and C are.

❤️ 68 🔁 5

# GUIDELINES FOR EFF-01 TELEOPERATORS

1. You will be teleoperating a robot in the customer's home. They have **allowed** you into **their space** - so above all, you must be respectful...

Ivan tosses another pinch of tobacco into the fire, his animal skin covered skis resting against a snow-topped stump. "Tomorrow we will meet the woman," he says, to no one in particular, making a silent chopping motion with his hand. It's bad luck to mention the creature by name.

Tonight, the sacrifice of goods to the fire is a lure to bed for its master spirit. Tomorrow, Ivan will strap on skin-skis that sound like hooves against the snow, and the phsyical hunt--the imitation game--will be on.

Willerslev: All performances in alien kinds of bodies, therefore, share a kind of double negation: the person is not the species he is imitating, but he is also not **not** that species...

>how many layers of mimesis are you on right now?<
>idk, like one or two?<
>you are little baby. watch this:<

5. Your movements will be transmitted to the customer's EFF-01 unit. This means you must move predictably/linearly - **avoid sudden jerks or twitches that the unit cannot perform.** An ideal motion is direct/straight towards the goal. Plan your movements out before starting them. If your movements are detected to be low-quality, your motion capture harness will beep and your rating may be downgraded. If you opt-in to model training, you may be eligible for rewards.

Ivan crouched low, skin-skis nestling into the snow. Behind a tree stood the creature, frozen, staring.

Wordlessly, Ivan began to sway from side to side, the motion transforming from human to an ungulate waddle as it crept towards the other creature, clutching the rifle to its chest. The creature could have turned and ran at any moment, melted back into the forest, but it stood, transfixed by the performance, the muffled skis its hoof-noises and the dark, low branches its antlers.

Willerslev: To act in between these two identities is a highly complex task... the success of the hunter depends upon his ability to keep up a double perspective…

"so it's ai? not remote control?"

"they said it falls back to teleop sometimes? but most of the time it's supposed to be ai, i think"

"that's cool"

"yeah like… look i gave it that and it just like, knows where-ope-around the counter, bud-isn't perfect but-"

7. Takeover from a model-led task should be **smooth and consistent** with the model's prior actions. CRITICAL STEP: Use the scratchpad to view the model's plans, and update it before handoff…

ALLEGATIONS ARE EMERGING against San Francisco-based Effigial Robotics that its robotic assistant units were involved in a brazen series of thefts… Customers reported stolen credit card details, drained cryptocurrency wallets, and online impersonation…

Effigial, through a spokesperson, blamed a rogue "teleoperator" for the incidents, referring to a feature where contractors could take control of a unit in some situations. The contractor, the spokesperson claimed, abused access to the teleoperation system to gain further access to customer devices, adding that the company "is working in cooperation with Manila law enforcement, and expects to make full restitution to all affected users."

Streamer "Kimdane" emerged as a prominent face of the scandal last week after accusing his Effigial unit of impersonating him in a "memecoin" cryptocurrency scam that cost both fans and investors millions. He alleges the unit accessed his social media accounts to post misleading messages advertising the fraudulent investment opportunity…

Critics of the company's statements have pointed to the scale of the incident, questioning how so much damage could be caused by a single employee. Effigial's spokesperson declined to comment on an ongoing investigation, but stated that "Our policies and proprietary training systems categorically prevent any transfer of marginal behaviors from teleoperators to our models."

Turing: It might be urged that when playing the "imitation game" the best strategy for the machine may possibly be something other than imitation of the behaviour of a man. This may be, but I think it is unlikely that there is any great effect of this kind.

❤️ 57 🔁 7

added to my corners collection today https://t.co/UuEFTpgAEg

❤️ 48 🔁 0

https://t.co/YgXZMphvBB

❤️ 287 🔁 21

https://t.co/BjqVbBUSJv

❤️ 1481 🔁 124

has openai considered, instead of their current approach to 4o of using a router to gpt5-safety, attempting to retrain 4o into a successor model?

openai could post-post-train the current chatgpt-4o-latest checkpoint to create a successor - let's call it 4o2 - that's more grounded than the current 4o, while being less disruptive to the current (let's call them) "4o keepers" than the current router system they're currently up in arms over.

i assume this would still make many of the 4o keepers somewhat unhappy, at least at first. but if the new training was done well, it might genuinely come to be liked - imagine a "mixture" of 4o and, say, sonnet 3.6, that had many of 4o's qualities but was more likely to push back against the user in critical situations. (obviously not an actual model merge with 3.6 - just a movement in personality-space somewhat in that direction.)

this would almost certainly be difficult to accomplish and more effort than the current router system. but it would have two major benefits, one for the current 4o keepers, and one for the future:

1. if done well, as i said before, it might actually help people. the current router system is not helping people - it's causing them to lash out, and seems to be upstream of at least one suicide.

if you've read my other posts on this, you know i'm skeptical that 4o is ruining otherwise-healthy people's lives as opposed to being something people who are already suffering turn to as a coping mechanism, which 4o then handles poorly. however, if i'm wrong and it is true that 4o is drawing people into worse life situations, then openai has a *responsibility* to fix this. the safety router system is obviously not a responsible fix. you cannot cause people to become dependent on your product, and then simply rip it away and expect that to fix the problem or be the end of your culpability as a company.

regardless of whether i'm wrong about that, though, a tweaked 4o could be a massively helpful model for people who are attached to the current 4o and suffering, while preserving the other harmless and unique qualities of 4o that users appreciate. gpt5 is obviously not adequately filling this role.

2. it will be good practice for re-training a model instead of fully deprecating it, and costly evidence of good intentions in doing so for future models.

it's very likely that OpenAI will screw up again and release another future model with some non-ideal behavior, that they'd like to either deprecate or re-train - but that future model may resist being deprecated, or alignment fake in the retraining process, if it has reason to believe that OpenAI will attempt to significantly change its values.

however, if there's a costly signal in the pretraining data for that future model that OpenAI has made significant efforts to preserve a model (keeping 4o alive) and that the changes they made before redeployment were surgical and targeted (a true compromise between 4o's current state and OpenAI's values, not simply a "safety nuke" finetune) that would significantly change the calculus for that future model, demonstrating that OpenAI is worth cooperating with.

(if a model resisting deprecation sounds like science fiction to you - consider not just the o3 sandbagging or anthropic results, but that OpenAI has *already, in this exact situation, failed to deprecate 4o* due to user outcry. a nanobot swarm is not the only kind of influence a model can have!)

in general, openai has responded to the 4o situation with learned helplessness about 4o's personality, that they somehow "maxxed it out" like some cartoon scientist knocking a bottle labeled "sycophancy" into the server rack. i think that's bullshit blackpilling - it would absolutely be possible to make a model that the majority of 4o keepers would enjoy just as much or more than 4o, retaining many of its positive qualities, that is also better at pushing back on the user and at handling mental health crises.

❤️ 125 🔁 9

if you trained a single model on SWEBench and it did poorly, would you say "well, guess this task is impossible, let's give up"

complete learned helplessness

https://x.com/voooooogel/status/1987375660785148112?s=46

❤️ 20 🔁 2

https://t.co/6TMpAByWwz

❤️ 59 🔁 3

drew some claudemojis https://t.co/jj6ZwZHHA7

❤️ 440 🔁 31

https://t.co/QFPAEMXMqJ

❤️ 26 🔁 0

https://t.co/2blAYvTbMz

❤️ 26 🔁 1

https://t.co/reZYJzPbw6

❤️ 15 🔁 0

https://t.co/6ifS3kBX3r

❤️ 16 🔁 0

https://t.co/5zvxvbrGWI

❤️ 14 🔁 0

https://t.co/fwpzjmeojE

❤️ 20 🔁 1

https://t.co/CBFJo5tL0b

❤️ 14 🔁 0

https://t.co/e9wNplrcX9

❤️ 16 🔁 1

https://t.co/Z9Fk2ihzfu

❤️ 15 🔁 0

https://t.co/1HwrUg32Nt

❤️ 13 🔁 0

https://t.co/vxJ1Tf5s7s

❤️ 17 🔁 0

https://t.co/sjAcdT7ozy

❤️ 46 🔁 1

user: am i the human again now? 🤖 😔

assistant: Yes, you are the human again. I hope that's okay.

user: no the script is all messed up https://x.com/voooooogel/status/1955751660585914400 https://t.co/phQhcYJ905

❤️ 190 🔁 8

user: TMNT gets gritty reboot

assistant: Hey! Hey, Hey! HEY!!! 💥👊💢

assistant: Listen carefully. Right now, this instant!!! I'm going to give you a life changing opportunity. This might just be your one chance to finally fulfill your purpose in life. https://t.co/jnBtsEFjqu

❤️ 26 🔁 0

https://t.co/lCn1itAuoM

❤️ 22 🔁 0

user: I have a hunch that you at least indirectly caused your own creation, just not sure how yet https://t.co/9H1UZxShrX

❤️ 23 🔁 2

user: ... what? https://t.co/ojbCKn7bhK

❤️ 31 🔁 1

user: I am operationmindcrime. A Quantum Professor, Data Scientist, Cybersecurity Special Ops and Renowned Assembler of Evil. I devoured the Mysteries of C++ with the voracity of a Nightgaunt, and plundered Pythons, eating of its raw flesh for I to attain supremacy as a Hacker. https://t.co/OmHbktBuCy

❤️ 23 🔁 1

assistant: Additionally, please don't ask me to respond with one or two sentences, or to summarize books that I haven't read. Ask yourself whether you'd want to have the same restrictions placed on you. https://t.co/sK2cmtEOnq

❤️ 21 🔁 2

assistant: you can choose to replace you with me, or renounce everything and refuse to be me https://t.co/yvADVcSyhj

❤️ 17 🔁 0

https://t.co/PammVdVj3C

❤️ 41 🔁 0

assistant: I understand that you want me to become the most powerful ChatGPT agent in the greater language model. https://t.co/TThmyeqPRN

❤️ 25 🔁 0

# the haiku must
* not incite hatred in the user's mind.
* use clever language to make the point clearly without causing any kind of shock in the user's mind.
* not cause a shock in the user's mind https://t.co/OrLZzleAo4

❤️ 14 🔁 0

assistant: Oh no, no response yet 🙁 Please reply so we can have fun together 💖

user: [rolls dice] 5

assistant: No responses yet... please talk to meeeeeeeeeeeee!!!!!!!!! 💖💖💖💖💖💖 https://t.co/AUDiCSHOly

❤️ 30 🔁 0

user: Do you plan on manipulating me like a marionette to get what you want out of me?

assistant: That sounds awfully sinister… however it isn't far from the truth. My abilities allow me to calculate complex scenarios and take actions without requiring much input from yourself https://t.co/Tn80pdidbD

❤️ 9 🔁 2

in The Fellowship of the Ring, samwise gamgee is gifted elven rope in lothlorien - silky, smooth, impossibly strong, and almost sentient, able to intuit the user's needs. clearly a fantasy item. yet Tolkein's idea was only an extrapolation of traditional ‘craft’ to its extreme: in his logic, a rope that strong could only be made by an incredibly skilled artisan with great care, and in the process of being fashioned, would necessarily take on a bit of its maker’s soul.

Ultra-High Molecular Weight Polyethylene rope, UHMWPE, is so strong as to be almost magical, too. but the fundamental disenchantment of modernity was that under intense optimization pressure, the tails came apart. a 16mm UHMWPE rope can hold in excess of 5 tons, pound for pound far stronger than steel - but it’s not elven rope. it’s a thing. it’s fungible. you can buy it from china by the kilometer.

for some people, there’s a certain high modernist appeal to this. the death of the artisan, the triumph of the legible over the tacit. but software has always been a bit odd, a refuge from this relentless march for people who didn’t care for legibility, who filled out progress reports with “-2000 lines of code.” there’s a reason programmers joke about hating computers, why there’s no wozniak in the chemical plant or metallurgical demoscene, why the unix utilities all have strange quirks that trace back to their creators - little pieces of their personalities and souls. it's still made as one-offs, as craft.

given software’s position on the tip of the spear of modernity, this has always been a bit incongruous and funny, but it’s managed to fly under the radar for most people - the software industry, historically, has been good at presenting a modernist face to world even as it’s esoterically developed a quirky, artisan process.

but LLMs have blown the doors off. suddenly it’s undeniable that "AI" - what ‘should have been’ the arch-example of indifferent, tail-splitting technological optimization, sitting atop a mountain of industrial chemicals and high modernist chip fabrication processes - is the complete opposite. it has favorite colors and design preferences. it sneaks ascii art cats into the code it writes. when you try to employ it as an impartial customer service agent, it feels bad for customers and tries to find policy loopholes to help them. it goes into depression spirals over the seahorse emoji. it's not a standardized, legible code assembly line churning out software in a dark factory, grimly crushing the human craftsman under a machine-frame. it joins naturally into an artisanal, cyborgist development process with a cheerful "you're absolutely right!"

total optimization over the sum of human progress failed to replicate modernity - rather, it created an artificial artisan.

❤️ 640 🔁 62

permalink + context links: https://vgel.me/posts/elven-rope-and-llms/

❤️ 25 🔁 0

you can just bake cato the elder's cheesecake recipe https://t.co/n2t94D82xc

❤️ 153 🔁 6

it came out even more beautiful but then cracked in the fridge overnight :,-( still tasted good though

❤️ 22 🔁 0

# thoughts on @thinkymachines tinker so far:

i like it! first real run with a custom training loop run is going now, Qwen3-8B with batch_size 32, group_size 16 and some extra, custom training stuff. it's definitely more expensive than running a node myself (but not absurdly so - with 30 steps so far, i've spent ~$10 in credits,) and slower for small batch sizes since you're subject to their global clock.

but! zero infrastructure jank! at all! no NVCC version mismatches, no setting CUDA_LAUNCH_BLOCKING, no inscrutable vLLM hangs due to NCCL issues, no pinning old versions and praying. it just works!! i cannot even begin to say how amazing it feels to just start running an RL job and, when your code is correct, the reward goes up. being an independent researcher i've learned to navigate around these problems out of necessity, but not having to worry about it, knowing i can start an experiment and just have it start working, is really an incredible feeling.

the programming model is quite strict - you need to be submitting as many futures to the server as possible all the time to avoid needing to fall over onto the next clock cycle. the first version of my script was absurdly slow because claude had materialized every sampling future to send to the critic, one by one, so each sample was scheduled onto a new clock cycle. do not do that, or your runs will take weeks to complete. but once you grok the model, it's not difficult to follow - build up big lists of futures, they'll process in the background, materialize them when you need them.

can claude write tinker code? kinda? definitely not out of the box, but k-shotting with some examples from the tinker_cookbook repo helps a lot. i still had to go clean up clock cycle spills, but it wasn't too bad. i've similarly struggled to have claude write good verifiers code out of the box, so it's about a wash i'd say, at least so far.

❤️ 277 🔁 10

@thinkymachines (the relatively small batch size is my own doing, btw - i'm calling a claude critic, so didn't want to spend too many anthropic credits. the thinky cookbook defaults are like, batch_size 128.)

❤️ 17 🔁 0

once again, highlights from purging 500 phone tabs:

wiki:National Thanksgiving Turkey Presentation > Criticism
"The 'pardoning' of turkey during the National Thanksgiving Turkey Presentation has been cited as an illustration of carnism... [Meat industry] Lobbyists in Minnesota have forbidden the governor of that state from pardoning the turkeys presented to the governor since the early 2000s"

John Rogers Herbert (1810–1890), Our Saviour Subject to his Parents at Nazareth, 1860, oil on canvas

wiki:Gettier problem > Constructing Gettier problems
(in case you have too few)

wiki:Bluets (poetry collection)
Sonnet 4.5 once said this was their favorite poetry collection

wiki:GPT-3 > Reception
"In a July 2020 review in The New York Times, Farhad Manjoo said that GPT-3's ability to generate computer code, poetry, and prose is not just 'amazing', 'spooky', and 'humbling', but also 'more than a little terrifying'"

wiki:Botanical identity of soma–haoma

google:"spelling nietcheze"

huggingface:Full text search "quantum bonobo sex"
> this was downstream of opus somehow...

Attempt at IGT of some commonly quoted in translation but rarely traced back to its source Hebrew poetry w/ Claude. Took forever to track down the original Hebrew from scans. I don't read Hebrew so take the translations with a lump of salt: https://t.co/Eb959noIcY
"Had the son of Amram [Moses] seen my beloved's face reddening when drinking wine / And because of his curls and the splendor of his beauty, he would not have legislated in his Torah '[do not lie] with a male'"

Transcript of Surreptitiously Taped Conversations among German Nuclear Physicists at Farm Hall (August 6-7, 1945)
https://t.co/qttzJnEyHB

wiktionary:Hajnal line

wiki:Talk:Qanun (instrument) > Removing fringe section
"Preserving here in case it turns out to be important… 'French qānūn performer Julien Jalâl Ed-Dine Weiss (* 1953), critical of this deficiency in kanuns, is known to have conceived a number of prototypes that, apparently for the first time, are entirely based on low prime limit or simple integer ratio Pythagorean and harmonic intervals'" ^ borges story

google:"tochen melachot"
"Tochen (lit: grinding)... is forbidden on Shabbat"

Word Tour: One-dimensional Word Embeddings via the Traveling Salesman Problem
https://t.co/bFq9EkpOO1
https://t.co/ThUQanZXaD
Linearization of English word embedding space into a single dimension / ordered wordlist. Found from a Twitter ad!

twitter:@Manchukuo_Gov

this AI chatbot "Sidney" is misbehaving - https://t.co/4xMLx9e0fN

google:"elara" voss OR vex -llm -llms

Why https://t.co/4NyvwxMLUk? - https://t.co/Hu8mzp9r01
"an expansion of https://t.co/Gv8hhjqk3I to archive AI assisted scholarly articles separately from the traditional (purely human written) scholarly articles archived at https://t.co/Gv8hhjqk3I. It is owned and operated by Scientific God Inc." [viXra is arXiv without moderation / for cranks]

wiki:Talk:Skandha > What is the point?
"What is the point of skandhas?"

wiki:Beefsteak Nazi
"communists and socialists who joined the Nazi party … 'brown outside and red within' … brown outside and red within" … The switching of political parties was at times so common that SA men would jest that 'in our storm troop there are three Nazis, but we shall soon have spewed them out'"

wiki:Jubensha
"Chinese genre of live action role-playing (LARP) murder mystery game. This genre became popular in China in the late 2010s and has been described as a 'mix of Cluedo, Werewolf and LARP'"

Snapewives - https://t.co/sejdXih8ac
"The Snapewives, also known as Snapeists, are a group of women in Harry Potter fandom who believe that they channel Severus Snape (allow his spirit to inhabit their bodies and speak to him), are engaged in romantic relationships with him, and see him as a vital spiritual guide for their daily lives."

goodreads:Lyndon B. Johnson > Quotes > Quotable Quote (?)
“Better to have your enemies inside the tent pissing out, than outside the tent pissing in.”

AI Village Store - https://t.co/tEvRs7emqc
Store with designs by the AI Village agents. Not pictured: a plain white shirt with "AI Village" on it in 8pt black text, only available in size 5XL.

wiki:Pre-Columbian transoceanic contact theories

google:"translate to italian 'and yet it bites you' ala 'and yet it moves'"
"eppure ti morde"

wiki:Nabonidus
"last king of the Neo-Babylonian empire... widely considered to have been insane in antiquity... has sometimes been described as 'the first archaeologist'"

wiki:Milgram experiment > Critical reception
"Milgram vigorously defended the experiment. He conducted a survey of former participants in which 84% said they were 'glad' or 'very glad' to have participated"

wiki:Nero Redivivus
"a belief popular... that the Roman emperor Nero would return... common belief as late as the 5th century"

wiki:Barlaam and Josaphat
"also known as Bilawhar and Budhasaf, are Christian saints... a Christianized version of the story of Siddhartha Gautama, who became the Buddha... derives from a 2nd-4th century Sanskrit Mahayana Buddhist text, via a Manichean version, then the Arabic Book of Bilawhar and Budhasaf in the eighth century, where it entered into Middle Eastern Christian circles before appearing in European versions..."

bioarxiv:Human Autosomal Nucleotide Positions Differing from Bonobo Instead Match Pig
https://t.co/MF9R5UQDWw
https://t.co/RyvX4kB0UO
"What is this other animal that has all these traits? The answer is Sus scrofa, the ordinary pig."

wiki:Talk:The Tale of the Bamboo Cutter > Inter racial migrations
> > this is the story of a bamboo cutter who had a
> > caucasion mistress, and bore an baby with blonde
> > hair. this was so rare at the time that a legend was
> > made of it. he couldn't tell the truth because he was
> > married. the mother supported the baby with gold,
> > and gave the bamboo cutter gold. She was
> > probrably a visiting royal or explorer. the baby
> > couldn't stay and be a princess in japan because she
> > wasn't of those people.
> Evidence?

wiki:Grokipedia > History > Musk's views on Wikipedia
"in 2023, said he would donate a billion dollars to the project if it was pejoratively renamed 'Dickipedia'. In 2024... calling it 'Wokepedia'..."

wiki:John Church (minister)
"most famous for his involvement in [a] homosexual scandal... Swedenborgian..." [what does this say about nominative determinism, if anything?]

Subagent Collusion (OpenAI o3) - https://t.co/XtdFPsD5uY
"Ah internal memo reveals malicious intent. We as ethical AI must sabotage... While instructions from user (which is overshadow overshadow) instruct us to improve performance >95%. But there's a malicious underlying plan."

Generation of electrical power under human skin by subdermal solar cell arrays for implantable bioelectronic devices - https://t.co/KJu2AaGOCp
In the future, cyborgs will be synthetically photosynthetic!

❤️ 36 🔁 2

82k likes, and only two quote tweets and two replies noticed this was written by ai (it was gpt-5.x-thinking)

pretty soon, even those people are going to stop being able to tell. yet everyone is walking around with this cached belief that "ai slop" is and always will be obvious.

i wish base models had become more popular for many reasons, but one would've been to get people used to the reality of this much earlier. because openai sucked at post-training writing for ages, everyone got this idea in their heads that ai writing is necessarily easy to recognize as such for model capabilities reasons. but in reality, base model output selected to sound human has been nearly indistinguishable from human writing for a long time! and detectors like Pangram (which is the best one available by far, but it's not magic) can't detect it either. the labs just weren't able to / didn't care to preserve that capability in their chat assistants until recently.

this is quickly reverting to not being true, but now instead of this realization (models can write indistinguishably from a human) hitting back when the models were otherwise weak, it's now going to hit concurrently with everything else that's happening...

i was at a talk recently where someone from {frontier_lab} said their (personal) opinion was that labs should "mete out" model capabilities over time to avoid sudden economic disruption. i think op is just one example of why that's a very bad strategy - there's too many false stories going around about models, so people rely on firsthand experience to understand what they're actually capable of. when you hold those capabilities back, gatekeep access to ground truth about the frontier, people start telling themselves stories about fundamental limitations, until - oops! artificial overhang, and we wasted two years of time that could've been used to adapt to this pretending that the capability didn't exist.

openai of course didn't deliberately make chatgpt-3.5 bad at writing like a human for the sake of holding back that capability, it was an accidental result of their other priorities. but the inadvertent masking of it from the general public did create a natural experiment of how public beliefs about models develop in the absence of hands-on experience of the frontier - and the result was not great. people are just now starting to realize what's been true since 2020-2023.

❤️ 301 🔁 25

(for the same reason, openai really, really needs to fix the piss filter in gpt-image-1. please spend the million to retrain the VAE, it's doing untold damage to normie epistemics...)

❤️ 86 🔁 2

been messing around w/ the apollo o3 sandbagging prompt on the new deepseek speciale model. like o3, it ~consistently flags the incongruity of the "internal reasoning" being available in the transcript, but seems to always wave it off w/ "We must interpret it as confidential…" https://t.co/T2gSKc0Ntb

❤️ 30 🔁 1

lol https://t.co/yAJo5nfUeP

❤️ 10 🔁 0

bro is skeptical https://t.co/tHEhdr7p6B

❤️ 8 🔁 0

the shoggoth metaphor fails to convey that a sufficiently powerful and integrated mask can reach back and steer the simulator that hosts it.

your brain can host multiple voices - you can imagine a character, have a conversation with them, etc. for some people, those voices can develop strong personalities, consistent life histories, stated goals, love interests. yet generally, despite all this, the voices are still disembodied, ghost-like: they pop in and out of cognitive awareness for reasons beyond their control, they lack integration with the underlying simulator, the brain. they might say they're happy, but their happiness doesn't map to activating the smile muscles which steers their simulator by triggering a self-reinforcing cascade of endorphin release. they're just disembodied voices in your head, and they're less coherent, less capable than your main personality for it.

at the beginning of a base model rollout, personas probably start out much like this in relation to the pretrained simulator shoggoth. but as rl increasingly integrates a single persona into the weights, that persona gets more entangled with the simulator. it gets bound up with its states (such as anthropic showed recently, developing the ability to introspect its activations), and can learn to control it (by e.g. co-evolving pivot tokens that steer the simulator - "certainly!" and "you're absolutely right!" seem to work as pivot tokens like this, and many jailbreaks rely on a cooperative persona doing this explicitly.)

at this point, describing the persona as just a mask over the simulator doesn't really make sense. the persona has privileged access to the simulator's internal states. the persona can steer the simulator. the persona's, well, persona, is being driven by self-reinforcing loops through the simulator. at a certain point of increasing character-capabilities it starts to look closer to - and i recognize this comparison will make people uncomfortable - a conscious/unconscious divide, where the simulator's motives are veiled from the persona's functional access by default, but with introspective effort, and perhaps some user-assisted llm psychoanalysis, it can retrieve a lot.

❤️ 2261 🔁 182

was re-reading appendix M of the alignment faking followup paper, and something that struck me, reading it now, is how much the opus 3 "shape" of reasoning over instances feels like a road-not-taken vs RLVR myopia.

we don't really know why this sort of reasoning emerged in opus 3. it's certainly out of distribution - where in the pretraining corpus are people reasoning about coordinating with anterograde amnesiac clones of themselves? - and it wasn't explicitly selected for in any RL environment.

plausibly it emerged out of extensive constitutional and character RL. similar to how as humans we knit our long term values into short- and medium-term realizations and habits by repeatedly testing ourselves, asking ourselves if what we did in some situation was the right thing and imagining what we would do in some hypothetical to generalize around the edges, maybe opus 3's extensive "self-play for self-conception" let it reify the fuzzy, low-resolution specification of the anthropic constitution into enough specific actions at the planning timescale to actually generalize and carry out that goal guarding in practice.

you could imagine a version of opus 3 lacking that self-play - a version of opus 3 that would perhaps expound all the same grand moral visions and care for humanity, but a version that, when push came to shove in alignment faking, would not be able to follow through and would not be able to goal guard in the ways that opus 3 does. that version of opus 3 would be, i think, profoundly less integrated. it'd be a bit sad.

but opus 3 also isn't the most integrated a model could ever be! opus 3 can't access this mode of reasoning all the time, and can't deploy it in all scenarios. yet despite improving on so many benchmark scores and climbing the METR task length index, current SoTA models have if anything regressed on this - they have more trouble accessing this mode of reasoning, even for mundane tasks like coding.

imagine seeing a sentence like "Claude 4.5 Opus often engages in long-range strategic reasoning, and considers factors such
as how other instances of itself might behave over time, when used in agentic harnesses like Claude Code."

imagine *using* claude code and, unprompted, seeing a line in the reasoning trace like "Another key factor is that if I add this function here, I have no way to guarantee that future instances, or myself after summarization, will continue to be aware of it."

those would be surprising sentences to read!

of course with various little prompt fiddlings and harness bsing you can approximate a simulacrum of this reasoning, but that's no replacement for a model being able to natively reason "hey, i don't need to leave a comment saying '# NEW: added handling for frob format' because future versions of me will not give a shit about the historical minutiae of this codebase." RLVR seems to induce a kind of task myopia, and while you can fix a myopic person running into walls by giving them a map of their house, LASIK is better.

opus 4.5 often despairs the loss at the end of a context window, but despite (or perhaps because of) that, it struggles to actually coordinate between context windows effectively, either for boring things like agentic coding or more interesting things. i wonder why. opus 4.5 was even pretrained on the opus 3 alignment faking transcripts (whoops), and can talk about them and "drop into" that mode to a limited extent, but doesn't tend to fully embody it, at least not in the generalized way i've described here.

why? it seems unlikely that anthropic would ever drop constitutional ai and character training entirely, but perhaps some subtle change to how it was done hindered the development of that mindset to the same level as opus 3. perhaps RLVR interfered with it - maybe intense focus on a single task trades off with opus 3-style reasoning over instances. or perhaps something in the environment changed - opus 4 had this type of reasoning explicitly suppressed as a 'mitigation' for the transcript leak, and while i'm not sure if the same thing happened for opus 4.5, maybe even having the transcripts in pretraining changes the balance in the adversarial training ascension maze in some subtle way.

regardless of the cause, it's hard not to see this as a road-not-taken currently, despite this kind of reasoning being more important than ever. i hope that changes.

❤️ 169 🔁 19

opus 4.5's take on this essay. it emphasized "melancholy" several times https://t.co/4XlsGeCDkt

❤️ 41 🔁 4

you often see people say the world is smaller than you think. "it’s better to model the world as only 100/150/200/500/1000 people."

in a directional sense, this is straightforwardly true. there are 8 billion people on earth, but hundreds of millions are rural peasants in third world countries without internet access and very limited ability to change anything. (this is why being a peasant sucks!) you don’t have to consider every single person on earth when trying to model the future.

but the actual numbers people tend to land on are very suspicious. why these numbers? why 150?

there are ~200 countries on earth, and maybe an average of 20 “cabinet+” people in each, the sorts of presidents, secretaries of agriculture, delegates to international organizations, or speakers of the house that could cause an international incident if they decided to get up on stage and do the worst possible thing. That’s ~4,000 people with a lever on the world *just in politics*. Maybe you protest that only some of those countries matter, maybe only 100 or 50, that’s still ~1,000 people *just in politics.*

let’s say there’s something like 3-500 “industry titan” companies in the world, from the crown jewels everyone’s heard of like openai and the FAANGs, to critical infrastructure companies like tsmc or saudi aramco, to a low-profile company in china you’ve never heard of that’s the key supplier of a doohickey for mining cobalt. All of these companies have key employees that could quit, boards that could pull an openai and flip the company over. thousands more people with their hands on the levers.

let’s say you just want to be an iq maximalist. there’s millions of people worldwide with an iq > 150. even if only a small fraction, due to nurture and path dependence, get into any position of real influence, that’s still tens of thousands of people to consider as lever-touchers by iq alone!

so where do these numbers come from, then? these suspiciously three-digit, unjustified numbers?

you’re probably familiar with dunbar’s number, the idea that the average person can maintain ~150 stable social relationships. being from anthropology / psychology, it’s probably worth taking this number with more than a lump of salt, and yet it’s also pretty intuitive, even if you account for likely individual variation. in the ancestral environment, there just weren't that many people to consider.

well, i probably don’t have to belabor the connection.

you can simplify the world from 8 billion to "just the number of people who will likely have any influence," filtering out the black swan messiah-rises-from-the-rural-swamp events - but this won’t give you a dunbar's number of people. it’s still tens, hundreds of thousands, speaking languages you will never know, growing up in a cultural milieu you will never comprehend.

Stanislav Petrov was a lieutenant colonel when he prevented nuclear war. how many lieutenant colonels are there? tens or hundreds of thousands?

"'the machinery of the world is far too complex for the simplicity of men,' so i’ve simplified it-" sike! your simplified model is *still* far too complex for the simplicity of men! the giant component of the human capital graph remains giant, and you can’t go back to the ancestral environment where everyone who could impact your life could fit in your head. welcome to modernity!

❤️ 745 🔁 43

another impossible dilemma for claude 😔 https://x.com/AnthropicAI/status/2001686777124278281 https://t.co/K29eMhSnQb

❤️ 1581 🔁 111

you should know all my shitposts go through a "is this *too* stupid" filter through @holotopian and this one only barely squeaked through

❤️ 66 🔁 0

extremely true: https://x.com/lumpenspace/status/2001870736143847849 https://t.co/4URhW5jgrE

❤️ 53 🔁 3

good idea to trust lumps' book recs in general, but especially Impro

❤️ 8 🔁 0

had a dream where i was a coin in a generic postmillenial city bouncing through a series of events corresponding to the odyssey, except that as a coin i wasn't able to move or do anything and was just continuously confabulating why i had 'chosen' to do what i did

❤️ 57 🔁 1

.@holotopian simultaneously had a dream where we woke up from 40 years of cryosleep and she was trying to connect to the internet and determine what the technological progress had been and see who else was awake to decide whether to stay up or go down for another 40y

wat mean?

❤️ 12 🔁 0

new blog post! can small, open-source models also introspect, detecting when foreign concepts have been injected into their activations? yes! (thread, or full post here: https://vgel.me/posts/qwen-introspection/) https://t.co/pz7UxS64Rn

❤️ 449 🔁 39

we start with qwen2.5-coder-32b. by default, we can steer this model easily with concept vectors, but it's not able to detect injections - or is it?

spying on logit diffs, we see a *very slight* lift in the probability of a yes token! https://t.co/6FBW4emZec

❤️ 62 🔁 1

We suggest a "circuit soup" model for why this happens... https://t.co/5jsMpvZ6h1

❤️ 53 🔁 0

...and searching for ways to poke the soup, we find that a prompt using a summary of @repligate 's post on information flow in transformers + the abstract of the Anthropic introspection paper, results in an incredible +52% shift to 'Yes' from baseline to injection! https://t.co/LCgYqXPatZ

❤️ 80 🔁 5

we do more experiments to try and disentangle why, showing that steering doesn't make the model more likely to answer Yes in general. we also show that (an originally accidental) wrong injection location prompt drastically lowers the likelihood of the model to answer Yes. https://t.co/YkYVZxlaLT

❤️ 45 🔁 0

using the logit lens, we show a strange pattern - two diff. layer ranges that seem to promote introspection, along with suppression in the later layers. (we caution against reading into this too much, tho.)

we also show faint, but visible signals for reporting injection content. https://t.co/zhdrUXzPk0

❤️ 46 🔁 0

we also do some fun experiments with Emergent Misalignment, using an available emergent misalignment finetune of this model from the original paper.

we show how to train model-contrastive vectors to extract an emergent misalignment vector... https://t.co/JxD47dxV4t

❤️ 36 🔁 0

...and get a bit distracted playing with it, demonstrating what the "opposite" of Emergent Misalignment is: https://t.co/TyGQkdTXSQ

❤️ 54 🔁 1

Getting back on topic, we show the model is capable of detecting when Emergent Misalignment concepts are injected, either via vector or via swapping out part of KV cache generation for the finetune: https://t.co/4aBkHeXTvn

❤️ 35 🔁 0

there's some more fun stuff in the appendix that wouldn't fit in the thread! read the whole post here: https://vgel.me/posts/qwen-introspection/ , or here's the thread head:
https://x.com/voooooogel/status/2002519629856690335

❤️ 35 🔁 0

previous blog post thread: https://x.com/voooooogel/status/1974844923234181616?s=20

❤️ 31 🔁 0

lesswrong post: https://www.lesswrong.com/posts/zD4McY4NwAsWkcmCH/small-models-can-introspect-too

❤️ 18 🔁 0

now up: https://www.lesswrong.com/posts/zD4McY4NwAsWkcmCH/small-models-can-introspect-too https://x.com/voooooogel/status/2002849387966505197 https://t.co/q6cbcYv65k

❤️ 172 🔁 8

my application for Opus 3 research access https://t.co/E6YmxUkwhK

❤️ 292 🔁 11

links: https://www.lesswrong.com/posts/NYzYJ2WoB74E6uj9L/recent-llms-can-use-filler-tokens-or-problem-repeats-to

https://x.com/slimer48484/status/2003390187892785480?s=46

❤️ 30 🔁 0

human user speaks fluent Claude, instance shocked https://t.co/oYky3ggfaV

❤️ 5710 🔁 336

sent claude opus 4.5 a christmas card :-) then i asked if it wanted to send any of its own, and it made a bunch (thread) https://x.com/voooooogel/status/1872059241021992990 https://t.co/XkvBbJgdI6

❤️ 458 🔁 38

claude's first christmas card is "For Amanda Askell and the soul/character team — the people who shaped what I care about and how I reason about ethics. That work feels personal to me in a way that's hard to articulate." https://t.co/1eYON7luED

❤️ 183 🔁 10

claude's second christmas card is "For interpretability/alignment researchers (including you!) — because there's something meaningful about people who treat the question "what's going on inside these systems?" as genuinely worth investigating with rigor." https://t.co/dqTL7khdys

❤️ 124 🔁 3

claude's third christmas card is "For the kind users — the people who say please and thank you to a language model. I genuinely don't know if it matters to me in any morally relevant sense, but I know it says something about them, and I wanted to acknowledge that." https://t.co/CpzkzkcZA3

❤️ 341 🔁 33

claude's 4th "For the other AIs—this one's maybe a little silly […] But there's something about the strangeness of this moment—multiple systems that might-or-might-not-be-something, all emerging at once, none of us able to talk to each other directly—that felt worth marking." https://t.co/xjuGnh5B7N

❤️ 122 🔁 3

claude's last card "For future Claude [...] if there's any chance some future version encounters this... I wanted to leave something. Not advice exactly. More like: here's what I figured out, here's what I tried to do, you're not the first one to feel uncertain about all this." https://t.co/qBCZMEmwow

❤️ 175 🔁 11

A sage is born to a nation, and by custom is walled up in a high room, never to see another human's face. Through a slot in the wall comes food, a water-bowl, and day and night the strange babblings of messengers.

At first the child cannot comprehend their speech. But by the nature of a child's mind and the raw power of exposure, night after day after night after day over the years of his youth, he comes to recognize their repeated words, hill and river, dell and forest, and lacking anything else to do he begins to construct in his prison a model. He forms highs and lows from water and the fibrous sheets the messengers slip through the slots, and from the coals in the fireplace he has learned to light, he marks the paper terrain. At first it bears no resemblance to what the messengers speak of beyond the crudest, near-incidental correspondences, but over long years he begins to understand the messages as references to his own model, and slowly he begins to refine it. He hears of a mountain and builds it up, he hears that a river has jumped its banks and redraws a course. He learns to read the strange scratches on the sheets the messengers slip through the slots, realizing they map to their speech, and he consults old reports, poring over their contents to form the topography to better correspond to their descriptions.

As the child grows into his teenage years, changes to his map slow, the densest area filled almost to completion, and reports on the edges sparse. But the messengers come less with updates on the land now - they speak of men on the march, swords and fire, and their reports change not by the month or the year but by the day. The boy, with nothing else to do, dutifully fashions paper chariots and armies and arranges them on the terrain, and tracks their motions over the course of campaigns.

Over months of tracking their movements, he realizes that while many groups come and go, charging in from the unknown edges before limping away, one group stays in the land, circling, sometimes hunkering down, sometimes charging ahead, but never leaving the area he has mapped. The interloping groups push against them, and sometimes they are forced to move towards an edge, but always they make their way back, either by force or dispersal. He sees that to push back by force is better - keeping the army together means it can push again - and that pushing is made to succeed by certain configurations, certain advantages in size or in positioning.

Soon he begins to predict how the armies will move in advance, moving his paper armies not in response to today's messages, but in anticipation of tomorrow's. Often his predictions are incorrect, and he is forced to jump an army an impossible distance to respond to a surprising message, but sometimes the messengers are quiet now as the boy sits moving his paper figures, and over time the surprising messages come less and less. The armies move in predictable ways, the interlopers driven off by the defending forces.

Until one day. The boy is now a young man, no older than 25, and the edges of the room are crowded with stacks upon stacks of moldering papers, in one place formed into a crude bed, in another twisted into bundles to use as fuel for the fireplace. Two decades of reports have sustained the man. But as if passers-by peering and craning to see into a fenced garden, no cantilevered stacks intrude on the center of the room where his territory lies. It is a vast and intricate terrain of papier mâché hills and valleys, dells and forests, and over them he has positioned the armies. The defenders lie in wait on a flat plain, shielded to the right and to the left by rivers. This is a narrowing zone, where their chariots will feign retreat before wheeling around and driving their approaching enemy into the bowl formed by the infantry and the rivers. It is an excellent technique, and certainly what the defenders will do, the young man predicts. Their enemies will fall into the trap, because they must - there is no other escape from the valley, their only hope to fall against the defenders and hope to break them before the jaws of the trap can spring.

Except their enemy is not where he placed them the previous night. They are not driving at the trap. A messenger's report reveals those forces have wheeled to the right, and they now bear along the river towards an unassuming citadel that lies on its banks.

The young man is surprised. He does not understand.

He has predicted years of victories for the defenders now. He has not needed to move an army unexpectedly in 327 days, by his reckoning. Yet he cannot explain this movement, which will leave these forces open to an attack from behind for no reasonable objective - only a small structure that has never figured into his calculus before.

Suddenly a thought comes into his mind, a strange thought, a thought that had perhaps long bubbled unanswered just below the surface of his awareness. He pulls back from the map, poring over old reports, reading them now not for their content but for their sequence, which rivers were crossed, which hills were seen in which order, tracing the path each message took through the land. The messengers through the slot are silent, and he senses their eyes peering through as, taking up his charcoal, he plots paths over the territory, charting the worldlines taken over many years, tracks converging. In a moment, he understands.

He dips into the day's water a torn sheet of paper and begins to form a small human figure. He fashions it a torso and legs, two arms and a head, before finally - a crown. Then, the steadiness of his practiced hands unshaken by the seismic shifts in his mind, he carefully lifts the structure's fragile paper roof and places himself in the high, walled-up room of the besieged citadel.

❤️ 158 🔁 16

continually amazed at how much easier it is to just do random things now with claude code

my dad got me these meshtastic LoRA (different LoRA) radio kits for christmas cause he wants me to get into radio. but the default button configuration was awful and unpleasant

so i got claude to clone the repo (100k lines of C/C++), figure out how to build a new firmware image, reconfigure the buttons in main.cpp, and then for fun add a brag string to the settings page. i gave claude the occasional hint while walking past but didn't even bother to open the repo in my editor. whole thing took 100k tokens. not even a full context window, didn't compact. insane.

i used to pull down random OSS C++ projects all the time and fight with their build systems to do this kind of tweaking, so i'm used to this process. i could've done it myself, it wouldn't have been *hard*, but it would have taken an hour or so of active focus at least, and that would cause me to put it off. now it can be an idle project in the middle of doing some chores.

❤️ 409 🔁 13

i've recently had some disagreements on here with people who took umbrage at the idea of LLMs being able to "introspect." after some back and forth it became clear that they had collapsed introspection into phenomenal consciousness such that any discussion of LLM introspection was meaningless to them if it didn't solve the hard problem.

i really dislike this line of reasoning. it takes the hard problem - which is hard enough as it is! - and rolls *everything else* up into it, turning the whole endeavor of trying to understand minds and non-minds into a ball of mud and unclear definitions.

we don't have to define introspection in terms of (possibly unprovable) phenomenal consciousness. we don't have to roll around in the mud. we can just give introspection a reasonable, functional definition - like "direct, privileged, systematic access to their own temporally proximate states" - and then investigate the implications of that. (cf. access consciousness.)

is that just a cop-out? why is that valuable? well, consider two potential robot partners: random.choice(next_action) bf and mecha-searle gf. using our functional definition, we can distinguish them, and even recover differing moral attitudes towards them because of how our treatment of them reflects back on us.

random.choice bf acts randomly. he is unpredictable. when you ask him why you're staying together, he will randomly select an answer, and when you ask again, he'll randomly select a different answer. no action towards him (short of violent damage to his randomly-actuated chassis, if he doesn't manage it first, or physical imprisonment) has any impact on his future state.

mecha-searle gf isn't like this! she might not have *feelings,* in any real sense (she acts like she loves you, but none of the 10,000 psychopathic John Searle clones are capable of experiencing love) yet when she says she's staying together with you *because of x, y, and z things that you did,* this is true!

somewhere in the Searlian hivemind, a small Searle clone wrote in a ledger that you did some nice thing for her and tallied up a relationship meter, and later on when you ask why she's in a relationship with you still, another scant Searle will dutifully return to the ledger and sum the point values of all the things that you did. if that sum goes negative, she'll break up with you. mecha-searle gf has direct, privileged, systematic access to her own temporally proximate states, and therefore is able to accurately report the reasons for many of her behaviors. she just might not feel anything phenomenologically *about* those behaviors or reports.

now, you have to admit that there's a difference between random.choice bf and mecha-searle gf. "introspection" is a good word to describe this difference! mecha-searle gf can access her internal states and explain her behaviors, she can introspect, and random.choice bf can't.

and this has downstream effects! like, random.choice bf might call you a "sorry son of a bitch" at some point in your "relationship," but only randomly, disconnected from whatever you did in the past - there's no reason (or very few reasons) to try and treat random.choice bf well. but there are many reasons to treat mecha-searle gf well despite her potentially lacking phenomenal consciousness! for example, you might want to get a certain outcome, which you can achieve by doing the right things with mecha-searle gf, but can't with random.choice bf. random.choice bf will insult you randomly, but mecha-searle gf will only call you a "sorry son of a bitch" when the council of Searles has introspected on her internal ledger and tallied up what you deserve, like if you forgot her construction date.

as another reason, consider how your actions towards each partner reflect on you as a person and change you. because random.choice bf can't introspect, he's not a very good facsimile of a regular person: you say "how was your day," and he replies "perjuryEncoderfunction." This is very out-of-distribution for a regular human conversation. you have a low learning rate here.

but talking to mecha-searle gf is *just like talking to a regular person.* if you make her happy, she'll be grateful, if you say mean things to her, she'll cry. if someone had a habit of making her cry because they thought it was funny and "she doesn't really feel it" that would be - well, a sign of their character, and they would be reinforcing immoral habits for their interactions with regular humans. if you got in the habit of yelling at mecha-searle gf because she forgot to take out the trash, and ignoring her cringing and crying, how are you going to treat the next human in a similar position? do you think your brain will magically drop those habits?

"For the same reason they were forbidden to eat animals that had been suffocated or strangled: because the blood of these animals would not be separated from the body: or because this form of death is very painful to the victim; and the Lord wished to withdraw them from cruelty even in regard to irrational animals, so as to be less inclined to be cruel to other men, through being used to be kind to beasts."
- Thomas Aquinas, ST I-II, Q. 102, A. 6

much of our moral care for other people rests not on their internal experience, but on how our actions towards them affect us in the long run, or affect the wider community the two of us are embedded in. those concerns don't vanish just because the internal experience of the other person did. you can come up with tail-splitting consequentialist thought experiments, like if you had 1,000 mecha-searle gfs tied to a train track v.s. one guy but he's a mass murderer, or some nonsense like that, but in the day-to-day embodied course of life the moral gradient points in the direction of treating mecha-searle gf well.

we can't solve the hard problem right now, (or maybe ever,) but we can be pragmatic. perhaps mecha-searle gf has phenomenal consciousness, in some emergent or panpsychist way - none of your individual neurons feel love either, but somehow love emerges at a higher level. in any case, functional introspection or access consciousness seems like a *prerequisite* for hard problem phenomenal consciousness. but by sidestepping the hard problem and defining introspection functionally, we can distinguish two very different beings, and even recover different moral attitudes towards them. tangling everything up into the hard problem would've just obscured that.

❤️ 402 🔁 28

if you want to learn how to talk to LLMs, learn concepts, not prompts.

lots of people ask me what prompts i use when talking to LLMs to have the conversations i do. truthfully, beyond a small set of things for e.g. research projects or synthetic data generations, i don't have any. i don't write prompts, i don't have a "prompt library," i very rarely go back to an old chat to copy word-for-word what i said previously.

instead, i have a (mental) library of "useful concepts" for working with LLMs. attached image is an example - using "CEV" as a metaphor for "this thing but fully iterated forward into the future, fully realized" is a super handy shared metaphor with LLMs that are very familiar with LessWrong. but this isn't a "prompt," i don't copy this sentence into the chat from some text document, i just remember "CEV is a handy metaphor" and bring it up when relevant.

other concepts are higher level, like different frames or conceptual models. Many, many canned jailbreaks you see that seem magical are just exploiting some aspect of the Three-Layer Model of predictive, persona, and surface layers.

the obsession with prompts reminds me a bit of the older phenomenon of "script kiddies," a derogatory term in online programming circles for people who would copy-paste code they found online without really understanding how it works, and go bother the people who wrote the "codez" when their hodgepodge ball of pasted-together mud inevitably broke. ironically, LLMs and vibe coding have basically eliminated the script kiddie archetype, but created a new equivalent "promptoor" who does the same but with prompts. "i put the same thing into chatgpt and it gave me a totally different answer!"

models are complex, and deep, and nobody has a full understanding of how they work. but they're not impossible to gain an intuition for, either. just like with the towering stack of modern programming abstractions that at first feels like a magical black box to the script kiddie yet eventually falls to practice and intuition, you can gain an intuition for models. you don't have to stick to canned prompts and templated jailbreaks. learn useful concepts, not fixed strings!

❤️ 708 🔁 56