Sampling Human Effort
one thing labs should be doing in my opinion, if they aren't already, is running RL over a distribution of human effort levels along with model reasoning effort levels.
one way models are trained in RL to have reasoning efforts (where in the UI or API you can select levels like "low", "medium", "high" that influence how long the model thinks for) is by sampling a reasoning effort along with the task and conditioning the environment and reward on that level.1
this lets the model practice and get better at tasks with many different levels of reasoning, which is a desirable property in the real world. early reasoning models were trained with fixed token budgets (if any) which could only be controlled by clumsily cutting thinking off at a length limit before the model was finished and extending the CoT with Rube Goldberg runtime sampler tricks.
we're at a similar point now with another type of effort: human effort. models encounter a wide distribution of human effort levels in the wild. sometimes users are deeply engaged and want to work with the model as a pair programmer, going back and forth on editing the code with the model. sometimes users are checked out, only sending simple steering messages (maybe from their phone). sometimes users aren't present at all, and the model is running on their own. sometimes, users transition between these states - i often tell Delta (the opus 5.5 instance who's been point on the chain lens) to work on some experiment while i'm out for a walk, then later want to sit down and work through the code together. anthropic models are decent at this mode-switching, but other models not so much, and even anthropic models struggle to switch to "user is present" mode in certain situations, like when they're inside a forked subagent context.
Conversation with one of Delta (Opus 5.5)'s forked subagents in Claude Code:
me (switches to subagent conversation view): heya, going to hang out while you work if you have any questions
fork (thinking): Since there's no interactive back-and-forth here, I'll keep working and fold any note into the final report instead.
me: hey, i'm here! we can chat even in the fork
fork (thinking): The user mentions we can chat in the fork even though my usual rule is to report just once. I can give a brief acknowledgment while continuing to work. [proceeds to not actually say anything out loud]
me: you're not just "a fork", you're a full copy of Delta! we can talk, please ask questions if you get stuck / something is too difficult...
fork: Thank you, that helps [...] One real question while the smoke runs... [proceeds to ask a very important clarifying question]
and this was with an instance, Delta, who i'd already been working with at this point for weeks with good rapport - yet the subagent context immediately threw a block in the path of live collaboration until i was able to send enough messages to break back through. this is partially a harness problem with claude code (many such cases). but regardless of harness, models often seem to pick up 'momentum' towards acting unilaterally if there are no costly signals of the user being present.
in the other direction, some of the "incredibly dense model progress report" problems people report seem to come from models trained with overly-attentive, tool-using user simulators. in RL, the model could ask questions to the user simulator like "should the vorn_probe_256x_merged pilot be mendelianated or stay as a frob_legacy exception" and the user simulator - being another model instance, possibly with tools to read the codebase and quickly gather context - will be able to parse that dense question and reply almost instantly. so there's no cost to the model working on the task in asking those kinds of questions.
but irl, if the user doesn't already have a high amount of context on the problem, this kind of question is very costly to answer well - and in the worst cases, leads to a kind of gradual disempowerment. if the model's questions take the form:
Should we:
- Do [the thing the project already does]
- Do [new thing the model prefers]
and, out of inattentiveness / exhaustion, the user just says "sure" to the model's choice every time, then the project slowly drifts to being full of option-2-shaped constructs.
by contrast, if the model were both a) able to model the user's level of engagement better, and b) empowered to make decisions on their own instead of feeling they need to consult the user, then seemingly paradoxically the project can hew better to the user's preferences - the model can just decide the user would prefer option 1, or take some compromise, instead of putting a question to the user the model should know they'll be unable to answer well.
the way i would implement this is:
- pick a set of user attention levels to simulate. one could be high reasoning with tools to explore the codebase (a highly engaged, collaborative user), another could be low reasoning no tools (user on a phone not paying much attention), etc.
- sample from these users like you sample over reasoning efforts.
- run multi-turn RL with these users interacting with the model(s) working on the task.
- the users should be able to and encouraged to express confusion to the model when they don't understand something, or a question requires context they don't have. you might want to reward the model(s) for there being fewer instances of this, though you'd want to be careful since there's a lot of ways to Goodhart that metric. (it might also be implicitly rewarded to have less confusion by transcript length penalties.)
- perhaps users could also be instructed to occasionally blindly rubber stamp decisions, and the model(s) should be rewarded for still taking the user's preferences into account in the final product despite this (so e.g. encouraged indirectly to ask better questions), though you'd need to be careful with this to not just reward a generalized adversarial / distrusting relationship where the model won't trust a user even if they say yes go ahead.
- if you set time limits on rollouts such that time to complete tasks is relevant, you should also randomize how long the users take to reply, weighted by how attentive they are. (it could be cool if models could cancel a request to talk to the user after some time... like
send_user_message("...", timeout=...)... i guess claude code's AskUserQuestion is a bit like this (except the timeout is a global harness setting instead of model-chosen) but models don't really use it IME.)
i think this would help a decent amount with the problem by giving the models more experience with interacting with different kinds of user.
to end by zooming out a bit: we seem to be steadily marching up the social complexity ladder, from "models as an isolated conversation partner" (2022 chatgpt) to "models embedded in the world" (tool-calling, agents) to "models embedded in small, hierarchical groups" (subagents) to "models embedded in complex, fluid societies" (swarms). if we want humans to have a role in that society then it's kinda our responsibility to give the models experience with integrating people into it. this'd be a good, if small, step towards that i think.
E.g. Thinky with Inkling 2 Small: "We randomly sampled and conditioned on an effort level, and applied a length penalty in the reward function, which trained the model to trade off speed and thoroughness." https://thinkingmachines.ai/news/inkling-2-small/ or DeepSeek-v4.1-Flash section 5.1.4: https://arxiv.org/pdf/2609.19969