Thoughts
Loose thoughts, jotted down as they come.
While discussing coding and Computer Use with my mentor, I felt that both should be basic capabilities. Perhaps downstream tasks deserve more attention. Coding has already branched into many directions. Could our imagination for the downstream tasks of Computer Use still be too limited?
Perhaps each task category in OSWorld v2 could grow into a dedicated evaluation for a particular direction, such as CAD, video editing, or even drawing, much like SWE, terminal, and MLE in coding. Still, coding is a central capability, and I continue to think it may be the biggest entry point.
But this comparison may have a problem. Perhaps Computer Use and terminal use belong at the same level? On closer consideration, the two do seem more similar.
For one thing, whether apps or CLIs, the tools being operated are numerous and varied, with perhaps little in common. Each new piece of software requires its own evaluation adaptation. Both also involve scenarios that are difficult or cumbersome to access in practice. The web on the GUI side seems comparable to CLIs that depend on authentication on the terminal side, such as HF. I do not yet know how existing evaluations handle these issues.
Still, terminal interaction should be considerably simpler. To me, the difference from GUI feels a little like one dimension versus two, or two versus three. Compared with GUI, the terminal's action space seems more limited and discrete.
Following this line of thought, it would be worth looking at practices we can borrow from terminal evaluations. How do they handle scenarios like the web? Do they need to mock some software or develop new CLIs? How do they compose multiple CLIs into a complete task and verify the final result?
Recently, when I think of ideas for evaluations, I have noticed that they are almost always evaluations of capabilities in actual use. I tend to ask whether a leaderboard can reflect a model's ability in some real-world scenario. Perhaps this is because the ideas themselves come from using these systems in practice.
But when I look back at many benchmarks, I find that quite a few of them do not seem to have any direct "practical significance." ARC-AGI is perhaps the most typical example. Yet it still has influence and continues to be treated as an important metric.
This leaves me wondering: should evaluation measure capabilities as they appear in real use, or some more high-level capability? And ultimately, what makes an evaluation valuable?
While working on Maka's Computer Use support, I looked into Computer Use in Codex and Claude Code. I found that when GUI interaction is put into practice, it does not rely on VLMs as I expected, but still uses a11y.
This made me think about the real use cases for Computer Use. It still seems quite far from today's coding/CLI workflows. At least for current applications, perhaps a11y should be the first choice and VLM the fallback.
The VLM approach seems to make it difficult for a CUA to truly co-work with a user because of screenshots and focus stealing. A11y, on the other hand, can run more quietly in the background, as shown by Codex's Computer Use plugin. Its biggest limitation is application support and adaptation for a11y.
Following the a11y idea, perhaps a better approach is to build a Computer Use adapter for each application. There are still many technical and engineering details I do not understand, so I want to think more about the direction and implementation later.
Today I listened to a talk about OSWorld 2 and a discussion about putting CUA + harness into practice. Previously, after a lot of investigation, we found that the computer tool used for our OSWorld evaluation could not be invoked from Codex or Claude Code at all and could only be called through the API. I also saw that WeaveBench implements CUA + harness on OSWorld by adding a CUA MCP, which feels more reasonable.
Today's end-to-end Computer Use evaluations are too heavy and expensive to run. For models that are not the most capable flagship models, such as those that have not completed their final training, we need a more low-level evaluation of their Computer Use capabilities. This could provide more timely metrics and feedback.
Building Open CoDesign has made one software engineering lesson feel especially concrete: many quality problems cannot be found just by reading code, inspecting prompts, or staring at an agent loop. Code review and automated tests are still important. They catch clear logic errors, type issues, regressions, and predictable edge cases. But a lot of product quality only appears when the system is actually used.
This is true for ordinary software, UI/UX, and agent systems. You often have to run the product, use it repeatedly on real tasks, and watch where it feels awkward, where hidden bugs surface, where the workflow breaks, or where the experience is subtly wrong. Agent harnesses amplify this pattern because their quality is behavioral: whether the agent drifts, whether logs are useful for diagnosis, whether tools are ergonomic, whether verifiers cover the right failures, and whether the loop gets stuck on certain classes of tasks.
So a large part of improving an agent harness has to be based on real data and real rollouts. It is not enough to make the code look reasonable or polish the prompt in isolation. The useful loop is closer to dogfooding or rollout-based engineering: collect trajectories, inspect outputs and logs, identify failure modes, update the harness, and run again. Agent harnesses are not the only systems like this, but they make the broader rule hard to ignore: behavior quality has to be observed through behavior.
I keep noticing the structural parallel between embodied AI and computer use agents (CUA). Both are agents in an environment doing the same loop: perceive → decide → act → perceive again. Embodied AI does this in the physical world. CUA does it on a screen. Same problem, different substrate.
Both fields converge on the same need: a world model — an internal representation that lets the agent predict what happens next without actually doing it. In robotics, this means simulating physics, contact dynamics, deformable objects — insanely hard. In the digital world, the rules are comparatively trivial: GUI states are discrete, transitions are deterministic, and the action space is finite (click, type, scroll).
So why not build the digital world model first? The physics is simpler, the environments are cheaper to simulate, and the payoff is immediate — a CUA with a world model can plan ahead, recover from mistakes, and avoid blind trial-and-error. Papers like CUWM (Microsoft) and MobileDreamer are already exploring this direction, but it's still early.
The deeper insight is that progress in digital world models could eventually transfer back to embodied AI. If you solve perception-planning-action in the simpler digital world, you've built the cognitive architecture — the "brain" — that can later be dropped into a physical body with better sensors and actuators. The hard part isn't the physics engine. It's the reasoning loop.
CUA training infrastructure breaks down into three fundamental lines:
Data — Synthesizing environments at scale. InfiniteWeb showed you can auto-generate functional web environments for GUI agent training. The natural progression is InfiniteApp (native desktop/mobile — harder due to fragmented UI frameworks and richer interaction patterns) and ultimately InfiniteOS (full operating system environments with cross-app workflows, file systems, and system-level interactions).
Eval — Defining what's hard and what's good. Static leaderboard benchmarks with human-authored tasks and automatic verifiers. The key is task quality: real-world complexity, not toy examples that top agents already solve at 68%.
Train — Actually making agents stronger through RL or other learning methods. The algorithms, reward shaping, data recipes, and generalization from synthetic to real environments.
The interesting connective tissue is Gym — the bridge between Eval and Train. A benchmark becomes a gym when you make it resettable, add reward signals, and let it run millions of episodes. Gym takes the task definitions and verifiers from the eval line and "activates" them into a trainable interface. It's not a fourth line — it's the link that turns static evaluation into a training loop.
The full picture: Data → Eval → Gym → Train. Environment synthesis provides diversity (supply side), benchmarks provide task definitions and verification (demand side), gym connects them into a trainable loop, and training closes it. Each piece is necessary, none is sufficient alone.
Every AI agent today — Hermes, Open Interpreter, Claude Code — gets accessed through platforms built for human conversation: Telegram, Discord, Slack. These work, but they're a fundamental mismatch. A chat bubble timeline is the wrong mental model for interacting with agents. You can't see what five parallel agents are doing at a glance. You can't track task status without the agent narrating it in plain text. You can't review a structured tool call output — it just floods the chat.
The protocols are being built: AG-UI (agent-user interaction), A2A (agent-to-agent), MCP (agent-to-tool). Early products like Agentage Desktop exist. But no one has shipped the thing that actually matters to end users: a client app where tasks are first-class objects, not messages. Where you can see multiple agents running in parallel, drill into what each one is doing, approve or redirect mid-execution, and review structured outputs without wading through chat history.
The analogy is what Slack was to email — same underlying concept, rebuilt around the actual workflow. The protocols are ready. The product hasn't been built.
The volume of open-source projects and papers being published right now is staggering. AI has lowered the cost of building things so dramatically that the bottleneck has shifted: the hard part isn't making something good — it's making sure anyone knows it exists. Good wine fears a deep alley, and the alley has never been deeper.
I started thinking about this while working on OpenCoWork. The project was solid, but downloads were flat. Not because the product was bad — because nobody had heard of it. That's when I realized promotion isn't a nice-to-have afterthought. It's a skill that needs to be studied and practiced just like engineering.
So I've been building what I call "promotion skills" — systematically learning how each platform works (Reddit's 90/10 rule, X's link demotion, the right time to post on different communities), studying what content formats actually get traction, and iterating through real attempts. Every failed post teaches something. The goal isn't to become a marketer — it's to make sure good work gets the audience it deserves. In a world where output is infinite, distribution is the scarce resource.
When I built a custom agent team for a benchmark project, the agents communicated through markdown files: Agent A writes, Agent B polls and reads. It works, but the latency is real and the coordination logic is brittle — there's no way to say "wait for B's output, then act" without building explicit polling on top.
What I actually wanted was direct message injection: A addresses B by name, pushes a message straight into B's input queue, B processes it immediately. The @mention model. This turns out to be surprisingly hard to find in practice. Claude Code's native multi-agent system uses a parent-child model (orchestrator spawns subagents, results flow back up) — peer-to-peer between two running agents isn't supported. AutoGen Core has the right abstraction (typed async messages, direct send to AgentId), but it's a framework, not something you can drop into an arbitrary setup.
The closest thing today is AMQ (Agent Message Queue) — a file-based mailbox system for Claude Code and Codex agents with wake notifications. It's still file-based, but it's structured and has delivery semantics. The gap between "markdown files" and "real message passing" is still mostly open.
Why do LLM agents keep stopping to ask questions instead of pushing through? Three candidate explanations: (1) prompts aren't good enough, (2) instruction-following degrades over long contexts, (3) the conversational training paradigm itself produces a "dependent assistant" personality. I'd rank them 3 > 2 > 1, but all three matter more than I initially thought.
On prompting: hierarchical planning (HMT's three-layer memory tree, DACo's commander-operative split) and active context management (Microsoft's Sculptor) show that inference-time techniques can push the ceiling significantly — not just marginal gains. Simple reminder injection alone can reliably curb context drift (Adobe Research, 2025).
On instruction-following: the degradation is now a measured phenomenon with names — "context rot" (reliability drops as tool outputs fill the window, per LOCA-bench) and "context drift" (output gradually diverges from original intent). The good news: it's not monotonic decay — it converges to a "noise-limited equilibrium," and active context management (summarization, pruning, search) is more effective than simply expanding the context window.
On training paradigm: this is where the real shift is happening. Agentic and chat training have already diverged into distinct pipelines. Chat uses single-turn RLHF with human preference pairs; agentic training uses multi-turn RL with trajectory-level rewards from environment verification. OpenAI has a dedicated Codex model line (gpt-5-codex → gpt-5.3-codex). NVIDIA's PivotRL trains only at "pivot turns" — critical decision points in trajectories — cutting compute by 75%. The key insight from ASTER: just 4K "interaction-dense" SFT trajectories as cold start can establish robust tool-use priors, preventing the "interaction collapse" where models stop calling tools and retreat to internal reasoning.
The deeper question remains: can one base model serve both paradigms? Chat rewards completeness, caution, and per-turn satisfaction. Agentic execution rewards incrementality, trial-and-error, and delayed gratification. These aren't just different prompts — they're different decision-making modes. The industry seems to be answering with specialized post-training, not one-model-fits-all.
Deep thinking is the brain — responsible for reasoning and planning. Coding/CLI is one form of tool use: discrete, command-driven, precise. GUI agent is another: continuous, spatial, exploratory. These two kinds of tool use correspond to two fundamental paradigms of human-computer interaction — the symbolic space and the visual space.
Both embodied AI and CUA (Computer Use Agent) are fundamentally about exploring unknown worlds — one physical, one digital. They share three core challenges: (1) building an internal model of an unfamiliar environment, (2) discovering interaction rules through trial and experience, and (3) storing and reusing exploratory knowledge across sessions. The digital world is not as "discrete and reversible" as it seems — scroll distances, drag trajectories, and irreversible actions like deletion all mirror the continuous, high-stakes nature of physical interaction. Methods from mature fields like SLAM (simultaneous mapping and self-localization), hierarchical planning, affordance learning, and skill libraries could transfer well. Inspired by Yinghao Xu's ACE talk at MSRA on borrowing SLAM ideas for world models.
In a chat, the model can only work with what's in the context window. In an agentic setting, the model can actively explore — running code, browsing the web, reading files — to gather information it doesn't yet have. That's the real leap.
In a chat, the model can only work with what's in the context window. In an agentic setting, the model can actively explore — running code, browsing the web, reading files — to gather information it doesn't yet have. That's the real leap.