TL;DR
Real coding work involves feedback and evolving requirements; single-turn correctness alone misses how well agents collaborate with users. We contribute a benchmark and an interaction evaluation framework for coding agents.
- Benchmark: 141 executable tasks across six coding domains, with instructions, environments, verifiers, and reference solutions.
- Interaction evaluation: Runs multi-turn episodes in persistent workspaces, generates adaptive user feedback, and records progress. Functional tests assess task completion; an LLM judge assesses the interaction.
Why evaluate beyond the first turn
Coding work often continues after an initial answer. A user may report a failure, clarify a requirement, or ask for stronger tests. A useful agent must carry earlier context forward and make the next revision work.
A good user experience means completing the task with little unnecessary effort, understanding complex and changing instructions, raising important questions or risks at the right time, and giving useful, clear and complete responses. UXInteract currently captures part of this picture through first-turn and final task success, success across repeated runs, and an overall judged interaction-quality score. These measures help assess task outcomes and the interaction, but do not separately quantify every aspect of user experience. More detailed evaluation of interaction efficiency, intent understanding, initiative and response quality would require additional measures.
The example below shows a user LLM asking a coding agent to build and refine a 3D chair, with follow-up requests even after a passing check.
The Pelican Test
The same brief calls for a pelican riding a bicycle across different art styles. See what changes when the agent can revise its work after feedback.
Single turn
Initial responseMulti-turn
After feedback and revisionHow we evaluate interactions
An episode follows one attempt at a task, from the initial request through any follow-ups. The agent continues in the same conversation session and workspace, while we evaluate both the code it produces and how it responds to feedback.
The agent receives an instruction and an isolated workspace, then edits code and runs tools. A verifier checks the result after each round. If more work is needed, the user LLM gives a follow-up based on the current progress, and the agent revises its work. By default, the loop stops on a full verifier pass or at the turn limit.
Scroll horizontally to view the full figure.
Feedback responds to the agent’s work. The simulated user receives summaries of earlier turns, code changes and a high-level verification outcome. It can request targeted debugging, stronger tests or requirement clarifications. Hidden tests and raw verifier logs stay private, and a configurable turn budget limits the interaction.
A separate judge assesses interaction quality. After the episode, an LLM reviews the recorded interaction and verification evidence. It considers correctness, the latest user requirements, context continuity, corrections, testing and user effort. Saved rounds can also be judged individually, using only the interaction up to that point.
Read the metrics together
- Pass₁ (First-turn success): The average per-task success rate after the initial request, using the task verifier.
- PassF (Final success): The same verifier-based success rate at the end of the episode.
- Pass² (Success in both runs): How often two rollouts both meet the correctness judge’s success threshold. With two attempts per task, both must succeed.
- Mean IQ (Judged interaction quality): The average overall interaction-quality score from the episode judge, on a 0–1 scale.
How we built the benchmark
We anchor each trajectory to its repository and commit, recover the starting state and reference change, and reconstruct the instruction, environment and tests. A candidate becomes a Harbor task only when the environment builds, the verifier fails on the starting state, and it passes after the reference change.
Complete, executable tasks
Package the instruction, starting workspace, environment and verifier. Keep the reference solution hidden from the evaluated agent.
The UXInteract 0.1 catalog contains 141 executable tasks across application development, frontend development, build and deployment, data analysis, machine learning, and UI optimization.
Each task combines an instruction, a starting environment, a verifier, and a reference solution. The verifier supplies an executable check of the result; the reference solution is kept separate from the agent’s workspace.
task_id/
├── instruction.md # Task instructions
├── task.toml # Configuration
├── environment/
│ ├── Dockerfile # Container definition
│ └── code/ # Initial workspace
├── solution/
│ └── solve.sh # Reference solution
└── tests/
└── test.sh # Verification entry pointWhat the reported results show
The results below cover 13 models evaluated on a 139-task subset. The UXInteract 0.1 release contains 141 tasks. These scores retain the denominator of the reported runs.
+3.9 pp
Average improvement
All 13 models improve from first-turn to final success.
66.2%
Highest final success
Claude Opus 5.5 also leads Pass² (60.4%) and Mean IQ (0.705).
+16.2 pp
Largest improvement
GLM-5.3 rises from 32.0% first-turn success to 48.2% final success.
pp = percentage points. Improvements are calculated from the displayed, rounded percentages; the average gives each model equal weight.
Equal completion rates can hide different interaction scores. DeepSeek V4 Pro and MiMo-V2.6-Pro both finish at 44.6%, but their Mean IQ scores are 0.533 and 0.502. Their Pass² scores also differ: 33.8% and 28.8%.
Repeated success is a separate challenge. Every model’s Pass² is below its final success rate in this snapshot. Solving a task once does not establish that the agent will solve it consistently.
View all 13 models Four metrics · ordered by final success
On narrow screens, scroll the table horizontally to see every metric.
| Model | Pass₁ | PassF | Pass² | Mean IQ |
|---|---|---|---|---|
| Claude Opus 5.5 | 65.8% | 66.2% | 60.4% | 0.705 |
| GPT-6.1 Sol | 56.1% | 57.9% | 48.2% | 0.657 |
| Claude Opus 5 | 51.1% | 52.9% | 42.5% | 0.612 |
| Kimi K3 | 49.3% | 52.5% | 41.0% | 0.565 |
| GLM-5.3 | 32.0% | 48.2% | 29.5% | 0.493 |
| Claude Opus 4.8 | 44.6% | 45.7% | 36.7% | 0.494 |
| DeepSeek V4 Pro | 41.7% | 44.6% | 33.8% | 0.533 |
| MiMo-V2.6-Pro | 40.7% | 44.6% | 28.8% | 0.502 |
| DeepSeek V4 Flash | 37.8% | 43.2% | 32.4% | 0.483 |
| GPT-5.6 Sol | 39.6% | 42.5% | 33.8% | 0.533 |
| GLM-5.2 | 34.5% | 41.4% | 28.8% | 0.479 |
| GPT-5.5 | 36.0% | 38.9% | 23.0% | 0.493 |
| Pangu-505B-LabExp | 35.6% | 37.4% | 30.2% | 0.498 |
Scroll horizontally to view the full figure.
Why do strong models still fail on UXInteract?
The evaluation runs we reviewed reveal three challenges for coding agents: implementing the logic behind the requested behavior, enforcing business rules across modules, and turning user feedback into a verified fix.
1. Trace the requested behavior back to the logic that makes it work. Users typically describe the result they want to see. Stronger models identify which logic produces that result, what data must change, and when the change must persist. Weaker models may stop once the visible result looks right, for example by changing the HTML or updating only the current page state. In the paragraph editor task, GPT-6.1 Sol and Opus 5.5 could close the editor and display the edited text, but the text disappeared on reload. GLM-5.3 passed in one run. The results looked alike immediately after editing. Reloading the page showed which implementations actually saved the text. task_017
2. Enforce business rules across module boundaries. A status, identifier, or validation result must retain its meaning as it moves between modules and govern what downstream components do. In the checkpoint and rollback task, the planner must refuse to generate a plan for a corrupt snapshot. In our follow-up check of that behavior, both GPT-6.1 Sol implementations detected the corruption and still generated a plan. Both Opus 5.5 implementations checked the validation result before planning and refused to proceed. The difference was whether the rule was enforced where the decision was made. task_088
3. Use feedback to make and verify a fix. User feedback often describes where a problem shows up. The model needs to reproduce the failure, trace it to the responsible code, and rerun the original workflow after making a change. In the LaTeX build task, the second-round feedback asked the agent to confirm that the build produced a valid PDF. GPT-6.1 Sol checked that the command exited successfully and produced a nonempty file, but the file contained plain text. In one run, Opus 5.5 identified the format problem but did not fix the required build entry point. None of the reviewed runs passed. A separate check generated a valid PDF with the LaTeX compiler in the same environment. To verify the repair, the agent needed to run the specified build command and open the PDF it produced. task_085
What to take away
We contribute both an executable coding benchmark and an interaction evaluation framework. Together, they measure task completion, consistency across runs and interaction quality, showing how agents respond to feedback and evolving requirements.