TL;DR

Real coding work involves feedback and evolving requirements; single-turn correctness alone misses how well agents collaborate with users. We contribute a benchmark and an interaction evaluation framework for coding agents.

  • Benchmark: 141 executable tasks across six coding domains, with instructions, environments, verifiers, and reference solutions.
  • Interaction evaluation: Runs multi-turn episodes in persistent workspaces, generates adaptive user feedback, and records progress. Functional tests assess task completion; an LLM judge assesses the interaction.

Why evaluate beyond the first turn

Coding work often continues after an initial answer. A user may report a failure, clarify a requirement, or ask for stronger tests. A useful agent must carry earlier context forward and make the next revision work.

A good user experience means completing the task with little unnecessary effort, understanding complex and changing instructions, raising important questions or risks at the right time, and giving useful, clear and complete responses. UXInteract currently captures part of this picture through first-turn and final task success, success across repeated runs, and an overall judged interaction-quality score. These measures help assess task outcomes and the interaction, but do not separately quantify every aspect of user experience. More detailed evaluation of interaction efficiency, intent understanding, initiative and response quality would require additional measures.

The example below shows a user LLM asking a coding agent to build and refine a 3D chair, with follow-up requests even after a passing check.

An example conversation
User LLM
Round 3 of 3Complete
Coding agent
3D preview
The chair gains thicker legs, then two armrests.Armrests added
Coding agent

Added both armrests and kept the thicker legs.

The Pelican Test

The same brief calls for a pelican riding a bicycle across different art styles. See what changes when the agent can revise its work after feedback.

Single turn

Initial response

Multi-turn

After feedback and revision

How we evaluate interactions

An episode follows one attempt at a task, from the initial request through any follow-ups. The agent continues in the same conversation session and workspace, while we evaluate both the code it produces and how it responds to feedback.

The agent receives an instruction and an isolated workspace, then edits code and runs tools. A verifier checks the result after each round. If more work is needed, the user LLM gives a follow-up based on the current progress, and the agent revises its work. By default, the loop stops on a full verifier pass or at the turn limit.

Stateful multi-turn evaluation with adaptive user feedback.
Figure 1. Multi-turn evaluation.

Scroll horizontally to view the full figure.

Feedback responds to the agent’s work. The simulated user receives summaries of earlier turns, code changes and a high-level verification outcome. It can request targeted debugging, stronger tests or requirement clarifications. Hidden tests and raw verifier logs stay private, and a configurable turn budget limits the interaction.

A separate judge assesses interaction quality. After the episode, an LLM reviews the recorded interaction and verification evidence. It considers correctness, the latest user requirements, context continuity, corrections, testing and user effort. Saved rounds can also be judged individually, using only the interaction up to that point.

Read the metrics together

  • Pass₁ (First-turn success): The average per-task success rate after the initial request, using the task verifier.
  • PassF (Final success): The same verifier-based success rate at the end of the episode.
  • Pass² (Success in both runs): How often two rollouts both meet the correctness judge’s success threshold. With two attempts per task, both must succeed.
  • Mean IQ (Judged interaction quality): The average overall interaction-quality score from the episode judge, on a 0–1 scale.

How we built the benchmark

We anchor each trajectory to its repository and commit, recover the starting state and reference change, and reconstruct the instruction, environment and tests. A candidate becomes a Harbor task only when the environment builds, the verifier fails on the starting state, and it passes after the reference change.

From trajectories to tasks

Complete, executable tasks

Package the instruction, starting workspace, environment and verifier. Keep the reference solution hidden from the evaluated agent.

The UXInteract 0.1 catalog contains 141 executable tasks across application development, frontend development, build and deployment, data analysis, machine learning, and UI optimization.

Each task combines an instruction, a starting environment, a verifier, and a reference solution. The verifier supplies an executable check of the result; the reference solution is kept separate from the agent’s workspace.

task_id/
├── instruction.md       # Task instructions
├── task.toml            # Configuration
├── environment/
│   ├── Dockerfile       # Container definition
│   └── code/            # Initial workspace
├── solution/
│   └── solve.sh         # Reference solution
└── tests/
    └── test.sh          # Verification entry point
Core task layout; additional files omitted.

What the reported results show

The results below cover 13 models evaluated on a 139-task subset. The UXInteract 0.1 release contains 141 tasks. These scores retain the denominator of the reported runs.

+3.9 pp

Average improvement

All 13 models improve from first-turn to final success.

66.2%

Highest final success

Claude Opus 5.5 also leads Pass² (60.4%) and Mean IQ (0.705).

+16.2 pp

Largest improvement

GLM-5.3 rises from 32.0% first-turn success to 48.2% final success.

pp = percentage points. Improvements are calculated from the displayed, rounded percentages; the average gives each model equal weight.

Equal completion rates can hide different interaction scores. DeepSeek V4 Pro and MiMo-V2.6-Pro both finish at 44.6%, but their Mean IQ scores are 0.533 and 0.502. Their Pass² scores also differ: 33.8% and 28.8%.

Repeated success is a separate challenge. Every model’s Pass² is below its final success rate in this snapshot. Solving a task once does not establish that the agent will solve it consistently.

View all 13 models Four metrics · ordered by final success

On narrow screens, scroll the table horizontally to see every metric.

139-task evaluation · higher is better
ModelPass₁PassFPass²Mean IQ
Claude Opus 5.565.8%66.2%60.4%0.705
GPT-6.1 Sol56.1%57.9%48.2%0.657
Claude Opus 551.1%52.9%42.5%0.612
Kimi K349.3%52.5%41.0%0.565
GLM-5.332.0%48.2%29.5%0.493
Claude Opus 4.844.6%45.7%36.7%0.494
DeepSeek V4 Pro41.7%44.6%33.8%0.533
MiMo-V2.6-Pro40.7%44.6%28.8%0.502
DeepSeek V4 Flash37.8%43.2%32.4%0.483
GPT-5.6 Sol39.6%42.5%33.8%0.533
GLM-5.234.5%41.4%28.8%0.479
GPT-5.536.0%38.9%23.0%0.493
Pangu-505B-LabExp35.6%37.4%30.2%0.498
139-task evaluation results: first-turn and final pass rates, success in both runs, and Mean IQ across thirteen models.
Figure 2. Results across thirteen models on the evaluated 139-task subset. The left panel shows first-turn and final success; the middle and right panels show Pass² and Mean IQ. Models are ordered by final success.

Scroll horizontally to view the full figure.

Why do strong models still fail on UXInteract?

The evaluation runs we reviewed reveal three challenges for coding agents: implementing the logic behind the requested behavior, enforcing business rules across modules, and turning user feedback into a verified fix.

1. Trace the requested behavior back to the logic that makes it work. Users typically describe the result they want to see. Stronger models identify which logic produces that result, what data must change, and when the change must persist. Weaker models may stop once the visible result looks right, for example by changing the HTML or updating only the current page state. In the paragraph editor task, GPT-6.1 Sol and Opus 5.5 could close the editor and display the edited text, but the text disappeared on reload. GLM-5.3 passed in one run. The results looked alike immediately after editing. Reloading the page showed which implementations actually saved the text. task_017

2. Enforce business rules across module boundaries. A status, identifier, or validation result must retain its meaning as it moves between modules and govern what downstream components do. In the checkpoint and rollback task, the planner must refuse to generate a plan for a corrupt snapshot. In our follow-up check of that behavior, both GPT-6.1 Sol implementations detected the corruption and still generated a plan. Both Opus 5.5 implementations checked the validation result before planning and refused to proceed. The difference was whether the rule was enforced where the decision was made. task_088

3. Use feedback to make and verify a fix. User feedback often describes where a problem shows up. The model needs to reproduce the failure, trace it to the responsible code, and rerun the original workflow after making a change. In the LaTeX build task, the second-round feedback asked the agent to confirm that the build produced a valid PDF. GPT-6.1 Sol checked that the command exited successfully and produced a nonempty file, but the file contained plain text. In one run, Opus 5.5 identified the format problem but did not fix the required build entry point. None of the reviewed runs passed. A separate check generated a valid PDF with the LaTeX compiler in the same environment. To verify the repair, the agent needed to run the specified build command and open the PDF it produced. task_085

What to take away

We contribute both an executable coding benchmark and an interaction evaluation framework. Together, they measure task completion, consistency across runs and interaction quality, showing how agents respond to feedback and evolving requirements.