How agent evaluation should evolve
Benchmarks that measure single-turn correctness miss the point. We need to evaluate agents across episodes, not just turns.
Most agent benchmarks today look like question-answering datasets with extra steps. Did the agent complete the task? How many steps did it take? Did it hallucinate? These are useful but incomplete.
A real agent evaluation should measure: robustness to distribution shift, recovery from mistakes, consistency over long episodes, cost of action, side effects on shared state, and alignment with operator intent over time.
This is closer to systems evaluation than to ML evaluation. It requires simulation environments, counterfactual analysis, and human-in-the-loop validation. I think the field will converge on a few standard 'agent operating environments' the way we have standard datasets for vision and NLP.