Benchmarks & Datasets

Route Anything presents a unified theory and omni-modal trajectory matrix for routing among models, tools, and policies, finding a 5.4-point per-input oracle gap, modest learned-router gains, and 8.82× average efficiency from cheaper routes.

MoviEdit is an end-to-end framework for long-form multi-shot video editing that combines scalable paired-data generation, a cascaded bidirectional–autoregressive editor, and a dedicated benchmark.

RoboColosseum is a distributed in-the-wild robot benchmark that evaluates VLA policies and frontier agents across four embodiments via Open and Fine-tune tracks, revealing weak cross-embodiment ranking agreement and sample-efficient active matchmaking.

YANAMI introduces a 400K-dialogue multimodal dataset and a 400-dialogue benchmark for multiturn stateful interaction, where models must track, update, and visually realize evolving dialogue states.

WebHarbor is a suite of 153 dockerized, executable reconstructions of real websites, built through task-driven human-agent collaboration, providing 3,518 tasks for evaluating and RL-training GUI web agents under deterministic control.

ReTrace is a scalable evaluation framework that formalizes transition-consistent recovery and uses goal-oriented action replay after controlled error injection to systematically study model-based recovery in computer-use agents.

IdeaTaste benchmarks LLM research taste by comparing proposals written from a 2025 literature snapshot with papers human communities published in 2026, finding LLM agents match citations and ideas far below human levels and mainly fail to turn directions into concrete approaches.

CrossSWE is a 171-instance benchmark for cross-repository joint development, where tasks require coordinated changes across interdependent repositories and are verified by joint-necessity testing.

The paper finds that frozen video-world-model representations expose modest physical-label information and action sensitivity, but native prediction errors are strongly affected by appearance and static content and can rank impossible events below possible ones.

The paper introduces ReGap-Bench, a paired benchmark that holds scene, question, and answer fixed while varying visual references, and finds that task-matched references improve multimodal reasoning by up to 25% while mismatched references give little benefit or hurt.

The paper proposes Avatar Intelligence: persistent personal agents that represent a user's goals, values, and constraints, with a user-centric architecture, an L0-L3 delegation roadmap, and the Agentic Oversight Index for evaluating delivered value against human oversight burden.

The paper functionally probes ten frozen visual foundation models across four spatial capabilities and finds that spatial readout, use of history and motion, predictive transfer, and goal-conditioned action utility dissociate across tasks, layers, and prefix lengths.

The paper recasts a benchmark as an empirical measure over item difficulty, formalizes measure mismatch and coverage gaps as two failure modes with diagnostic metrics and theory, diagnoses 25 benchmarks, and proposes BenchFill to fill large difficulty gaps.

Co-Design Bench evaluates coding agents on 273 real-user application revisions with hidden acceptance criteria, finding that strong models satisfy many criteria per task but fully complete only a minority of tasks.

IntReal introduces a real-world interleaved dataset and a coarse-to-fine expert-model evaluation suite for multimodal interleaved generation.

WebAnything automatically reconstructs source-grounded executable web environments from real URLs using an Explorer–Builder and independent verifier, yielding 244 artifacts and 12,984 accepted tasks and improving agent benchmarks after SFT.

AskStream introduces an ask action and a multi-turn benchmark with Ask-F1 and StreamHarness for evaluating when and what streaming video models should ask under video and user uncertainty.

S2 is a training-free offline metric that measures nearest-neighbor overlap between sensory features and end-effector trajectories and correlates with robot task success across model families.

RealCLI is a paired CLI benchmark that evaluates autonomous execution and scripted human-agent collaboration on matched tasks, revealing low joint task-collaboration performance among frontier agents.

PersonaCUABench holds 74 tasks fixed while varying the requesting user across 44 personas, and across nine backbones persona requests lower mean task completion from 0.689 to 0.441 while user satisfaction and completion diverge.

TauIndianBankBench is a 1,000-task Indian retail-banking benchmark that scores both required procedure steps and final records for tool-using conversational agents.

V4.1 Flash-1017 shows strong safety improvements over DeepSeek V4 models in evaluation integrity, loss of control, and harmful manipulation, but retains weaknesses under sustained pressure, partial oversight, and assigned deceptive goals.

Injected metaorders elicit identifiable but too-steep impact laws in five neural LOB generators on five stocks, with no credible post-execution relaxation, while replay and kernel baselines behave as designed.

The paper instruments atomic single-token word-replacement rules and a binned logit model to measure instruction capacity and degradation laws, reporting a GPT-4o mini passport with capacities in rules and uncertainties.