VideoMemBench: How Well Do MLLMs Remember in Video Conversations?

VideoMemBench evaluates in-context video memory with 326 videos paired with multi-turn dialogues whose turns are linked by verified Episodic and Answer dependencies, finding the best of 25 tested MLLMs reaches only 45.7% overall versus 86.1% for humans.

MLLMs can now process hour-long videos, making extended multi-turn dialogue over a single video a natural mode of use, which requires retrieving and reusing what was established earlier in the conversation. Existing video QA benchmarks decompose evaluation into mutually independent questions that never reference one another and are each answerable from the video alone, so their dialogue histories are never load-bearing. In-context memory has been systematically evaluated for LLMs in text-only settings, but extending this to video requires distinguishing questions that genuinely depend on prior dialogue from those answerable from the video and current question. The paper therefore introduces VideoMemBench to measure this ability, termed in-context video memory.

Each sample pairs one video with a long multi-turn dialogue whose turns are connected by typed cross-turn dependencies: an Episodic Dependency refers back to a visual episode described in an earlier turn (memory of what the model has seen), and an Answer Dependency builds on the model's own earlier answer (memory of what it has said). Every dependency is verified for necessity (unanswerable without parent turns, even with the video and remaining dialogue) and non-leakage (the question must not restate information from its parents), plus visual grounding of every turn. Construction uses three LLM roles: a planner that samples keyframes and produces a synopsis, dialogue guidelines, a typed dependency graph and per-turn intents; a generator that builds the dialogue turn by turn using referring expressions and records evidence windows; and a judger that checks necessity, non-leakage and visual grounding, returning failed turns for regeneration (two consecutive failures trigger replanning). Eight human experts then review all dialogues against the videos. Evaluation gives all models the same 128 timestamped frames up front, presents questions in dialogue order with the model's own previous responses retained in history, and reports TurnAcc, EpiDepAcc (scored independently of parent correctness) and AnsDepAcc (chain crediting over Answer-dependency ancestry), with Overall as their arithmetic mean.

• The best of 25 evaluated models, Claude Opus 5, reaches only 45.7% overall, far below the 86.1% human baseline. §Benchmark Results
• On the best model, answer-dependent and episodic-dependent turns score below independent turns. §Introduction
• Mean accuracy on dependent turns decreases with dependency distance, and model-specific trends differ in steepness. §Analysis
• Accuracy declines consistently as the number of parent turns increases for all four compared models. §Analysis
• Replacing prior model responses with ground-truth answers (oracle histories) substantially improves answer-dependent accuracy, though it stays below independent-turn accuracy. §Analysis
• Increasing sampled frames generally improves Overall scores, but gains vary widely across models. §Effect of Frame Count

• The paper states no explicit limitations section; the main experiments use a single protocol in which every model receives the same fixed 128 frames rather than the full video, so behavior under denser or full-video input is left open.
• Answer Dependency Accuracy uses chain crediting, which the paper notes makes the gap to independent-turn accuracy partly reflect the stricter scoring rule rather than memory failure alone, so the metric conflates current-turn errors with upstream failures.
• The oracle-history analysis in Figure 4(d) covers only three models (Qwen3-VL-8B, Qwen3-VL-32B, Gemini 3.5 Flash), leaving open whether the same pattern holds for all 25 evaluated models.
• Open-ended responses are graded by a GPT-5.4 mini judge, so correctness of open-ended turns depends on that judge rather than on exact-match scoring.

• Two dependency types are defined: Episodic Dependency (recalling a visual episode described in an earlier turn) and Answer Dependency (recalling the model's own earlier answer), plus multi-parent dependencies and long-range call-back. §Introduction
• Necessity requires a dependent question to become unanswerable when parent turns are removed even with the video and remaining dialogue; an implicit reference alone can fail this if the video disambiguates it. §Design Requirements
• Non-leakage forbids restating even partially the episode descriptions or answers of parents, and a turn can satisfy necessity while violating non-leakage. §Design Requirements
• The planner fixes the dependency structure before any question is written; dependencies must arise from recurring video content and planned turns must collectively cover the full video. §Construction Pipeline
• VideoMemBench contains 326 dialogues over 326 videos with 2,476 turns, of which 1,089 (43.98%) are independent and 1,387 (56.02%) dependent, with 1,794 typed dependencies (1,200 Episodic, 594 Answer). §Dataset Statistics
• Among dependent turns, 348 have multiple parents (289 double-parent and 59 triple-parent); dialogues span 4-21 turns averaging 7.6 turns, with dependency distance up to nine turns and maximum chain depth seven. §Dataset Statistics
• Videos come from Video-MME, VSI-Bench, Video-MMMU, LongVideoBench, the web, and EgoLife recordings, from an initial pool of about 3.2k videos, covering 31 semantic domains and lasting up to 23.47 hours. §Construction Pipeline
• Twenty annotators evaluated all 326 dialogues in pairs, yielding 652 complete dialogue evaluations under the same 128-frame inputs as models, with reference answers and dependency labels hidden. §Human Baseline Evaluation