- Who it helps
- Researchers and developers building video question-answering systems.
- Availability
- The October 13 abstract is public. Dataset files and evaluator access could not be verified from the linked project during this run.
The comparison
The benchmark pairs references while asking questions about identity, place and time. The authors report substantial inconsistencies across the tested models.
A useful evaluation question
When testing your own assistant, keep the clip and intended target fixed while changing only the reference. Record both answers and whether each identifies the intended evidence. This isolates a failure that an overall accuracy number can hide.
What you can do
- Read the paper’s task definition before adapting the paired-reference idea to a small set of your own permitted clips. Open reference ↗
Where the evidence stops
The abstract was read; the dataset, scoring code and reported results were not independently reproduced.
Sources read. Primary sources were read on October 17, 2026. The product or benchmark was not executed in this research session.
Sources & further reading
Paper · paper
OmniCompass visual-reference benchmark ↗
Abstract read. Paired references and reported evaluation scope checked; dataset and evaluator not inspected.
OmniCompass authors · page checked 2026-10-17 · Published 2026-10-13
Coverage note: Independent usage evidence and cross-platform discussion remain limited in this sample.
The record over time (1)
Dated source checks and published explanations. An observation date can differ from the original release date.
OmniCompass evaluates whether a model answers consistently when a video target is described in text or shown visually.