- Who it helps
- Researchers and operations teams evaluating multi-step business agents.
- Availability
- The October 16 paper is public and links research artifacts. Those artifacts were not retrieved or executed during this review.
The measured task
Four open models were evaluated over thirty simulated days and three seeds. The authors report that even the strongest configuration earned roughly one eighth of the scripted baseline’s profit.
A crucial comparison limit
The scripted baseline receives direct state information and a correct supplier interaction format. That advantage prevents treating the profit ratio as a controlled estimate of model capability. Reporting accuracy also differs by model.
What you can do
- Before comparing results, write down which observations and action formats each participant receives. Treat unequal access as a separate experimental condition. Open reference ↗
Where the evidence stops
This is a small simulation, not evidence that a real shop can operate autonomously. Baseline privileges and environment assumptions affect the conclusions.
Sources read. Original sources read October 23, 2026 UTC. No external project, model inference or reported experiment was executed.
Sources & further reading
Paper · paper
KiranaBench ↗
Read the October 16 paper on multi-agent general-store simulation. The scripted baseline has privileged state access, which limits direct model-versus-baseline comparisons.
KiranaBench authors · page checked 2026-10-23
Paper · paper
KiranaBench v1 evaluation and limitations ↗
Read setup, reported results and privileged-baseline limitations. Artifact links were not downloaded and business outcomes were not reproduced.
KiranaBench authors · page checked 2026-10-23
Coverage note: No independent end-to-end reproduction or complete video demonstration was verified in this bounded sample.
The record over time (1)
Dated source checks and published explanations. An observation date can differ from the original release date.
- · Added to Current
KiranaBench studies four agent roles running a simulated Indian general store. It connects agent decisions to business outcomes and exposes how a comparison can favor a baseline with better access to state.