- Who it helps
- Teams evaluating customer-facing agents and developers building task-specific tests.
- Availability
- The public Apache-2.0 repository includes three cases. The paper’s larger evaluation covers 95 tasks across 20 scenarios; access to that broader collection is requested separately.
What the evaluation asks
Agents interact with simulated stakeholders and working systems. Messages have costs and incidents carry penalties. The authors report that discovering constraints remains a bottleneck across their tested model configurations.
Read a case by separating its starting request, hidden requirements and scoring rules. That separation makes it easier to see whether an agent solved the intended problem or simply optimized an easy visible metric.
How a team can use the idea
For an original internal exercise, give the agent a fictional support migration. The visible request is to move ten tickets. Keep a separate evaluator sheet requiring preserved IDs, unchanged owners and an explicit list of failed imports. Score those fields directly after the run.
The repository’s judge setup uses Docker, Python 3.9 or later and model access for simulated stakeholders. Model calls can cost money. Inspect the local-service and account setup before running it; the first account has administrative significance.
What you can do
- Inspect one of the three released cases and its judge before deciding whether the benchmark matches your own agent task. Open reference ↗
Where the evidence stops
Reported aggregate results come from the authors. The judge and example cases were not executed here, and the full task collection was not inspected.
Sources read. The author abstract and repository instructions were read on October 17 UTC. This is a source-based explanation with an original evaluation example.
Sources & further reading
GitHub · implementation
FDE-Gym repository ↗
README distinguishes three open cases from the larger benchmark and lists judge dependencies.
Deeplethe · page checked 2026-10-17
Hugging Face · paper
FDE-Gym author abstract ↗
Author abstract describes customer-scenario evaluation and its reported failure modes.
FDE-Gym authors · page checked 2026-10-17 · Published 2026-10-15
Coverage note: Primary and linked sources are limited to those listed. Independent execution, original X coverage and comparable cross-platform measurements were not established in this run.
The record over time (1)
Dated source checks and published explanations. An observation date can differ from the original release date.
FDE-Gym evaluates agents on customer problems with incomplete requirements. A convincing answer can still fail when the agent misses a stakeholder constraint or damages the simulated system.