- Who it helps
- Researchers and developers evaluating memory in personal assistants.
- Availability
- The October 17 paper, revised October 20, has a public evaluator and downloadable synthetic histories. The documented evaluation environment uses Python 3.11, Linux, CUDA and a separately configured model server.
Weak evidence over time
The benchmark includes 114 synthetic users and 3,328 four-choice probes. It tests exceptions, changing habits and nonbinding suggestions. The authors report that the best tested method reached 33.61%, against a 25% chance baseline.
Keep answers outside memory
The repository separates visible histories from evaluator-only gold labels. A useful original check is a dinner assistant that remembers repeated vegetarian orders but does not turn a friend’s unaccepted steak suggestion into the user’s preference.
What you can do
- Inspect a probe’s history cutoff and answer annotations before comparing memory methods. Keep evaluator labels outside the assistant input. Open reference ↗
Where the evidence stops
Synthetic histories and four-choice questions do not establish performance with real personal data. Reported scores were not reproduced.
Sources read. Public sources checked October 28, 2026. No software installation, inference or complete workflow was executed.
Sources & further reading
Paper · paper
HABIT-Bench paper ↗
Direct evidence for the subject, its availability and the limits of its claims.
Project authors · page checked 2026-10-28
GitHub · implementation
HABIT-Bench evaluator ↗
Direct evidence for the subject, its availability and the limits of its claims.
Project authors · page checked 2026-10-28
Coverage note: Original X access was unavailable. Independent execution, comparable social measurements and complete lookback coverage were not established.
The record over time (1)
Dated source checks and published explanations. An observation date can differ from the original release date.
- · Added to Current
HABIT-Bench tests whether an assistant can infer a user’s habits from repeated clues without treating every suggestion as a preference.