Review an agent’s optimization before trusting the speedup

1. Define what must stay true

Our input is a list of event dictionaries. Each event has a step number, a payload and a state object. Several events intentionally share the same state object. The function returns events at or above a minimum step, preserving their order. The output must be independent of the input, while preserving shared identity inside the output.

The baseline copies the entire list before filtering it. The candidate filters first, then copies the selected list in one operation. When only a few events survive, it can avoid copying much of the payload. This optimization is valid for the plain dictionaries used here; arbitrary Python objects can have custom copy behavior, so the same transformation needs a new contract in another application.

Before running it, write down the expected result for a three-event input with steps zero, one and two, and a minimum of one. You should retain events one and two in that order. Changing their copied state should affect both returned events because they share that state, while leaving the original input unchanged.

2. Save and run the complete fixture

Create a new empty folder for the exercise. Save the following code as optimization-check.py. It uses only Python’s standard library and creates its own synthetic data. It reads no personal files and makes no network requests.

The verification loop checks empty inputs, boundary values and several sizes. The timing loop runs each function repeatedly and reports the median seconds per call, which reduces the effect of one unusually slow measurement. Keep the assertions enabled: run ordinary python3, without its optimization flag.

import copy
import json
import statistics
import sys
import timeit


def baseline(events, minimum):
    return [e for e in copy.deepcopy(events) if e['step'] >= minimum]


def candidate(events, minimum):
    return copy.deepcopy([e for e in events if e['step'] >= minimum])


def fixture(size):
    shared = {'labels': ['original']}
    return [{'step': i, 'state': shared, 'payload': list(range(40))}
            for i in range(size)]


def verify(fn):
    for size in (0, 1, 20):
        events = fixture(size)
        for minimum in (-1, 0, size // 2, size, size + 1):
            expected = baseline(events, minimum)
            actual = fn(events, minimum)
            assert actual == expected, 'Wrong values or order'
            if actual:
                assert actual[0]['state'] is not events[0]['state'], 'Input alias leaked'
                assert all(e['state'] is actual[0]['state'] for e in actual), 'Shared identity lost'
                actual[0]['state']['labels'].append('edited')
                assert events[0]['state']['labels'] == ['original'], 'Input was mutated'
    return 'PASS'


if __name__ == '__main__':
    result = {'python': sys.version.split()[0], 'checks': verify(candidate), 'timings': []}
    for size in (50, 2000):
        events = fixture(size)
        minimum = size - 5
        times = {fn.__name__: statistics.median(timeit.repeat(
            lambda: fn(events, minimum), number=10, repeat=5)) / 10
            for fn in (baseline, candidate)}
        result['timings'].append({'events': size, 'seconds_per_call': times})
    print(json.dumps(result, indent=2))

python3 optimization-check.py > optimization-result.json
cat optimization-result.json

3. Read correctness before timing

A successful run prints a JSON object with your Python version, checks set to PASS and two timing records. The records cover fifty and two thousand events. Both retain the last five events. Their exact durations depend on your machine; no fixed millisecond number is a pass criterion.

The pass condition is that every assertion succeeds. A traceback is a failed check, even if you observed a speedup in another run. The two timing records describe this synthetic workload only. They do not measure Gemini, an agent’s reasoning, a production application or the cost of using an API.

Keep the JSON beside the source file. Record the machine, Python version and any changes you make to the fixture. For a decision on real software, add a realistic input distribution: a filter that retains nearly everything can have a very different performance profile from one that retains five events.

4. Make the checks reject a bad change

A check is more convincing when it catches a plausible mistake. In a copy of the file, replace the candidate body with a plain filtered list. That returns references into the original input. Run the file again and expect Input alias leaked. Restore the working candidate before continuing.

A second plausible mistake deep-copies each selected event independently. Values still look correct, but the shared state becomes several separate objects. Replacing the body with the second example below should produce Shared identity lost. These two negative controls were executed during preparation of this guide.

If either broken candidate passes, inspect the fixture and assertions before trusting any performance result. A reduced test that removes shared state can accidentally remove the exact behavior you intended to protect.

return [e for e in events if e['step'] >= minimum]

return [copy.deepcopy(e) for e in events if e['step'] >= minimum]

5. Give an agent a bounded optimization task

You can now use an assistant you already have to propose another change. Supply the complete fixture and contract, and keep any edit in this disposable folder. The prompt below is a suggested brief; it was not sent to an external model during this check. Model access may involve provider charges.

Ask for the final diff and the raw command output. Independently run the checks after reviewing the diff. A report that says tests passed is incomplete evidence when you cannot see which source version and command produced it. For a real repository, require its own integration checks and representative workload in addition to this small example.

Optimize candidate(events, minimum) in optimization-check.py.
Preserve values, order, input isolation and shared identity in the returned list.
Keep baseline, fixture and verify unchanged. Work only in this disposable folder.
Explain the proposed change and its assumptions before editing.
Return the diff, exact verification command, raw JSON result and any failures.
Do not claim a production speedup from this synthetic benchmark.
Stop if a change requires broadening file or network access.

6. Recover from failures and make a decision

If Python reports that the file cannot be found, check that your terminal is in the folder where you saved it. If your editor inserted smart quotation marks, replace them with ordinary code quotes. The script requires no package installation; an import error for a standard-library module usually points to a broken Python installation or a local file shadowing the module name.

If correctness fails, restore the baseline behavior and identify the violated assertion. Do not weaken the check merely to make an optimization pass. If timing varies substantially, close heavy background work, keep the input fixed and rerun the same source. Persist the results separately instead of selecting only the fastest run.

Accept the candidate for this exercise when all checks pass and its behavior is understandable. A production decision also needs representative data, concurrency behavior, memory observations and the application’s existing tests. Decline a change whose extra complexity exceeds the measured benefit. Preserve the prior implementation so a regression can be reversed.

This exercise costs local CPU time and produces no external charges. It deliberately leaves model accuracy, account permissions, API cost and end-to-end agent reliability untested. Those questions require their own measured workflow, with an explicit record of what was actually run.

Guide history

Revisions preserve the original instructions. A source recheck alone does not create a new revision.

Revision 1 · New original exercise with a complete standard-library fixture, tested correctness checks, two rejected negative controls and a bounded agent brief.