Verified coding tasks

Tasks that prove their own tests

Each task is what a coding agent would receive: a problem statement and starting code, plus hidden tests and a reference solution. Our sandbox ran every one: the tests fail on the starting code, pass twice on the reference, and have no network access.

12 tasks
Python 3.12, TypeScript, JavaScript
33 fail→pass tests
Bug fixes, features, two security fixes

Download JSONL

Retry only transient errors, with capped backoff

code-03 · what the agent sees is the problem statement and the starting code

mediumPython 3.12 · pytest

Problem statement

net/retry.py has a retry helper used around HTTP calls. It has three problems in production: 1. It retries every exception, so bugs such as ValueError are retried and hidden. 2. After the last attempt it returns None instead of raising, so callers crash later with confusing errors. 3. The delay doubles without limit, so a long outage sleeps for minutes. Change retry so that it only retries exceptions listed in retry_on, raises the last error when attempts run out (without sleeping after the final attempt), and never sleeps longer than max_delay. The delay before retry i (starting at 0) is base_delay * 2**i, capped at max_delay. Keep the signature; tests pass a fake sleep.

Starting code

net/retry.py

import time


def retry(func, attempts=3, base_delay=0.5, max_delay=8.0, retry_on=(ConnectionError,), sleep=time.sleep):
    for i in range(attempts):
        try:
            return func()
        except Exception:
            sleep(base_delay * 2 ** i)
    return None

Reference solution

net/retry.py

import time


def retry(func, attempts=3, base_delay=0.5, max_delay=8.0, retry_on=(ConnectionError,), sleep=time.sleep):
    for i in range(attempts):
        try:
            return func()
        except retry_on:
            if i == attempts - 1:
                raise
            sleep(min(base_delay * 2 ** i, max_delay))

Tests (hidden from the agent)

tests/test_retry.py

import pytest

from net.retry import retry


def flaky(failures, error=ConnectionError, value="ok"):
    calls = {"n": 0}

    def func():
        calls["n"] += 1
        if calls["n"] <= failures:
            raise error("boom")
        return value

    return func, calls


def test_returns_first_success():
    delays = []
    func, _ = flaky(0)
    assert retry(func, sleep=delays.append) == "ok"
    assert delays == []


def test_retries_then_succeeds():
    delays = []
    func, calls = flaky(2)
    assert retry(func, sleep=delays.append) == "ok"
    assert calls["n"] == 3
    assert delays == [0.5, 1.0]


def test_raises_after_the_last_attempt():
    delays = []
    func, calls = flaky(99)
    with pytest.raises(ConnectionError):
        retry(func, attempts=3, sleep=delays.append)
    assert calls["n"] == 3
    assert delays == [0.5, 1.0]


def test_does_not_retry_other_errors():
    delays = []
    func, calls = flaky(1, error=ValueError)
    with pytest.raises(ValueError):
        retry(func, sleep=delays.append)
    assert calls["n"] == 1
    assert delays == []


def test_delay_is_capped():
    delays = []
    func, _ = flaky(99)
    with pytest.raises(ConnectionError):
        retry(func, attempts=6, base_delay=1, max_delay=4, sleep=delays.append)
    assert delays == [1, 2, 4, 4, 4]

Notes

The fake sleep records delays, so the tests are exact and take no time.

Sandbox run

Recorded by pnpm samples:verify. Our tests fail the build if this stops matching the task.

Checks passed5 tests pass on the reference solution; 3 of them fail on the starting code.

Test	Starting code	Reference	Second run
tests/test_retry.py::test_returns_first_success	pass	pass	pass	pass → pass
tests/test_retry.py::test_retries_then_succeeds	pass	pass	pass	pass → pass
tests/test_retry.py::test_raises_after_the_last_attempt Failed: DID NOT RAISE ConnectionError	fail	pass	pass	fail → pass
tests/test_retry.py::test_does_not_retry_other_errors Failed: DID NOT RAISE ValueError	fail	pass	pass	fail → pass
tests/test_retry.py::test_delay_is_capped Failed: DID NOT RAISE ConnectionError	fail	pass	pass	fail → pass

Python 3.12 · pytest · canonset-sandbox-python:1 · 1.2 s · checked 2026-09-29 23:30 UTC

Output: Starting code (exit 1, 0.3 s)

..FFF                                                                    [100%]
=================================== FAILURES ===================================
______________________ test_raises_after_the_last_attempt ______________________

    def test_raises_after_the_last_attempt():
        delays = []
        func, calls = flaky(99)
>       with pytest.raises(ConnectionError):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       Failed: DID NOT RAISE ConnectionError

tests/test_retry.py:36: Failed
_______________________ test_does_not_retry_other_errors _______________________

    def test_does_not_retry_other_errors():
        delays = []
        func, calls = flaky(1, error=ValueError)
>       with pytest.raises(ValueError):
             ^^^^^^^^^^^^^^^^^^^^^^^^^
E       Failed: DID NOT RAISE ValueError

tests/test_retry.py:45: Failed
_____________________________ test_delay_is_capped _____________________________

    def test_delay_is_capped():
        delays = []
        func, _ = flaky(99)
>       with pytest.raises(ConnectionError):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       Failed: DID NOT RAISE ConnectionError

tests/test_retry.py:54: Failed
=========================== short test summary info ============================
FAILED tests/test_retry.py::test_raises_after_the_last_attempt - Failed: DID ...
FAILED tests/test_retry.py::test_does_not_retry_other_errors - Failed: DID NO...
FAILED tests/test_retry.py::test_delay_is_capped - Failed: DID NOT RAISE Conn...
3 failed, 2 passed in 0.04s

Output: Reference solution (exit 0, 0.3 s)

.....                                                                    [100%]
5 passed in 0.02s

Output: Reference, second run (exit 0, 0.3 s)

.....                                                                    [100%]
5 passed in 0.02s