Verified coding tasks

Tasks that prove their own tests

Each task is what a coding agent would receive: a problem statement and starting code, plus hidden tests and a reference solution. Our sandbox ran every one: the tests fail on the starting code, pass twice on the reference, and have no network access.

12 tasks
Python 3.12, TypeScript, JavaScript
33 fail→pass tests
Bug fixes, features, two security fixes

Download JSONL

Stop the rate limiter from over-refilling

code-07 · what the agent sees is the problem statement and the starting code

mediumPython 3.12 · pytest

Problem statement

limits/bucket.py implements a token bucket for our public API: rate tokens per second, up to capacity. Under steady traffic it lets far more requests through than configured, and after an idle period it allows unlimited bursts. Fix TokenBucket.allow so that tokens refill only for the time since the previous call and never exceed capacity. Keep the constructor (rate, capacity, clock) and allow(cost=1) -> bool; tests inject a fake clock.

Starting code

limits/bucket.py

import time


class TokenBucket:
    def __init__(self, rate: float, capacity: int, clock=time.monotonic):
        self.rate = rate
        self.capacity = capacity
        self.clock = clock
        self.tokens = capacity
        self.start = clock()

    def allow(self, cost: int = 1) -> bool:
        self.tokens += (self.clock() - self.start) * self.rate
        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False

Reference solution

limits/bucket.py

import time


class TokenBucket:
    def __init__(self, rate: float, capacity: int, clock=time.monotonic):
        self.rate = rate
        self.capacity = capacity
        self.clock = clock
        self.tokens = float(capacity)
        self.last = clock()

    def allow(self, cost: int = 1) -> bool:
        now = self.clock()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False

Tests (hidden from the agent)

tests/test_bucket.py

from limits.bucket import TokenBucket


class FakeClock:
    def __init__(self):
        self.now = 0.0

    def __call__(self):
        return self.now


def test_burst_up_to_capacity():
    clock = FakeClock()
    bucket = TokenBucket(rate=1, capacity=3, clock=clock)
    assert [bucket.allow() for _ in range(4)] == [True, True, True, False]


def test_cost_above_one():
    bucket = TokenBucket(rate=1, capacity=5, clock=FakeClock())
    assert bucket.allow(cost=5)
    assert not bucket.allow()


def test_refills_only_for_elapsed_time():
    clock = FakeClock()
    bucket = TokenBucket(rate=1, capacity=3, clock=clock)
    for _ in range(3):
        bucket.allow()
    clock.now = 1.0
    assert bucket.allow()
    assert not bucket.allow()


def test_never_exceeds_capacity_after_idle():
    clock = FakeClock()
    bucket = TokenBucket(rate=10, capacity=3, clock=clock)
    clock.now = 100.0
    assert [bucket.allow() for _ in range(4)] == [True, True, True, False]

Notes

A fake clock keeps the tests deterministic; the second-run check in the sandbox would flag any timing dependence.

Sandbox run

Recorded by pnpm samples:verify. Our tests fail the build if this stops matching the task.

Checks passed4 tests pass on the reference solution; 2 of them fail on the starting code.

Test	Starting code	Reference	Second run
tests/test_bucket.py::test_burst_up_to_capacity	pass	pass	pass	pass → pass
tests/test_bucket.py::test_cost_above_one	pass	pass	pass	pass → pass
tests/test_bucket.py::test_refills_only_for_elapsed_time assert not True + where True = allow() + where allow = <limits.bucket.TokenBucket object at 0x7d30b418fe60>.allow	fail	pass	pass	fail → pass
tests/test_bucket.py::test_never_exceeds_capacity_after_idle assert [True, True, True, True] == [True, True, True, False] At index 3 diff: True != False Use -v to get more diff	fail	pass	pass	fail → pass

Python 3.12 · pytest · canonset-sandbox-python:1 · 1.2 s · checked 2026-09-29 23:30 UTC

Output: Starting code (exit 1, 0.3 s)

..FF                                                                     [100%]
=================================== FAILURES ===================================
______________________ test_refills_only_for_elapsed_time ______________________

    def test_refills_only_for_elapsed_time():
        clock = FakeClock()
        bucket = TokenBucket(rate=1, capacity=3, clock=clock)
        for _ in range(3):
            bucket.allow()
        clock.now = 1.0
        assert bucket.allow()
>       assert not bucket.allow()
E       assert not True
E        +  where True = allow()
E        +    where allow = <limits.bucket.TokenBucket object at 0x7d30b418fe60>.allow

tests/test_bucket.py:31: AssertionError
____________________ test_never_exceeds_capacity_after_idle ____________________

    def test_never_exceeds_capacity_after_idle():
        clock = FakeClock()
        bucket = TokenBucket(rate=10, capacity=3, clock=clock)
        clock.now = 100.0
>       assert [bucket.allow() for _ in range(4)] == [True, True, True, False]
E       assert [True, True, True, True] == [True, True, True, False]
E         
E         At index 3 diff: True != False
E         Use -v to get more diff

tests/test_bucket.py:38: AssertionError
=========================== short test summary info ============================
FAILED tests/test_bucket.py::test_refills_only_for_elapsed_time - assert not ...
FAILED tests/test_bucket.py::test_never_exceeds_capacity_after_idle - assert ...
2 failed, 2 passed in 0.03s

Output: Reference solution (exit 0, 0.3 s)

....                                                                     [100%]
4 passed in 0.02s

Output: Reference, second run (exit 0, 0.3 s)

....                                                                     [100%]
4 passed in 0.02s