Verified coding tasks
Tasks that prove their own tests
Each task is what a coding agent would receive: a problem statement and starting code, plus hidden tests and a reference solution. Our sandbox ran every one: the tests fail on the starting code, pass twice on the reference, and have no network access.
- 12 tasks
- Python 3.12, TypeScript, JavaScript
- 33 fail→pass tests
- Bug fixes, features, two security fixes
Fix an SQL injection in the user lookup
code-04 · what the agent sees is the problem statement and the starting code
mediumPython 3.12 · pytest
Problem statement
app/users.py builds its query by pasting the email into the SQL text. A security review found that an email such as ' OR '1'='1 returns another user's record, and customers with an apostrophe in their email (o'brien@example.com) cannot log in at all. Fix find_user so that the email is passed as a query parameter. Emails are case-insensitive: ANA@EXAMPLE.COM must find ana@example.com. Return the same dictionary shape, or None when nobody matches.
Starting code
app/users.py
import sqlite3
def find_user(conn: sqlite3.Connection, email: str):
row = conn.execute(f"SELECT id, email, name FROM users WHERE email = '{email}'").fetchone()
return None if row is None else {"id": row[0], "email": row[1], "name": row[2]}Reference solution
app/users.py
import sqlite3
def find_user(conn: sqlite3.Connection, email: str):
row = conn.execute(
"SELECT id, email, name FROM users WHERE lower(email) = lower(?)",
(email,),
).fetchone()
return None if row is None else {"id": row[0], "email": row[1], "name": row[2]}Tests (hidden from the agent)
tests/test_users.py
import sqlite3
import pytest
from app.users import find_user
@pytest.fixture
def conn():
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, email TEXT NOT NULL, name TEXT NOT NULL)")
db.executemany(
"INSERT INTO users (email, name) VALUES (?, ?)",
[("ana@example.com", "Ana"), ("o'brien@example.com", "Siobhan O'Brien")],
)
return db
def test_finds_a_user(conn):
assert find_user(conn, "ana@example.com") == {"id": 1, "email": "ana@example.com", "name": "Ana"}
def test_missing_user_is_none(conn):
assert find_user(conn, "nobody@example.com") is None
def test_apostrophe_in_email(conn):
assert find_user(conn, "o'brien@example.com")["name"] == "Siobhan O'Brien"
def test_injection_matches_nobody(conn):
assert find_user(conn, "' OR '1'='1") is None
def test_email_is_case_insensitive(conn):
assert find_user(conn, "ANA@EXAMPLE.COM")["name"] == "Ana"Notes
The injection test fails on the starting code by returning a user, not by crashing, which is the dangerous case.
Sandbox run
Recorded by pnpm samples:verify. Our tests fail the build if this stops matching the task.
Checks passed5 tests pass on the reference solution; 3 of them fail on the starting code.
| Test | Starting code | Reference | Second run | |
|---|---|---|---|---|
tests/test_users.py::test_finds_a_user | pass | pass | pass | pass → pass |
tests/test_users.py::test_missing_user_is_none | pass | pass | pass | pass → pass |
tests/test_users.py::test_apostrophe_in_email sqlite3.OperationalError: near "brien": syntax error | fail | pass | pass | fail → pass |
tests/test_users.py::test_injection_matches_nobody assert {'id': 1, 'email': 'ana@example.com', 'name': 'Ana'} is None
+ where {'id': 1, 'email': 'ana@example.com', 'name': 'Ana'} = find_user(<sqlite3.Connection object at 0x7388bcbf74c0>, "' OR '1'='1") | fail | pass | pass | fail → pass |
tests/test_users.py::test_email_is_case_insensitive TypeError: 'NoneType' object is not subscriptable | fail | pass | pass | fail → pass |
Python 3.12 · pytest · canonset-sandbox-python:1 · 1.2 s · checked 2026-09-29 23:30 UTC
Output: Starting code (exit 1, 0.3 s)
..FFF [100%]
=================================== FAILURES ===================================
___________________________ test_apostrophe_in_email ___________________________
conn = <sqlite3.Connection object at 0x7388bcbf7b50>
def test_apostrophe_in_email(conn):
> assert find_user(conn, "o'brien@example.com")["name"] == "Siobhan O'Brien"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/test_users.py:28:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
conn = <sqlite3.Connection object at 0x7388bcbf7b50>
email = "o'brien@example.com"
def find_user(conn: sqlite3.Connection, email: str):
> row = conn.execute(f"SELECT id, email, name FROM users WHERE email = '{email}'").fetchone()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E sqlite3.OperationalError: near "brien": syntax error
app/users.py:5: OperationalError
________________________ test_injection_matches_nobody _________________________
conn = <sqlite3.Connection object at 0x7388bcbf74c0>
def test_injection_matches_nobody(conn):
> assert find_user(conn, "' OR '1'='1") is None
E assert {'id': 1, 'email': 'ana@example.com', 'name': 'Ana'} is None
E + where {'id': 1, 'email': 'ana@example.com', 'name': 'Ana'} = find_user(<sqlite3.Connection object at 0x7388bcbf74c0>, "' OR '1'='1")
tests/test_users.py:32: AssertionError
________________________ test_email_is_case_insensitive ________________________
conn = <sqlite3.Connection object at 0x7388bcb345e0>
def test_email_is_case_insensitive(conn):
> assert find_user(conn, "ANA@EXAMPLE.COM")["name"] == "Ana"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E TypeError: 'NoneType' object is not subscriptable
tests/test_users.py:36: TypeError
=========================== short test summary info ============================
FAILED tests/test_users.py::test_apostrophe_in_email - sqlite3.OperationalErr...
FAILED tests/test_users.py::test_injection_matches_nobody - assert {'id': 1, ...
FAILED tests/test_users.py::test_email_is_case_insensitive - TypeError: 'None...
3 failed, 2 passed in 0.04s
Output: Reference solution (exit 0, 0.3 s)
..... [100%] 5 passed in 0.02s
Output: Reference, second run (exit 0, 0.3 s)
..... [100%] 5 passed in 0.02s