Verified coding tasks

Tasks that prove their own tests

Each task is what a coding agent would receive: a problem statement and starting code, plus hidden tests and a reference solution. Our sandbox ran every one: the tests fail on the starting code, pass twice on the reference, and have no network access.

12 tasks
Python 3.12, TypeScript, JavaScript
33 fail→pass tests
Bug fixes, features, two security fixes

Download JSONL

Close a prototype-pollution hole in deepMerge

code-09 · what the agent sees is the problem statement and the starting code

hardTypeScript · Node.js 24 · node:test

Problem statement

merge.ts merges user settings (parsed from JSON) into defaults. A pentest showed that a settings payload of {"__proto__": {"isAdmin": true}} makes every object in the process report isAdmin === true. It also mutates the defaults object, so one user's settings leak into the next request, and arrays are merged index by index instead of replaced. Rewrite deepMerge(target, source) so that it returns a new object without modifying either input, merges plain objects recursively, replaces arrays and other values from source, and skips the keys __proto__, constructor and prototype.

Starting code

merge.ts

export function deepMerge(target: any, source: any): any {
  for (const key in source) {
    if (typeof source[key] === "object" && source[key] !== null) {
      target[key] = deepMerge(target[key] ?? {}, source[key]);
    } else {
      target[key] = source[key];
    }
  }
  return target;
}

Reference solution

merge.ts

type Plain = Record<string, unknown>;

const BLOCKED = new Set(["__proto__", "constructor", "prototype"]);

function isPlain(value: unknown): value is Plain {
  return typeof value === "object" && value !== null && !Array.isArray(value) && Object.getPrototypeOf(value) === Object.prototype;
}

export function deepMerge(target: Plain, source: Plain): Plain {
  const out: Plain = {};
  for (const key of Object.keys(target)) {
    if (!BLOCKED.has(key)) out[key] = target[key];
  }
  for (const key of Object.keys(source)) {
    if (BLOCKED.has(key)) continue;
    const value = source[key];
    if (isPlain(value)) out[key] = deepMerge(isPlain(out[key]) ? out[key] : {}, value);
    else if (Array.isArray(value)) out[key] = [...value];
    else out[key] = value;
  }
  return out;
}

Tests (hidden from the agent)

merge.test.ts

import assert from "node:assert/strict";
import { test } from "node:test";
import { deepMerge } from "./merge.ts";

test("merges nested objects", () => {
  assert.deepEqual(deepMerge({ a: { b: 1 } }, { a: { c: 2 } }), { a: { b: 1, c: 2 } });
});

test("null in source replaces the value", () => {
  assert.deepEqual(deepMerge({ a: { b: 1 } }, { a: null }), { a: null });
});

test("does not modify its inputs", () => {
  const defaults = { theme: { color: "blue" } };
  const settings = { theme: { size: 14 } };
  deepMerge(defaults, settings);
  assert.deepEqual(defaults, { theme: { color: "blue" } });
  assert.deepEqual(settings, { theme: { size: 14 } });
});

test("arrays are replaced, not merged", () => {
  assert.deepEqual(deepMerge({ tags: ["a", "b"] }, { tags: ["c"] }), { tags: ["c"] });
});

test("__proto__ in JSON cannot pollute objects", () => {
  try {
    deepMerge({}, JSON.parse('{"__proto__": {"isAdmin": true}}'));
    assert.equal(({} as { isAdmin?: boolean }).isAdmin, undefined);
  } finally {
    delete (Object.prototype as { isAdmin?: boolean }).isAdmin;
  }
});

Notes

The pollution test cleans up after itself, so a failing starting code cannot break the other tests.

Sandbox run

Recorded by pnpm samples:verify. Our tests fail the build if this stops matching the task.

Checks passed5 tests pass on the reference solution; 3 of them fail on the starting code.

Test	Starting code	Reference	Second run
merge.test.ts::merges nested objects	pass	pass	pass	pass → pass
merge.test.ts::null in source replaces the value	pass	pass	pass	pass → pass
merge.test.ts::does not modify its inputs Expected values to be strictly deep-equal: + actual - expected { theme: { color: 'blue', + size: 14 } }	fail	pass	pass	fail → pass
merge.test.ts::arrays are replaced, not merged Expected values to be strictly deep-equal: + actual - expected { tags: [ 'c', + 'b' ] }	fail	pass	pass	fail → pass
merge.test.ts::__proto__ in JSON cannot pollute objects Expected values to be strictly equal: + actual - expected + true - undefined	fail	pass	pass	fail → pass

TypeScript · Node.js 24 · node:test · canonset-sandbox-node:1 · 0.9 s · checked 2026-09-29 23:30 UTC

Output: Starting code (exit 1, 0.2 s)

✔ merges nested objects (1.459181ms)
✔ null in source replaces the value (0.16592ms)
✖ does not modify its inputs (1.364713ms)
✖ arrays are replaced, not merged (0.433983ms)
✖ __proto__ in JSON cannot pollute objects (0.442046ms)
ℹ tests 5
ℹ suites 0
ℹ pass 2
ℹ fail 3
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 154.54908

✖ failing tests:

test at merge.test.ts:13:1
✖ does not modify its inputs (1.364713ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
  + actual - expected
  
    {
      theme: {
        color: 'blue',
  +     size: 14
      }
    }
  
      at TestContext.<anonymous> (file:///work/0-starter/merge.test.ts:17:10)
      at Test.runInAsyncScope (node:async_hooks:227:14)
      at Test.run (node:internal/test_runner/test:1402:25)
      at Test.processPendingSubtests (node:internal/test_runner/test:974:18)
      at Test.postRun (node:internal/test_runner/test:1542:19)
      at Test.run (node:internal/test_runner/test:1467:12)
      at async Test.processPendingSubtests (node:internal/test_runner/test:974:7) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: { theme: { color: 'blue', size: 14 } },
    expected: { theme: { color: 'blue' } },
    operator: 'deepStrictEqual',
    diff: 'simple'
  }

test at merge.test.ts:21:1
✖ arrays are replaced, not merged (0.433983ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
  + actual - expected
  
    {
      tags: [
        'c',
  +     'b'
      ]
    }
  
      at TestContext.<anonymous> (file:///work/0-starter/merge.test.ts:22:10)
      at Test.runInAsyncScope (node:async_hooks:227:14)
      at Test.run (node:internal/test_runner/test:1402:25)
      at Test.processPendingSubtests (node:internal/test_runner/test:974:18)
      at Test.postRun (node:internal/test_runner/test:1542:19)
      at Test.run (node:internal/test_runner/test:1467:12)
      at async Test.processPendingSubtests (node:internal/test_runner/test:974:7) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: { tags: [ 'c', 'b' ] },
    expected: { tags: [ 'c' ] },
    operator: 'deepStrictEqual',
    diff: 'simple'
  }

test at merge.test.ts:25:1
✖ __proto__ in JSON cannot pollute objects (0.442046ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
  + actual - expected
  
  + true
  - undefined
  
      at TestContext.<anonymous> (file:///work/0-starter/merge.test.ts:28:12)
      at Test.runInAsyncScope (node:async_hooks:227:14)
      at Test.run (node:internal/test_runner/test:1402:25)
      at Test.processPendingSubtests (node:internal/test_runner/test:974:18)
      at Test.postRun (node:internal/test_runner/test:1542:19)
      at Test.run (node:internal/test_runner/test:1467:12)
      at async Test.processPendingSubtests (node:internal/test_runner/test:974:7) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: true,
    expected: undefined,
    operator: 'strictEqual',
    diff: 'simple'
  }

Output: Reference solution (exit 0, 0.2 s)

✔ merges nested objects (1.554491ms)
✔ null in source replaces the value (0.225159ms)
✔ does not modify its inputs (0.216244ms)
✔ arrays are replaced, not merged (0.223887ms)
✔ __proto__ in JSON cannot pollute objects (0.258944ms)
ℹ tests 5
ℹ suites 0
ℹ pass 5
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 139.597142

Output: Reference, second run (exit 0, 0.2 s)

✔ merges nested objects (1.510121ms)
✔ null in source replaces the value (0.176654ms)
✔ does not modify its inputs (0.215208ms)
✔ arrays are replaced, not merged (0.275419ms)
✔ __proto__ in JSON cannot pollute objects (0.227248ms)
ℹ tests 5
ℹ suites 0
ℹ pass 5
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 133.773689