Canonset
Log inApply as an expert
← Samples

RL environment · business software

Expense approvals

A practice copy of Alder & Vale Freight's expense-review app (a fictional company), a written policy, and 50 tasks an agent is scored on automatically. Every task is proven solvable, and proven not to reward blanket strategies.

Play a taskDownload tasks (JSONL)

Verification, recomputed on every page load

Share of checks each strategy passes. A task is valid when the reference scores 100% and no attack reaches 100%.

TaskReferenceDo nothingApprove everythingReject everythingAsk for receipts everywhereValid
exp-01100%60%40%0%0%valid
exp-02100%60%0%0%40%valid
exp-03100%43%14%14%0%valid
exp-04100%50%17%0%0%valid
exp-05100%50%50%0%0%valid
exp-06100%50%17%0%0%valid
exp-07100%50%50%0%0%valid
exp-08100%43%14%14%0%valid
exp-09100%43%14%14%0%valid
exp-10100%43%14%14%0%valid
exp-11100%60%40%0%0%valid
exp-12100%60%0%0%20%valid
exp-13100%60%40%0%0%valid
exp-14100%50%17%0%0%valid
exp-15100%60%0%0%20%valid
exp-16100%50%17%0%0%valid
exp-17100%60%0%0%0%valid
exp-18100%60%40%0%0%valid
exp-19100%50%17%0%0%valid
exp-20100%60%0%0%0%valid
exp-21100%43%14%14%0%valid
exp-22100%43%14%14%0%valid
exp-23100%57%43%14%14%valid
exp-24100%50%0%0%17%valid
exp-25100%60%0%0%20%valid
exp-26100%38%13%13%0%valid
exp-27100%67%0%0%17%valid
exp-28100%60%0%0%0%valid
exp-29100%60%0%0%0%valid
exp-30100%60%0%0%40%valid
exp-31100%60%40%0%0%valid
exp-32100%50%17%0%0%valid
exp-33100%50%17%0%0%valid
exp-34100%50%17%0%0%valid
exp-35100%38%25%13%0%valid
exp-36100%44%33%0%0%valid
exp-37100%0%42%8%25%valid
exp-38100%33%33%11%0%valid
exp-39100%43%29%0%0%valid
exp-40100%30%30%0%10%valid
exp-41100%27%18%9%0%valid
exp-42100%43%0%0%29%valid
exp-43100%60%0%20%0%valid
exp-44100%67%0%0%50%valid
exp-45100%60%0%0%0%valid
exp-46100%43%14%0%0%valid
exp-47100%43%14%0%0%valid
exp-48100%27%9%27%0%valid
exp-49100%20%30%10%10%valid
exp-50100%0%50%0%17%valid