Eight datasets where the label is computed by code, then re-derived from the prompt text alone by a verifier that does not import the generator. A row that cannot be reproduced from what the reader is given does not ship.
| dataset | task | train | val | test | total | strongest shortcut (test) |
|---|---|---|---|---|---|---|
| Contradiction Ledger | Find facts stated one way then another | 40,000 | 2,000 | 4,000 | 46,000 | 0.719 |
| WitnessDrift | Reconstruct an event from partly-wrong accounts | 50,000 | 2,500 | 5,000 | 57,500 | 0.384 |
| FraudTrail | Name the scheme in a ledger, or say there is none | 35,000 | 2,000 | 4,000 | 41,000 | 0.316 |
| TimelineForge | Rebuild an absolute timeline told out of order | 60,000 | 3,000 | 6,000 | 69,000 | 0.643 |
| DeEscalate | Pick the reply that cools a conflict | 45,000 | 2,500 | 5,000 | 52,500 | 0.687 |
| Urdu Legal Reason | Apply a statute to a complaint, trilingual | 30,000 | 2,000 | 4,000 | 36,000 | 1.000 |
| Interrogation Logic | Name answers that contradict the record | 40,000 | 2,500 | 5,000 | 47,500 | 0.307 |
| ProtocolCheck | Audit an execution log against its procedure | 45,000 | 2,500 | 5,000 | 52,500 | 0.676 |
| Total | 345,000 | 19,000 | 38,000 | 402,000 |
Each dataset ships its own baseline table, including the baselines a model could exploit instead of doing the task. The column above is the strongest of those shortcuts on the test split, so lower means more of the task is actually left to do. Applying the full stated rules scores 1.000 on every one of the eight, which is what makes them verified rather than merely generated.
Urdu Legal Reason is the honest exception. A word-overlap baseline that knows no law is right every time, because each provision has one fact template per language and train and test share templates. Treat it as trilingual parallel legal text with a fixed label set, not as a test of legal reasoning. That limit is stated on its Kaggle page as a number rather than left for someone to discover.
No prompt appears in two splits in any of the eight. All rows are synthetic, generated and labelled by code, with no personal data. Licence CC BY 4.0.
Index and build notes: github.com/hammasbuilds/llm-agentic-datasets · Computer vision & image generation · Notebooks · Code