A test set for AI extraction of council contracts

Synthetic PDF in, expected CSV out, plus the wrong extraction. Each case folder has document.pdf, gold.json / expected.csv (what an extractor should return), notes.md (why the case is hard) and the expected rule-engine flags. A deliberately naive pdftotext + regex baseline is scored on every case so you can see exactly where a text-dump approach breaks.

Every figure, authority and supplier is fictional and each page carries a visible "synthetic test document" marking. Real council notices can be pulled from Find a Tender and Contracts Finder (Open Government Licence) with crtest.collect, which records source and licence per file.

git clone https://github.com/Marxist-Leninist/contractradar-testset
cd contractradar-testset
pip install -r requirements.txt

# score your own extractor: write predictions in the
# results/ format, then
python -m crtest.score --pred results/my_extractor.json

# or run the built-in LLM extractor against any
# OpenAI-compatible endpoint (no keys in the repo)
export LLM_BASE_URL=https://api.deepseek.com/v1 LLM_API_KEY=... LLM_MODEL=deepseek-chat
python -m crtest.extract_llm --out results/deepseek.json --vision --ocr
python -m crtest.score --pred results/deepseek.json

Add a case. Add a spec to crtest/gen.py (gold fields + layout style + hazards) and run make cases. The PDF and gold labels are generated from the same spec, so they cannot disagree. Good targets from the pilot discussion: tables without set boundaries, grouped ID rows such as dates, and page boundaries - we have one of each, more variants welcome.

Score an extractor. Write predictions as {"extractor": "name", "cases": {"CR-001": {"fields": {...}, "flags": [...]}}} and run the scorer. Commit the predictions and the .scored.json to results/ and the leaderboard picks them up.

Label real documents. Use crtest.collect to fetch council notices, copy the OCDS values into a gold.json with the same field names, and attach the published document. Keep the _provenance block: the pilot's data rules require source and licence for everything uploaded.

Rules. crtest/pa2023.py holds the reference checks (thresholds from 1 Jan 2026, 10%/15% modification limits, 8-working-day standstill, 30-day notices, 3 KPIs over £5m). If you think a check is wrong, open an issue with the section of the Act.

Discussion: pilot thread on Discord. Generated 31 October 2026.