Commodities 7 -> 21 and a district_commodity table holding a per-district
basket of 60 pairs, replacing the district x commodity cross product that would
have been 252 mostly-impossible calls a night. District, not zone: Kolar and
Belagavi are both SOUTH and trade almost disjoint baskets.
Only the four original cereals had verified Agmarknet spellings, so the rest
carry sourceVerified = false until POST /admin/ingestion/probe confirms them.
The probe separates a wrong spelling from an absent local market by
cross-referencing which districts and commodities returned rows anywhere — a
distinction that matters because both fail silently and identically.
Three district names were costing us every price in their district: Nasik ->
Nashik, Ahmadnagar -> Ahilyanagar, Belgaum -> Belagavi. The last is the
nastiest: Belgaum still answers with 1000 rows, but they stop in May 2026 while
Belagavi is current, so a retired name serving a frozen archive reads as a
healthy feed. Only the arrival dates give it away.
data.gov.in meters by quota per window, so the limit is reached by the run and
not by any one call: a 62-pair run straight after a 62-pair probe returned 429
on all 62 and recorded the whole ingestion FAILED with zero rows. DataGovClient
now retries with a 45s backoff, long because the window is minutes not seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
