Egg, milk and fodder have no usable API for the pilot districts — NECC
publishes HTML, dairy rates exist only as annual state averages, and every
recent Agmarknet fodder row is one market in Ambala. This is how a bank
gets that data in.
Upload stages a PREVIEW batch and writes nothing. It reports what would
land, what was rejected and why, and which rows would overwrite an
existing value. Only commit touches market_signal.
Effective dating: effective_date is the date the fact describes and comes
from the file; observedAt is when we learned it and is stamped at commit.
The uploader cannot reach observedAt — a March rate uploaded in August
must stay invisible to a March scoring run, or the model trains on data
that did not exist yet.
Rejections name themselves: UNKNOWN_DISTRICT, UNKNOWN_COMMODITY,
FUTURE_DATE, BAD_DATE, BAD_NUMBER, UNIT_MISMATCH, PRICE_ORDER,
DUPLICATE_IN_FILE, MISSING_REQUIRED. A price far from last week warns but
is accepted — a genuine price shock is the event this system exists to
catch, and rejecting it would be backwards.
Rows are written with source=MANUAL, which the unique index keeps separate
from a feed's own row for the same week, so manual data never silently
replaces a government feed.
Also distinguishes INSUFFICIENT_HISTORY from NO_SIGNAL in the price
derivations. A newly onboarded manual feed spends its first weeks below
the four-week minimum, and reporting NO_SIGNAL there reads as a broken
integration rather than a series still filling up.
Result: margin_squeeze and output_price_change compute for the first time
— 57 of 142 enterprises, exactly the DAIRY (30) and POULTRY (27) sectors
that have an output commodity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First end-to-end run against Postgres and the real government APIs, which
surfaced four defects that static checks could not.
- Agmarknet pointed at 9ef84268-…, "Current Daily Price": a today-only
snapshot holding one arrival_date, in which none of the four pilot
districts appear. Switched to 35985678-…, the historical resource.
- Filter fields were lowercase. This resource matches exactly on the
capitalised name; the lowercase form is tokenised, so "Madhya Pradesh"
also matched Andhra, Himachal and Uttar Pradesh — wrong-state prices
written under a pilot district, with freshness looking healthy.
- Rows arrive unordered and the lookback filter is client side, so a page
was mostly years-old data and every commodity wrote zero weeks. Added
sort[Arrival_Date]=desc.
- ExternalFeedConfig injected a RestClient.Builder bean Boot does not
expose, so the context never built and the app could not start.
Also: Open-Meteo meters by request cost, so the 30-year baseline call
exhausted the minutely allowance and 429'd two districts out of their
climate data. Rate-limited calls now retry with backoff, while genuine
failures still fail once and are reported.
FODDER moves to MANUAL — Agmarknet carries Green Fodder, but every recent
row is a single market in Ambala, Haryana.
Result: 16,000 rows fetched, 202 market weeks and 52 climate weeks written,
0 failed calls; 142 scored, 33 trip-wires fired.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Spring Boot 4 / Java 17 / Postgres. Masters and serving (A), Agmarknet and
climate ingestion (B), feature store with 14 derivations (C), SpEL rule
engine and risk scoring (D).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
