The CBS/LMS extract is the deepest gap in the system: emi_delay and
emi_to_inflow are computed from the loan book, and DPD history is the
training label without which the model cannot be trained at all. This is
how a real book gets in before a lender builds a nightly feed.
Two schema problems had to be fixed first.
loan_record had a unique index on accountRef alone, so the table could
hold only each loan's current state — every monthly extract overwriting
the evidence the model is meant to learn from. The key is now
(accountRef, asOfDate): one row per loan per extract. The old constraint
must be dropped by hand, as ddl-auto adds indexes but never removes them.
emi_delay and emi_to_inflow took the worst DPD across every row at or
before the scoring date, which with snapshots present would report a loan
sixty days down in March as sixty days down today. Both now read the
newest extract per accountRef. The history is there to train on, not to
be summarised as the present.
The no-PII promise is enforced at the parse boundary. A file whose headers
carry a name, phone, Aadhaar, PAN, account number or address is rejected
outright with the offending columns named, rather than having the column
quietly dropped — a bank must learn the data is unwanted before it builds
a nightly job that keeps sending it.
ManualEntryLine now stores the row as supplied, so commit reads the file's
own columns rather than a re-parse of a file nobody kept, and an auditor
sees what the bank sent rather than our reading of it.
Verified: a six-month, two-loan book uploads as 12 snapshots; a file with
customer_name and mobile_no is refused whole; and emi_delay reports 2 for
a loan that peaked at 3 and is now 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dscr_p10 is the feature TW-01 fires on, and so the origin of nearly every
alert an officer sees — but nothing had written forecast_point since the
seeder. A headline number computed from demo data is worse than no
headline number, because it looks identical to a real one.
ForecastStrategy is a seam shaped like ScoringStrategy: the LightGBM
quantile heads implement it later and the swap is one property. The
forecast and the score answer different questions and fail differently,
so they get separate seams.
The method is fully explainable. Baseline is the median whole-month net
position from the ledger, median so one festival month does not become
the expectation, and the partial current month is discarded. Drift comes
from margin_squeeze, else feed_cost_change, capped so one sharp month
does not extrapolate into a cliff. Spread widens with the square root of
the horizon and again for every input we did not have.
Missing inputs widen the band, never lower the centre. A pessimistic P50
built on ignorance would manufacture breaches.
An enterprise with too little ledger gets no path rather than a fabricated
one, so dscr_p10 resolves NO_FORECAST visibly.
forecast_point gains engine/model_version. Seeded rows carry neither, and
udyam.forecast.trust-seeded decides whether dscr_p10 may fall back to
them. Left true so the demo portfolio stays populated; with it false the
picture is honest — 6 enterprises forecast, 136 NO_FORECAST, and
trip-wires firing drops from 34 to 5.
The forecast reads margin features and dscr_p10 reads the forecast, so
features are computed twice per enterprise rather than letting either half
read a previous run's answer.
Seeder now writes 150 days of ledger, not 31: the baseline needs whole
months and one month leaves nothing to take a median of. Ledger joins
loans in the backfill pass, keyed on clientId.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Egg, milk and fodder have no usable API for the pilot districts — NECC
publishes HTML, dairy rates exist only as annual state averages, and every
recent Agmarknet fodder row is one market in Ambala. This is how a bank
gets that data in.
Upload stages a PREVIEW batch and writes nothing. It reports what would
land, what was rejected and why, and which rows would overwrite an
existing value. Only commit touches market_signal.
Effective dating: effective_date is the date the fact describes and comes
from the file; observedAt is when we learned it and is stamped at commit.
The uploader cannot reach observedAt — a March rate uploaded in August
must stay invisible to a March scoring run, or the model trains on data
that did not exist yet.
Rejections name themselves: UNKNOWN_DISTRICT, UNKNOWN_COMMODITY,
FUTURE_DATE, BAD_DATE, BAD_NUMBER, UNIT_MISMATCH, PRICE_ORDER,
DUPLICATE_IN_FILE, MISSING_REQUIRED. A price far from last week warns but
is accepted — a genuine price shock is the event this system exists to
catch, and rejecting it would be backwards.
Rows are written with source=MANUAL, which the unique index keeps separate
from a feed's own row for the same week, so manual data never silently
replaces a government feed.
Also distinguishes INSUFFICIENT_HISTORY from NO_SIGNAL in the price
derivations. A newly onboarded manual feed spends its first weeks below
the four-week minimum, and reporting NO_SIGNAL there reads as a broken
integration rather than a series still filling up.
Result: margin_squeeze and output_price_change compute for the first time
— 57 of 142 enterprises, exactly the DAIRY (30) and POULTRY (27) sectors
that have an output commodity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Spring Boot 4 / Java 17 / Postgres. Masters and serving (A), Agmarknet and
climate ingestion (B), feature store with 14 derivations (C), SpEL rule
engine and risk scoring (D).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
