A fresh deployment came up with a populated portfolio, an empty feeds screen and
no alerts, because everything downstream of the seeder was an admin call whose
order lived only in a runbook. POST /admin/bootstrap/demo now runs the six steps
in dependency order and reports each one.
The price simulator widens from the four commodities with no feed to every
active basket pair, so price_z_score gets its three-year seasonal mean and TW-05
can fire with no DATAGOV_API_KEY present at all. Levels come from a per-crop
table of ordinary mandi rates and the shape from the commodity group, because
the group already encodes perishability - a vegetable moves on a weather event
within days and a cereal does not, and flattening that would erase the one
distinction the z-score exists to read.
A pair holding any non-simulated row is skipped unless forced. That ordering is
the safety property: run this after a real ingestion and it fills the gaps
around the real series rather than over them.
The orchestrator is deliberately not transactional. The seeder holds one
transaction across its whole run, and when the machine slept mid-run nothing
committed until morning while the console confidently served pre-seed state. A
bootstrap generating three years of prices across every pair is long enough to
hit exactly that, so each step commits alone and a failure in one leaves the
others standing.
Still not trainable, and coverage still says so. The seeder derives DPD from the
band and the UPI simulator derives inflow from the same band, so a model fitted
here recovers the seeder rule and reports fine metrics for doing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PortfolioPlan makes the demo book explicit — 263 enterprises across six states,
one per NABARD zone, over eight partner banks and sixteen branches — and
PortfolioPurge is the destructive escape hatch for the one boot where that shape
changes. The seeder never rewrites an existing enterprise, so a plan change
without a purge leaves orphans rather than corrupting rows.
udyam.forecast.trust-seeded now defaults to false. It was true only because
ledger coverage reached six enterprises of 142, so without the fallback the
console had almost no forecasts and showed 34 trip-wires of which 5 were real.
The seeder now writes five months of ledger for every enterprise, band-coherent
— a RED enterprise's monthly net genuinely sits below its instalment — so the
computed forecast is the one worth showing and the fallback is no longer
load-bearing.
jwt.secret becomes UDYAM_JWT_SECRET-overridable. The default stays committed so
a fresh clone boots without setup, which is precisely why it is not a secret:
any deployment holding real borrower data must set it.
Two traps this run surfaced, both silent:
- dealBands() now deals round-robin across sectors. Appending HORTICULTURE to
the sector list put it last and the fixed-seed band shuffle was drained by
the five packs ahead of it, so all 19 seeded STANDARD.
- Anything added to the seeder after a database was first populated never
reaches it, because of the count() > 0 guards. backfillPortfolio() now
covers loans and ledger and FeatureCatalog.seed() reconciles tolerances, but
trip-wire rules and sector weights still carry the hazard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine of fifteen features now reach 263/263 and all six trip-wires fire for the
first time — TW-02 and TW-05 had never fired in the project's life. That
coverage comes partly from simulation, and every simulated row says so:
source = SIMULATED, upserted on a key that includes source, and refused if the
existing row is real even with overwrite=true. Real and simulated coexist; the
simulator can never overwrite a fact.
UpiSimulationService writes 26 weeks of upi_summary for each enterprise and
respects consentActive — a simulator that ignored consent would erase the very
distinction NO_CONSENT exists to hold.
NoFeedPriceSimulationService covers milk, egg, fodder and sugarcane, which have
no usable API anywhere: data.gov.in carries only annual state averages, NECC
publishes HTML, and cane is sold to mills at a set price and never reaches a
mandi. Milk's level is anchored on a real 2018 cooperative procurement figure
carried forward.
Calibrating a generated series took three passes, each plausible as a chart and
wrong as an input to a specific rule. Real Agmarknet series average a
price_z_score of 0.44, which is what exposed all of it:
- Too smooth. Index series at +/-1.5% weekly noise gave a tiny seasonal stdev,
so the newest value was an outlier by construction and TW-05 fired for 197
of 263. Noise must be large relative to trend.
- Monotonic. Even at +/-4% noise, a 5%/yr trend compounding over three years
swamped a +/-9% year shift, so every newest week was a record. Trend cut to
1.5-2.5%/yr and the year shift widened to +/-13%, so year-over-year
variation dominates. Portfolio mean z 1.39 -> 0.91, TW-05 197 -> 72.
- Wrong window. A linear decline over 26 weeks moves an 8-vs-8-week comparison
by only a third of its total, so a 40% drift showed as 13% and never reached
TW-02's -20% threshold. The decline now starts at 40% of the window and
curves. Check a generated series against the rule's own window, not against
its total drift.
Alerts are cumulative, so stale rows from the miscalibrated runs had to be
cleared: risk_alert held 206 TW-05 rows while the corrected run raised one.
The four output indices are categorically weaker than the rest and are labelled
separately in the console. Milk and egg are real prices lacking an API; a
finished sari has no price series to lack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Commodities 7 -> 21 and a district_commodity table holding a per-district
basket of 60 pairs, replacing the district x commodity cross product that would
have been 252 mostly-impossible calls a night. District, not zone: Kolar and
Belagavi are both SOUTH and trade almost disjoint baskets.
Only the four original cereals had verified Agmarknet spellings, so the rest
carry sourceVerified = false until POST /admin/ingestion/probe confirms them.
The probe separates a wrong spelling from an absent local market by
cross-referencing which districts and commodities returned rows anywhere — a
distinction that matters because both fail silently and identically.
Three district names were costing us every price in their district: Nasik ->
Nashik, Ahmadnagar -> Ahilyanagar, Belgaum -> Belagavi. The last is the
nastiest: Belgaum still answers with 1000 rows, but they stop in May 2026 while
Belagavi is current, so a retired name serving a frozen archive reads as a
healthy feed. Only the arrival dates give it away.
data.gov.in meters by quota per window, so the limit is reached by the run and
not by any one call: a 62-pair run straight after a 62-pair probe returned 429
on all 62 and recorded the whole ingestion FAILED with zero rows. DataGovClient
now retries with a 45s backoff, long because the window is minutes not seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a fourth supervisory level between apex and bank — HO -> RO -> Bank ->
Branch -> Enterprise — as 25 offices with a state mapping, plus /nabard/
regional-offices and its bank drill.
An RO is Tier.APEX carrying a regionalOfficeId, not a tier of its own, so it
inherits mayReadBorrowers() == false rather than restating it. The office is
resolved branch -> district -> state -> office and never stored on the bank:
six of eight partner banks are state-bound RRBs, so a bank-level column would
look right today and break the moment an SFB opens a second-state branch.
The two new endpoints scoped correctly from the start while the eight
pre-existing apex reads ignored the office claim entirely — the supervisory
boundary was decorative, though no borrower leaked. Every read now passes
through ApexService.scopeDistricts(tenant), one choke point rather than eight
guards. All six offices sum exactly to the national 263 units / 8 banks /
342 alerts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The book held one snapshot per loan, so there was no history at all and
nothing a model could learn from. Two routes now fill it, and the
difference between them is stated rather than implied: generate
synthetic monthly snapshots for the prototype, or backfill month-end
extracts from the lender's core, which is the real path — every core can
answer "what did the book look like at each month end last year",
because it is the query behind their regulatory returns.
LoanRecord.source is stamped at every write site: SEEDED,
SEEDED_HISTORY, UPLOAD, or the connector profile code. Coverage reports
three buckets — synthetic, real, and unknown provenance — and never
folds unknown into real, because an unlabelled row is not evidence of a
lender extract. With no real extract on the book it reports
trainable:false and says why in words. Generation never overwrites an
existing row, so a real extract survives a regeneration untouched, and
purge removes only what was generated.
The generator's arithmetic had to be got right and was not at first.
Real DPD steps: pay and cure to zero, or miss and roll forward one
instalment period. The first version drew the day-of-month slack per
month, so consecutive months differed by anything from 1 to 59 days — a
shape no loan book produces, and one that would have taught a model a
transition matrix nothing real generates. The slack is now fixed per
delinquency episode, since a due date does not move between two missed
payments: an account 5 days down that misses again is 35 days down, not
12. For the run reaching the anchor it is derived from the anchor's own
DPD, so the series joins today's number exactly rather than near it.
DpdWalk is shared with the simulated adapters, which previously returned
today's DPD for every requested date — a backfill through them would
have been a flat line, which is exactly the shape that makes a training
label useless and which a chart would have hidden.
Verified against a running instance: 3,256 snapshots across 142 loans,
24 monthly extract dates spanning 2024-08 to 2026-07, 100% of
enterprises now carrying history, every generated series rolling by
exactly 30 — the only non-30 steps being joins to pre-existing rows the
generator correctly refused to overwrite. A 6-month Finacle backfill
pulled 264 rows and staged them for review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loan book is the training label and the one input no public feed
supplies. Integrating with a lender's core is not a technical unknown —
it is an onboarding process per bank, with a schema agreed in a room.
What can be built before that room is the seam, and this is it.
CoreBankingConnector fetches and returns rows; it never writes. Mapped
rows go to LoanBookEntryService.stage — the same validator an uploaded
spreadsheet goes through, landing in the same manual_entry_batch with
the same preview. A parallel validator for the automated path would be
the obvious shape and would eventually enforce the no-PII rule
differently.
FileDropConnector is real: it reads the newest matching file from a
directory a bank delivers to, and leaves it there so a failed run is
repeatable. Finacle, FLEXCUBE, BaNCS and an in-house REST LMS are
simulated, each emitting the same portfolio in its own column dialect —
DPD_DAYS, OVERDUE_DAYS, DAYS_OVERDUE, daysPastDue — which is what makes
the case for a mapping layer rather than asserting it.
The mapping is {canonical: vendor}, keyed by canonical name, so an
unmapped vendor column has no route through and is dropped at the
boundary. CUST_NAME and MOBILE_NO stay out structurally rather than by
anyone remembering. Auto-commit is off on every profile: a sync stages
and stops, keeping the rule that nothing is written until someone has
seen what will be. simulated is stamped per run, not read from the
profile, so switching a profile live cannot make old runs look real.
Fixes a real defect this surfaced: FORBIDDEN_COLUMNS matched on the
exact normalised name, so CUST_NAME — what Finacle actually calls it —
was missed, and a directly uploaded bank file would have kept the
borrower's name in manual_entry_line.raw_row. Matching now splits
separators and camelCase, catching borrowerMobile too, while
loan_account_ref still splits clean.
Verified against a running instance: Finacle 44/44 accepted with 4
columns dropped, the file drop 2 accepted and 1 UNKNOWN_ENTERPRISE
rejected from a real CSV, and both CUST_NAME and borrowerMobile refused
at mapping save.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
