feature_value held three as-of dates. A point-in-time training set needs
history, so three paths now generate it:
POST /admin/features/backfill?from=&to= loops computeAll(monthEnd) across
a date range — 23 months, ~6 min
POST /admin/bootstrap/ledger-history fills the gap before the earliest
entry only, stamped SEEDED_HISTORY
so generated rows stay
distinguishable from recorded ones
ClimateIngestionService honours the lookbackWeeks the
controller was already passing and
had been ignoring, so the
Open-Meteo archive can be pulled
Together these took the store to 21 monthly as-of dates over 263 enterprises,
which is the 5,523-row dataset the published metrics were measured on.
Documentation catches up with the code: API-SPEC lists the new endpoints,
ARCHITECTURE §7 records what was deliberately not built and why.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The offline promise was only half kept. Ledger entries were queued on the
phone and marked PENDING, and the client flipped them to SYNCED without
sending anything — so an owner could record a sale with no signal, which was
the point, but it never reached the lender and the officer's screen never saw
it. POST /app/ledger/sync now has a caller; it dedupes on the clientId the
phone minted, so a retry after a half-failed sync writes nothing twice.
Adds GET /app/reasons: the borrower's own scored reason codes, newest run
only. A borrower told their limit is under review is entitled to know which
measurement drove it, and the app had been generating sentences from a demo
sector table rather than reading the ones the model produced. Both surfaces
now render the identical string.
PUT /app/consent was already here but nothing called it; the profile toggle
was local state. Consent is the artefact that makes reading a borrower's
inflows lawful, so it has to live on the server and be revocable from the
owner's own device.
Separately: the connector layer no longer names third-party core banking
products. Finacle, FLEXCUBE and BaNCS are replaced by CBSNexora, LincPay's
own core, with an in-place migration in ConnectorProfileSeeder.renameLegacy()
that also rewrites existing sync runs and manual entry batch labels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both repositories were queried by enterprise alone, so every reason code and
every forecast point ever written came back. A run-287 model score was being
rendered beside run-145 rule sentences, and one enterprise's six-month chart
drew 102 points from seventeen different runs.
Now read through findByEnterpriseIdAndRunId..., resolved from the newest run.
The same unscoped query existed in EnterpriseAppService and is fixed there
too.
Also adds GET /officer/app-enrolled: which borrowers on this caller's book
have a phone login. Not every borrower is enrolled and that is the design —
the platform scores a book from core banking and UPI whether or not the owner
installs anything — so the console needs to know rather than offering to open
an app that does not exist. Scoped by intersecting with the caller's own book,
so it cannot be used to discover that an id exists on another bank's.
EnterpriseRow gains firstBreachHorizon, firstBreachMonth and topDriver so the
dashboard can rank by lead time instead of by composite score.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds MlClient and the two model-backed strategies behind the existing
ScoringStrategy / ForecastStrategy seams, so switching engines stays a config
value rather than a code change.
MlClient never throws. Every call returns Optional.empty() on any failure —
timeout, connection refused, malformed body — and ModelScoringStrategy falls
back to the rule engine for that enterprise. A nightly run must complete, and
a demo must survive the Python process dying mid-presentation.
The important part is what the fallback records. A run configured MODEL used
to stamp every score MODEL even when the ML service was down and rules had
actually produced the number, which made the provenance column a lie exactly
when it mattered. ScoreOutcome now carries an `engine` field and the fallback
returns .attributedTo("RULE_FALLBACK"). Verified by killing the service
mid-run: run 285 MODEL, run 286 RULE_FALLBACK, 260 of 263 scores differ.
ModelMetric and its read-only controller expose the measured backtest at
GET /admin/model/metrics. syntheticLabelShare is stored on the same row as
every metric so accuracy cannot be quoted without the caveat travelling with
it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces a hash of the commodity code with a table of the week each commodity's
price actually peaks. The peak is the lean season, not the harvest: wheat comes
in during March and April and is cheapest then, and dearest in November when
last season's stock has run down and the new crop is still in the ground. Wheat
is therefore week 48, not week 14, and the same inversion runs down the list —
rabi peaks in autumn, kharif in early summer, and winter vegetables peak through
the monsoon when they cannot be grown at all.
This is not cosmetic. margin_squeeze reads a sector's OUTPUT price against its
INPUT price, and a squeeze is only real when the two peak at different times of
year. A hashed peak scattered them arbitrarily, so a poultry unit could show
feed and egg peaking in the same week and a dairy could show them six months
apart, both by accident. The seasonal relationship between an input and an
output is precisely what the derivation measures, so it has to come from a
calendar rather than from a hash.
Approximate and national, and both caveats matter: harvest windows shift by
weeks across fourteen districts and six zones, and this carries one figure each.
Better than a hash, not a district crop calendar. A commodity with no entry logs
a warning and falls back to the hash rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A fresh deployment came up with a populated portfolio, an empty feeds screen and
no alerts, because everything downstream of the seeder was an admin call whose
order lived only in a runbook. POST /admin/bootstrap/demo now runs the six steps
in dependency order and reports each one.
The price simulator widens from the four commodities with no feed to every
active basket pair, so price_z_score gets its three-year seasonal mean and TW-05
can fire with no DATAGOV_API_KEY present at all. Levels come from a per-crop
table of ordinary mandi rates and the shape from the commodity group, because
the group already encodes perishability - a vegetable moves on a weather event
within days and a cereal does not, and flattening that would erase the one
distinction the z-score exists to read.
A pair holding any non-simulated row is skipped unless forced. That ordering is
the safety property: run this after a real ingestion and it fills the gaps
around the real series rather than over them.
The orchestrator is deliberately not transactional. The seeder holds one
transaction across its whole run, and when the machine slept mid-run nothing
committed until morning while the console confidently served pre-seed state. A
bootstrap generating three years of prices across every pair is long enough to
hit exactly that, so each step commits alone and a failure in one leaves the
others standing.
Still not trainable, and coverage still says so. The seeder derives DPD from the
band and the UPI simulator derives inflow from the same band, so a model fitted
here recovers the seeder rule and reports fine metrics for doing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PortfolioPlan makes the demo book explicit — 263 enterprises across six states,
one per NABARD zone, over eight partner banks and sixteen branches — and
PortfolioPurge is the destructive escape hatch for the one boot where that shape
changes. The seeder never rewrites an existing enterprise, so a plan change
without a purge leaves orphans rather than corrupting rows.
udyam.forecast.trust-seeded now defaults to false. It was true only because
ledger coverage reached six enterprises of 142, so without the fallback the
console had almost no forecasts and showed 34 trip-wires of which 5 were real.
The seeder now writes five months of ledger for every enterprise, band-coherent
— a RED enterprise's monthly net genuinely sits below its instalment — so the
computed forecast is the one worth showing and the fallback is no longer
load-bearing.
jwt.secret becomes UDYAM_JWT_SECRET-overridable. The default stays committed so
a fresh clone boots without setup, which is precisely why it is not a secret:
any deployment holding real borrower data must set it.
Two traps this run surfaced, both silent:
- dealBands() now deals round-robin across sectors. Appending HORTICULTURE to
the sector list put it last and the fixed-seed band shuffle was drained by
the five packs ahead of it, so all 19 seeded STANDARD.
- Anything added to the seeder after a database was first populated never
reaches it, because of the count() > 0 guards. backfillPortfolio() now
covers loans and ledger and FeatureCatalog.seed() reconciles tolerances, but
trip-wire rules and sector weights still carry the hazard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine of fifteen features now reach 263/263 and all six trip-wires fire for the
first time — TW-02 and TW-05 had never fired in the project's life. That
coverage comes partly from simulation, and every simulated row says so:
source = SIMULATED, upserted on a key that includes source, and refused if the
existing row is real even with overwrite=true. Real and simulated coexist; the
simulator can never overwrite a fact.
UpiSimulationService writes 26 weeks of upi_summary for each enterprise and
respects consentActive — a simulator that ignored consent would erase the very
distinction NO_CONSENT exists to hold.
NoFeedPriceSimulationService covers milk, egg, fodder and sugarcane, which have
no usable API anywhere: data.gov.in carries only annual state averages, NECC
publishes HTML, and cane is sold to mills at a set price and never reaches a
mandi. Milk's level is anchored on a real 2018 cooperative procurement figure
carried forward.
Calibrating a generated series took three passes, each plausible as a chart and
wrong as an input to a specific rule. Real Agmarknet series average a
price_z_score of 0.44, which is what exposed all of it:
- Too smooth. Index series at +/-1.5% weekly noise gave a tiny seasonal stdev,
so the newest value was an outlier by construction and TW-05 fired for 197
of 263. Noise must be large relative to trend.
- Monotonic. Even at +/-4% noise, a 5%/yr trend compounding over three years
swamped a +/-9% year shift, so every newest week was a record. Trend cut to
1.5-2.5%/yr and the year shift widened to +/-13%, so year-over-year
variation dominates. Portfolio mean z 1.39 -> 0.91, TW-05 197 -> 72.
- Wrong window. A linear decline over 26 weeks moves an 8-vs-8-week comparison
by only a third of its total, so a 40% drift showed as 13% and never reached
TW-02's -20% threshold. The decline now starts at 40% of the window and
curves. Check a generated series against the rule's own window, not against
its total drift.
Alerts are cumulative, so stale rows from the miscalibrated runs had to be
cleared: risk_alert held 206 TW-05 rows while the corrected run raised one.
The four output indices are categorically weaker than the rest and are labelled
separately in the console. Milk and egg are real prices lacking an API; a
finished sari has no price series to lack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Commodities 7 -> 21 and a district_commodity table holding a per-district
basket of 60 pairs, replacing the district x commodity cross product that would
have been 252 mostly-impossible calls a night. District, not zone: Kolar and
Belagavi are both SOUTH and trade almost disjoint baskets.
Only the four original cereals had verified Agmarknet spellings, so the rest
carry sourceVerified = false until POST /admin/ingestion/probe confirms them.
The probe separates a wrong spelling from an absent local market by
cross-referencing which districts and commodities returned rows anywhere — a
distinction that matters because both fail silently and identically.
Three district names were costing us every price in their district: Nasik ->
Nashik, Ahmadnagar -> Ahilyanagar, Belgaum -> Belagavi. The last is the
nastiest: Belgaum still answers with 1000 rows, but they stop in May 2026 while
Belagavi is current, so a retired name serving a frozen archive reads as a
healthy feed. Only the arrival dates give it away.
data.gov.in meters by quota per window, so the limit is reached by the run and
not by any one call: a 62-pair run straight after a 62-pair probe returned 429
on all 62 and recorded the whole ingestion FAILED with zero rows. DataGovClient
now retries with a 45s backoff, long because the window is minutes not seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a fourth supervisory level between apex and bank — HO -> RO -> Bank ->
Branch -> Enterprise — as 25 offices with a state mapping, plus /nabard/
regional-offices and its bank drill.
An RO is Tier.APEX carrying a regionalOfficeId, not a tier of its own, so it
inherits mayReadBorrowers() == false rather than restating it. The office is
resolved branch -> district -> state -> office and never stored on the bank:
six of eight partner banks are state-bound RRBs, so a bank-level column would
look right today and break the moment an SFB opens a second-state branch.
The two new endpoints scoped correctly from the start while the eight
pre-existing apex reads ignored the office claim entirely — the supervisory
boundary was decorative, though no borrower leaked. Every read now passes
through ApexService.scopeDistricts(tenant), one choke point rather than eight
guards. All six offices sum exactly to the national 263 units / 8 banks /
342 alerts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The book held one snapshot per loan, so there was no history at all and
nothing a model could learn from. Two routes now fill it, and the
difference between them is stated rather than implied: generate
synthetic monthly snapshots for the prototype, or backfill month-end
extracts from the lender's core, which is the real path — every core can
answer "what did the book look like at each month end last year",
because it is the query behind their regulatory returns.
LoanRecord.source is stamped at every write site: SEEDED,
SEEDED_HISTORY, UPLOAD, or the connector profile code. Coverage reports
three buckets — synthetic, real, and unknown provenance — and never
folds unknown into real, because an unlabelled row is not evidence of a
lender extract. With no real extract on the book it reports
trainable:false and says why in words. Generation never overwrites an
existing row, so a real extract survives a regeneration untouched, and
purge removes only what was generated.
The generator's arithmetic had to be got right and was not at first.
Real DPD steps: pay and cure to zero, or miss and roll forward one
instalment period. The first version drew the day-of-month slack per
month, so consecutive months differed by anything from 1 to 59 days — a
shape no loan book produces, and one that would have taught a model a
transition matrix nothing real generates. The slack is now fixed per
delinquency episode, since a due date does not move between two missed
payments: an account 5 days down that misses again is 35 days down, not
12. For the run reaching the anchor it is derived from the anchor's own
DPD, so the series joins today's number exactly rather than near it.
DpdWalk is shared with the simulated adapters, which previously returned
today's DPD for every requested date — a backfill through them would
have been a flat line, which is exactly the shape that makes a training
label useless and which a chart would have hidden.
Verified against a running instance: 3,256 snapshots across 142 loans,
24 monthly extract dates spanning 2024-08 to 2026-07, 100% of
enterprises now carrying history, every generated series rolling by
exactly 30 — the only non-30 steps being joins to pre-existing rows the
generator correctly refused to overwrite. A 6-month Finacle backfill
pulled 264 rows and staged them for review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loan book is the training label and the one input no public feed
supplies. Integrating with a lender's core is not a technical unknown —
it is an onboarding process per bank, with a schema agreed in a room.
What can be built before that room is the seam, and this is it.
CoreBankingConnector fetches and returns rows; it never writes. Mapped
rows go to LoanBookEntryService.stage — the same validator an uploaded
spreadsheet goes through, landing in the same manual_entry_batch with
the same preview. A parallel validator for the automated path would be
the obvious shape and would eventually enforce the no-PII rule
differently.
FileDropConnector is real: it reads the newest matching file from a
directory a bank delivers to, and leaves it there so a failed run is
repeatable. Finacle, FLEXCUBE, BaNCS and an in-house REST LMS are
simulated, each emitting the same portfolio in its own column dialect —
DPD_DAYS, OVERDUE_DAYS, DAYS_OVERDUE, daysPastDue — which is what makes
the case for a mapping layer rather than asserting it.
The mapping is {canonical: vendor}, keyed by canonical name, so an
unmapped vendor column has no route through and is dropped at the
boundary. CUST_NAME and MOBILE_NO stay out structurally rather than by
anyone remembering. Auto-commit is off on every profile: a sync stages
and stops, keeping the rule that nothing is written until someone has
seen what will be. simulated is stamped per run, not read from the
profile, so switching a profile live cannot make old runs look real.
Fixes a real defect this surfaced: FORBIDDEN_COLUMNS matched on the
exact normalised name, so CUST_NAME — what Finacle actually calls it —
was missed, and a directly uploaded bank file would have kept the
borrower's name in manual_entry_line.raw_row. Matching now splits
separators and camelCase, catching borrowerMobile too, while
loan_account_ref still splits clean.
Verified against a running instance: Finacle 44/44 accepted with 4
columns dropped, the file drop 2 accepted and 1 UNKNOWN_ENTERPRISE
rejected from a real CSV, and both CUST_NAME and borrowerMobile refused
at mapping save.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
