feat(loan-history): DPD history the model can be trained against, with provenance

The book held one snapshot per loan, so there was no history at all and
nothing a model could learn from. Two routes now fill it, and the
difference between them is stated rather than implied: generate
synthetic monthly snapshots for the prototype, or backfill month-end
extracts from the lender's core, which is the real path — every core can
answer "what did the book look like at each month end last year",
because it is the query behind their regulatory returns.

LoanRecord.source is stamped at every write site: SEEDED,
SEEDED_HISTORY, UPLOAD, or the connector profile code. Coverage reports
three buckets — synthetic, real, and unknown provenance — and never
folds unknown into real, because an unlabelled row is not evidence of a
lender extract. With no real extract on the book it reports
trainable:false and says why in words. Generation never overwrites an
existing row, so a real extract survives a regeneration untouched, and
purge removes only what was generated.

The generator's arithmetic had to be got right and was not at first.
Real DPD steps: pay and cure to zero, or miss and roll forward one
instalment period. The first version drew the day-of-month slack per
month, so consecutive months differed by anything from 1 to 59 days — a
shape no loan book produces, and one that would have taught a model a
transition matrix nothing real generates. The slack is now fixed per
delinquency episode, since a due date does not move between two missed
payments: an account 5 days down that misses again is 35 days down, not
12. For the run reaching the anchor it is derived from the anchor's own
DPD, so the series joins today's number exactly rather than near it.

DpdWalk is shared with the simulated adapters, which previously returned
today's DPD for every requested date — a backfill through them would
have been a flat line, which is exactly the shape that makes a training
label useless and which a chart would have hidden.

Verified against a running instance: 3,256 snapshots across 142 loans,
24 monthly extract dates spanning 2024-08 to 2026-07, 100% of
enterprises now carrying history, every generated series rolling by
exactly 30 — the only non-30 steps being joins to pre-existing rows the
generator correctly refused to overwrite. A 6-month Finacle backfill
pulled 264 rows and staged them for review.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
LincPay
2026-08-07 18:57:56 +05:30
co-authored by Claude Opus 5
parent b79b92fe8d
commit 8efd9fd6ed
12 changed files with 755 additions and 5 deletions
+18
View File
@@ -191,6 +191,7 @@ The loan book is the one input no public feed supplies, and no two lenders expos
| `PUT` | `/admin/connectors/profiles/{code}` | mapping and settings; **an invalid mapping is refused here**, not at the next nightly run |
| `POST` | `/admin/connectors/profiles/{code}/test` | reaches the source and shows the columns. **Stages nothing.** |
| `POST` | `/admin/connectors/profiles/{code}/sync` | pulls and stages; writes only if the profile has `autoCommit` |
| `POST` | `/admin/connectors/profiles/{code}/backfill?from=&to=` | one extract per month across a range, staged as **one** batch |
| `GET` | `/admin/connectors/runs` | every attempt, successful or not |
| `GET` | `/admin/connectors/runs/{runId}/preview` | the staged batch behind a run |
@@ -200,6 +201,23 @@ The loan book is the one input no public feed supplies, and no two lenders expos
**Auto-commit is off by default.** A sync stages and stops, preserving the manual path's rule that nothing is written until someone has seen what will be. A bank that wants the nightly feed unattended turns it on per profile, knowingly.
### DPD history — the training label
| Method | Path | Notes |
|---|---|---|
| `GET` | `/admin/loan-history/coverage` | three buckets and a verdict in words |
| `GET` | `/admin/loan-history/series/{accountRef}` | one account's snapshots, oldest first |
| `POST` | `/admin/loan-history/generate?months=24` | synthetic monthly snapshots, stamped `SEEDED_HISTORY` |
| `DELETE` | `/admin/loan-history/generated` | removes generated rows only |
**Every `loan_record` carries a `source`:** `SEEDED`, `SEEDED_HISTORY`, `UPLOAD`, or the connector profile code. Coverage reports **three** buckets — synthetic, real, and unknown provenance — and never folds unknown into real. A row whose origin is unlabelled is not evidence of a lender extract, and treating it as one is exactly the mistake the column exists to prevent.
**Generated history is shaped like a loan book, not like noise.** Real DPD steps: an account either pays and cures to zero, or misses and rolls forward one instalment period. So the miss/pay sequence is drawn first and DPD derived from it (`DpdWalk`), and the day-of-month slack is **fixed per delinquency episode** — a due date does not move between two missed payments, so an account 5 days down that misses again is 35 days down, not 12. Drawing the slack per month made consecutive months differ by 1 to 59 days, which is not a roll-forward and would have taught a model a transition matrix no book produces.
The series is walked **backwards from the state already on the book**, and the trailing episode's slack is derived from the anchor's own DPD, so the generated history joins today's number exactly. Generation never overwrites an existing row: a real extract survives a regeneration untouched.
The simulated adapters walk the same `DpdWalk`, so a connector backfill returns a real-shaped series rather than today's DPD repeated for every month.
A failed run is still a run. Nothing to read is `SUCCESS` with a message, not `FAILED` — a bank that has not delivered tonight's file is a normal state, and conflating it with a broken connector trains everyone to ignore the status.
---