feat(loan-history): DPD history the model can be trained against, with provenance
The book held one snapshot per loan, so there was no history at all and nothing a model could learn from. Two routes now fill it, and the difference between them is stated rather than implied: generate synthetic monthly snapshots for the prototype, or backfill month-end extracts from the lender's core, which is the real path — every core can answer "what did the book look like at each month end last year", because it is the query behind their regulatory returns. LoanRecord.source is stamped at every write site: SEEDED, SEEDED_HISTORY, UPLOAD, or the connector profile code. Coverage reports three buckets — synthetic, real, and unknown provenance — and never folds unknown into real, because an unlabelled row is not evidence of a lender extract. With no real extract on the book it reports trainable:false and says why in words. Generation never overwrites an existing row, so a real extract survives a regeneration untouched, and purge removes only what was generated. The generator's arithmetic had to be got right and was not at first. Real DPD steps: pay and cure to zero, or miss and roll forward one instalment period. The first version drew the day-of-month slack per month, so consecutive months differed by anything from 1 to 59 days — a shape no loan book produces, and one that would have taught a model a transition matrix nothing real generates. The slack is now fixed per delinquency episode, since a due date does not move between two missed payments: an account 5 days down that misses again is 35 days down, not 12. For the run reaching the anchor it is derived from the anchor's own DPD, so the series joins today's number exactly rather than near it. DpdWalk is shared with the simulated adapters, which previously returned today's DPD for every requested date — a backfill through them would have been a flat line, which is exactly the shape that makes a training label useless and which a chart would have hidden. Verified against a running instance: 3,256 snapshots across 142 loans, 24 monthly extract dates spanning 2024-08 to 2026-07, 100% of enterprises now carrying history, every generated series rolling by exactly 30 — the only non-30 steps being joins to pre-existing rows the generator correctly refused to overwrite. A 6-month Finacle backfill pulled 264 rows and staged them for review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+18
@@ -191,6 +191,7 @@ The loan book is the one input no public feed supplies, and no two lenders expos
|
||||
| `PUT` | `/admin/connectors/profiles/{code}` | mapping and settings; **an invalid mapping is refused here**, not at the next nightly run |
|
||||
| `POST` | `/admin/connectors/profiles/{code}/test` | reaches the source and shows the columns. **Stages nothing.** |
|
||||
| `POST` | `/admin/connectors/profiles/{code}/sync` | pulls and stages; writes only if the profile has `autoCommit` |
|
||||
| `POST` | `/admin/connectors/profiles/{code}/backfill?from=&to=` | one extract per month across a range, staged as **one** batch |
|
||||
| `GET` | `/admin/connectors/runs` | every attempt, successful or not |
|
||||
| `GET` | `/admin/connectors/runs/{runId}/preview` | the staged batch behind a run |
|
||||
|
||||
@@ -200,6 +201,23 @@ The loan book is the one input no public feed supplies, and no two lenders expos
|
||||
|
||||
**Auto-commit is off by default.** A sync stages and stops, preserving the manual path's rule that nothing is written until someone has seen what will be. A bank that wants the nightly feed unattended turns it on per profile, knowingly.
|
||||
|
||||
### DPD history — the training label
|
||||
|
||||
| Method | Path | Notes |
|
||||
|---|---|---|
|
||||
| `GET` | `/admin/loan-history/coverage` | three buckets and a verdict in words |
|
||||
| `GET` | `/admin/loan-history/series/{accountRef}` | one account's snapshots, oldest first |
|
||||
| `POST` | `/admin/loan-history/generate?months=24` | synthetic monthly snapshots, stamped `SEEDED_HISTORY` |
|
||||
| `DELETE` | `/admin/loan-history/generated` | removes generated rows only |
|
||||
|
||||
**Every `loan_record` carries a `source`:** `SEEDED`, `SEEDED_HISTORY`, `UPLOAD`, or the connector profile code. Coverage reports **three** buckets — synthetic, real, and unknown provenance — and never folds unknown into real. A row whose origin is unlabelled is not evidence of a lender extract, and treating it as one is exactly the mistake the column exists to prevent.
|
||||
|
||||
**Generated history is shaped like a loan book, not like noise.** Real DPD steps: an account either pays and cures to zero, or misses and rolls forward one instalment period. So the miss/pay sequence is drawn first and DPD derived from it (`DpdWalk`), and the day-of-month slack is **fixed per delinquency episode** — a due date does not move between two missed payments, so an account 5 days down that misses again is 35 days down, not 12. Drawing the slack per month made consecutive months differ by 1 to 59 days, which is not a roll-forward and would have taught a model a transition matrix no book produces.
|
||||
|
||||
The series is walked **backwards from the state already on the book**, and the trailing episode's slack is derived from the anchor's own DPD, so the generated history joins today's number exactly. Generation never overwrites an existing row: a real extract survives a regeneration untouched.
|
||||
|
||||
The simulated adapters walk the same `DpdWalk`, so a connector backfill returns a real-shaped series rather than today's DPD repeated for every month.
|
||||
|
||||
A failed run is still a run. Nothing to read is `SUCCESS` with a message, not `FAILED` — a bank that has not delivered tonight's file is a normal state, and conflating it with a broken connector trains everyone to ignore the status.
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user