diff --git a/API-SPEC.md b/API-SPEC.md index 7cf77b6..efb501f 100644 --- a/API-SPEC.md +++ b/API-SPEC.md @@ -191,6 +191,7 @@ The loan book is the one input no public feed supplies, and no two lenders expos | `PUT` | `/admin/connectors/profiles/{code}` | mapping and settings; **an invalid mapping is refused here**, not at the next nightly run | | `POST` | `/admin/connectors/profiles/{code}/test` | reaches the source and shows the columns. **Stages nothing.** | | `POST` | `/admin/connectors/profiles/{code}/sync` | pulls and stages; writes only if the profile has `autoCommit` | +| `POST` | `/admin/connectors/profiles/{code}/backfill?from=&to=` | one extract per month across a range, staged as **one** batch | | `GET` | `/admin/connectors/runs` | every attempt, successful or not | | `GET` | `/admin/connectors/runs/{runId}/preview` | the staged batch behind a run | @@ -200,6 +201,23 @@ The loan book is the one input no public feed supplies, and no two lenders expos **Auto-commit is off by default.** A sync stages and stops, preserving the manual path's rule that nothing is written until someone has seen what will be. A bank that wants the nightly feed unattended turns it on per profile, knowingly. +### DPD history — the training label + +| Method | Path | Notes | +|---|---|---| +| `GET` | `/admin/loan-history/coverage` | three buckets and a verdict in words | +| `GET` | `/admin/loan-history/series/{accountRef}` | one account's snapshots, oldest first | +| `POST` | `/admin/loan-history/generate?months=24` | synthetic monthly snapshots, stamped `SEEDED_HISTORY` | +| `DELETE` | `/admin/loan-history/generated` | removes generated rows only | + +**Every `loan_record` carries a `source`:** `SEEDED`, `SEEDED_HISTORY`, `UPLOAD`, or the connector profile code. Coverage reports **three** buckets — synthetic, real, and unknown provenance — and never folds unknown into real. A row whose origin is unlabelled is not evidence of a lender extract, and treating it as one is exactly the mistake the column exists to prevent. + +**Generated history is shaped like a loan book, not like noise.** Real DPD steps: an account either pays and cures to zero, or misses and rolls forward one instalment period. So the miss/pay sequence is drawn first and DPD derived from it (`DpdWalk`), and the day-of-month slack is **fixed per delinquency episode** — a due date does not move between two missed payments, so an account 5 days down that misses again is 35 days down, not 12. Drawing the slack per month made consecutive months differ by 1 to 59 days, which is not a roll-forward and would have taught a model a transition matrix no book produces. + +The series is walked **backwards from the state already on the book**, and the trailing episode's slack is derived from the anchor's own DPD, so the generated history joins today's number exactly. Generation never overwrites an existing row: a real extract survives a regeneration untouched. + +The simulated adapters walk the same `DpdWalk`, so a connector backfill returns a real-shaped series rather than today's DPD repeated for every month. + A failed run is still a run. Nothing to read is `SUCCESS` with a message, not `FAILED` — a bank that has not delivered tonight's file is a normal state, and conflating it with a broken connector trains everyone to ignore the status. --- diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 0bea7a3..529bd87 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -218,6 +218,7 @@ believes. | Risk engine + rule engine + admin API | ✅ Phase D — see §6.3 | | Console wire-up: rules screen, market panel, freshness | ✅ Phase E — see §6.4 | | Core banking / LMS connector seam | ✅ Phase F — see §6.5 | +| DPD history: generated + connector backfill | ✅ Phase F — see §6.6 | | LightGBM quantile heads + SHAP | ○ R1 milestone | Phases B, C and D each terminate in a table you can inspect, so each is testable @@ -379,6 +380,41 @@ That fix belongs to the manual upload path as much as to the connector. synced by something the bank already runs), scheduled execution (`scheduleCron` is recorded and read by nothing), and incremental watermarks. +### 6.6 DPD history — the training label, and where it comes from + +Package `com.lincfusion.udyamlinc.loanhistory`, plus a fourth tab on +`/officer/integration`. + +The book held **one snapshot per loan**, so there was no history at all and +nothing a model could be trained against. Two routes now fill it, and the +difference between them is stated rather than implied. + +| Route | What it is | +|---|---| +| `LoanHistoryService.generate` | Synthetic monthly snapshots, every row stamped `SEEDED_HISTORY`. Exercises the feature store, the point-in-time joins and the label plumbing before a bank has delivered anything. | +| `ConnectorSyncService.backfill` | One extract per month across a range, mapped and staged as a single batch. This is the real path: every core can answer "what did the book look like at each month end last year". | + +**Provenance is the whole design.** `LoanRecord.source` is stamped at every write +site, and coverage reports **three** buckets — synthetic, real, unknown — never +folding unknown into real. Generation refuses to overwrite an existing row, so a +lender's extract survives a regeneration. + +**The generator's arithmetic had to be got right, and was not at first.** Real DPD +steps: pay and cure to zero, or miss and roll forward one instalment period. The +first version added day-of-month slack **per month**, so consecutive months +differed by anything from 1 to 59 days — a shape no loan book produces, and one +that would have taught a model a nonsense transition matrix. The slack is now +fixed per delinquency episode (a due date does not move between two missed +payments), and for the run reaching the anchor it is derived from the anchor's own +DPD so the series joins today's number exactly. Verified against a running +instance: every generated series now rolls by exactly 30, and the only steps that +are not are the joins to pre-existing rows the generator correctly refused to +touch. + +**What this does not claim.** With no lender extract on the book, coverage reports +`trainable: false` and says so in words. Training on generated history and +reporting the accuracy would be measuring the generator. + --- ## 7. Explicitly not being built yet diff --git a/src/main/java/com/lincfusion/udyamlinc/connector/ConnectorController.java b/src/main/java/com/lincfusion/udyamlinc/connector/ConnectorController.java index 7b9b510..5d341bc 100644 --- a/src/main/java/com/lincfusion/udyamlinc/connector/ConnectorController.java +++ b/src/main/java/com/lincfusion/udyamlinc/connector/ConnectorController.java @@ -185,6 +185,34 @@ public class ConnectorController { return ApiResponse.ok(result.run().getMessage(), body); } + /** + * Pulls one extract per month across a range and stages them as one batch. + * + *

This is how DPD history arrives from a lender that has it. Every core can + * answer "what did the book look like at each month end last year" — it is the + * same query behind their regulatory returns — so the honest way to get the + * training label is to ask once, rather than wait a year for a nightly sync to + * accumulate it. + */ + @PostMapping("/profiles/{code}/backfill") + public ApiResponse> backfill( + @PathVariable String code, + @RequestParam(required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate from, + @RequestParam(required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate to, + Authentication authentication) { + + ConnectorSyncService.SyncResult result = + syncService.backfill(code, from, to, "MANUAL", username(authentication)); + + Map body = new LinkedHashMap<>(); + body.put("run", result.run()); + body.put("droppedColumns", result.droppedColumns()); + body.put("extractLabel", result.extractLabel()); + if (result.preview() != null) body.put("preview", describe(result.preview())); + + return ApiResponse.ok(result.run().getMessage(), body); + } + @GetMapping("/runs") public ApiResponse> runs(@RequestParam(required = false) String profileCode, @RequestParam(defaultValue = "30") int limit) { diff --git a/src/main/java/com/lincfusion/udyamlinc/connector/ConnectorSyncService.java b/src/main/java/com/lincfusion/udyamlinc/connector/ConnectorSyncService.java index b1db0ce..2e9a74a 100644 --- a/src/main/java/com/lincfusion/udyamlinc/connector/ConnectorSyncService.java +++ b/src/main/java/com/lincfusion/udyamlinc/connector/ConnectorSyncService.java @@ -177,6 +177,132 @@ public class ConnectorSyncService { return (int) Duration.between(startedAt, LocalDateTime.now()).toMillis(); } + // ── historical backfill ───────────────────────────────── + + /** + * Pulls one extract per month across a range and stages them as a single batch. + * + *

Why this exists. DPD history is the training label, and a nightly + * sync started today produces its first useful month in a month's time. Every + * core can answer "what did the book look like at each month end last year" — + * it is the same query the bank already runs for its regulatory returns — so + * the honest way to get history is to ask for it once, not to wait a year or + * to invent it. + * + *

One batch rather than one per month, because each row already carries its + * own {@code as_of_date} and the point-in-time key is (loan, extract date). + * That makes a two-year backfill reviewable in a single preview, which is the + * only way anyone would actually review it. + */ + public SyncResult backfill(String profileCode, LocalDate from, LocalDate to, + String trigger, String username) { + ConnectorProfile profile = profileRepository.findByCode(profileCode) + .orElseThrow(() -> new IllegalArgumentException("No connector profile '" + profileCode + "'")); + CoreBankingConnector connector = catalog.find(profile.getVendor()) + .orElseThrow(() -> new IllegalStateException("No adapter for vendor " + profile.getVendor())); + + LocalDate end = to == null ? LocalDate.now().withDayOfMonth(1).minusDays(1) : to; + LocalDate start = from == null ? end.minusMonths(23) : from; + if (start.isAfter(end)) { + throw new IllegalArgumentException("The range starts after it ends: " + start + " to " + end); + } + + LocalDateTime startedAt = LocalDateTime.now(); + ConnectorSyncRun run = ConnectorSyncRun.builder() + .profileCode(profile.getCode()) + .vendor(profile.getVendor()) + .institutionCode(profile.getInstitutionCode()) + .trigger(trigger) + .triggeredBy(username) + .simulated(connector.simulated()) + .asOfDate(end) + .startedAt(startedAt) + .committed(false) + .build(); + + try { + Map mapping = fieldMapper.read(profile); + List headers = List.of(); + List> allRows = new ArrayList<>(); + Set dropped = new LinkedHashSet<>(); + int months = 0; + + // Month ends, oldest first, capped so a mistyped range cannot ask a + // core for a decade. + LocalDate cursor = monthEnd(start); + while (!cursor.isAfter(end) && months < 60) { + CoreBankingConnector.Extract extract = connector.fetch(profile, cursor); + if (!extract.rows().isEmpty()) { + FieldMapper.Mapped mapped = fieldMapper.apply(extract, mapping); + if (!mapped.missingRequired().isEmpty()) { + throw new IllegalStateException("Mapping is incomplete — " + + String.join(", ", mapped.missingRequired()) + " not mapped"); + } + if (headers.isEmpty()) headers = mapped.headers(); + allRows.addAll(mapped.rows()); + dropped.addAll(mapped.droppedColumns()); + } + months++; + cursor = monthEnd(cursor.plusMonths(1)); + } + + if (allRows.isEmpty()) { + run.setStatus("SUCCESS"); + run.setRowsFetched(0); + run.setRowsAccepted(0); + run.setRowsRejected(0); + run.setDurationMs(elapsedMs(startedAt)); + run.setMessage("Nothing to read across " + months + " months from " + start + " to " + end); + runRepository.save(run); + stampProfile(profile, run); + return new SyncResult(run, null, List.of(), null); + } + + String filename = "%s · backfill %s to %s (%d months)".formatted( + profile.getCode(), start, end, months); + ManualEntryPreview preview = loanBookEntryService.stage( + new TabularParser.Sheet(headers, allRows), filename, username, profile.getCode()); + + boolean committed = false; + if (Boolean.TRUE.equals(profile.getAutoCommit()) && !preview.accepted().isEmpty()) { + preview = loanBookEntryService.commit(preview.batch().getId(), username); + committed = true; + } + + run.setRowsFetched(allRows.size()); + run.setRowsAccepted(preview.accepted().size()); + run.setRowsRejected(preview.rejected().size()); + run.setRowsOverwriting(preview.overwriting().size()); + run.setColumnsDropped(dropped.size()); + run.setBatchId(preview.batch().getId()); + run.setCommitted(committed); + run.setDurationMs(elapsedMs(startedAt)); + run.setStatus(preview.rejected().isEmpty() ? "SUCCESS" : "PARTIAL"); + run.setMessage("Backfill %d months (%s to %s) · %d accepted · %d rejected%s%s".formatted( + months, start, end, preview.accepted().size(), preview.rejected().size(), + committed ? " · written" : " · staged, awaiting commit", + connector.simulated() ? " · SIMULATED extract, not a live core" : "")); + runRepository.save(run); + stampProfile(profile, run); + + return new SyncResult(run, preview, List.copyOf(dropped), filename); + + } catch (Exception e) { + log.warn("Connector backfill {} failed: {}", profileCode, e.toString()); + run.setStatus("FAILED"); + run.setRowsFetched(0); + run.setDurationMs(elapsedMs(startedAt)); + run.setMessage(e.getMessage() == null ? e.toString() : e.getMessage()); + runRepository.save(run); + stampProfile(profile, run); + return new SyncResult(run, null, List.of(), null); + } + } + + private LocalDate monthEnd(LocalDate date) { + return date.withDayOfMonth(1).plusMonths(1).minusDays(1); + } + // ── connection test ───────────────────────────────────── /** diff --git a/src/main/java/com/lincfusion/udyamlinc/connector/SimulatedCoreConnector.java b/src/main/java/com/lincfusion/udyamlinc/connector/SimulatedCoreConnector.java index b9507b2..fa9a3f9 100644 --- a/src/main/java/com/lincfusion/udyamlinc/connector/SimulatedCoreConnector.java +++ b/src/main/java/com/lincfusion/udyamlinc/connector/SimulatedCoreConnector.java @@ -5,11 +5,13 @@ import com.lincfusion.udyamlinc.enterprise.EnterpriseRepository; import com.lincfusion.udyamlinc.enterprise.LoanRecord; import com.lincfusion.udyamlinc.enterprise.LoanRecordRepository; import com.lincfusion.udyamlinc.institution.InstitutionRepository; +import com.lincfusion.udyamlinc.loanhistory.DpdWalk; import lombok.RequiredArgsConstructor; import java.math.BigDecimal; import java.math.RoundingMode; import java.time.LocalDate; +import java.time.temporal.ChronoUnit; import java.util.*; /** @@ -160,14 +162,39 @@ public class SimulatedCoreConnector implements CoreBankingConnector { asOf.plusDays(12), asOf.minusDays(Math.max(1, dpd)), dpd, asOf); } + /** + * The account as it stood at {@code asOf}. + * + *

Walked back from the anchor snapshot with {@link DpdWalk}, the same + * arithmetic {@code LoanHistoryService} uses. Returning today's DPD for every + * requested date would make a historical backfill a flat line — which is + * exactly the shape that would make the training label useless, and which a + * chart would hide until someone looked at the numbers. + */ private Map rowFrom(Enterprise enterprise, LoanRecord loan, LocalDate asOf) { - int base = loan.getDpd() == null ? 0 : loan.getDpd(); - int dpd = Math.max(0, base + drift(loan.getAccountRef(), enterprise.getBand()) - 2); + int anchorDpd = loan.getDpd() == null ? 0 : loan.getDpd(); + int monthsBack = (int) ChronoUnit.MONTHS.between( + asOf.withDayOfMonth(1), loan.getAsOfDate().withDayOfMonth(1)); + + int dpd; + if (monthsBack <= 0) { + // At or after the anchor: today's state, with a little movement so a + // nightly sync is not identical to yesterday's. + dpd = Math.max(0, anchorDpd + drift(loan.getAccountRef(), enterprise.getBand()) - 2); + } else { + int window = Math.max(2, monthsBack + 2); + boolean[] missed = DpdWalk.missSequence( + loan.getAccountRef() + ":history", enterprise.getBand(), anchorDpd, window); + dpd = DpdWalk.dpdAt(missed, window - 1 - monthsBack, anchorDpd); + } return emit(enterprise, loan.getAccountRef(), loan.getProduct(), loan.getSanctionedAmount(), loan.getOutstandingAmount(), loan.getInstalmentAmount(), - loan.getNextDueDate() == null ? asOf.plusDays(10) : loan.getNextDueDate(), - loan.getLastPaidDate() == null ? asOf.minusDays(Math.max(1, dpd)) : loan.getLastPaidDate(), + // Both dates are derived from the walked DPD rather than copied off + // the anchor: a row claiming an August due date at a March extract + // date would contradict its own DPD. + asOf.plusDays(loan.getNextDueDate() == null ? 10 : loan.getNextDueDate().getDayOfMonth()), + asOf.minusDays(Math.max(1, dpd)), dpd, asOf); } diff --git a/src/main/java/com/lincfusion/udyamlinc/enterprise/LoanRecord.java b/src/main/java/com/lincfusion/udyamlinc/enterprise/LoanRecord.java index 6718f89..4e05e4f 100644 --- a/src/main/java/com/lincfusion/udyamlinc/enterprise/LoanRecord.java +++ b/src/main/java/com/lincfusion/udyamlinc/enterprise/LoanRecord.java @@ -73,9 +73,29 @@ public class LoanRecord { private LocalDateTime observedAt; + /** + * Where this row came from: {@code SEEDED}, {@code SEEDED_HISTORY}, + * {@code UPLOAD}, or the connector profile code that pulled it. + * + *

Not decoration. DPD history is the training label, and a + * generated one is a demonstration of shape, not evidence. Once a real + * lender's extract lands beside it, the only thing distinguishing the two is + * this column — so it is stamped at every write site and it is what + * {@code isSynthetic} reads. A model trained without filtering on it would be + * learning the seeder's own arithmetic. + */ + @Column(length = 40) + private String source; + @PrePersist void onCreate() { if (observedAt == null) observedAt = LocalDateTime.now(); if (closed == null) closed = Boolean.FALSE; + if (source == null) source = "UNKNOWN"; + } + + /** True for anything this system generated rather than received from a lender. */ + public boolean isSynthetic() { + return source != null && source.startsWith("SEEDED"); } } diff --git a/src/main/java/com/lincfusion/udyamlinc/enterprise/LoanRecordRepository.java b/src/main/java/com/lincfusion/udyamlinc/enterprise/LoanRecordRepository.java index 9ef47db..a08ffb7 100644 --- a/src/main/java/com/lincfusion/udyamlinc/enterprise/LoanRecordRepository.java +++ b/src/main/java/com/lincfusion/udyamlinc/enterprise/LoanRecordRepository.java @@ -1,6 +1,7 @@ package com.lincfusion.udyamlinc.enterprise; import org.springframework.data.jpa.repository.JpaRepository; +import org.springframework.data.jpa.repository.Query; import java.time.LocalDate; import java.util.List; @@ -19,4 +20,47 @@ public interface LoanRecordRepository extends JpaRepository { Optional findByAccountRefAndAsOfDate(String accountRef, LocalDate asOfDate); long countByEnterpriseId(String enterpriseId); + + // ── history ───────────────────────────────────────────── + // DPD history is the training label, so "how many snapshots does this loan + // have" and "how many of them did we invent" are first-class questions. + + List findByEnterpriseIdOrderByAsOfDateAsc(String enterpriseId); + + List findByAccountRefOrderByAsOfDateAsc(String accountRef); + + long countBySourceStartingWith(String prefix); + + /** + * Rows written before provenance was stamped. + * + *

Counted separately and never folded into either side. A row whose origin + * is unknown is not evidence of a real extract, and quietly treating it as one + * is the exact failure this column was added to prevent. + */ + @Query("select count(l) from LoanRecord l where l.source is null or l.source = 'UNKNOWN'") + long countUnknownProvenance(); + + void deleteBySourceStartingWith(String prefix); + + /** Distinct extract dates on the book — the shape of the history, cheaply. */ + @Query("select count(distinct l.asOfDate) from LoanRecord l") + long countDistinctAsOfDates(); + + /** + * Enterprises with more than one snapshot — the ones a model could learn from. + * + *

A correlated subquery rather than a derived table: JPQL has no + * {@code FROM (subselect)}, and the obvious phrasing silently fails to parse + * only at first use. + */ + @Query("select count(distinct l.enterpriseId) from LoanRecord l where l.enterpriseId in " + + "(select l2.enterpriseId from LoanRecord l2 group by l2.enterpriseId having count(l2) > 1)") + long countEnterprisesWithHistory(); + + @Query("select min(l.asOfDate) from LoanRecord l") + LocalDate earliestAsOfDate(); + + @Query("select max(l.asOfDate) from LoanRecord l") + LocalDate latestAsOfDate(); } diff --git a/src/main/java/com/lincfusion/udyamlinc/loanhistory/DpdWalk.java b/src/main/java/com/lincfusion/udyamlinc/loanhistory/DpdWalk.java new file mode 100644 index 0000000..5ccf671 --- /dev/null +++ b/src/main/java/com/lincfusion/udyamlinc/loanhistory/DpdWalk.java @@ -0,0 +1,115 @@ +package com.lincfusion.udyamlinc.loanhistory; + +import java.util.Random; + +/** + * How a delinquent account actually moves, as arithmetic. + * + *

Real DPD steps; it does not drift. Each month an account either pays + * and cures to zero, or misses and rolls forward roughly one instalment period. + * A genuine series is therefore a staircase with cliffs, and the bucket + * transitions between snapshots are a roll-rate matrix. A generator producing + * smooth noise would look plausible on a chart and teach a model to expect + * something no loan book contains — so the miss/pay sequence is drawn first and + * DPD is derived from it, never the other way round. + * + *

Shared by {@link LoanHistoryService}, which writes a synthetic history + * directly, and by the simulated core-banking adapters, which have to answer + * "what did this account look like at that month end" when a backfill walks + * backwards. Both must agree: an account whose history is generated one way and + * fetched the other would otherwise contradict itself in the same table. + * + *

Pure and deterministic — seeded from the account reference, so the same + * account always produces the same past. A preview that changed each time it was + * asked would be worthless. + */ +public final class DpdWalk { + + /** One missed instalment moves an account this far along. */ + public static final int ROLL_DAYS = 30; + + private DpdWalk() { + } + + /** + * Which months an account missed, ending in the run that produces its current + * DPD. + * + *

The trailing block is forced: an account sitting at 67 days past due + * missed its last three instalments, and no other history is consistent with + * that. The month immediately before the block is forced to a payment — + * without it the run would be longer and today's DPD higher than the book + * says. Everything earlier is drawn against a band-driven miss rate, which is + * what gives the series earlier delinquency episodes to learn from. + * + * @param months window length; index {@code months - 1} is the anchor month + */ + public static boolean[] missSequence(String key, String band, int currentDpd, int months) { + int window = Math.max(2, months); + boolean[] missed = new boolean[window]; + + int trailing = currentDpd <= 0 ? 0 : (int) Math.ceil(currentDpd / (double) ROLL_DAYS); + trailing = Math.min(trailing, window - 1); + for (int i = 0; i < trailing; i++) { + missed[window - 1 - i] = true; + } + + int cureAt = window - 1 - trailing; + if (cureAt >= 0) missed[cureAt] = false; + + double missRate = switch (band == null ? "STANDARD" : band) { + case "RED" -> 0.42; + case "WATCH" -> 0.22; + default -> 0.06; + }; + Random random = new Random((key == null ? "" : key).hashCode()); + for (int i = 0; i < cureAt; i++) { + missed[i] = random.nextDouble() < missRate; + } + return missed; + } + + /** + * DPD at one month: how far into an unbroken run of misses that month sits. + * + *

A few days of slack are added so buckets are not all exact multiples of + * thirty, which no real book is. The slack is fixed per episode, not per + * month. It represents where in the month the instalment fell due, and a + * due date does not move between two missed payments — so an account 5 days + * down that misses again is 35 days down, not 12. Drawing it per month made + * consecutive months differ by anything from 1 to 59 days, which is not a + * roll-forward and would have taught a model a transition matrix no loan book + * produces. + * + *

For the run that reaches the anchor month, the slack is derived from the + * anchor's own DPD, so the generated series joins the book's current state + * exactly rather than near it. + */ + public static int dpdAt(boolean[] missed, int index, int anchorDpd) { + if (index < 0 || index >= missed.length || !missed[index]) return 0; + + int start = index; + while (start > 0 && missed[start - 1]) start--; + + int end = index; + while (end < missed.length - 1 && missed[end + 1]) end++; + + int run = index - start + 1; + boolean reachesAnchor = end == missed.length - 1; + + int offset = reachesAnchor + ? Math.floorMod(anchorDpd - 1, ROLL_DAYS) + : Math.floorMod(start * 17 + 7, ROLL_DAYS); + + return (run - 1) * ROLL_DAYS + 1 + offset; + } + + /** 0 | 1-30 | 31-60 | 61-90 | 90+ */ + public static String bucketOf(int dpd) { + if (dpd <= 0) return "0"; + if (dpd <= 30) return "1-30"; + if (dpd <= 60) return "31-60"; + if (dpd <= 90) return "61-90"; + return "90+"; + } +} diff --git a/src/main/java/com/lincfusion/udyamlinc/loanhistory/LoanHistoryController.java b/src/main/java/com/lincfusion/udyamlinc/loanhistory/LoanHistoryController.java new file mode 100644 index 0000000..022eb4b --- /dev/null +++ b/src/main/java/com/lincfusion/udyamlinc/loanhistory/LoanHistoryController.java @@ -0,0 +1,62 @@ +package com.lincfusion.udyamlinc.loanhistory; + +import com.lincfusion.udyamlinc.common.ApiResponse; +import com.lincfusion.udyamlinc.enterprise.LoanRecord; +import lombok.RequiredArgsConstructor; +import org.springframework.web.bind.annotation.*; + +import java.util.List; + +/** + * The loan book seen as a training set. + * + *

Under {@code /admin}, so the existing split holds: any bank user may read + * the coverage, only a role carrying CONFIG:MANAGE may generate or purge. Reading + * how much of the DPD history is invented is exactly the thing a field officer + * should be able to check before believing a score. + */ +@RestController +@RequestMapping("/admin/loan-history") +@RequiredArgsConstructor +public class LoanHistoryController { + + private final LoanHistoryService loanHistoryService; + + /** + * How much history exists, how much of it is generated, and whether the book + * is trainable — stated as a verdict rather than left as a number to interpret. + */ + @GetMapping("/coverage") + public ApiResponse coverage() { + return ApiResponse.ok(loanHistoryService.coverage()); + } + + /** One account's DPD series, oldest first. */ + @GetMapping("/series/{accountRef}") + public ApiResponse> series(@PathVariable String accountRef) { + return ApiResponse.ok(loanHistoryService.series(accountRef)); + } + + /** + * Backfills monthly snapshots behind the extract already on the book. + * + *

Never overwrites an existing row, so a real uploaded or connector-fetched + * extract survives a regeneration untouched. + */ + @PostMapping("/generate") + public ApiResponse generate( + @RequestParam(defaultValue = "24") int months) { + LoanHistoryService.GenerateResult result = loanHistoryService.generate(months); + return ApiResponse.ok( + "%d snapshots written across %d loans — all stamped %s".formatted( + result.snapshotsWritten(), result.loans(), LoanHistoryService.SOURCE), + result); + } + + /** Removes generated snapshots only. Real extracts are left alone. */ + @DeleteMapping("/generated") + public ApiResponse purge() { + long removed = loanHistoryService.purgeGenerated(); + return ApiResponse.ok(removed + " generated snapshots removed; real extracts untouched", removed); + } +} diff --git a/src/main/java/com/lincfusion/udyamlinc/loanhistory/LoanHistoryService.java b/src/main/java/com/lincfusion/udyamlinc/loanhistory/LoanHistoryService.java new file mode 100644 index 0000000..1589d71 --- /dev/null +++ b/src/main/java/com/lincfusion/udyamlinc/loanhistory/LoanHistoryService.java @@ -0,0 +1,270 @@ +package com.lincfusion.udyamlinc.loanhistory; + +import com.lincfusion.udyamlinc.enterprise.Enterprise; +import com.lincfusion.udyamlinc.enterprise.EnterpriseRepository; +import com.lincfusion.udyamlinc.enterprise.LoanRecord; +import com.lincfusion.udyamlinc.enterprise.LoanRecordRepository; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.math.BigDecimal; +import java.math.RoundingMode; +import java.time.LocalDate; +import java.util.*; + +/** + * Generates the DPD history the model needs, until a lender supplies a real one. + * + *

What this is honestly claiming. The book currently holds one snapshot + * per loan, so there is no history at all and nothing a model could be trained + * against. This fills that in — and every row it writes is stamped + * {@code SEEDED_HISTORY}, which is what makes it a demonstration of shape rather + * than evidence. A real extract arriving through a connector lands beside it + * carrying the profile code instead, and the two never have to be told apart by + * eye. Training on this and reporting the accuracy as a result would be + * self-deception; the point is that the pipeline, the point-in-time joins and the + * label plumbing can be exercised before a bank has delivered anything. + * + *

Why the arithmetic is shaped the way it is. Real DPD does not drift + * smoothly — it steps. An account either pays and cures to zero, or misses and + * rolls forward roughly one instalment period, so a genuine series looks like a + * staircase with cliffs, not a random walk. A generator that produced smooth + * noise would teach a model to expect something no loan book contains, and the + * bucket transitions (0 → 1-30 → 31-60) would never look like a real roll-rate + * matrix. So the miss/pay sequence is generated first and DPD is derived from it. + * + *

The series is built backwards from the state already on the book, so + * the last month reproduces the DPD the console is showing today rather than + * contradicting it. Every draw is seeded from the account reference, so + * regenerating gives the same history — a preview that changed each time it was + * asked would be worthless. + */ +@Service +@RequiredArgsConstructor +@Slf4j +public class LoanHistoryService { + + /** Stamped on every generated row. {@code LoanRecord.isSynthetic} reads this. */ + public static final String SOURCE = "SEEDED_HISTORY"; + + private final LoanRecordRepository loanRepository; + private final EnterpriseRepository enterpriseRepository; + + public record GenerateResult(int enterprises, int loans, int snapshotsWritten, int monthsRequested, + LocalDate from, LocalDate to, String note) { + } + + /** + * What the book looks like as a training set, which is the only thing that + * matters here. + * + *

Three buckets, not two. {@code unknownProvenance} is rows written before + * the {@code source} column existed, and it is never folded into + * {@code realRecords} — an unlabelled row is not evidence of a real extract, + * and treating it as one is precisely the mistake provenance was added to stop. + */ + public record Coverage(long totalRecords, long syntheticRecords, long realRecords, + long unknownProvenance, + long enterprisesTotal, long enterprisesWithAnyLoan, long enterprisesWithHistory, + long distinctAsOfDates, LocalDate earliest, LocalDate latest, + double historyCoveragePct, boolean trainable, String verdict) { + } + + // ── generation ────────────────────────────────────────── + + /** + * Writes {@code months} monthly snapshots per open loan, ending at the extract + * date already on the book. + * + *

Existing rows are left alone rather than overwritten — a real uploaded or + * connector-fetched extract must never be replaced by a generated one, and + * making that the default behaviour is safer than making it a flag someone has + * to remember to set. + */ + @Transactional + public GenerateResult generate(int months) { + int requested = Math.max(2, Math.min(months, 60)); + List enterprises = enterpriseRepository.findAll(); + + int loans = 0; + int written = 0; + LocalDate earliest = null; + LocalDate latest = null; + + for (Enterprise enterprise : enterprises) { + List existing = loanRepository.findByEnterpriseIdOrderByAsOfDateAsc( + enterprise.getEnterpriseId()); + if (existing.isEmpty()) continue; + + // One series per account, anchored on that account's latest snapshot. + Map anchors = new LinkedHashMap<>(); + for (LoanRecord record : existing) { + anchors.merge(record.getAccountRef(), record, + (a, b) -> a.getAsOfDate().isAfter(b.getAsOfDate()) ? a : b); + } + + for (LoanRecord anchor : anchors.values()) { + loans++; + List series = seriesFor(enterprise, anchor, requested); + for (LoanRecord snapshot : series) { + if (loanRepository.findByAccountRefAndAsOfDate( + snapshot.getAccountRef(), snapshot.getAsOfDate()).isPresent()) { + continue; + } + loanRepository.save(snapshot); + written++; + if (earliest == null || snapshot.getAsOfDate().isBefore(earliest)) { + earliest = snapshot.getAsOfDate(); + } + if (latest == null || snapshot.getAsOfDate().isAfter(latest)) { + latest = snapshot.getAsOfDate(); + } + } + } + } + + log.info("Generated {} synthetic DPD snapshots across {} loans ({} months requested)", + written, loans, requested); + + return new GenerateResult(enterprises.size(), loans, written, requested, earliest, latest, + written == 0 + ? "Nothing written — every snapshot in that window already exists" + : "Every row is stamped " + SOURCE + " and is excluded from anything " + + "reported as a real-data result"); + } + + /** + * The months before {@code anchor}, as that account would actually have moved. + * + *

The miss/pay sequence is drawn first and DPD derived from it, so the + * result steps and cures like a loan book instead of wandering like noise. + */ + private List seriesFor(Enterprise enterprise, LoanRecord anchor, int months) { + int currentDpd = anchor.getDpd() == null ? 0 : anchor.getDpd(); + boolean[] missed = DpdWalk.missSequence( + anchor.getAccountRef() + ":history", enterprise.getBand(), currentDpd, months); + + BigDecimal outstanding = anchor.getOutstandingAmount() == null + ? BigDecimal.ZERO : anchor.getOutstandingAmount(); + BigDecimal instalment = anchor.getInstalmentAmount() == null + ? BigDecimal.ZERO : anchor.getInstalmentAmount(); + BigDecimal sanctioned = anchor.getSanctionedAmount(); + + List out = new ArrayList<>(); + for (int i = 0; i < months - 1; i++) { + // i counts back from the anchor: i = 0 is the month before it. + int monthsBack = months - 1 - i; + LocalDate asOf = anchor.getAsOfDate().minusMonths(monthsBack); + int index = i; + + int dpd = DpdWalk.dpdAt(missed, index, currentDpd); + // Going back in time the balance is higher, because the instalments + // since then have not been paid yet. Only paid months amortise. + BigDecimal owedThen = outstanding.add( + instalment.multiply(BigDecimal.valueOf(paidBetween(missed, index, months - 1) * 0.72))); + if (sanctioned != null && owedThen.compareTo(sanctioned) > 0) owedThen = sanctioned; + + out.add(LoanRecord.builder() + .accountRef(anchor.getAccountRef()) + .enterpriseId(anchor.getEnterpriseId()) + .product(anchor.getProduct()) + .sanctionedAmount(sanctioned) + .outstandingAmount(owedThen.setScale(2, RoundingMode.HALF_UP)) + .instalmentAmount(instalment) + .nextDueDate(asOf.plusDays(dueDayOffset(anchor))) + .lastPaidDate(lastPaidBefore(missed, index, asOf)) + .dpd(dpd) + .bucket(DpdWalk.bucketOf(dpd)) + .closed(false) + .asOfDate(asOf) + .source(SOURCE) + .build()); + } + return out; + } + + private int paidBetween(boolean[] missed, int from, int to) { + int paid = 0; + for (int i = from; i <= to && i < missed.length; i++) { + if (!missed[i]) paid++; + } + return paid; + } + + private LocalDate lastPaidBefore(boolean[] missed, int index, LocalDate asOf) { + for (int i = index; i >= 0; i--) { + if (!missed[i]) return asOf.minusMonths(index - (long) i).withDayOfMonth( + Math.min(28, 4 + (Math.abs(i * 11) % 22))); + } + return asOf.minusMonths(index + 1L); + } + + private int dueDayOffset(LoanRecord anchor) { + return anchor.getNextDueDate() == null ? 12 : Math.max(1, anchor.getNextDueDate().getDayOfMonth()); + } + + // ── coverage ──────────────────────────────────────────── + + /** Reads the book as a training set and says plainly whether it is one yet. */ + public Coverage coverage() { + long total = loanRepository.count(); + long synthetic = loanRepository.countBySourceStartingWith("SEEDED"); + long enterprises = enterpriseRepository.count(); + long withHistory = loanRepository.countEnterprisesWithHistory(); + + long withAnyLoan = enterpriseRepository.findAll().stream() + .filter(e -> loanRepository.countByEnterpriseId(e.getEnterpriseId()) > 0) + .count(); + + double pct = enterprises == 0 ? 0 : (withHistory * 100.0) / enterprises; + long unknown = loanRepository.countUnknownProvenance(); + long real = Math.max(0, total - synthetic - unknown); + + boolean trainable = real > 0 && withHistory > 0; + String verdict; + if (real == 0 && unknown == 0) { + verdict = "Not trainable. Every snapshot on the book is generated — the pipeline can be " + + "exercised end to end, but any accuracy measured against this would be " + + "measuring the generator, not the model."; + } else if (real == 0) { + verdict = ("Not trainable. %d rows predate provenance stamping, so their origin is " + + "unknown and cannot be counted as a real extract. Purge the generated " + + "history and re-run a connector sync for a clean split.").formatted(unknown); + } else { + verdict = ("%d snapshots came from a real extract — train on those. The %d generated " + + "rows exist to keep the pipeline exercisable and must be filtered out of any " + + "reported result.").formatted(real, synthetic); + } + + return new Coverage(total, synthetic, real, unknown, enterprises, withAnyLoan, withHistory, + loanRepository.countDistinctAsOfDates(), + loanRepository.earliestAsOfDate(), loanRepository.latestAsOfDate(), + Math.round(pct * 10) / 10.0, trainable, verdict); + } + + /** One loan's series, for the console to draw. */ + public List series(String accountRef) { + return loanRepository.findByAccountRefOrderByAsOfDateAsc(accountRef); + } + + // ── removal ───────────────────────────────────────────── + + /** + * Drops every generated snapshot, leaving real ones untouched. + * + *

Exists because the moment a lender delivers a genuine extract, the + * generated history stops being scaffolding and becomes contamination. + * Deleting by {@code source} prefix is only safe because the column is stamped + * at every write site. + */ + @Transactional + public long purgeGenerated() { + long before = loanRepository.countBySourceStartingWith(SOURCE); + loanRepository.deleteBySourceStartingWith(SOURCE); + log.info("Purged {} generated DPD snapshots", before); + return before; + } + +} diff --git a/src/main/java/com/lincfusion/udyamlinc/manualentry/LoanBookEntryService.java b/src/main/java/com/lincfusion/udyamlinc/manualentry/LoanBookEntryService.java index bc02468..d225dd1 100644 --- a/src/main/java/com/lincfusion/udyamlinc/manualentry/LoanBookEntryService.java +++ b/src/main/java/com/lincfusion/udyamlinc/manualentry/LoanBookEntryService.java @@ -307,8 +307,11 @@ public class LoanBookEntryService { loan.setDpd(line.getModalPrice() == null ? null : line.getModalPrice().intValue()); loan.setBucket(bucketOf(loan.getDpd(), TabularParser.get(row, "bucket"))); loan.setClosed(parseBoolean(TabularParser.get(row, "closed"))); - // As with market signals, when we learned it is ours to stamp. + // As with market signals, when we learned it is ours to stamp — and so + // is where it came from. A row that arrived through a connector must + // stay distinguishable from one this system generated for itself. loan.setObservedAt(observedAt); + loan.setSource(batch.getSource() == null ? SOURCE_UPLOAD : batch.getSource()); loanRepository.save(loan); } diff --git a/src/main/java/com/lincfusion/udyamlinc/seed/UdyamDataSeeder.java b/src/main/java/com/lincfusion/udyamlinc/seed/UdyamDataSeeder.java index ad8bcd3..96aa9e5 100644 --- a/src/main/java/com/lincfusion/udyamlinc/seed/UdyamDataSeeder.java +++ b/src/main/java/com/lincfusion/udyamlinc/seed/UdyamDataSeeder.java @@ -485,6 +485,7 @@ public class UdyamDataSeeder implements CommandLineRunner { .bucket(bucketOf(dpd)) .closed(false) .asOfDate(asOf) + .source("SEEDED") .build()); }