Author SHA1 Message Date
LincPayandClaude Opus 5 7d192dfe9a feat(features): backfill the feature store so a model can be trained on it
feature_value held three as-of dates. A point-in-time training set needs
history, so three paths now generate it:

  POST /admin/features/backfill?from=&to=   loops computeAll(monthEnd) across
                                            a date range — 23 months, ~6 min
  POST /admin/bootstrap/ledger-history      fills the gap before the earliest
                                            entry only, stamped SEEDED_HISTORY
                                            so generated rows stay
                                            distinguishable from recorded ones
  ClimateIngestionService                    honours the lookbackWeeks the
                                            controller was already passing and
                                            had been ignoring, so the
                                            Open-Meteo archive can be pulled

Together these took the store to 21 monthly as-of dates over 263 enterprises,
which is the 5,523-row dataset the published metrics were measured on.

Documentation catches up with the code: API-SPEC lists the new endpoints,
ARCHITECTURE §7 records what was deliberately not built and why.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 13:44:54 +05:30
LincPayandClaude Opus 5 1c42376a92 feat(app): complete the borrower PWA contract, and drop vendor core names
The offline promise was only half kept. Ledger entries were queued on the
phone and marked PENDING, and the client flipped them to SYNCED without
sending anything — so an owner could record a sale with no signal, which was
the point, but it never reached the lender and the officer's screen never saw
it. POST /app/ledger/sync now has a caller; it dedupes on the clientId the
phone minted, so a retry after a half-failed sync writes nothing twice.

Adds GET /app/reasons: the borrower's own scored reason codes, newest run
only. A borrower told their limit is under review is entitled to know which
measurement drove it, and the app had been generating sentences from a demo
sector table rather than reading the ones the model produced. Both surfaces
now render the identical string.

PUT /app/consent was already here but nothing called it; the profile toggle
was local state. Consent is the artefact that makes reading a borrower's
inflows lawful, so it has to live on the server and be revocable from the
owner's own device.

Separately: the connector layer no longer names third-party core banking
products. Finacle, FLEXCUBE and BaNCS are replaced by CBSNexora, LincPay's
own core, with an in-place migration in ConnectorProfileSeeder.renameLegacy()
that also rewrites existing sync runs and manual entry batch labels.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 13:44:41 +05:30
LincPayandClaude Opus 5 e7c93e29ea fix(officer): scope reasons and forecasts to the run that produced the score
Both repositories were queried by enterprise alone, so every reason code and
every forecast point ever written came back. A run-287 model score was being
rendered beside run-145 rule sentences, and one enterprise's six-month chart
drew 102 points from seventeen different runs.

Now read through findByEnterpriseIdAndRunId..., resolved from the newest run.
The same unscoped query existed in EnterpriseAppService and is fixed there
too.

Also adds GET /officer/app-enrolled: which borrowers on this caller's book
have a phone login. Not every borrower is enrolled and that is the design —
the platform scores a book from core banking and UPI whether or not the owner
installs anything — so the console needs to know rather than offering to open
an app that does not exist. Scoped by intersecting with the caller's own book,
so it cannot be used to discover that an id exists on another bank's.

EnterpriseRow gains firstBreachHorizon, firstBreachMonth and topDriver so the
dashboard can rank by lead time instead of by composite score.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 13:44:22 +05:30
LincPayandClaude Opus 5 6b4aec468d feat(risk): bind the scoring engine to the ML service, with honest attribution
Adds MlClient and the two model-backed strategies behind the existing
ScoringStrategy / ForecastStrategy seams, so switching engines stays a config
value rather than a code change.

MlClient never throws. Every call returns Optional.empty() on any failure —
timeout, connection refused, malformed body — and ModelScoringStrategy falls
back to the rule engine for that enterprise. A nightly run must complete, and
a demo must survive the Python process dying mid-presentation.

The important part is what the fallback records. A run configured MODEL used
to stamp every score MODEL even when the ML service was down and rules had
actually produced the number, which made the provenance column a lie exactly
when it mattered. ScoreOutcome now carries an `engine` field and the fallback
returns .attributedTo("RULE_FALLBACK"). Verified by killing the service
mid-run: run 285 MODEL, run 286 RULE_FALLBACK, 260 of 263 scores differ.

ModelMetric and its read-only controller expose the measured backtest at
GET /admin/model/metrics. syntheticLabelShare is stored on the same row as
every metric so accuracy cannot be quoted without the caveat travelling with
it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 13:44:07 +05:30
LincPay bfbb933fe1 build: package an executable war, and document runtime UI config 2026-08-15 17:01:48 +05:30
LincPay 351da2f67d docs(deploy): restore the dump rather than seeding fresh, with the measured difference 2026-08-15 16:40:43 +05:30
LincPayandClaude Opus 5 bbf226506b feat(simulation): a real crop calendar for the simulated price peaks
Replaces a hash of the commodity code with a table of the week each commodity's
price actually peaks. The peak is the lean season, not the harvest: wheat comes
in during March and April and is cheapest then, and dearest in November when
last season's stock has run down and the new crop is still in the ground. Wheat
is therefore week 48, not week 14, and the same inversion runs down the list —
rabi peaks in autumn, kharif in early summer, and winter vegetables peak through
the monsoon when they cannot be grown at all.

This is not cosmetic. margin_squeeze reads a sector's OUTPUT price against its
INPUT price, and a squeeze is only real when the two peak at different times of
year. A hashed peak scattered them arbitrarily, so a poultry unit could show
feed and egg peaking in the same week and a dairy could show them six months
apart, both by accident. The seasonal relationship between an input and an
output is precisely what the derivation measures, so it has to come from a
calendar rather than from a hash.

Approximate and national, and both caveats matter: harvest windows shift by
weeks across fourteen districts and six zones, and this carries one figure each.
Better than a hash, not a district crop calendar. A commodity with no entry logs
a warning and falls back to the hash rather than failing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 16:17:14 +05:30
LincPayandClaude Opus 5 81e66a9b57 feat(admin): one-call demo bootstrap, and price simulation for the whole market
A fresh deployment came up with a populated portfolio, an empty feeds screen and
no alerts, because everything downstream of the seeder was an admin call whose
order lived only in a runbook. POST /admin/bootstrap/demo now runs the six steps
in dependency order and reports each one.

The price simulator widens from the four commodities with no feed to every
active basket pair, so price_z_score gets its three-year seasonal mean and TW-05
can fire with no DATAGOV_API_KEY present at all. Levels come from a per-crop
table of ordinary mandi rates and the shape from the commodity group, because
the group already encodes perishability - a vegetable moves on a weather event
within days and a cereal does not, and flattening that would erase the one
distinction the z-score exists to read.

A pair holding any non-simulated row is skipped unless forced. That ordering is
the safety property: run this after a real ingestion and it fills the gaps
around the real series rather than over them.

The orchestrator is deliberately not transactional. The seeder holds one
transaction across its whole run, and when the machine slept mid-run nothing
committed until morning while the console confidently served pre-seed state. A
bootstrap generating three years of prices across every pair is long enough to
hit exactly that, so each step commits alone and a failure in one leaves the
others standing.

Still not trainable, and coverage still says so. The seeder derives DPD from the
band and the UPI simulator derives inflow from the same band, so a model fitted
here recovers the seeder rule and reports fine metrics for doing it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 15:56:41 +05:30
LincPay e28ac52803 docs(deploy): the demo data sequence a fresh database needs 2026-08-15 15:47:14 +05:30
LincPay 88702f2b43 docs: deployment notes for nginx, Apache httpd and Tomcat 2026-08-15 15:33:19 +05:30
LincPay 053a1cee05 fix(config): make CORS origins overridable by environment 2026-08-15 15:30:33 +05:30
37 changed files with 2073 additions and 122 deletions
+1 -1
View File
@@ -195,7 +195,7 @@ The loan book is the one input no public feed supplies, and no two lenders expos
| `GET` | `/admin/connectors/runs` | every attempt, successful or not |
| `GET` | `/admin/connectors/runs/{runId}/preview` | the staged batch behind a run |
**One adapter is real.** `FILE_DROP` reads an actual extract from a directory a bank delivers to (`udyam.connector.file-drop.dir`, default `./data/cbs-drop`), newest file matching the profile's glob, left where it was found so a failed run is repeatable. `FINACLE`, `FLEXCUBE`, `BANCS` and `CUSTOM_API` are **simulated**: they generate an extract in each core's own column dialect so the mapping layer is exercisable end to end. `simulated` is stamped per run, not read from the profile — switching a profile to a live adapter must not retroactively make its old runs look real.
**One adapter is real.** `FILE_DROP` reads an actual extract from a directory a bank delivers to (`udyam.connector.file-drop.dir`, default `./data/cbs-drop`), newest file matching the profile's glob, left where it was found so a failed run is repeatable. `CBSNEXORA` is **simulated**: no CBSNexora instance feeds this deployment, so it generates an extract in that platform’s own field names to exercise the mapping layer end to end. No third-party core banking product is named anywhere — a lender on a different core is absorbed by the per-profile field mapping. `simulated` is stamped per run, not read from the profile — switching a profile to a live adapter must not retroactively make its old runs look real.
**The mapping is `{canonical_column: vendor_column}`, and the direction matters.** Keyed by the canonical name, so a vendor column nobody mapped has no route through: unmapped columns are **dropped at the boundary** and counted on the run. That is how `CUST_NAME` and `MOBILE_NO` stay out of a system that promised not to hold them — structurally, rather than by anyone remembering.
+2 -2
View File
@@ -355,7 +355,7 @@ ahead of it is the shape of the seam, and that is what this is.
|---|---|
| `CoreBankingConnector` | Fetches and returns rows. Never writes — an adapter that could write would be an adapter that could bypass the no-PII rule. |
| `FileDropConnector` | **The one that is real.** Reads the newest matching file from a drop directory a bank delivers to. Leaves the file where it found it, so a failed run is repeatable. |
| `SimulatedCoreConnector` × 4 | Finacle, FLEXCUBE, BaNCS and an in-house REST LMS, each emitting the same portfolio in its own column dialect. Rows derived from real enterprises and loan records, drift seeded from the account ref so an extract is reproducible. |
| `SimulatedCoreConnector` | CBSNexora — LincPay's own core banking platform — emitting the real portfolio in its own field names. Rows derived from real enterprises and loan records, drift seeded from the account ref so an extract is reproducible. **No third-party core banking product is named anywhere in this system;** where a lender runs someone else's core, the per-profile field mapping absorbs their column names. |
| `FieldMapper` | `{canonical_column: vendor_column}`. Validated before it is stored, not at the next nightly run. |
| `ConnectorSyncService` | Fetch → map → **`LoanBookEntryService.stage`** → run record. |
| `ConnectorProfile`, `ConnectorSyncRun` | Per-bank configuration and per-attempt audit. |
@@ -371,7 +371,7 @@ profile, so switching a profile to a live adapter cannot retroactively make its
old runs look real.
**One real defect this surfaced.** `FORBIDDEN_COLUMNS` was matched by exact
normalised name, so `CUST_NAME` — what Finacle actually calls it — was not
normalised name, so `CUST_NAME` — a column name real extracts do carry — was not
caught, and a directly uploaded bank file would have kept the borrower's name in
`manual_entry_line.raw_row`. Matching now also splits separators and camelCase.
That fix belongs to the manual upload path as much as to the connector.
+385
View File
@@ -0,0 +1,385 @@
# Deploying UdyamLinc
Two artifacts, one host.
| Artifact | Built from | Output |
|---|---|---|
| API | `UdyamLinc-Backend` | `target/udyamlinc-backend.war` (~80 MB) |
| UI | `UdyamLinc-UI` | `dist/` (static files, ~880 kB) |
**The war is also executable.** `spring-boot-maven-plugin` repackages it with the
provided-scope Tomcat under `WEB-INF/lib-provided`, so one artifact both runs on its own
and deploys into a container:
```bash
java -jar udyamlinc-backend.war # embeds Tomcat 11 — recommended
# or drop it into a servlet container — see §2c
```
### Pointing the UI at the API without rebuilding
The bundle reads its API base from **`config.js`**, served unbundled from the static
root. Deploy the backend first, then edit one line on the server — no rebuild, no
redeploy of the UI:
```js
// config.js, at the web root next to index.html
window.__UDYAM_CONFIG__ = {
apiBaseUrl: '', // empty = same-origin /api/v1/udyam (the default)
}
```
Set it only when the API is on another host:
```js
apiBaseUrl: 'https://udyamlinc-api.lincpay.tech/api/v1/udyam'
```
Resolution order is `config.js` → the build-time `VITE_UDYAM_API_BASE_URL` →
`/api/v1/udyam`. An empty string means "unset", so the shipped file needs no edit for a
same-origin deployment.
Two things that bite:
- **`config.js` must never be cached.** The nginx and httpd configs below send
`no-store` for it. Cached, it pins the old URL for everyone who has already loaded
the app, and nothing on the server side shows it.
- **A cross-origin value needs `UDYAM_CORS_ALLOWED_ORIGINS`** set to the UI's origin, or
the browser blocks every call while the API itself looks perfectly healthy.
The UI bundle is otherwise built with a **relative** default. It therefore contains no hostname, serves correctly from any domain, and is
same-origin with the API. That last point is the one worth keeping: same-origin means
CORS never applies, and CORS is the single largest source of "worked locally, broken on
the server" in a split deployment.
The only requirement on the web server is that it does two things:
1. serve `dist/` with an SPA fallback — unknown paths return `index.html`, because
React Router owns `/officer/**`, `/nabard/**` and `/app/**`, not the file system;
2. pass `/api/**` through to the Spring Boot process on port 8082.
Any of nginx, Apache httpd or Tomcat can do both. Pick the section that matches your box.
---
## 1. The API (identical in all three cases)
The jar embeds Tomcat 11. It does not need an application server installed, and it does
not go in `webapps/`.
```bash
java -jar udyamlinc-backend.jar
```
Configuration is entirely by environment variable — nothing needs editing inside the jar:
```bash
export UDYAM_DB_URL=jdbc:postgresql://localhost:5432/UdyamLinc_DEMO
export UDYAM_DB_USERNAME=postgres
export UDYAM_DB_PASSWORD=<the real one>
# Set this. The default is committed to the repository, so it is public, and a
# signing key anyone can read signs anyone's token.
export UDYAM_JWT_SECRET=<a long random string>
```
`spring.jpa.hibernate.ddl-auto=update` builds the schema on first boot, so the database
only has to exist and be empty:
```sql
CREATE DATABASE "UdyamLinc_DEMO";
```
### As a systemd service
```ini
[Unit]
Description=UdyamLinc API
After=network.target postgresql.service
[Service]
User=udyamlinc
WorkingDirectory=/opt/udyamlinc
Environment=UDYAM_DB_URL=jdbc:postgresql://localhost:5432/UdyamLinc_DEMO
Environment=UDYAM_DB_USERNAME=postgres
Environment=UDYAM_DB_PASSWORD=change-me
Environment=UDYAM_JWT_SECRET=change-me
ExecStart=/usr/bin/java -jar /opt/udyamlinc/udyamlinc-backend.jar
SuccessExitStatus=143
Restart=on-failure
[Install]
WantedBy=multi-user.target
```
### First boot takes a while, and the failure mode is silent
`UdyamDataSeeder` populates 263 enterprises across 8 partner banks and 16 branches, and
it runs as **one transaction**. Nothing is visible in the database until it commits.
This has bitten us once already: the machine slept mid-seed, the transaction stayed open
overnight, and every figure read from the console during that window was the pre-seed
state — reported confidently, with no warning anywhere. **Watch for the seeder's final
log line before trusting any number on screen**, and check its timestamp rather than
just its text.
Verify the API is up:
```bash
curl -s localhost:8082/api/v1/udyam/auth/regional-offices
```
That endpoint is deliberately public — the login page reads it before anyone has signed
in — so it is a clean liveness check.
---
## 2a. Front door: nginx
`UdyamLinc-UI/nginx.conf` is ready to use as-is. Copy `dist/` to the document root and
the conf to `/etc/nginx/conf.d/`. It already carries the `/api/` proxy block, the SPA
fallback, gzip, and asset caching.
Adjust `listen 5173` to `80`/`443` for a real deployment.
---
## 2b. Front door: Apache httpd
Enable the modules first — on a stock install `proxy_http` and `rewrite` are usually off:
```bash
a2enmod proxy proxy_http rewrite headers
```
Virtual host:
```apache
<VirtualHost *:80>
ServerName udyamlinc.lincpay.tech
DocumentRoot /var/www/udyamlinc
# ── API ─────────────────────────────────────────────
# The backend's context-path is already /api/v1/udyam, so the prefix passes
# through unchanged rather than being rewritten.
ProxyPreserveHost On
ProxyPass /api/ http://127.0.0.1:8082/api/
ProxyPassReverse /api/ http://127.0.0.1:8082/api/
# Ingestion and the three-year backfill are long-running admin calls. The
# default proxy timeout cuts them off mid-run and reports a gateway error
# over a job that is still working.
ProxyTimeout 300
<Directory /var/www/udyamlinc>
Require all granted
Options -Indexes
# ── SPA fallback ────────────────────────────────
# Anything that is not a real file, not a real directory, and not the
# API is handed to index.html for React Router to resolve. Without the
# !^/api/ guard this rule swallows API calls and returns HTML to axios,
# which surfaces as a JSON parse error rather than a routing problem.
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/api/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.html [L]
</Directory>
# Fingerprinted assets are immutable; index.html must never be cached or a
# deploy cannot reach browsers that already have it.
<LocationMatch "^/assets/">
Header set Cache-Control "public, max-age=2592000, immutable"
</LocationMatch>
<LocationMatch "^/(index\.html|config\.js)$">
Header set Cache-Control "no-cache, no-store, must-revalidate"
</LocationMatch>
</VirtualHost>
```
---
## 2c. Front door: Tomcat
Two things are worth separating here, because "Tomcat" can mean either.
### Serving the UI from Tomcat — fine
Copy `dist/` into `$CATALINA_HOME/webapps/ROOT/` and add the SPA fallback via the
rewrite valve. In `conf/Catalina/localhost/ROOT.xml` (or the `<Context>` element):
```xml
<Context>
<Valve className="org.apache.catalina.valves.rewrite.RewriteValve" />
</Context>
```
with `webapps/ROOT/WEB-INF/rewrite.config`:
```
RewriteCond %{REQUEST_URI} !^/api/
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-f
RewriteRule ^(.*)$ /index.html
```
The API still runs as its own `java -jar` process on 8082, and Tomcat proxies to it —
or you put httpd/nginx in front of both.
### Deploying the API as a WAR into Tomcat
Supported — the build produces a war and `UdyamLincApplication` extends
`SpringBootServletInitializer`. Two conditions, and both are hard:
**1. Tomcat 11 or newer.** Spring Boot 4.0.1 is built on Jakarta EE 11 / Servlet 6.1.
Tomcat 9 is `javax.*` and cannot load a `jakarta.*` application at all; Tomcat 10 is
short of the servlet level. Check before deploying:
```bash
$CATALINA_HOME/bin/version.sh # or version.bat
```
**2. Deploy it as `api#v1#udyam.war`.** Under a container `server.servlet.context-path`
is ignored — the file name decides the path, and Tomcat reads `#` as a separator. So:
```bash
cp udyamlinc-backend.war $CATALINA_HOME/webapps/api#v1#udyam.war
```
serves `/api/v1/udyam`, which is what the UI's relative API base expects. Deployed under
any other name every endpoint moves and the UI gets 404s that look like a broken build.
Environment variables are read the same way; set them in `setenv.sh` or the service
definition rather than per-shell.
**Still, running the war directly is the lower-risk option** — it embeds Tomcat 11, so
neither condition can bite, and the same file works both ways.
---
## Checklist before showing anyone
- [ ] `curl localhost:8082/api/v1/udyam/auth/regional-offices` returns JSON
- [ ] The seeder's final log line has appeared, with a plausible timestamp
- [ ] The site loads, and a hard refresh on a deep link such as `/nabard/geography`
returns the app rather than a 404 — this is the SPA fallback, and it is the
single most common thing to get wrong
- [ ] The browser network tab shows `/api/v1/udyam/...` calls returning 200, not HTML
- [ ] `UDYAM_JWT_SECRET` is not the committed default
## Restoring the demo database — do this rather than seeding fresh
Verified on 2026-08-15: a freshly seeded database and the working demo database are
**not the same demo**. The seeder writes the plan, which is 129 enterprises; the working
database carries 263, along with three years of real Agmarknet prices that only a
`DATAGOV_API_KEY` and a long backfill can produce.
| | Restored dump | Fresh boot + bootstrap |
|---|---|---|
| Enterprises | 263 | 129 |
| RED | 50 | 14 |
| Trip-wires firing | **6** | **5** — TW-05 silent |
| Open alerts | 344 | 70 |
| Real Agmarknet rows | 8,292 | 0 |
TW-05 reads a price anomaly against a three-year seasonal mean. Generated series are
calibrated to sit near the real data's typical z of 0.44, so they legitimately do not
breach a 1.5 threshold — the trip-wire is not broken on a fresh database, it simply has
nothing anomalous to find.
```bash
createdb -U postgres UdyamLinc_DEMO
pg_restore -U postgres -d UdyamLinc_DEMO --no-owner --no-privileges udyamlinc_demo.dump
```
Then start the jar. The seeder is idempotent and will leave the restored book alone.
`--no-owner --no-privileges` matters when the demo server's Postgres role is not
`postgres`; without them the restore fails on every `ALTER ... OWNER TO` line.
Verify:
```sql
select count(*) from enterprise; -- expect 263
select source, count(*) from market_signal group by source; -- expect an AGMARKNET row
```
The sequence below is the fallback for a server that has no dump to restore.
## Populating a fresh database — the demo sequence
The seeder writes the portfolio, the ledger and the forecasts. It does **not** write
loan history, UPI inflows or the no-feed price series: those are admin calls, so a fresh
database comes up without them and several screens look thinner than they should.
**One call does all of it.** After the seeder's final log line:
```bash
BASE=http://localhost:8082/api/v1/udyam
TOKEN=<a supervisor token from POST $BASE/auth/login>
curl -X POST -H "Authorization: Bearer $TOKEN" "$BASE/admin/bootstrap/demo?months=24"
```
That runs six steps in dependency order and reports each one:
| Step | What it writes |
|---|---|
| `loan-history` | 24 monthly DPD snapshots per loan, `SEEDED_HISTORY` |
| `upi-inflows` | 26 weeks per enterprise, `SIMULATED` — the Account Aggregator stand-in |
| `prices-no-feed` | milk, egg, fodder, sugarcane — the four with no API anywhere |
| `prices-all` | three years for **every remaining basket pair**, so `price_z_score` has a seasonal mean and TW-05 can fire without a data.gov.in key |
| `features` | the 15 derivations across the book |
| `scoring` | stress scores, bands, trip-wires, alerts |
It is **idempotent** — re-running is safe, and each step commits on its own, so a
failure in one leaves the others intact and the response names which failed.
The individual endpoints remain if you want to run a stage on its own:
`/admin/loan-history/generate`, `/admin/upi/simulate`,
`/admin/ingestion/simulate/no-feed`, `/admin/ingestion/simulate/all`,
`/admin/features/compute`, `/admin/scoring/run`.
Verify with the coverage endpoints, both of which report synthetic and real separately:
```bash
curl -H "Authorization: Bearer $TOKEN" "$BASE/admin/loan-history/coverage"
curl -H "Authorization: Bearer $TOKEN" "$BASE/admin/upi/coverage"
```
### All of it is labelled, and none of it can overwrite a fact
Every row written above carries its provenance in the table — `SEEDED_HISTORY` on loan
records, `SIMULATED` on UPI and price rows — and the simulators refuse to overwrite a
non-simulated row even when asked to. Real data can land beside this at any time and
stays distinguishable forever. The console surfaces the distinction rather than leaving
it to a slide.
Two consequences to state out loud rather than discover later:
- **`loan-history/coverage` reports `trainable: false`** while no real extract exists,
and that is correct. The seeder derives DPD from the risk band and the UPI simulator
derives inflow drift from the same band, so a model trained on this would learn the
seeder's own rule and validate beautifully while knowing nothing. The synthetic data
is here to exercise the pipeline and fill the console, not to train.
- **No external dependency at all.** The bootstrap needs no `DATAGOV_API_KEY`, no
Account Aggregator consent and no bank connection: every screen fills from generated
data. If a key *is* present, run the real ingestion first and the price simulator will
skip any pair that already holds real rows, filling the gaps around the real series
rather than across it.
## What is deliberately off
`udyam.ingestion.enabled` is `false`, so a demo boot never calls a government endpoint,
and `DATAGOV_API_KEY` is unset. The portfolio is seeded, which is enough for every
screen. To refresh from live sources, set the key and run, in order:
```
POST /admin/ingestion/run
POST /admin/features/compute
POST /admin/scoring/run
```
Space bulk ingestion runs a couple of minutes apart — data.gov.in meters by quota per
window, so a run started too soon after another returns 429 on every pair at once and
records the whole ingestion as failed with zero rows.
+23 -1
View File
@@ -14,7 +14,18 @@
<groupId>com.lincfusion</groupId>
<artifactId>udyamlinc-backend</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<!--
war, but still executable. spring-boot-maven-plugin repackages a war with
the provided-scope container under WEB-INF/lib-provided, so the one
artifact both runs with `java -jar` and deploys into a servlet container.
Nothing is given up by choosing war here.
If deploying into Tomcat rather than running it directly, that container
must be Tomcat 11+: Spring Boot 4 is Jakarta EE 11 / Servlet 6.1, and
Tomcat 9 (javax.*) cannot load it. Deploy the file as api#v1#udyam.war so
the container path matches the context path this app expects.
-->
<packaging>war</packaging>
<name>udyamlinc-backend</name>
<description>UdyamLinc — predictive cash-flow intelligence and early warning for rural microenterprises (NABARD Hackathon)</description>
@@ -31,6 +42,17 @@
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!--
provided, so a servlet container supplies its own Tomcat instead of
the app carrying a second one into it. The repackaged war still holds
these under WEB-INF/lib-provided, so `java -jar` keeps working.
-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>
<!-- Security -->
<dependency>
<groupId>org.springframework.boot</groupId>
@@ -2,6 +2,8 @@ package com.lincfusion.udyamlinc;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;
/**
* UdyamLinc — predictive cash-flow intelligence and early warning for rural
@@ -13,9 +15,36 @@ import org.springframework.boot.autoconfigure.SpringBootApplication;
* ingestion pipeline are separate components and are not implemented here.
*/
@SpringBootApplication
public class UdyamLincApplication {
public class UdyamLincApplication extends SpringBootServletInitializer {
public static void main(String[] args) {
SpringApplication.run(UdyamLincApplication.class, args);
}
/**
* Entry point when a servlet container starts this instead of {@code main}.
*
* <p>The artifact is packaged as a war so it can be deployed to Tomcat, but
* {@code spring-boot-maven-plugin} still repackages it as executable — the
* provided-scope container lands in {@code WEB-INF/lib-provided}, which the
* launcher puts on the classpath. So {@code java -jar udyamlinc-backend.war}
* and dropping the same file into {@code webapps/} are both supported, from
* one build.
*
* <p><b>Tomcat 11 or newer.</b> Spring Boot 4 is built on Jakarta EE 11 and
* Servlet 6.1. Tomcat 9 is {@code javax.*} and cannot load this at all;
* Tomcat 10 is short of the servlet level. Running the war directly embeds
* Tomcat 11 and sidesteps the question entirely, which is why that remains
* the recommendation.
*
* <p><b>The context path moves.</b> {@code server.servlet.context-path} is
* ignored under a container — the deployed file name decides it. Tomcat reads
* {@code #} as a path separator, so the war must be deployed as
* {@code api#v1#udyam.war} to serve {@code /api/v1/udyam}. Deployed under any
* other name every endpoint moves and the UI's relative API base misses.
*/
@Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
return builder.sources(UdyamLincApplication.class);
}
}
@@ -0,0 +1,65 @@
package com.lincfusion.udyamlinc.admin;
import com.lincfusion.udyamlinc.common.ApiResponse;
import com.lincfusion.udyamlinc.seed.LedgerBackfillService;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/**
* One call to make a fresh deployment demonstrable.
*
* <p>Supervisor-only under the existing {@code /admin/**} write rule in
* SecurityConfig — this generates across the whole book and is not something a
* field officer should be able to trigger.
*/
@RestController
@RequestMapping("/admin/bootstrap")
@RequiredArgsConstructor
public class DemoBootstrapController {
private final DemoBootstrapService bootstrapService;
private final LedgerBackfillService ledgerBackfillService;
/**
* Generate loan history, UPI inflows and price series, then compute features
* and score the book.
*
* <p>Idempotent, and safe to re-run: the simulators upsert on a key that
* includes the source, and skip any pair a real feed has already written to.
*
* @param months months of DPD history per loan, default 24
* @param force regenerate price series even where real data exists. Leave
* off unless you are rebuilding a demo database on purpose —
* the default is what protects a real ingested series.
*/
@PostMapping("/demo")
public ApiResponse<DemoBootstrapService.Result> demo(
@RequestParam(defaultValue = "24") int months,
@RequestParam(defaultValue = "false") boolean force) {
DemoBootstrapService.Result result = bootstrapService.bootstrap(months, force);
return ApiResponse.ok(result.message(), result);
}
/**
* Extend the demo ledger backwards so features can be computed at a past
* as-of date.
*
* <p>Separate from {@code /demo} because it is the one bootstrap step that
* <b>invents</b> rather than derives: a ledger has no archive to recover, so
* reaching further back means generating. Every row is stamped
* {@code SEEDED_HISTORY} and can only be written into the gap before the
* earliest entry an enterprise already has.
*
* @param days how far back the ledger should reach, default 760 (~25 months)
*/
@PostMapping("/ledger-history")
public ApiResponse<LedgerBackfillService.Result> ledgerHistory(
@RequestParam(defaultValue = "760") int days) {
LedgerBackfillService.Result result = ledgerBackfillService.backfill(days);
return ApiResponse.ok(result.message(), result);
}
}
@@ -0,0 +1,140 @@
package com.lincfusion.udyamlinc.admin;
import com.lincfusion.udyamlinc.enterprise.UpiSimulationService;
import com.lincfusion.udyamlinc.features.FeatureEngineeringService;
import com.lincfusion.udyamlinc.ingestion.NoFeedPriceSimulationService;
import com.lincfusion.udyamlinc.loanhistory.LoanHistoryService;
import com.lincfusion.udyamlinc.risk.RiskEngineService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.time.LocalDate;
import java.util.ArrayList;
import java.util.List;
/**
* Brings a freshly seeded database up to a demonstrable state without any
* external dependency.
*
* <p>The seeder writes the portfolio, the ledger and the institutions. Everything
* downstream of that — loan history, UPI inflows, price series, features, scores —
* has always been an admin call, which is correct for a pilot and wrong for a demo
* server: a fresh deployment came up with a populated portfolio, an empty feeds
* screen and no alerts, and the sequence to fix it lived only in a runbook.
*
* <p>Every row this produces is generated, and every row says so in its own
* source column — {@code SEEDED_HISTORY} on loan records, {@code SIMULATED} on
* UPI and price rows. The simulators refuse to overwrite a non-generated row, so
* this can be run against a database holding real data and will fill the gaps
* around it rather than across it.
*
* <p><b>What this does not give you is a trainable dataset.</b> The seeder derives
* DPD from the risk band and the UPI simulator derives inflow drift from the same
* band, so label and features share a cause we invented; a model fitted to this
* would recover the seeder's own rule and report excellent metrics for doing it.
* {@code /admin/loan-history/coverage} reports {@code trainable: false} for that
* reason and should keep saying so until a real lender extract exists.
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class DemoBootstrapService {
private final LoanHistoryService loanHistoryService;
private final UpiSimulationService upiSimulationService;
private final NoFeedPriceSimulationService priceSimulationService;
private final FeatureEngineeringService featureEngineeringService;
private final RiskEngineService riskEngine;
public record Step(String name, boolean ok, String detail) {
}
public record Result(List<Step> steps, boolean ok, String message) {
}
/**
* Runs the whole sequence in dependency order.
*
* <p><b>Deliberately not {@code @Transactional}.</b> Each step commits on its
* own, for a reason this project has already paid for once: the seeder holds a
* single transaction across its whole run, and when the machine slept mid-run
* nothing committed until the next morning — every figure read from the console
* in between was pre-seed state, reported confidently, with no warning
* anywhere. A bootstrap that generates three years of prices across every
* basket pair is long enough to hit exactly that. Committing per step also
* means a failure in scoring does not discard the price series that succeeded.
*
* <p>The cost is that a partial failure leaves partial state. That is the
* better trade here — every step is idempotent, so the fix is to run it again.
*
* @param months months of DPD history per loan
* @param force regenerate price series even where real data exists
*/
public Result bootstrap(int months, boolean force) {
List<Step> steps = new ArrayList<>();
LocalDate today = LocalDate.now();
// Loan history and UPI first: both are inputs to the feature derivations,
// and computing features before them would write a snapshot full of nulls
// that the scoring run would then faithfully score as unknown.
steps.add(run("loan-history", () -> {
LoanHistoryService.GenerateResult r = loanHistoryService.generate(months);
return "%d loans · %d snapshots over %d months".formatted(
r.loans(), r.snapshotsWritten(), r.monthsRequested());
}));
steps.add(run("upi-inflows", () -> {
UpiSimulationService.SimulationResult r = upiSimulationService.simulate(false);
return "%d enterprises · %d weeks · %d skipped for consent".formatted(
r.enterprisesWritten(), r.weeksWritten(), r.skippedNoConsent());
}));
// The four commodities with no feed anywhere, then every remaining basket
// pair. Running the narrow one first keeps its hand-anchored levels — a
// milk rate with a citation behind it — rather than the group defaults.
steps.add(run("prices-no-feed", () -> priceSimulationService.simulate().message()));
steps.add(run("prices-all", () -> priceSimulationService.simulateAll(force).message()));
steps.add(run("features", () -> {
FeatureEngineeringService.Summary s = featureEngineeringService.computeAll(today, null);
return "%d snapshots · %d null values".formatted(s.snapshots(), s.nullValues());
}));
steps.add(run("scoring", () -> {
RiskEngineService.Result r = riskEngine.scoreAll(today);
return "%d scored · %d trip-wires fired · %d alerts".formatted(
r.scored(), r.tripWiresFired(), r.alertsRaised());
}));
boolean ok = steps.stream().allMatch(Step::ok);
String message = ok
? "Demo data generated. All figures are simulated and labelled as such."
: "Bootstrap finished with failures — see steps. Every step is idempotent; re-running is safe.";
log.info("Demo bootstrap — {}", message);
return new Result(steps, ok, message);
}
/**
* One step, with its failure contained.
*
* <p>A step that throws must not abort the ones after it: prices do not depend
* on loan history, and losing the whole run because one generator failed would
* leave a demo server in a worse state than the partial one.
*/
private Step run(String name, StepBody body) {
try {
String detail = body.execute();
log.info("Demo bootstrap · {} — {}", name, detail);
return new Step(name, true, detail);
} catch (Exception e) {
log.error("Demo bootstrap · {} failed", name, e);
return new Step(name, false, e.getClass().getSimpleName() + ": " + e.getMessage());
}
}
@FunctionalInterface
private interface StepBody {
String execute() throws Exception;
}
}
@@ -47,6 +47,13 @@ public class EnterpriseAppController {
return ApiResponse.ok("Synced", appService.sync(enterpriseId, batch));
}
/** Why the owner's score is what it is — the newest run's sentences. */
@GetMapping("/reasons")
public ApiResponse<List<com.lincfusion.udyamlinc.risk.ReasonCode>> reasons(
@AuthenticationPrincipal String enterpriseId) {
return ApiResponse.ok(appService.reasons(enterpriseId));
}
@GetMapping("/alerts")
public ApiResponse<List<RiskAlert>> alerts(@AuthenticationPrincipal String enterpriseId) {
return ApiResponse.ok(appService.alerts(enterpriseId));
@@ -10,8 +10,11 @@ import com.lincfusion.udyamlinc.enterprise.LedgerEntry;
import com.lincfusion.udyamlinc.enterprise.LedgerEntryRepository;
import com.lincfusion.udyamlinc.master.Sector;
import com.lincfusion.udyamlinc.master.SectorRepository;
import com.lincfusion.udyamlinc.risk.ForecastPoint;
import com.lincfusion.udyamlinc.risk.ForecastPointRepository;
import com.lincfusion.udyamlinc.risk.RiskAlert;
import com.lincfusion.udyamlinc.risk.ReasonCode;
import com.lincfusion.udyamlinc.risk.ReasonCodeRepository;
import com.lincfusion.udyamlinc.risk.RiskAlertRepository;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
@@ -38,6 +41,7 @@ public class EnterpriseAppService {
private final LedgerEntryRepository ledgerRepository;
private final ForecastPointRepository pointRepository;
private final RiskAlertRepository alertRepository;
private final ReasonCodeRepository reasonRepository;
private final SectorRepository sectorRepository;
public AppHomeResponse home(String enterpriseId) {
@@ -47,13 +51,22 @@ public class EnterpriseAppService {
.findFirst()
.orElse(null);
// Newest scoring run only. Unscoped, this returned every point ever forecast
// for the enterprise — seventeen runs deep on the demo book — and the owner's
// six-month strip drew a hundred overlapping months.
Long latestRun = pointRepository.findByEnterpriseIdOrderByHorizonAsc(enterpriseId).stream()
.map(ForecastPoint::getRunId).filter(java.util.Objects::nonNull)
.max(Long::compareTo).orElse(null);
List<ForecastPoint> forecast = latestRun == null ? List.of()
: pointRepository.findByEnterpriseIdAndRunIdOrderByHorizonAsc(enterpriseId, latestRun);
List<LedgerEntry> today = ledgerRepository.findByEnterpriseIdAndEntryDate(enterpriseId, LocalDate.now());
Map<String, String> advice = adviceFor(sectorCode(enterprise));
return new AppHomeResponse(
enterprise,
alert,
pointRepository.findByEnterpriseIdOrderByHorizonAsc(enterpriseId),
forecast,
sum(today, "IN"),
sum(today, "OUT"),
advice.get("en"),
@@ -65,6 +78,22 @@ public class EnterpriseAppService {
return ledgerRepository.findByEnterpriseIdOrderByEntryDateDesc(enterpriseId);
}
/**
* The owner's own reasons, newest run only.
*
* <p>The same sentences the officer sees. A borrower who is told their limit
* is under review is entitled to know which measurement drove it, and reading
* across runs would pair today's score with last month's explanation.
*/
public List<ReasonCode> reasons(String enterpriseId) {
requireEnterprise(enterpriseId);
List<ReasonCode> all = reasonRepository.findByEnterpriseIdOrderByRankAsc(enterpriseId);
Long latestRun = all.stream().map(ReasonCode::getRunId)
.filter(java.util.Objects::nonNull).max(Long::compareTo).orElse(null);
if (latestRun == null) return List.of();
return all.stream().filter(r -> latestRun.equals(r.getRunId())).toList();
}
public List<RiskAlert> alerts(String enterpriseId) {
return alertRepository.findByEnterpriseIdOrderByRaisedAtDesc(enterpriseId);
}
@@ -9,4 +9,7 @@ public interface UdyamUserRepository extends JpaRepository<UdyamUser, Long> {
Optional<UdyamUser> findByUsernameIgnoreCase(String username);
Optional<UdyamUser> findByEnterpriseIdAndScopeIgnoreCase(String enterpriseId, String scope);
java.util.List<UdyamUser> findByScopeIgnoreCaseAndEnterpriseIdIn(
String scope, java.util.Collection<String> enterpriseIds);
}
@@ -12,25 +12,25 @@ import java.util.*;
/**
* Every adapter the platform can run, and the dialect each one speaks.
*
* <p>The four simulated dialects are not decoration. Read the {@code columns}
* maps side by side and the case for a mapping layer makes itself: the days-past
* due field is {@code DPD_DAYS}, {@code OVERDUE_DAYS}, {@code DAYS_OVERDUE} and
* {@code daysPastDue} depending on whose core it is, and a bank will not be
* renaming it for us. Column names are drawn from the public documentation and
* common extract conventions for each core; a given bank's actual export will
* differ again, which is exactly why the mapping is per-profile data rather than
* per-vendor code.
* <p>Two adapters, and the difference between them is the whole design. The file
* drop is real and reads whatever a bank chooses to deliver. {@link #cbsNexora()}
* is CBSNexora, LincPay's own core banking platform, simulated here because no
* instance feeds this deployment.
*
* <p>{@code extraColumns} on each dialect carries the fields a real export ships
* that this system has promised never to store — the borrower's name, a mobile
* number. They are emitted so that the mapping screen can show them arriving and
* being dropped unmapped at the boundary. An extract that were already clean
* would let a reviewer assume the promise is kept by luck.
* <p><b>No third-party core banking product is named anywhere in this system.</b>
* Where a lender runs someone else's core, the integration is described by its
* shape — a scheduled extract, a reporting view, a nightly position file — and
* the per-profile field mapping absorbs whatever that deployment calls its
* columns. Naming another vendor would imply a relationship with it that does not
* exist, and the mapping layer means the platform never needed the name.
*/
@Component
@RequiredArgsConstructor
public class ConnectorCatalog {
/** Vendor code for the CBSNexora adapter. Matches a profile's {@code vendor}. */
public static final String VENDOR = "CBSNEXORA";
private final FileDropConnector fileDropConnector;
private final EnterpriseRepository enterpriseRepository;
private final LoanRecordRepository loanRepository;
@@ -41,10 +41,7 @@ public class ConnectorCatalog {
@PostConstruct
void register() {
add(fileDropConnector);
add(simulated(finacle()));
add(simulated(flexcube()));
add(simulated(bancs()));
add(simulated(customApi()));
add(simulated(cbsNexora()));
}
private void add(CoreBankingConnector connector) {
@@ -64,70 +61,20 @@ public class ConnectorCatalog {
}
// ── dialects ────────────────────────────────────────────
private SimulatedCoreConnector.Dialect finacle() {
LinkedHashMap<String, String> columns = new LinkedHashMap<>();
columns.put("enterprise_ref", "ENTITY_REF");
columns.put("loan_account_ref", "CUST_ACCT_ID");
columns.put("product", "SCHM_CODE");
columns.put("sanctioned_amount", "SANCT_AMT");
columns.put("outstanding", "PRIN_OS");
columns.put("emi_amount", "INSTL_AMT");
columns.put("due_date", "NEXT_DUE_DT");
columns.put("last_paid_date", "LAST_PAID_DT");
columns.put("dpd", "DPD_DAYS");
columns.put("bucket", "ASSET_CLASS");
columns.put("closed", "ACCT_CLS_FLG");
columns.put("as_of_date", "EXTRACT_DT");
return new SimulatedCoreConnector.Dialect("FINACLE", "Infosys Finacle (simulated)", "FILE",
"Finacle exports through a scheduled CRM/Reports extract. A bank enables it, "
+ "agrees the column list, and the file lands on the drop path — at which point "
+ "this profile switches to the file-drop adapter with the mapping already made.",
columns, List.of("CUST_NAME", "MOBILE_NO", "BRANCH_CODE", "CCY"));
}
private SimulatedCoreConnector.Dialect flexcube() {
LinkedHashMap<String, String> columns = new LinkedHashMap<>();
columns.put("enterprise_ref", "CUSTOMER_REF");
columns.put("loan_account_ref", "ACCOUNT_NUMBER");
columns.put("product", "PRODUCT_CODE");
columns.put("sanctioned_amount", "AMOUNT_FINANCED");
columns.put("outstanding", "PRINCIPAL_OUTSTANDING");
columns.put("emi_amount", "INSTALMENT_AMOUNT");
columns.put("due_date", "NEXT_DUE_DATE");
columns.put("last_paid_date", "LAST_PAYMENT_DATE");
columns.put("dpd", "OVERDUE_DAYS");
columns.put("bucket", "NPA_STAGE");
columns.put("closed", "ACCOUNT_STATUS");
columns.put("as_of_date", "VALUE_DATE");
return new SimulatedCoreConnector.Dialect("FLEXCUBE", "Oracle FLEXCUBE (simulated)", "JDBC",
"FLEXCUBE is usually read through a reporting schema or an extract view rather than "
+ "the transaction tables. Needs a read-only account scoped to the loan module, "
+ "and the bank's DBA in the room.",
columns, List.of("CUSTOMER_NAME", "CONTACT_MOBILE", "BRANCH_CODE"));
}
private SimulatedCoreConnector.Dialect bancs() {
LinkedHashMap<String, String> columns = new LinkedHashMap<>();
columns.put("enterprise_ref", "PARTY_REF");
columns.put("loan_account_ref", "ARRANGEMENT_ID");
columns.put("product", "PRODUCT_GROUP");
columns.put("sanctioned_amount", "LIMIT_AMT");
columns.put("outstanding", "OUTSTANDING_BAL");
columns.put("emi_amount", "EMI");
columns.put("due_date", "DUE_DATE");
columns.put("last_paid_date", "PAID_DATE");
columns.put("dpd", "DAYS_OVERDUE");
columns.put("bucket", "CLASSIFICATION");
columns.put("closed", "STATUS");
columns.put("as_of_date", "POSITION_DATE");
return new SimulatedCoreConnector.Dialect("BANCS", "TCS BaNCS (simulated)", "FILE",
"BaNCS deployments here are typically cooperative-bank hosted. A nightly position "
+ "file is the realistic ask; a live API rarely is in year one.",
columns, List.of("PARTY_NAME", "MOBILE", "BRANCH"));
}
private SimulatedCoreConnector.Dialect customApi() {
/**
* CBSNexora — LincPay's own core banking platform, as the platform would read it.
*
* <p>Simulated here: no CBSNexora instance feeds this deployment, and every row
* an adapter of this class produces is stamped {@code simulated} per sync run so
* a profile later pointed at a real instance cannot make its old runs look real.
*
* <p>{@code extraColumns} carries the fields a real export ships that this system
* has promised never to store — the borrower's name, a mobile number. They are
* emitted so the mapping screen can show them arriving and being dropped unmapped
* at the boundary. An extract that were already clean would let a reviewer assume
* the promise is kept by luck.
*/
private SimulatedCoreConnector.Dialect cbsNexora() {
LinkedHashMap<String, String> columns = new LinkedHashMap<>();
columns.put("enterprise_ref", "borrowerRef");
columns.put("loan_account_ref", "loanId");
@@ -141,10 +88,10 @@ public class ConnectorCatalog {
columns.put("bucket", "assetClassification");
columns.put("closed", "isClosed");
columns.put("as_of_date", "asOfDate");
return new SimulatedCoreConnector.Dialect("CUSTOM_API", "In-house LMS over REST (simulated)", "REST",
"For a lender whose loan management system is their own. They expose one paged "
+ "read-only endpoint returning these fields; the mapping absorbs whatever they "
+ "happen to call them.",
return new SimulatedCoreConnector.Dialect(VENDOR, "CBSNexora (simulated)", "REST",
"CBSNexora exposes one paged, read-only loan endpoint. A lender enables it for "
+ "the platform's service account and agrees the field list; the mapping absorbs "
+ "whatever their deployment happens to call them.",
columns, List.of("borrowerName", "borrowerMobile", "branchCode"));
}
}
@@ -43,7 +43,7 @@ public class ConnectorProfile {
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
/** Unique per deployment — "MPGB-FINACLE". */
/** Unique per deployment — "MPGB-CBSNEXORA". */
@Column(nullable = false, unique = true, length = 40)
private String code;
@@ -1,7 +1,10 @@
package com.lincfusion.udyamlinc.connector;
import com.lincfusion.udyamlinc.manualentry.ManualEntryBatch;
import com.lincfusion.udyamlinc.manualentry.ManualEntryBatchRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.data.domain.Pageable;
import org.springframework.stereotype.Component;
import java.util.List;
@@ -15,10 +18,17 @@ import java.util.List;
* first populated never reaches it, and the symptom is a screen that is empty for
* everyone except whoever last dropped their schema.
*
* <p>The three lenders are on three different cores on purpose. It is also true
* of the sector — a regional rural bank, a district cooperative bank and a small
* finance bank will not be on one core — and a demo where every bank is on
* Finacle would quietly suggest that the mapping layer is optional.
* <p>The three lenders do not arrive the same way, on purpose. Two are on
* CBSNexora and read over its loan API; the cooperative bank delivers a file to a
* drop path. A demo where every bank arrived identically would quietly suggest
* that the mapping layer is optional, and it is the opposite of optional — it is
* what lets the platform onboard a lender whose core it has never seen.
*
* <p><b>CBSNexora is the only core banking product named anywhere in this
* system,</b> and it is named because LincPay builds it. A lender on someone
* else's core is absorbed by the per-profile field mapping instead: the platform
* never needs that product's name, and printing one would imply a relationship
* with its vendor that does not exist.
*/
@Component
@RequiredArgsConstructor
@@ -26,6 +36,8 @@ import java.util.List;
public class ConnectorProfileSeeder {
private final ConnectorProfileRepository profileRepository;
private final ConnectorSyncRunRepository runRepository;
private final ManualEntryBatchRepository batchRepository;
private final ConnectorCatalog catalog;
private final FieldMapper fieldMapper;
@@ -33,14 +45,90 @@ public class ConnectorProfileSeeder {
String endpoint, String cron) {
}
/**
* Codes and vendors this seeder used before the vendor names were removed.
*
* <p>Seeding is guarded per code, so without this a database populated under
* the old names would keep them for ever and quietly seed a second profile
* beside each one. Renaming in place also preserves every sync run already
* recorded against the profile, which deleting and reseeding would orphan.
*/
private record Legacy(String oldCode, String newCode, String oldVendor, String displayName) {
}
private static final List<Legacy> LEGACY = List.of(
new Legacy("MPGB-FINACLE", "MPGB-CBSNEXORA", "FINACLE",
"MP Gramin Bank — CBSNexora loan API"),
new Legacy("VSFB-LMS", "VSFB-CBSNEXORA", "CUSTOM_API",
"Vindhya SFB — CBSNexora loan API"));
/**
* Relabel, in three places, because the old name was denormalised into all
* three. A profile carries it; every sync run copies the code and vendor it
* ran under; and each staged upload batch carries the profile code plus a
* human label built from it at the time.
*
* <p>The one field deliberately left alone is {@code simulated} on a run. It
* records what the run actually was, and renaming an identity must never
* rewrite the provenance of work already done under it.
*/
private void renameLegacy() {
for (Legacy row : LEGACY) {
profileRepository.findByCode(row.oldCode()).ifPresent(profile -> {
profile.setCode(row.newCode());
profile.setVendor(ConnectorCatalog.VENDOR);
profile.setDisplayName(row.displayName());
profileRepository.save(profile);
log.info("Renamed legacy connector profile {} → {}", row.oldCode(), row.newCode());
});
// Runs are looked up by profile code, so leaving these would both show
// the old name in sync history and strand the runs from their profile.
List<ConnectorSyncRun> runs =
runRepository.findByProfileCodeOrderByStartedAtDesc(row.oldCode(), Pageable.unpaged());
if (!runs.isEmpty()) {
runs.forEach(run -> {
run.setProfileCode(row.newCode());
run.setVendor(ConnectorCatalog.VENDOR);
});
runRepository.saveAll(runs);
log.info("Repointed {} sync run(s) from {} to {}", runs.size(), row.oldCode(), row.newCode());
}
// The batch label is free text composed at sync time — "MPGB-FINACLE ·
// FINACLE simulated extract · 44 accounts". Both halves have to go.
List<ManualEntryBatch> batches = batchRepository.findAllByOrderByUploadedAtDesc(Pageable.unpaged())
.stream()
.filter(b -> row.oldCode().equals(b.getSource())
|| (b.getFilename() != null && b.getFilename().contains(row.oldCode())))
.toList();
if (batches.isEmpty()) continue;
batches.forEach(b -> {
if (row.oldCode().equals(b.getSource())) b.setSource(row.newCode());
if (b.getFilename() != null) {
b.setFilename(b.getFilename()
.replace(row.oldCode(), row.newCode())
.replace(row.oldVendor(), ConnectorCatalog.VENDOR));
}
});
batchRepository.saveAll(batches);
log.info("Relabelled {} upload batch(es) from {} to {}",
batches.size(), row.oldCode(), row.newCode());
}
}
public void seed() {
renameLegacy();
List<Seed> seeds = List.of(
new Seed("MPGB-FINACLE", "MP Gramin Bank — Finacle nightly extract",
"FINACLE", "MPGB", "loanbook-*.csv", "0 30 1 * * *"),
new Seed("MPGB-CBSNEXORA", "MP Gramin Bank — CBSNexora loan API",
ConnectorCatalog.VENDOR, "MPGB", "https://cbsnexora.mpgb.example/api/loans",
"0 30 1 * * *"),
new Seed("SEHDCCB-DROP", "Sehore DCCB — SFTP file drop",
FileDropConnector.VENDOR, "SEHDCCB", "sehdccb-loanbook-*.{csv,xlsx}", "0 0 2 * * *"),
new Seed("VSFB-LMS", "Vindhya SFB — in-house LMS over REST",
"CUSTOM_API", "VSFB", "https://lms.vindhyasfb.example/api/loans", "0 15 1 * * *"));
new Seed("VSFB-CBSNEXORA", "Vindhya SFB — CBSNexora loan API",
ConnectorCatalog.VENDOR, "VSFB", "https://cbsnexora.vindhyasfb.example/api/loans",
"0 15 1 * * *"));
int written = 0;
for (Seed seed : seeds) {
@@ -9,7 +9,8 @@ import java.util.Map;
* management system.
*
* <p><b>Why an interface and not an integration.</b> Every bank's core is
* different — Finacle, Flexcube, BaNCS, a cooperative bank's in-house system —
* different — a scheduled report extract, a reporting view, a nightly position
* file, a REST loan API such as CBSNexora —
* and each names the same eleven fields differently, exposes them over a
* different transport, and will only agree a schema during its own onboarding.
* There is no single integration to build ahead of that conversation. What can
@@ -31,7 +32,7 @@ import java.util.Map;
*/
public interface CoreBankingConnector {
/** Stable code — {@code FINACLE}, {@code FILE_DROP}. Matches a profile's vendor. */
/** Stable code — {@code CBSNEXORA}, {@code FILE_DROP}. Matches a profile's vendor. */
String vendor();
/** What a bank operations user would call it. */
@@ -15,17 +15,17 @@ import java.time.temporal.ChronoUnit;
import java.util.*;
/**
* A core banking system that does not exist, exporting in the dialect of one
* that does.
* A core banking instance that is not there, exporting in the dialect of one
* that would be.
*
* <p><b>What this is for, and what it is not.</b> Integrating with Finacle,
* Flexcube or BaNCS is not a technical unknown — it is an onboarding process per
* bank, with a schema agreed in a room. Until that room, the thing worth
* demonstrating is not a fake feed but the <i>shape of the seam</i>: that each
* core names the same eleven fields differently, that a mapping profile is what
* reconciles them, and that nothing after the mapping cares which core it was.
* Running the same portfolio out through four dialects makes that visible in a
* way a paragraph of intent does not.
* <p><b>What this is for, and what it is not.</b> Connecting a lender's core is
* not a technical unknown — it is an onboarding process per bank, with a schema
* agreed in a room. Until that room, the thing worth demonstrating is not a fake
* feed but the <i>shape of the seam</i>: that a core names the same eleven fields
* in its own way, that a mapping profile is what reconciles them, and that
* nothing after the mapping cares which core it was. Running the real portfolio
* out through a declared dialect makes that visible in a way a paragraph of
* intent does not.
*
* <p>Every row it produces is stamped simulated, all the way to the console badge
* and the sync-run record. That matters more than it sounds: the failure mode
@@ -54,7 +54,22 @@ public final class UdyamResponses {
Integer score,
String band,
List<Integer> trend,
String lastUpdated) {
String lastUpdated,
/**
* Months until the downside case first fails to cover the instalment,
* or null if it never does over the horizon.
*
* <p>Here because lead time is the product. A dashboard that ranks by
* score alone shows what a bureau score already shows; ranking by when
* trouble arrives is the thing this platform can do that they cannot.
*/
Integer firstBreachHorizon,
String firstBreachMonth,
/** The top-ranked reason sentence from the run that produced the score. */
String topDriver,
String topDriverHi) {
}
/** Dashboard KPI tiles plus the enterprises-by-risk register. */
@@ -107,6 +107,59 @@ public class FeatureController {
engineeringService.computeAll(date == null ? LocalDate.now() : date, null));
}
/**
* Recompute the book at every month end in a range — the feature store as a
* history rather than as a snapshot.
*
* <p><b>Why this exists.</b> {@code /compute} writes one as-of date, and the
* job runs nightly, so the store only ever held the present. A model cannot
* be trained on the present: it needs the features as they stood at each
* past date, joined to what happened next. Every derivation already takes
* {@code asOfDate} and filters on it, so this is a loop, not new logic.
*
* <p><b>What it is honest about.</b> This reconstructs the features by
* <em>fact date</em>, not by <em>knowledge date</em>. A mandi price for March
* was ingested in August, so a March row here uses a number a live March run
* would not yet have had. That is not label leakage — the label lives in
* {@code loan_record} ninety days later — but it does mean these rows are
* reconstructed rather than recorded, and anything trained on them should
* say so.
*
* <p>Idempotent: each row upserts on
* {@code (enterprise, feature, valid_for_date)}, so a re-run corrects rather
* than duplicates.
*/
@PostMapping("/backfill")
public ApiResponse<Map<String, Object>> backfill(
@RequestParam @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate from,
@RequestParam @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate to) {
if (from.isAfter(to)) {
throw new IllegalArgumentException("from must not be after to");
}
List<Map<String, Object>> passes = new ArrayList<>();
int computed = 0;
// Month ends, because the label in loan_record is stamped at month end and
// a training row whose feature date and label date disagree is a training
// row nobody can reproduce.
LocalDate cursor = from.withDayOfMonth(from.lengthOfMonth());
while (!cursor.isAfter(to)) {
FeatureEngineeringService.Summary summary = engineeringService.computeAll(cursor, null);
computed += summary.snapshots();
passes.add(Map.of("asOfDate", cursor.toString(),
"snapshots", summary.snapshots(),
"skippedForConsent", summary.skippedForConsent(),
"nullValues", summary.nullValues()));
cursor = cursor.plusMonths(1);
cursor = cursor.withDayOfMonth(cursor.lengthOfMonth());
}
return ApiResponse.ok("Backfilled %d month(s), %d snapshots".formatted(passes.size(), computed),
Map.of("months", passes.size(), "snapshots", computed, "passes", passes));
}
/** One enterprise, for when a single reason code needs explaining. */
@PostMapping("/compute/{enterpriseId}")
public ApiResponse<FeatureSnapshot> computeOne(
@@ -53,13 +53,30 @@ public class ClimateIngestionService {
/** Not one transaction, for the same reason as {@link MandiIngestionService#ingest()}. */
public IngestionResult ingest() {
return ingest(null);
}
/**
* @param lookbackWeeksOverride how far back to reach, or null for the configured default.
*
* <p>The override exists because this reads a <em>reanalysis archive</em>, not a live weather
* endpoint — the same request that returns last week returns 2019, at the same cost. The
* nightly job wants twelve weeks; a feature-store backfill wants two years, and there is no
* reason to make that a redeploy. Mandi ingestion already took this parameter; climate
* silently ignored the one the controller was passing it.
*/
public IngestionResult ingest(Integer lookbackWeeksOverride) {
List<District> districts = districtRepository.findByActiveTrue();
if (districts.isEmpty()) {
return new IngestionResult(0, 0, 0, 0, "No active districts");
}
int weeks = lookbackWeeksOverride != null && lookbackWeeksOverride > 0
? lookbackWeeksOverride
: lookbackWeeks;
LocalDate to = LocalDate.now().minusDays(lagDays);
LocalDate from = Weeks.startOf(to).minusWeeks(lookbackWeeks);
LocalDate from = Weeks.startOf(to).minusWeeks(weeks);
LocalDateTime observedAt = LocalDateTime.now();
int fetched = 0, written = 0, skipped = 0, failed = 0;
@@ -48,6 +48,27 @@ public class IngestionController {
return ApiResponse.ok(result.message(), result);
}
/**
* Three years of weekly prices for every active basket pair that has no real
* data — the whole market, not just the four commodities without a feed.
*
* <p>For a deployment with no {@code DATAGOV_API_KEY}. A pair that already
* holds real rows is skipped rather than overwritten, so running this against
* a database that has ingested Agmarknet fills the gaps around it and leaves
* the real series alone.
*
* @param force generate even over a real series. For rebuilding a demo
* database on purpose; the rows still carry {@code SIMULATED}
* and still sit beside the real ones rather than replacing them.
*/
@PostMapping("/simulate/all")
public ApiResponse<NoFeedPriceSimulationService.FullSimulationResult> simulateAll(
@RequestParam(defaultValue = "false") boolean force) {
NoFeedPriceSimulationService.FullSimulationResult result =
noFeedPriceSimulationService.simulateAll(force);
return ApiResponse.ok(result.message(), result);
}
/**
* One call per basket pair, reporting which spellings actually match the
* feed. A write only in the {@code persist=true} sense — it sets the
@@ -86,7 +86,7 @@ public class IngestionScheduler {
IngestionResult result = switch (source) {
case DataGovClient.SOURCE -> mandiService.ingest(lookbackWeeks);
case OpenMeteoClient.SOURCE -> climateService.ingest();
case OpenMeteoClient.SOURCE -> climateService.ingest(lookbackWeeks);
default -> throw new IllegalArgumentException("Unknown ingestion source: " + source);
};
return record(source, trigger, result.status(), result, started);
@@ -16,6 +16,18 @@ public interface MarketSignalRepository extends JpaRepository<MarketSignal, Long
List<MarketSignal> findByDistrictIdAndCommodityIdOrderByWeekStartDesc(Long districtId, Long commodityId);
/**
* How many rows for this pair came from somewhere other than the given source.
*
* <p>Used by the simulator to leave a pair alone when a real feed has already
* written to it. The test is deliberately "any row at all" rather than "a row
* this week": a district that ingested real Agmarknet prices and then went
* quiet for a fortnight still has a real series, and burying it under a
* generated one to fill the gap would be the one thing the source column
* exists to prevent.
*/
long countByDistrictIdAndCommodityIdAndSourceNot(Long districtId, Long commodityId, String source);
List<MarketSignal> findByWeekStartGreaterThanEqualOrderByWeekStartDesc(LocalDate from);
/**
@@ -3,6 +3,8 @@ package com.lincfusion.udyamlinc.ingestion;
import com.lincfusion.udyamlinc.master.Commodity;
import com.lincfusion.udyamlinc.master.CommodityRepository;
import com.lincfusion.udyamlinc.master.District;
import com.lincfusion.udyamlinc.master.DistrictCommodity;
import com.lincfusion.udyamlinc.master.DistrictCommodityRepository;
import com.lincfusion.udyamlinc.master.DistrictRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
@@ -130,13 +132,216 @@ public class NoFeedPriceSimulationService {
// and peaks with the post-harvest cotton arrival window.
"YARN_INDEX", new Profile(100.0, 0.11, 0.015, 6, 0.065));
/**
* Farm-gate levels in ₹/quintal for the commodities that normally arrive from
* Agmarknet, used only when no real row exists for a pair.
*
* <p>These are ordinary Indian mandi levels rather than invented numbers, for
* the same reason the profiles above are anchored: an officer sanity-checks a
* price on sight, and a plausible shape carrying an implausible level is worse
* than an obvious placeholder because it invites belief.
*
* <p>They are still approximate, and a simulated row says so in its source
* column. Nothing here is a substitute for the real feed — it is what keeps
* the console populated on a deployment that has no data.gov.in key.
*/
private static final Map<String, Double> CROP_BASE = Map.ofEntries(
// Cereals — procurement-anchored, so the least volatile group.
Map.entry("WHEAT", 2550.0), Map.entry("PADDY", 2320.0), Map.entry("RICE", 3800.0),
Map.entry("MAIZE", 2150.0), Map.entry("BAJRA", 2600.0), Map.entry("JOWAR", 3300.0),
// Pulses
Map.entry("GRAM", 5900.0), Map.entry("MASOOR", 6400.0),
Map.entry("MOONG", 8500.0), Map.entry("ARHAR", 7600.0),
// Oilseeds and cash crops
Map.entry("SOYA", 4900.0), Map.entry("MUSTARD", 5800.0), Map.entry("COTTON", 7500.0),
// Vegetables — the volatile end, and where margin_squeeze earns its keep.
Map.entry("ONION", 1900.0), Map.entry("TOMATO", 2100.0), Map.entry("POTATO", 1500.0),
Map.entry("BRINJAL", 2200.0), Map.entry("CAULIFLOWER", 1900.0), Map.entry("CABBAGE", 1400.0),
Map.entry("GREEN_CHILLI", 4600.0), Map.entry("OKRA", 3100.0), Map.entry("CAPSICUM", 4200.0),
Map.entry("CUCUMBER", 1700.0), Map.entry("GREEN_PEAS", 4400.0), Map.entry("PUMPKIN", 1250.0),
Map.entry("BEETROOT", 2050.0), Map.entry("BITTER_GOURD", 3300.0),
Map.entry("BOTTLE_GOURD", 1450.0), Map.entry("CARROT", 2300.0),
// Fruit
Map.entry("BANANA", 2100.0), Map.entry("MANGO", 5200.0), Map.entry("GRAPES", 6200.0),
Map.entry("POMEGRANATE", 9500.0), Map.entry("LITCHI", 7200.0),
Map.entry("PAPAYA", 1850.0), Map.entry("GUAVA", 3200.0),
// Spices
Map.entry("GARLIC", 9200.0), Map.entry("GINGER", 6800.0), Map.entry("TURMERIC", 8000.0));
/**
* Seasonal swing, annual trend and weekly noise by commodity group.
*
* <p>The group already encodes perishability — that is why the column exists
* rather than as a display nicety — and perishability is exactly what decides
* volatility. A vegetable price moves on a weather event within days; a cereal
* price does not, because procurement puts a floor under it. Giving both the
* same shape would flatten the one distinction {@code price_z_score} is there
* to read.
*
* <p>Noise is set relative to the trend throughout, per the calibration note
* on {@link Profile}: real Agmarknet series in this same database average a
* z of 0.44, and a generator that trends faster than it varies puts every
* newest observation at the top of its own distribution.
*/
private record GroupShape(double seasonalAmplitude, double trendPerYear, double weeklyNoise) {
}
private static final Map<String, GroupShape> GROUP_SHAPES = Map.of(
"CEREAL", new GroupShape(0.07, 0.020, 0.030),
"PULSE", new GroupShape(0.11, 0.020, 0.042),
"OILSEED", new GroupShape(0.10, 0.020, 0.040),
"CASH", new GroupShape(0.09, 0.020, 0.038),
"VEGETABLE", new GroupShape(0.26, 0.015, 0.085),
"FRUIT", new GroupShape(0.22, 0.015, 0.070),
"SPICE", new GroupShape(0.20, 0.020, 0.065));
private static final GroupShape DEFAULT_SHAPE = new GroupShape(0.12, 0.020, 0.045);
/**
* The ISO week each commodity's price <em>peaks</em> — which is not when it is
* harvested, and getting that backwards inverts every series.
*
* <p>A mandi price troughs at the arrival glut and peaks in the lean season
* before the next one. Wheat is harvested in March–April and is cheapest then;
* it is dearest in November–December, when last season's stock has run down and
* the new crop is still in the ground. So the entry for wheat is week 48, not
* week 14. The same inversion applies right down the list: rabi crops peak in
* autumn, kharif crops peak in early summer, and winter vegetables peak in the
* monsoon when they cannot be grown.
*
* <p>Why this matters beyond looking right. {@code margin_squeeze} reads a
* sector's OUTPUT price against its INPUT price, and a squeeze is real when the
* two peak at different times of year. Deriving the peak from a hash of the
* commodity code — which is what this replaces — put those peaks in arbitrary
* places, so a poultry unit could show feed and egg peaking in the same week and
* a dairy could show them six months apart, both by accident. The seasonal
* relationship between an input and an output is the thing the derivation exists
* to measure, so it has to come from a calendar rather than from a hash.
*
* <p>Approximate and national, and both caveats are real. Harvest windows shift
* by weeks across fourteen districts spanning six zones — Bihar's paddy and
* Karnataka's are not the same week — and this table carries one figure for
* each. It is a better approximation than a hash, not a crop calendar for a
* specific district. Anything reading these rows sees {@code SIMULATED}.
*/
private static final Map<String, Integer> PRICE_PEAK_WEEK = Map.ofEntries(
// Rabi cereals and pulses: harvested Mar–Apr, dearest in the autumn lean.
Map.entry("WHEAT", 48), Map.entry("GRAM", 42), Map.entry("MASOOR", 42),
Map.entry("MUSTARD", 42),
// Kharif: harvested Sep–Nov, dearest before the monsoon crop lands.
Map.entry("PADDY", 26), Map.entry("RICE", 26), Map.entry("MAIZE", 28),
Map.entry("SOYA", 30), Map.entry("BAJRA", 24), Map.entry("JOWAR", 32),
Map.entry("ARHAR", 36), Map.entry("MOONG", 12), Map.entry("COTTON", 30),
// Onion and potato are stored, so their peak is the far end of storage —
// Sep–Nov for onion, before the kharif crop, and autumn for potato.
Map.entry("ONION", 42), Map.entry("POTATO", 40),
// Winter vegetables: grown Nov–Feb, so dearest through the monsoon when
// the crop cannot be raised at all.
Map.entry("CAULIFLOWER", 24), Map.entry("CABBAGE", 24), Map.entry("GREEN_PEAS", 27),
Map.entry("CARROT", 30), Map.entry("BEETROOT", 28),
// Summer vegetables: the mirror image, dearest in the winter gap.
Map.entry("OKRA", 52), Map.entry("BITTER_GOURD", 50), Map.entry("BOTTLE_GOURD", 52),
Map.entry("CUCUMBER", 1), Map.entry("PUMPKIN", 10),
// Tomato peaks in the monsoon, when transit losses are worst — the
// annual spike that reaches the newspapers.
Map.entry("TOMATO", 27), Map.entry("BRINJAL", 28), Map.entry("GREEN_CHILLI", 32),
Map.entry("CAPSICUM", 28),
// Fruit peaks at the start of its own short season, then falls as
// arrivals build.
Map.entry("MANGO", 14), Map.entry("LITCHI", 20), Map.entry("GRAPES", 2),
Map.entry("BANANA", 20), Map.entry("PAPAYA", 18), Map.entry("POMEGRANATE", 30),
Map.entry("GUAVA", 30),
// Spices are stored and traded year-round, so the peak sits well after
// the harvest rather than opposite it.
Map.entry("GARLIC", 48), Map.entry("GINGER", 35), Map.entry("TURMERIC", 40));
private final CommodityRepository commodityRepository;
private final DistrictRepository districtRepository;
private final DistrictCommodityRepository districtCommodityRepository;
private final MarketSignalRepository signalRepository;
public record SimulationResult(int commodities, int districts, int weeksWritten, String message) {
}
public record FullSimulationResult(int pairsWritten, int pairsSkipped, int weeksWritten, String message) {
}
/**
* A three-year series for every active basket pair that has no real data.
*
* <p>This exists for a deployment with no {@code DATAGOV_API_KEY} — a demo
* server, most immediately. Without it a fresh database has twelve weeks of
* nothing: {@code price_z_score} needs a seasonal mean built from three years
* of the same week in the same district, so a short or empty series does not
* merely weaken the feature, it makes it null, and TW-05 never fires.
*
* <p><b>A pair with any non-simulated row is skipped</b> unless {@code force}
* is set. That ordering is the whole safety property: run this on a database
* that has already ingested real Agmarknet prices and it fills the gaps
* around them rather than papering over them. The rows it does write carry
* {@code source = SIMULATED} and key on the source, so even a forced run sits
* beside the real series instead of replacing it.
*
* @param force write even where a real series already exists — for rebuilding
* a demo database deliberately, never as a default
*/
@Transactional
public FullSimulationResult simulateAll(boolean force) {
LocalDate thisWeek = Weeks.startOf(LocalDate.now());
LocalDateTime observedAt = LocalDateTime.now();
int pairs = 0, skipped = 0, written = 0;
for (DistrictCommodity pair : districtCommodityRepository.findByActiveTrue()) {
District district = districtRepository.findById(pair.getDistrictId()).orElse(null);
Commodity commodity = commodityRepository.findById(pair.getCommodityId()).orElse(null);
if (district == null || commodity == null) {
continue;
}
if (!force && signalRepository.countByDistrictIdAndCommodityIdAndSourceNot(
district.getId(), commodity.getId(), SOURCE) > 0) {
skipped++;
continue;
}
written += writeSeries(district, commodity, profileFor(commodity), thisWeek, observedAt);
pairs++;
}
String message = "%d basket pairs simulated, %d left to real data · %d weeks written"
.formatted(pairs, skipped, written);
log.info("Full market simulation — {}", message);
return new FullSimulationResult(pairs, skipped, written, message);
}
/**
* The profile to generate a commodity's series from.
*
* <p>An explicit hand-anchored profile wins; otherwise the level comes from
* {@link #CROP_BASE} and the shape from the commodity's group.
*
* <p>The peak week comes from {@link #PRICE_PEAK_WEEK}, the lean-season table.
* A commodity missing from it falls back to a hash of its code, which keeps the
* series deterministic but puts its season somewhere arbitrary — acceptable for
* a commodity nothing reads as a sector's input or output, and a warning that a
* calendar entry is missing if it ever is.
*/
private Profile profileFor(Commodity commodity) {
Profile explicit = PROFILES.get(commodity.getCode());
if (explicit != null) {
return explicit;
}
GroupShape shape = GROUP_SHAPES.getOrDefault(commodity.getCommodityGroup(), DEFAULT_SHAPE);
double base = CROP_BASE.getOrDefault(commodity.getCode(), 3000.0);
Integer calendarPeak = PRICE_PEAK_WEEK.get(commodity.getCode());
if (calendarPeak == null) {
log.warn("No price-peak week for commodity {} — falling back to a hashed season", commodity.getCode());
}
int peakWeek = calendarPeak != null
? calendarPeak
: Math.floorMod(commodity.getCode().hashCode(), 52) + 1;
return new Profile(base, shape.seasonalAmplitude(), shape.trendPerYear(), peakWeek, shape.weeklyNoise());
}
@Transactional
public SimulationResult simulate() {
LocalDate thisWeek = Weeks.startOf(LocalDate.now());
@@ -0,0 +1,99 @@
package com.lincfusion.udyamlinc.ml;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.Optional;
/**
* The one call out to the Python inference service.
*
* <p><b>Never throws to the caller.</b> Every method returns an {@link Optional},
* empty on any failure — connection refused, timeout, a 500, a malformed body.
* That is deliberate: the caller is a scoring run, and a run that fails because
* an inference service is unreachable is worse than a run that scores on rules
* and says which engine it used. The failure is logged once per call, not per
* enterprise, so a service being down does not produce 263 identical stack traces.
*/
@Component
@Slf4j
public class MlClient {
private final MlProperties properties;
private final RestClient http;
public MlClient(MlProperties properties) {
this.properties = properties;
SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
factory.setConnectTimeout(Duration.ofMillis(properties.getTimeoutMs()));
factory.setReadTimeout(Duration.ofMillis(properties.getBatchTimeoutMs()));
this.http = RestClient.builder()
.baseUrl(properties.getBaseUrl())
.requestFactory(factory)
.build();
}
public boolean enabled() {
return properties.isEnabled();
}
/** What the service is serving — becomes {@code scoring_run.model_version}. */
public Optional<Map<String, Object>> modelInfo() {
return call(() -> http.get().uri("/model").retrieve().body(Map.class), "/model");
}
public Optional<MlScore> score(Map<String, Object> request) {
return call(() -> http.post().uri("/score").body(request).retrieve().body(MlScore.class), "/score");
}
/** One call for the nightly run, rather than one per enterprise. */
@SuppressWarnings("unchecked")
public Optional<List<MlScore>> scoreBatch(List<Map<String, Object>> items) {
return call(() -> http.post().uri("/score/batch")
.body(Map.of("items", items))
.retrieve()
.body(List.class), "/score/batch");
}
@SuppressWarnings("unchecked")
private <T> Optional<T> call(java.util.function.Supplier<Object> supplier, String what) {
if (!properties.isEnabled()) return Optional.empty();
try {
return Optional.ofNullable((T) supplier.get());
} catch (Exception e) {
log.warn("ML service {} unavailable ({}) — falling back to the rule engine",
what, e.getClass().getSimpleName());
return Optional.empty();
}
}
public Optional<MlForecast> forecast(Map<String, Object> request) {
return call(() -> http.post().uri("/forecast").body(request).retrieve().body(MlForecast.class),
"/forecast");
}
/** Mirrors the FastAPI response, field for field. */
public record MlScore(
String enterpriseId,
int score,
String band,
int confidence,
double probability,
Map<String, Integer> subScores,
List<MlDriver> drivers,
int familiesOnPrior,
String engine,
String modelVersion) {}
public record MlDriver(String family, int weight, String featureCode, Double value, boolean onPrior) {}
public record MlForecast(String enterpriseId, String engine, String modelVersion,
List<MlForecastPoint> points) {}
public record MlForecastPoint(int horizon, double p10, double p50, double p90) {}
}
@@ -0,0 +1,32 @@
package com.lincfusion.udyamlinc.ml;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
/**
* Where the inference service is, and how long we are willing to wait for it.
*
* <p>Timeouts are short on purpose. A scoring run over the whole book must finish
* inside its window, and an inference service that has become slow is
* operationally the same as one that is down — both mean "score this enterprise
* on rules and move on".
*/
@Component
@ConfigurationProperties(prefix = "udyam.ml")
@Getter
@Setter
public class MlProperties {
/** Off by default: a checkout with no Python running must still boot and score. */
private boolean enabled = false;
private String baseUrl = "http://127.0.0.1:8000";
/** Per-enterprise call. */
private int timeoutMs = 2000;
/** Batch call over the book — larger, because it is one call for 263 rows. */
private int batchTimeoutMs = 30000;
}
@@ -40,6 +40,14 @@ public class OfficerController {
return ApiResponse.ok(officerService.enterprises(tenant, block));
}
/** Ids on this book that have a phone-app login. */
@GetMapping("/app-enrolled")
public ApiResponse<List<String>> appEnrolled(
TenantContext tenant,
@RequestParam(defaultValue = "ALL") String block) {
return ApiResponse.ok(officerService.appEnrolled(tenant, block));
}
@GetMapping("/enterprises/{enterpriseId}")
public ApiResponse<EnterpriseDetailResponse> detail(TenantContext tenant, @PathVariable String enterpriseId) {
return ApiResponse.ok(officerService.detail(tenant, enterpriseId));
@@ -41,6 +41,7 @@ public class OfficerService {
/** Sentinel used by the console when no block filter is applied. */
public static final String ALL_BLOCKS = "ALL";
private final com.lincfusion.udyamlinc.auth.UdyamUserRepository userRepository;
private final EnterpriseRepository enterpriseRepository;
private final LedgerEntryRepository ledgerRepository;
private final SectorRepository sectorRepository;
@@ -66,6 +67,28 @@ public class OfficerService {
return toRows(enterprisesIn(tenant, block));
}
/**
* Which of this caller's borrowers can sign in to the phone app.
*
* <p>Not every borrower is enrolled, and that is the design rather than a gap:
* the platform scores a book from core banking and UPI whether or not the owner
* ever installs anything. A console that offered to open the app for a borrower
* who has none would be promising a screen that does not exist.
*
* <p>Scoped by intersecting with the caller's own book, so this cannot be used
* to discover that an id exists on another bank's book.
*/
public List<String> appEnrolled(TenantContext tenant, String block) {
List<String> mine = enterprisesIn(tenant, block).stream()
.map(Enterprise::getEnterpriseId).toList();
if (mine.isEmpty()) return List.of();
return userRepository.findByScopeIgnoreCaseAndEnterpriseIdIn("ENTERPRISE", mine).stream()
.map(com.lincfusion.udyamlinc.auth.UdyamUser::getEnterpriseId)
.filter(java.util.Objects::nonNull)
.sorted()
.toList();
}
public EnterpriseDetailResponse detail(TenantContext tenant, String enterpriseId) {
Enterprise enterprise = requireEnterprise(tenant, enterpriseId);
StressScore score = scoreRepository.findFirstByEnterpriseIdOrderByComputedAtDesc(enterpriseId).orElse(null);
@@ -75,8 +98,18 @@ public class OfficerService {
sectorCode(enterprise),
districtName(enterprise),
score,
reasonRepository.findByEnterpriseIdOrderByRankAsc(enterpriseId),
pointRepository.findByEnterpriseIdOrderByHorizonAsc(enterpriseId),
// Scoped to the run the score came from. Unscoped, this returned
// every reason ever stored for the enterprise, so tonight's model
// score arrived beside sentences a rule engine wrote months ago —
// one number, two engines explaining it. If the score predates the
// reason table, an empty list is the honest answer.
score == null || score.getRunId() == null
? List.of()
: reasonRepository.findByEnterpriseIdAndRunIdOrderByRankAsc(enterpriseId, score.getRunId()),
// Scoped to the score's run, for the same reason the reason codes are.
score == null || score.getRunId() == null
? List.of()
: pointRepository.findByEnterpriseIdAndRunIdOrderByHorizonAsc(enterpriseId, score.getRunId()),
alertRepository.findByEnterpriseIdOrderByRaisedAtDesc(enterpriseId),
visitRepository.findByEnterpriseIdOrderByDueDateAsc(enterpriseId),
ledgerRepository.findByEnterpriseIdOrderByEntryDateDesc(enterpriseId).stream().limit(60).toList());
@@ -238,6 +271,23 @@ public class OfficerService {
Map<String, List<StressScore>> history = scoreRepository.findAll().stream()
.collect(Collectors.groupingBy(StressScore::getEnterpriseId));
// Lead time and the leading reason, both from the newest run only. Three
// bulk queries rather than two per enterprise — this feeds a list of 263.
Long latestRun = scoreRepository.findAll().stream()
.map(StressScore::getRunId).filter(java.util.Objects::nonNull)
.max(Long::compareTo).orElse(null);
Map<String, ForecastPoint> firstBreach = latestRun == null ? Map.of()
: pointRepository.findByRunIdOrderByHorizonAsc(latestRun).stream()
.filter(p -> Boolean.TRUE.equals(p.getBreach()))
.collect(Collectors.toMap(ForecastPoint::getEnterpriseId, Function.identity(),
(a, b) -> a.getHorizon() <= b.getHorizon() ? a : b));
Map<String, ReasonCode> topReason = latestRun == null ? Map.of()
: reasonRepository.findByRunIdOrderByRankAsc(latestRun).stream()
.collect(Collectors.toMap(ReasonCode::getEnterpriseId, Function.identity(),
(a, b) -> a.getRank() <= b.getRank() ? a : b));
return rows.stream()
.map(e -> new EnterpriseRow(
e.getId(),
@@ -254,7 +304,15 @@ public class OfficerService {
e.getScore(),
e.getBand(),
trendOf(history.get(e.getEnterpriseId()), e.getScore()),
e.getLastScoredAt() == null ? null : e.getLastScoredAt().toString()))
e.getLastScoredAt() == null ? null : e.getLastScoredAt().toString(),
firstBreach.containsKey(e.getEnterpriseId())
? firstBreach.get(e.getEnterpriseId()).getHorizon() : null,
firstBreach.containsKey(e.getEnterpriseId())
? firstBreach.get(e.getEnterpriseId()).getMonth() : null,
topReason.containsKey(e.getEnterpriseId())
? topReason.get(e.getEnterpriseId()).getSentenceEn() : null,
topReason.containsKey(e.getEnterpriseId())
? topReason.get(e.getEnterpriseId()).getSentenceHi() : null))
.toList();
}
@@ -9,4 +9,15 @@ public interface ForecastPointRepository extends JpaRepository<ForecastPoint, Lo
List<ForecastPoint> findByRunIdOrderByHorizonAsc(Long runId);
List<ForecastPoint> findByEnterpriseIdOrderByHorizonAsc(String enterpriseId);
/**
* The path from one run.
*
* <p>The unscoped variant returns every point ever forecast for the
* enterprise — seventeen runs deep on the demo book — so a six-month chart
* drew a hundred points and repeated every month. A forecast is a statement
* made at a moment; reading them all at once is not a longer forecast, it is
* several contradictory ones overlaid.
*/
List<ForecastPoint> findByEnterpriseIdAndRunIdOrderByHorizonAsc(String enterpriseId, Long runId);
}
@@ -0,0 +1,194 @@
package com.lincfusion.udyamlinc.risk;
import com.lincfusion.udyamlinc.enterprise.Enterprise;
import com.lincfusion.udyamlinc.enterprise.LedgerEntry;
import com.lincfusion.udyamlinc.enterprise.LedgerEntryRepository;
import com.lincfusion.udyamlinc.features.FeatureCatalog;
import com.lincfusion.udyamlinc.features.FeatureValue;
import com.lincfusion.udyamlinc.master.Sector;
import com.lincfusion.udyamlinc.ml.MlClient;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.time.LocalDate;
import java.time.YearMonth;
import java.util.*;
/**
* The quantile heads, arriving through the seam built for them.
*
* <p>Selected by {@code udyam.forecast.engine=MODEL}. Answers the half of the
* problem statement the rule engine never could: "AI-Driven <b>Cash Flow
* Prediction</b>" was, until this existed, a median of past months with a band
* widening as sqrt(h) — defensible, explainable, and not a model.
*
* <h2>What crosses the boundary</h2>
*
* <p>The feature map, and five trailing statistics computed here from the
* ledger. The inference service holds no database credentials and cannot read a
* borrower's ledger even if asked — this class already reads it for the
* deterministic path, so the query is one that was being made anyway.
*
* <h2>Fallback</h2>
*
* <p>Unreachable service, or fewer than three months of ledger, and this returns
* an empty list — which {@code RiskEngineService} already treats as "not enough
* to forecast from", leaving {@code dscr_p10} null with {@code NO_FORECAST}. That
* is the honest answer and is visibly different from a forecast saying the
* enterprise is fine.
*/
@Component
@RequiredArgsConstructor
@Slf4j
public class ModelForecastStrategy implements ForecastStrategy {
/** Trailing window the trainer used. Changing it here silently degrades the model. */
private static final int TRAIL_MONTHS = 6;
private static final int MIN_MONTHS = 3;
private static final List<String> SENT = List.of(
FeatureCatalog.FEED_COST_CHANGE,
FeatureCatalog.OUTPUT_PRICE_CHANGE,
FeatureCatalog.MARGIN_SQUEEZE,
FeatureCatalog.PRICE_Z_SCORE,
FeatureCatalog.RAINFALL_ANOMALY,
FeatureCatalog.HEAT_ANOMALY,
FeatureCatalog.CASH_FLOW_CHANGE,
FeatureCatalog.EMI_DELAY,
FeatureCatalog.INVENTORY_DAYS,
FeatureCatalog.FEED_STALENESS_DAYS,
FeatureCatalog.DAYS_SINCE_LAST_ENTRY,
FeatureCatalog.EMI_TO_INFLOW);
private final MlClient client;
private final LedgerEntryRepository ledgerRepository;
private final DeterministicForecastStrategy fallback;
@Override
public String engine() {
return "MODEL";
}
@Override
public String version() {
return "lightgbm-forecast-v1";
}
@Override
public List<ForecastPoint> forecast(Enterprise enterprise, Sector sector,
Map<String, FeatureValue> features, LocalDate asOfDate) {
Trailing trail = trailing(enterprise.getEnterpriseId(), asOfDate);
if (trail.months() < MIN_MONTHS) {
// Not the model's decision to make — the same floor the deterministic
// strategy uses, so switching engines does not change who gets a forecast.
return List.of();
}
Optional<MlClient.MlForecast> answer = client.forecast(request(enterprise, features, trail));
if (answer.isEmpty()) {
log.debug("Forecasting {} deterministically — model unavailable", enterprise.getEnterpriseId());
return fallback.forecast(enterprise, sector, features, asOfDate);
}
BigDecimal obligation = enterprise.getMonthlyInstalment() == null
? BigDecimal.ZERO
: enterprise.getMonthlyInstalment();
List<ForecastPoint> points = new ArrayList<>();
for (MlClient.MlForecastPoint p : answer.get().points()) {
YearMonth month = YearMonth.from(asOfDate).plusMonths(p.horizon());
BigDecimal p10 = money(p.p10());
points.add(ForecastPoint.builder()
.enterpriseId(enterprise.getEnterpriseId())
.horizon(p.horizon())
.month(month.toString())
.p10(p10)
.p50(money(p.p50()))
.p90(money(p.p90()))
.obligation(obligation)
.breach(p10.compareTo(obligation) < 0)
.engine(engine())
.modelVersion(version())
.missingInputs(missingCount(features))
.build());
}
return points;
}
private Map<String, Object> request(Enterprise enterprise, Map<String, FeatureValue> features, Trailing t) {
Map<String, Object> values = new HashMap<>();
for (String code : SENT) {
FeatureValue row = features.get(code);
values.put(code, row == null || row.getValue() == null ? null : row.getValue().doubleValue());
}
Map<String, Object> body = new HashMap<>();
body.put("enterpriseId", enterprise.getEnterpriseId());
body.put("sectorId", enterprise.getSectorId());
body.put("districtId", enterprise.getDistrictId());
body.put("features", values);
body.put("trailMean", t.mean());
body.put("trailSd", t.sd());
body.put("trailMin", t.min());
body.put("trailInflow", t.inflow());
body.put("trailMonths", (double) t.months());
body.put("emi", enterprise.getMonthlyInstalment() == null
? null : enterprise.getMonthlyInstalment().doubleValue());
return body;
}
/**
* Monthly net position over the trailing window, point-in-time.
*
* <p>Whole months only, and never the month {@code asOfDate} falls in — a
* partial month reads as a collapse in income and would teach the model that
* every enterprise fails on the first of the month.
*/
private Trailing trailing(String enterpriseId, LocalDate asOfDate) {
YearMonth current = YearMonth.from(asOfDate);
YearMonth from = current.minusMonths(TRAIL_MONTHS);
Map<YearMonth, double[]> byMonth = new TreeMap<>();
for (LedgerEntry e : ledgerRepository.findByEnterpriseIdOrderByEntryDateDesc(enterpriseId)) {
if (e.getEntryDate() == null || e.getAmount() == null) continue;
YearMonth m = YearMonth.from(e.getEntryDate());
if (!m.isBefore(current) || m.isBefore(from)) continue;
double[] cell = byMonth.computeIfAbsent(m, k -> new double[2]);
double amount = e.getAmount().doubleValue();
if ("IN".equals(e.getKind())) {
cell[0] += amount;
cell[1] += amount;
} else if ("OUT".equals(e.getKind())) {
cell[0] -= amount;
}
}
if (byMonth.isEmpty()) return new Trailing(0, null, null, null, null);
double[] nets = byMonth.values().stream().mapToDouble(c -> c[0]).toArray();
double mean = Arrays.stream(nets).average().orElse(0);
double variance = Arrays.stream(nets).map(n -> (n - mean) * (n - mean)).average().orElse(0);
double inflow = byMonth.values().stream().mapToDouble(c -> c[1]).average().orElse(0);
return new Trailing(nets.length, mean, Math.sqrt(variance),
Arrays.stream(nets).min().orElse(0), inflow);
}
private int missingCount(Map<String, FeatureValue> features) {
int missing = 0;
for (String code : SENT) {
FeatureValue row = features.get(code);
if (row == null || row.getValue() == null) missing++;
}
return missing;
}
private BigDecimal money(double value) {
return BigDecimal.valueOf(value).setScale(2, RoundingMode.HALF_UP);
}
private record Trailing(int months, Double mean, Double sd, Double min, Double inflow) {}
}
@@ -0,0 +1,85 @@
package com.lincfusion.udyamlinc.risk;
import jakarta.persistence.*;
import lombok.*;
import java.math.BigDecimal;
import java.time.LocalDateTime;
/**
* One measured number about a model, and everything needed to read it honestly.
*
* <h2>Why this table exists</h2>
*
* <p>The Reports page used to render <code>0.87</code> AUC from a constant in
* <code>lib/demo/ops.ts</code>. Nothing computed it. A juror asking "how did you
* arrive at that?" would have been shown a literal. Every figure the console
* publishes about model quality now comes from a row here, written by the
* backtest that produced it.
*
* <h2>Provenance travels with the number</h2>
*
* <p>{@code syntheticLabelShare} is not decoration. Every DPD label on this book
* is generated, so a metric computed against it measures the generator as much
* as the borrowers. Storing the share on the same row makes it impossible to
* quote the accuracy without also quoting what it was measured on — the console
* reads both from one record.
*
* <h2>Baselines are rows too</h2>
*
* <p>{@code arm} names what was measured: an algorithm, or a single-feature
* baseline. AUC 0.76 is either a real result or a restatement of "a borrower who
* is late today is late in three months", and only the comparison says which.
* The baselines sit beside the model rather than in a separate table, so a chart
* cannot show one without the other being available.
*/
@Entity
@Table(name = "model_metric", indexes = {
@Index(name = "ix_model_metric_run", columnList = "computedAt"),
@Index(name = "ix_model_metric_arm", columnList = "arm")
})
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class ModelMetric {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
/** The artifact this describes — matches {@code scoring_run.model_version}. */
@Column(nullable = false, length = 64)
private String modelVersion;
/** lightgbm | xgboost | repayment_history_only | obligation_ratio_only */
@Column(nullable = false, length = 64)
private String arm;
/** True for a single-feature comparison arm rather than a trained model. */
@Column(nullable = false)
private boolean baseline;
/** auc_roc | pr_auc | precision_red | recall_red | brier */
@Column(nullable = false, length = 40)
private String metric;
@Column(nullable = false, precision = 10, scale = 5)
private BigDecimal value;
// ── what it was measured on ──────────────────────────────
private Integer datasetRows;
private Integer datasetPositives;
/** 1.0 = every label generated. Read this before reading the value above. */
@Column(precision = 5, scale = 4)
private BigDecimal syntheticLabelShare;
@Column(length = 120)
private String validation;
@Column(nullable = false)
private LocalDateTime computedAt;
}
@@ -0,0 +1,56 @@
package com.lincfusion.udyamlinc.risk;
import com.lincfusion.udyamlinc.common.ApiResponse;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* What the model actually scored, for the Reports page.
*
* <p>Serves the latest backtest only. Metrics from two runs are not comparable
* on one screen — a chart mixing them would show a model beating itself.
*/
@RestController
@RequestMapping("/admin/model")
@RequiredArgsConstructor
public class ModelMetricController {
private final ModelMetricRepository repository;
@GetMapping("/metrics")
public ApiResponse<Map<String, Object>> metrics() {
var latest = repository.findFirstByOrderByComputedAtDesc().orElse(null);
if (latest == null) {
// No backtest has been run. The console shows that in words rather
// than falling back to a number, which is how 0.87 survived so long.
return ApiResponse.ok(Map.of("available", false,
"message", "No backtest has been run against this database yet."));
}
List<ModelMetric> rows = repository.findByComputedAtOrderByArmAscMetricAsc(latest.getComputedAt());
Map<String, Map<String, Object>> byArm = new LinkedHashMap<>();
for (ModelMetric row : rows) {
Map<String, Object> arm = byArm.computeIfAbsent(row.getArm(), k -> new LinkedHashMap<>());
arm.put("baseline", row.isBaseline());
arm.put(row.getMetric(), row.getValue());
}
Map<String, Object> body = new LinkedHashMap<>();
body.put("available", true);
body.put("modelVersion", latest.getModelVersion());
body.put("computedAt", latest.getComputedAt());
body.put("validation", latest.getValidation());
body.put("datasetRows", latest.getDatasetRows());
body.put("datasetPositives", latest.getDatasetPositives());
body.put("syntheticLabelShare", latest.getSyntheticLabelShare());
body.put("arms", byArm);
return ApiResponse.ok(body);
}
}
@@ -0,0 +1,15 @@
package com.lincfusion.udyamlinc.risk;
import org.springframework.data.jpa.repository.JpaRepository;
import java.time.LocalDateTime;
import java.util.List;
import java.util.Optional;
public interface ModelMetricRepository extends JpaRepository<ModelMetric, Long> {
/** The most recent backtest. Metrics from two different runs must not be mixed. */
Optional<ModelMetric> findFirstByOrderByComputedAtDesc();
List<ModelMetric> findByComputedAtOrderByArmAscMetricAsc(LocalDateTime computedAt);
}
@@ -0,0 +1,128 @@
package com.lincfusion.udyamlinc.risk;
import com.lincfusion.udyamlinc.enterprise.Enterprise;
import com.lincfusion.udyamlinc.features.FeatureCatalog;
import com.lincfusion.udyamlinc.features.FeatureValue;
import com.lincfusion.udyamlinc.master.Sector;
import com.lincfusion.udyamlinc.ml.MlClient;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* The gradient-boosted engine, arriving through the seam that was built for it.
*
* <p>Registered as another {@link ScoringStrategy} bean; {@code RiskEngineService}
* resolves it by name from {@code udyam.risk.engine=MODEL}. Nothing downstream
* changes — the score, the sub-scores, the reason codes and the alerts are the
* same records the rule engine produces, because both write {@link ScoreOutcome}.
*
* <h2>Fallback is the point, not a nicety</h2>
*
* <p>If the inference service is unreachable, slow or wrong, this delegates to
* {@link RuleBasedScoringStrategy} for that enterprise and marks the outcome so a
* reader can tell which engine actually produced it. A nightly run must complete;
* a demo must survive the Python process dying mid-presentation. The alternative
* — a run that aborts because a sidecar is down — trades a slightly worse score
* for no score at all.
*
* <h2>What crosses the boundary</h2>
*
* <p>The feature map, the sector and district ids, and three loan-book figures.
* No name, no phone, no account reference — the schema does not hold them, and
* this service must never become the reason someone adds them.
*/
@Component
@RequiredArgsConstructor
@Slf4j
public class ModelScoringStrategy implements ScoringStrategy {
/** The features the model was trained on. Order is the service's business, not ours. */
private static final List<String> SENT = List.of(
FeatureCatalog.FEED_COST_CHANGE,
FeatureCatalog.OUTPUT_PRICE_CHANGE,
FeatureCatalog.MARGIN_SQUEEZE,
FeatureCatalog.PRICE_Z_SCORE,
FeatureCatalog.RAINFALL_ANOMALY,
FeatureCatalog.HEAT_ANOMALY,
FeatureCatalog.CASH_FLOW_CHANGE,
FeatureCatalog.EMI_DELAY,
FeatureCatalog.INVENTORY_DAYS,
FeatureCatalog.UPI_INFLOW_CHANGE,
FeatureCatalog.FEED_STALENESS_DAYS,
FeatureCatalog.DAYS_SINCE_LAST_ENTRY,
FeatureCatalog.EMI_TO_INFLOW);
private final MlClient client;
private final RuleBasedScoringStrategy fallback;
@Override
public String engine() {
return "MODEL";
}
@Override
public String modelVersion() {
return client.modelInfo()
.map(info -> String.valueOf(info.getOrDefault("modelVersion", "model-unknown")))
.orElse("model-unavailable");
}
@Override
public ScoreOutcome score(Enterprise enterprise, Sector sector, Map<String, FeatureValue> features) {
return client.score(request(enterprise, features))
.map(this::toOutcome)
.orElseGet(() -> {
log.debug("Scoring {} on rules — model unavailable", enterprise.getEnterpriseId());
// Attributed to RULE_FALLBACK, not MODEL. The run is a model run;
// this score is not, and the difference has to survive to the table.
return fallback.score(enterprise, sector, features).attributedTo("RULE_FALLBACK");
});
}
private Map<String, Object> request(Enterprise enterprise, Map<String, FeatureValue> features) {
Map<String, Object> values = new HashMap<>();
for (String code : SENT) {
FeatureValue row = features.get(code);
// A null goes across as a null. The model was trained with an explicit
// missing-indicator per column, so "we did not have this" is a state it
// has seen — dropping the key would silently make it a zero.
values.put(code, row == null || row.getValue() == null ? null : row.getValue().doubleValue());
}
Map<String, Object> body = new HashMap<>();
body.put("enterpriseId", enterprise.getEnterpriseId());
body.put("sectorId", enterprise.getSectorId());
body.put("districtId", enterprise.getDistrictId());
body.put("features", values);
// DPD at the as-of date is already in the feature store as emi_delay; reading
// it from there keeps the request to one source of truth.
body.put("dpdAtT", doubleOf(features.get(FeatureCatalog.EMI_DELAY)));
body.put("outstandingAtT", asDouble(enterprise.getLoanOutstanding()));
body.put("emiAtT", asDouble(enterprise.getMonthlyInstalment()));
return body;
}
private ScoreOutcome toOutcome(MlClient.MlScore ml) {
List<ScoreOutcome.Driver> drivers = new ArrayList<>();
for (MlClient.MlDriver d : ml.drivers()) {
drivers.add(new ScoreOutcome.Driver(d.family(), d.weight(), d.featureCode(), d.value(), d.onPrior()));
}
return new ScoreOutcome(
ml.score(), ml.band(), ml.confidence(), ml.subScores(), drivers, ml.familiesOnPrior());
}
private Double doubleOf(FeatureValue row) {
return row == null || row.getValue() == null ? null : row.getValue().doubleValue();
}
private Double asDouble(BigDecimal value) {
return value == null ? null : value.doubleValue();
}
}
@@ -9,4 +9,13 @@ public interface ReasonCodeRepository extends JpaRepository<ReasonCode, Long> {
List<ReasonCode> findByRunIdOrderByRankAsc(Long runId);
List<ReasonCode> findByEnterpriseIdOrderByRankAsc(String enterpriseId);
/**
* The drivers from one run, for one enterprise.
*
* <p>The unscoped variant returns every reason ever stored for the enterprise,
* so a detail screen could show a score from tonight's model run beside
* sentences a rule engine wrote months ago — two engines explaining one number.
*/
List<ReasonCode> findByEnterpriseIdAndRunIdOrderByRankAsc(String enterpriseId, Long runId);
}
@@ -240,7 +240,7 @@ public class RiskEngineService {
.enterpriseId(enterprise.getEnterpriseId())
.score(scored.score())
.band(band)
.engine(strategy.engine())
.engine(scored.engine() != null ? scored.engine() : strategy.engine())
.confidence(scored.confidence())
.liquidity(scored.subScores().get(RuleBasedScoringStrategy.LIQUIDITY))
.behavioural(scored.subScores().get(RuleBasedScoringStrategy.BEHAVIOURAL))
@@ -17,7 +17,31 @@ public record ScoreOutcome(
Map<String, Integer> subScores,
List<Driver> drivers,
/** Families that had no usable feature and fell back to the sector prior. */
int familiesOnPrior) {
int familiesOnPrior,
/**
* What actually produced this score, when that differs from the engine the
* run was configured with.
*
* <p>Null means "the configured engine", which is the normal case. It is
* set when {@link ModelScoringStrategy} cannot reach the inference service
* and answers from the rule engine instead: the run is still a MODEL run,
* but <em>this</em> score is not, and stamping the whole run MODEL would
* be the platform telling itself a provenance lie of exactly the kind the
* simulated/real distinction exists to prevent.
*/
String engine) {
/** Engine unset — the score came from the strategy the run is configured with. */
public ScoreOutcome(int score, String band, int confidence, Map<String, Integer> subScores,
List<Driver> drivers, int familiesOnPrior) {
this(score, band, confidence, subScores, drivers, familiesOnPrior, null);
}
/** The same outcome, attributed to a different engine. */
public ScoreOutcome attributedTo(String actualEngine) {
return new ScoreOutcome(score, band, confidence, subScores, drivers, familiesOnPrior, actualEngine);
}
/**
* One family's contribution to the composite. In the rule engine this is the
@@ -0,0 +1,181 @@
package com.lincfusion.udyamlinc.seed;
import com.lincfusion.udyamlinc.enterprise.Enterprise;
import com.lincfusion.udyamlinc.enterprise.EnterpriseRepository;
import com.lincfusion.udyamlinc.enterprise.LedgerEntry;
import com.lincfusion.udyamlinc.enterprise.LedgerEntryRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
/**
* Extends the demo ledger backwards, so the feature store has something to be
* computed <em>from</em> at a past as-of date.
*
* <h2>Why this exists</h2>
*
* <p>The seeder writes 150 days of ledger per enterprise and refuses to touch an
* enterprise that already has entries — correct, because it protects a real
* ledger from being overwritten by a generator. The consequence is that the
* cash-flow features can only be computed for the last five months, while the
* label in {@code loan_record} runs back two years. A model trained on that
* overlap has a few hundred rows and almost no positives.
*
* <h2>What it is, and what it is not</h2>
*
* <p><b>This is generated data.</b> A ledger is what an owner typed into their
* phone; there is no archive to recover it from, so extending it backwards means
* inventing it. Every row is stamped {@code SEEDED_HISTORY} in {@code note} and
* carries a {@code clientId} prefixed {@code backfill:}, so a generated entry can
* never be mistaken for one an owner made — the same rule
* {@code LoanRecord.source} and the simulated price series already follow.
*
* <p><b>It only ever fills the gap before the earliest existing entry.</b> It
* cannot overwrite, cannot duplicate, and re-running it is a no-op once the
* window is filled. An enterprise with no ledger at all is skipped rather than
* invented from nothing — that is the seeder's job, and an enterprise the seeder
* declined to give a ledger should stay without one.
*
* <p>The generated entries continue the enterprise's own rhythm: the same
* seeded {@link Random} stream keyed by enterprise id, and amounts scaled to the
* median of its existing months, so a backfilled month looks like the enterprise
* rather than like the average of the book. A mild seasonal term is applied
* because the whole premise of the platform is that rural income is seasonal, and
* a flat history would teach a model the opposite.
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class LedgerBackfillService {
/** Marks every row this service writes. Never applied to an owner's entry. */
public static final String SOURCE_NOTE = "SEEDED_HISTORY";
private static final String CLIENT_PREFIX = "backfill:";
private final EnterpriseRepository enterpriseRepository;
private final LedgerEntryRepository ledgerRepository;
public record Result(int enterprises, int skipped, int entries, LocalDate earliest, String message) {}
/**
* @param targetDays how far back the ledger should reach from today
*/
@Transactional
public Result backfill(int targetDays) {
LocalDate target = LocalDate.now().minusDays(targetDays);
List<Enterprise> all = enterpriseRepository.findAll();
int touched = 0, skipped = 0, written = 0;
LocalDate earliest = null;
for (Enterprise e : all) {
List<LedgerEntry> existing =
ledgerRepository.findByEnterpriseIdOrderByEntryDateDesc(e.getEnterpriseId());
// No ledger at all is the seeder's business, not ours.
if (existing.isEmpty()) {
skipped++;
continue;
}
LocalDate oldest = existing.get(existing.size() - 1).getEntryDate();
if (!oldest.isAfter(target)) {
skipped++;
continue;
}
List<LedgerEntry> generated = generate(e, target, oldest.minusDays(1), existing);
if (generated.isEmpty()) {
skipped++;
continue;
}
ledgerRepository.saveAll(generated);
touched++;
written += generated.size();
if (earliest == null || target.isBefore(earliest)) earliest = target;
}
String message = "%d enterprises extended to %s, %d entries written, %d skipped"
.formatted(touched, target, written, skipped);
log.info("Ledger backfill — {}", message);
return new Result(touched, skipped, written, earliest, message);
}
/**
* Entries for {@code [from, to]}, scaled to what this enterprise already
* trades rather than to a flat range.
*/
private List<LedgerEntry> generate(Enterprise e, LocalDate from, LocalDate to, List<LedgerEntry> existing) {
if (from.isAfter(to)) return List.of();
BigDecimal inScale = medianAmount(existing, "IN");
BigDecimal outScale = medianAmount(existing, "OUT");
if (inScale.signum() <= 0) return List.of();
if (outScale.signum() <= 0) outScale = inScale.multiply(BigDecimal.valueOf(0.7));
// Keyed on the enterprise so a re-run after a wipe reproduces the same
// history — a training set that changes between runs is not a baseline.
Random r = new Random(("backfill:" + e.getEnterpriseId()).hashCode());
LocalDateTime now = LocalDateTime.now();
String[] inCats = {"SALE", "UPI", "RECOVERY"};
String[] outCats = {"INPUT", "WAGES", "TRANSPORT", "EMI"};
List<LedgerEntry> out = new ArrayList<>();
for (LocalDate d = from; !d.isAfter(to); d = d.plusDays(1)) {
// Rural trade is seasonal; a flat history would teach the model that
// it is not, which is the opposite of the thing being predicted.
double season = 1 + 0.22 * Math.sin(2 * Math.PI * (d.getDayOfYear() / 365.0));
if (r.nextDouble() < 0.72) {
out.add(row(e, d, "IN", inCats[r.nextInt(inCats.length)],
jitter(inScale, season, r), now, out.size()));
}
if (r.nextDouble() < 0.58) {
out.add(row(e, d, "OUT", outCats[r.nextInt(outCats.length)],
jitter(outScale, 2 - season, r), now, out.size()));
}
}
return out;
}
private BigDecimal jitter(BigDecimal scale, double season, Random r) {
double factor = season * (0.6 + r.nextDouble() * 0.8);
return scale.multiply(BigDecimal.valueOf(factor)).setScale(0, RoundingMode.HALF_UP);
}
private LedgerEntry row(Enterprise e, LocalDate date, String kind, String category,
BigDecimal amount, LocalDateTime now, int seq) {
return LedgerEntry.builder()
.clientId(CLIENT_PREFIX + e.getEnterpriseId() + ":" + date + ":" + seq)
.enterpriseId(e.getEnterpriseId())
.kind(kind)
.amount(amount)
.category(category)
.note(SOURCE_NOTE)
.entryDate(date)
.syncedAt(now)
.createdAt(now)
.build();
}
private BigDecimal medianAmount(List<LedgerEntry> entries, String kind) {
List<BigDecimal> amounts = entries.stream()
.filter(x -> kind.equals(x.getKind()))
.map(LedgerEntry::getAmount)
.filter(java.util.Objects::nonNull)
.sorted()
.toList();
if (amounts.isEmpty()) return BigDecimal.ZERO;
return amounts.get(amounts.size() / 2);
}
}
+14 -2
View File
@@ -34,8 +34,11 @@ jwt.secret=${UDYAM_JWT_SECRET:55647961_6D4C_696E_6320_4E41_4241_5244_5F32_3032_3
# 24 hours
jwt.expiration=86400000
# The Vite dev server for UdyamLinc-UI (5180 is BharatLinc)
udyam.cors.allowed-origins=http://localhost:5182,http://127.0.0.1:5182,http://localhost:4173
# The Vite dev server for UdyamLinc-UI (5180 is BharatLinc). A deployment must
# add its own UI origin via UDYAM_CORS_ALLOWED_ORIGINS — this list is exhaustive,
# so an origin missing from it fails in the browser as an opaque network error
# with a healthy-looking backend behind it.
udyam.cors.allowed-origins=${UDYAM_CORS_ALLOWED_ORIGINS:http://localhost:5182,http://127.0.0.1:5182,http://localhost:4173}
# Forecast horizon in months — the deck's 3–6 month early-warning window
udyam.forecast.horizon-months=6
@@ -192,3 +195,12 @@ udyam.forecast.trust-seeded=${UDYAM_TRUST_SEEDED_FORECASTS:false}
# to the enterprise — at the 0.75 ceiling P10 is a quarter of P50 for everyone,
# so a worst-of-six rule separates nobody from anybody.
udyam.forecast.tripwire-horizon=3
# ── ML inference service ───────────────────────────────────────────
# Off by default: a checkout with no Python running must still boot and score.
# Turn both on together — udyam.risk.engine=MODEL without udyam.ml.enabled=true
# would resolve the model strategy and then fall back on every enterprise.
udyam.ml.enabled=${UDYAM_ML_ENABLED:false}
udyam.ml.base-url=${UDYAM_ML_BASE_URL:http://127.0.0.1:8000}
udyam.ml.timeout-ms=2000
udyam.ml.batch-timeout-ms=30000