udyamlinc-api/ARCHITECTURE.md

UdyamLinc — Technical Architecture

Team LincFusion · LincPay Solutions Pvt. Ltd. · NABARD Hackathon

This document is the build contract for UdyamLinc-Backend. It records what the system is, what is built, and what is deliberately deferred.

---

1. Shape of the system

                    Enterprise PWA / Field Officer Console
                                     │
                                     ▼
                       Spring Boot application (one deployable)
                                     │
   ┌──────────────┬──────────────────┼──────────────────┬──────────────┐
   ▼              ▼                  ▼                  ▼              ▼
enterprise   institutional       ingestion            rules           ai
(ledger)     (CBS, DPD, UPI)   (mandi, climate)    (trip-wires)   (LightGBM)
   │              │                  │                  │              │
   └──────────────┴────────┬─────────┴──────────────────┴──────────────┘
                           ▼
                    feature store  (tall = truth, wide = read)
                           │
                           ▼
                      risk engine  (rule-based now, model later)
                           │
                           ▼
              alerts · dashboard · reports · officer playbook

One deployable, not seven services

The module boundaries above are Java packages, not microservices. At pilot scale — four districts, 142 enterprises — separate deployables would buy distributed transactions, cross-service auth and five release pipelines in exchange for nothing. The package boundaries are enforced by convention (no cross-package entity imports; talk to a neighbouring module through its service), so ingestion or ai can be lifted out later when volume actually justifies it.

For the same reason there is no separate API gateway component: at this scale the application is the gateway.

---

2. Three principles that decide whether this survives real data

2.1 External feeds are never called in the request path

Mandi prices, egg rates and climate series are pulled by a scheduled job at 02:00, validated, and written to market_signal / climate_signal. Scoring and login read those tables and nothing else. A government endpoint being down degrades tomorrow's freshness; it never fails a user request.

2.2 The feature store is the only interface to the model

Ingestion writes signals. Feature engineering turns signals into indicators. The AI layer consumes only engineered features. This is what allows the model to be retrained or replaced without touching ingestion, storage or the API.

The store has two shapes:

Table	Shape	Purpose
feature_value	tall — one row per (enterprise, feature, date)	System of record. Auditable, schema-free as features are added.
feature_snapshot	wide — one row per (enterprise, scoring run)	What the engine and the trainer actually read.

Tall alone would mean ~5,700 row reads per scoring pass and far worse for training. Wide alone would mean a migration every time a feature is added.

2.3 Point-in-time correctness

Every feature row carries two timestamps:

• valid_for_date — the date the fact describes
• observed_at — when we learned it

A mandi price for 3 March scraped on 7 March must be invisible to a 4 March scoring run. Without this rule, backtests look excellent and production does not reproduce them. This is what makes the claim of "no forward leakage across folds" true rather than aspirational.

---

3. Data model

Master

Table	Notes
district	The join key for every external signal.
market	A mandi, belonging to a district.
commodity	Maize, soya, egg, milk, wheat…
sector	The five risk packs, carrying sub-score weights in basis points.
sector_commodity	Which commodities a sector consumes vs sells. The input/output distinction is what makes margin_squeeze computable.
role	RBAC for the console.

Enterprise & institutional

Table	Notes
enterprise	Opaque id, sector, district, lat/long. No owner name, phone, Aadhaar or account number — the no-PII promise is enforced by schema, not policy.
ledger_entry	Three-tap entries. client_id is generated on the phone so an offline entry keeps identity through sync.
loan_record	Per loan: sanctioned, outstanding, EMI, due date, DPD, bucket. DPD history is the training label.
upi_summary	Consented transaction aggregates. Source pluggable — Account Aggregator (needs FIU registration under Sahamati), PSP settlement reports, or manual.

Signals & features

Table	Notes
market_signal	(district, commodity, week) — modal/min/max price. ~40 rows a day for four districts, not 142 per-enterprise fetches.
climate_signal	(district, week) — rainfall, tmax, anomalies vs 30-year normal.
feature_definition	Feature catalogue: code, family, unit, staleness tolerance.
feature_value	Tall store, with valid_for_date + observed_at.
feature_snapshot	Wide store, one per enterprise per run.

Scoring & decision

Table	Notes
scoring_run	One pass. Carries model_version and source (SERVER | ON_DEVICE) so a historic alert traces to the exact engine that raised it.
forecast_point	P10 / P50 / P90 and the obligation due, per month, h = 1…6.
stress_score	Composite 0–100 plus the five stored sub-scores, engine, confidence.
reason_code	SHAP driver as a sentence in both languages. Stored, not regenerated, so the wording an officer acted on is preserved.
tripwire_rule	Configurable rule: SpEL expression over the feature snapshot.
risk_alert	Raised by the model or by a rule. References the rule and the snapshot, so it is reproducible.
field_visit	Playbook intervention and its outcome — the feedback edge of the loop.

Audit

api_log (outbound calls to external feeds), ingestion_run (scheduler results, counts, failures), and scoring_run doubling as the model prediction log.

---

4. External data — provenance

Signal	Source	Access	Cadence
Mandi prices	Agmarknet, via the data.gov.in Open Government Data API	API key, JSON	Daily
Egg realisation	NECC (e2necc.com)	No API — scrape the daily city rate table	Daily
Rainfall, temperature	NASA POWER or Open-Meteo	Free, no key, long history for anomalies	Daily
District advisories	IMD Agromet	District-level advisory text	Tue / Fri
Milk procurement rate	State dairy federation	No API anywhere — MoU or admin entry	Monthly
Institutional extract	Lender CBS / LMS	SFTP CSV or read-only DB view	Nightly
UPI aggregates	Account Aggregator (RBI / Sahamati)	FIU registration + consent artefact	Weekly

IMD supplies the advisory; NASA POWER or Open-Meteo supplies the numbers, because anomalies need decades of history that live endpoints do not serve.

Feature derivations

Raw prices are not features. An absolute maize price of ₹2,140 means nothing.

Feature	Derivation
feed_cost_change	modal price vs 8-week trailing mean
output_price_change	same, on the sale-side commodity
margin_squeeze	input change − output change
price_z_score	(current − 3yr seasonal mean) / stdev
rainfall_anomaly	season-to-date vs 30yr normal
heat_anomaly	days above threshold vs normal
cash_flow_change	ledger net position vs trailing mean
emi_delay	DPD from loan_record
inventory_days	stock entries vs sales velocity
upi_inflow_change	8-week rolling inflow delta
feed_staleness_days	age of the newest observation — guards all of the above

Degradation rules

• Carry the last known value forward, stamped with feed_staleness_days.
• Trip-wires refuse to fire on stale data. A rule firing on a three-week-old price is worse than no rule.
• Missing shock features fall back to the sector's seasonal prior. The forecast widens — P10 drops — which is the honest response to knowing less.
• Scoring never blocks on an external feed. The institutional record and the ledger are always present; market data is an enhancement.

---

5. Risk and rules

The risk engine is rule-based first: a sector-weighted combination of the five sub-score families, with weights held in the sector table so tuning needs no deploy. It sits behind a ScoringStrategy interface, so the LightGBM engine replaces it as a config change.

The rule engine is deliberately independent. tripwire_rule rows carry a SpEL expression evaluated against the feature snapshot:

emi_delay > 30 and cash_flow_change < -0.40

Business teams edit rows; nobody deploys. SpEL rather than Drools — seven rules do not justify a rules platform.

A trip-wire alert survives a model retrain. That is the entire point: every alert stays auditable and explainable independently of whatever the model currently believes.

---

6. Build status

Area	State
Field officer console — 8 screens	✅ Built
Enterprise PWA — 4 tabs, offline, Hindi-first	✅ Built
Ledger + offline queue + sync	✅ Built
Auth, JWT, two scoped surfaces	✅ Built
Serving entities, repositories, services	✅ Built
Master data, controllers, seeder	◐ Phase A
Ingestion (Agmarknet, climate, scheduler)	✅ Phase B — see §6.1
Feature store + derivations	✅ Phase C — see §6.2
Risk engine + rule engine + admin API	✅ Phase D — see §6.3
Console wire-up: rules screen, market panel, freshness	✅ Phase E — see §6.4
Core banking / LMS connector seam	✅ Phase F — see §6.5
LightGBM quantile heads + SHAP	○ R1 milestone

Phases B, C and D each terminate in a table you can inspect, so each is testable without any of the machine learning existing.

6.1 What Phase B delivered

Package com.lincfusion.udyamlinc.ingestion.

Piece	Notes
DataGovClient	Agmarknet via data.gov.in. Case-insensitive field reads, because the resource's own casing changes between versions. Returns null on a failed call and an empty list on a filter that matched nothing — an outage and a typo must not look alike.
OpenMeteoClient	Reanalysis archive for the district centroid. No key.
MandiIngestionService	Folds daily arrivals into one market_signal per (district, commodity, ISO week). Upserts, so re-running corrects a partial week.
ClimateIngestionService	Weekly climate_signal with the 30-year normal alongside, and climate_normal computed once per district from a single baseline request.
IngestionScheduler	The 02:00 cron. Off by default; one pass at a time; every outcome — including a skip — lands in ingestion_run.
IngestionController	/admin/ingestion/{runs,calls,signals/market,signals/climate,freshness}, plus POST /run and /run/{source}. Reads open to any officer, triggers supervisor-only under the existing /admin/** rule.

Both signal tables carry valid_for_date + observed_at, and each repository exposes a findTrailing(...) that filters on both — the point-in-time window Phase C's derivations will read. /admin/ingestion/freshness is the staleness number Phase D's trip-wires must consult before firing.

Configuration. udyam.ingestion.enabled is false by default so a demo boot never calls out. udyam.ingestion.datagov.api-key (env DATAGOV_API_KEY) is required for Agmarknet; without it the run records SKIPPED rather than failing. Climate needs no key.

Still deferred from §4. NECC egg rates (no API — needs an HTML scrape), IMD Agromet advisories, and the dairy federation rate (no API anywhere). Their commodities are seeded and their source values reserved; nothing pulls them yet.

6.2 What Phase C delivered

Package com.lincfusion.udyamlinc.features, plus the two institutional tables of §3 the derivations read from (loan_record, upi_summary).

Piece	Notes
feature_definition	The catalogue: family, unit, source kind, direction, and staleness_tolerance_days — the column a trip-wire consults before it is allowed to fire. Seeded idempotently at boot.
feature_value	The tall store. A feature that cannot be computed is written as null with a missing_reason, never skipped — NO_SIGNAL, NO_LEDGER, NO_CBS, NO_CONSENT, NO_FORECAST, INSUFFICIENT_HISTORY. "We did not know" and "we never looked" are different facts.
feature_snapshot	The wide store, one per (enterprise, as-of date), carrying feature_count / missing_count / max_staleness_days so a snapshot mostly made of holes cannot read as confident. asFeatureMap() exposes the same values under their snake_case codes — the shape a SpEL rule is written against.
FeatureEngineeringService	All fourteen derivations. Every read is point-in-time on both valid_for_date and observed_at, so recomputing last month uses only what was knowable last month.
FeatureController	/admin/features/{definitions,snapshot/{id},values/{id},coverage} and POST /compute, POST /compute/{id}. coverage reports the populated share per feature — a feature at 4 % coverage is a column, not a feature.

Catalogue. §4's eleven, plus three the seeded trip-wires already named: dscr_p10 (worst month of min(p10 / obligation) over the latest run), days_since_last_entry, and emi_to_inflow. A rule referring to a feature nobody computes is a rule that silently never fires. These three also give the OBLIGATION_LOAD family the inputs it otherwise lacked.

Demo inputs. The seeder now writes one loan_record per enterprise with DPD tracking the band, so emi_delay and emi_to_inflow are real without a CBS feed. upi_summary has an entity and a point-in-time query but no source until Account Aggregator registration — upi_inflow_change therefore resolves to null with NO_CONSENT, which is the honest answer and exercises the degradation path. Price and climate features stay null until Phase B has actually run.

6.3 What Phase D delivered

Packages rules and risk.

Piece	Notes
RuleEvaluator	SpEL over the feature rows, in a SimpleEvaluationContext restricted to property access — these expressions are authored by business users, so no method calls or bean lookups. Compiled expressions cached by rule code and version, so an edit invalidates its own entry; the admin write endpoints also clear the cache.
RuleOutcome	Every rule returns FIRED / NOT_FIRED / SKIPPED_STALE / SKIPPED_MISSING / SKIPPED_SECTOR / ERROR. "No alert because the enterprise is fine" and "no alert because the feed has been down a fortnight" must never look the same.
ScoringStrategy	The seam the model arrives through. udyam.risk.engine binds the implementation; LightGBM later is that one line. Implementations read features and nothing else — no reaching back into the signal tables.
RuleBasedScoringStrategy	Five families onto 0–100, combined with the sector's basis-point weights. Each family takes its worst signal, not the mean — averaging lets a calm dimension mask an acute one, and the acute one is where the officer needs to go.
RiskEngineService	Compute features → score → evaluate trip-wires against the same feature rows, never the score → persist score, top-3 reason codes, alerts.
RiskEngineController	POST /admin/scoring/run, /run/{id}, GET /admin/scoring/runs, and GET /admin/scoring/explain/{id} — every rule's outcome with no writes.
ScoringScheduler	03:00, an hour behind ingestion so it scores what that job wrote. Off by default.

The staleness gate. Before any expression runs, each of a rule's requiredFeatures must have a value and be within the stricter of the rule's maxStalenessDays and the catalogue's per-feature tolerance — a supervisor tightening a rule must not be overridden by a generous default. This is the promise of §4's degradation rules, enforced rather than documented.

Band versus score. A fired trip-wire escalates the band; it never rewrites the score. They answer different questions — "how stressed is this enterprise" and "has a specific agreed line been crossed" — and an audit needs both. A family with no usable feature falls back to udyam.risk.family-prior rather than scoring zero, because zero reads as "safe here" when the truth is "unknown", and each family on a prior costs confidence.

What the seeded rules can actually do today. TW-01 (dscr_p10) and TW-03 (days_since_last_entry) evaluate from seeded data. TW-04 (emi_to_inflow) evaluates once loan records exist, which the seeder now writes. TW-02 (upi_inflow_change) and TW-05 (price_z_score, heat_anomaly) will report SKIPPED_MISSING until Account Aggregator consent and Phase B ingestion respectively — visibly skipped, which is the intended behaviour rather than a silent no-op.

Not built here. The quantile forecast is still the seeded path; this engine scores and alerts but does not produce forecast_point rows. dscr_p10 reads whatever the latest run stored. Generating forecasts is part of the R1 model milestone.

6.4 What Phase E delivered

In UdyamLinc-UI, two console screens and one piece of chrome — all three about provenance, because the failure mode that matters is silent. A score computed from a three-week-old mandi price looks exactly like one computed this morning.

Piece	Notes
features/officer/DataFeedsPage.tsx	/officer/feeds — freshness per district first, then the mandi panel, the rainfall/heat panel with the 30-year normal beside each observation, and the ingestion run log. A skipped run appears as a row, so "nothing ran last night" never reads as "it ran and found nothing".
features/officer/RulesPage.tsx	/officer/rules — every trip-wire with its expression and tolerance, its outcome for a chosen enterprise, and the feature snapshot underneath with each null's reason. A banner counts rules that could not run.
FreshnessChip in OfficerShell	Data age in the header on every screen, amber past ten days, linking to the feeds page.
lib/demo/feeds.ts	Demo fallback that applies the same two guards in the same order as RuleEvaluator — missing, then stale, then the expression — so the offline demo cannot claim a rule fired that the real engine would have skipped.

Both screens follow the house pattern: useQuery against the real endpoint, in-file demo fallback, and a "Demo data" pill when the backend is unreachable. npm run build and oxlint are clean.

Deliberately not added: rule editing from the console. The backend accepts POST/PUT /admin/rules and bumps the version on every edit, but a supervisor changing a live threshold from a demo build with no audit trail in front of it is a worse idea than the round trip through an API client.

---

6.5 What Phase F delivered

Package com.lincfusion.udyamlinc.connector, plus /officer/integration.

The problem it answers. The loan book is the training label and the one input no public feed supplies. Integrating with a lender's core is not a technical unknown — it is an onboarding process per bank, with a schema agreed in a room. There is no single integration to build ahead of that room. What can be built ahead of it is the shape of the seam, and that is what this is.

Piece	Notes
CoreBankingConnector	Fetches and returns rows. Never writes — an adapter that could write would be an adapter that could bypass the no-PII rule.
FileDropConnector	The one that is real. Reads the newest matching file from a drop directory a bank delivers to. Leaves the file where it found it, so a failed run is repeatable.
SimulatedCoreConnector × 4	Finacle, FLEXCUBE, BaNCS and an in-house REST LMS, each emitting the same portfolio in its own column dialect. Rows derived from real enterprises and loan records, drift seeded from the account ref so an extract is reproducible.
FieldMapper	{canonical_column: vendor_column}. Validated before it is stored, not at the next nightly run.
ConnectorSyncService	Fetch → map → LoanBookEntryService.stage → run record.
ConnectorProfile, ConnectorSyncRun	Per-bank configuration and per-attempt audit.
features/officer/IntegrationPage.tsx	Connections, field mapping, sync history. Simulated adapters badged everywhere they appear.

Three decisions worth keeping. The mapped rows go through the validator the manual upload already had rather than a second one written for the automated path — the obvious shape would be a parallel validator, and it would eventually enforce the no-PII rule differently. Unmapped vendor columns are dropped because the map is keyed by canonical name, which makes the promise structural rather than remembered. And simulated is stamped per run rather than read from the profile, so switching a profile to a live adapter cannot retroactively make its old runs look real.

One real defect this surfaced. FORBIDDEN_COLUMNS was matched by exact normalised name, so CUST_NAME — what Finacle actually calls it — was not caught, and a directly uploaded bank file would have kept the borrower's name in manual_entry_line.raw_row. Matching now also splits separators and camelCase. That fix belongs to the manual upload path as much as to the connector.

Still not built: SFTP polling with credentials (the drop is assumed mounted or synced by something the bank already runs), scheduled execution (scheduleCron is recorded and read by nothing), and incremental watermarks.

---

7. Explicitly not being built yet

Kafka, Neo4j, Elasticsearch, Keycloak, MinIO, and separate deployables. All appear on the submission's technology slide; none earn their keep at four districts. Postgres plus a scheduled job carries the pilot comfortably. Kafka arrives when ingestion volume or a second consumer demands it — not before.