udyamlinc-api/ARCHITECTURE.md

# UdyamLinc — Technical Architecture

Team LincFusion · LincPay Solutions Pvt. Ltd. · NABARD Hackathon

This document is the build contract for `UdyamLinc-Backend`. It records what the
system is, what is built, and what is deliberately deferred.

---

## 1. Shape of the system

```
                    Enterprise PWA / Field Officer Console
                                     │
                                     ▼
                       Spring Boot application (one deployable)
                                     │
   ┌──────────────┬──────────────────┼──────────────────┬──────────────┐
   ▼              ▼                  ▼                  ▼              ▼
enterprise   institutional       ingestion            rules           ai
(ledger)     (CBS, DPD, UPI)   (mandi, climate)    (trip-wires)   (LightGBM)
   │              │                  │                  │              │
   └──────────────┴────────┬─────────┴──────────────────┴──────────────┘
                           ▼
                    feature store  (tall = truth, wide = read)
                           │
                           ▼
                      risk engine  (rule-based now, model later)
                           │
                           ▼
              alerts · dashboard · reports · officer playbook
```

### One deployable, not seven services

The module boundaries above are **Java packages, not microservices**. At pilot
scale — four districts, 142 enterprises — separate deployables would buy
distributed transactions, cross-service auth and five release pipelines in
exchange for nothing. The package boundaries are enforced by convention (no
cross-package entity imports; talk to a neighbouring module through its service),
so `ingestion` or `ai` can be lifted out later when volume actually justifies it.

For the same reason there is no separate API gateway component: at this scale the
application *is* the gateway.

---

## 2. Three principles that decide whether this survives real data

### 2.1 External feeds are never called in the request path

Mandi prices, egg rates and climate series are pulled by a scheduled job at
02:00, validated, and written to `market_signal` / `climate_signal`. Scoring and
login read those tables and nothing else. A government endpoint being down
degrades tomorrow's freshness; it never fails a user request.

### 2.2 The feature store is the only interface to the model

Ingestion writes signals. Feature engineering turns signals into indicators. The
AI layer consumes **only** engineered features. This is what allows the model to
be retrained or replaced without touching ingestion, storage or the API.

The store has two shapes:

| Table | Shape | Purpose |
|---|---|---|
| `feature_value` | tall — one row per (enterprise, feature, date) | System of record. Auditable, schema-free as features are added. |
| `feature_snapshot` | wide — one row per (enterprise, scoring run) | What the engine and the trainer actually read. |

Tall alone would mean ~5,700 row reads per scoring pass and far worse for
training. Wide alone would mean a migration every time a feature is added.

### 2.3 Point-in-time correctness

Every feature row carries **two** timestamps:

- `valid_for_date` — the date the fact describes
- `observed_at` — when we learned it

A mandi price for 3 March scraped on 7 March must be invisible to a 4 March
scoring run. Without this rule, backtests look excellent and production does not
reproduce them. This is what makes the claim of "no forward leakage across folds"
true rather than aspirational.

---

## 3. Data model

### Master

| Table | Notes |
|---|---|
| `district` | The join key for every external signal. |
| `market` | A mandi, belonging to a district. |
| `commodity` | Maize, soya, egg, milk, wheat… |
| `sector` | The five risk packs, carrying sub-score weights in basis points. |
| `sector_commodity` | Which commodities a sector consumes vs sells. The **input/output** distinction is what makes `margin_squeeze` computable. |
| `role` | RBAC for the console. |

### Enterprise & institutional

| Table | Notes |
|---|---|
| `enterprise` | Opaque id, sector, district, lat/long. No owner name, phone, Aadhaar or account number — the no-PII promise is enforced by schema, not policy. |
| `ledger_entry` | Three-tap entries. `client_id` is generated on the phone so an offline entry keeps identity through sync. |
| `loan_record` | Per loan: sanctioned, outstanding, EMI, due date, **DPD**, bucket. DPD history is the training label. |
| `upi_summary` | Consented transaction aggregates. Source pluggable — Account Aggregator (needs FIU registration under Sahamati), PSP settlement reports, or manual. |

### Signals & features

| Table | Notes |
|---|---|
| `market_signal` | `(district, commodity, week)` — modal/min/max price. ~40 rows a day for four districts, not 142 per-enterprise fetches. |
| `climate_signal` | `(district, week)` — rainfall, tmax, anomalies vs 30-year normal. |
| `feature_definition` | Feature catalogue: code, family, unit, staleness tolerance. |
| `feature_value` | Tall store, with `valid_for_date` + `observed_at`. |
| `feature_snapshot` | Wide store, one per enterprise per run. |

### Scoring & decision

| Table | Notes |
|---|---|
| `scoring_run` | One pass. Carries `model_version` and `source` (`SERVER` \| `ON_DEVICE`) so a historic alert traces to the exact engine that raised it. |
| `forecast_point` | P10 / P50 / P90 and the obligation due, per month, h = 1…6. |
| `stress_score` | Composite 0–100 plus the five stored sub-scores, `engine`, `confidence`. |
| `reason_code` | SHAP driver as a sentence in both languages. Stored, not regenerated, so the wording an officer acted on is preserved. |
| `tripwire_rule` | Configurable rule: SpEL expression over the feature snapshot. |
| `risk_alert` | Raised by the model or by a rule. References the rule and the snapshot, so it is reproducible. |
| `field_visit` | Playbook intervention and its outcome — the feedback edge of the loop. |

### Audit

`api_log` (outbound calls to external feeds), `ingestion_run` (scheduler results,
counts, failures), and `scoring_run` doubling as the model prediction log.

---

## 4. External data — provenance

| Signal | Source | Access | Cadence |
|---|---|---|---|
| Mandi prices | Agmarknet, via the data.gov.in Open Government Data API | API key, JSON | Daily |
| Egg realisation | NECC (e2necc.com) | No API — scrape the daily city rate table | Daily |
| Rainfall, temperature | NASA POWER or Open-Meteo | Free, no key, long history for anomalies | Daily |
| District advisories | IMD Agromet | District-level advisory text | Tue / Fri |
| Milk procurement rate | State dairy federation | No API anywhere — MoU or admin entry | Monthly |
| Institutional extract | Lender CBS / LMS | SFTP CSV or read-only DB view | Nightly |
| UPI aggregates | Account Aggregator (RBI / Sahamati) | FIU registration + consent artefact | Weekly |

IMD supplies the advisory; NASA POWER or Open-Meteo supplies the numbers, because
anomalies need decades of history that live endpoints do not serve.

### Feature derivations

Raw prices are not features. An absolute maize price of ₹2,140 means nothing.

| Feature | Derivation |
|---|---|
| `feed_cost_change` | modal price vs 8-week trailing mean |
| `output_price_change` | same, on the sale-side commodity |
| `margin_squeeze` | input change − output change |
| `price_z_score` | (current − 3yr seasonal mean) / stdev |
| `rainfall_anomaly` | season-to-date vs 30yr normal |
| `heat_anomaly` | days above threshold vs normal |
| `cash_flow_change` | ledger net position vs trailing mean |
| `emi_delay` | DPD from `loan_record` |
| `inventory_days` | stock entries vs sales velocity |
| `upi_inflow_change` | 8-week rolling inflow delta |
| `feed_staleness_days` | age of the newest observation — guards all of the above |

### Degradation rules

- Carry the last known value forward, stamped with `feed_staleness_days`.
- **Trip-wires refuse to fire on stale data.** A rule firing on a three-week-old
  price is worse than no rule.
- Missing shock features fall back to the sector's seasonal prior. The forecast
  widens — P10 drops — which is the honest response to knowing less.
- Scoring never blocks on an external feed. The institutional record and the
  ledger are always present; market data is an enhancement.

---

## 5. Risk and rules

The **risk engine** is rule-based first: a sector-weighted combination of the five
sub-score families, with weights held in the `sector` table so tuning needs no
deploy. It sits behind a `ScoringStrategy` interface, so the LightGBM engine
replaces it as a config change.

The **rule engine** is deliberately independent. `tripwire_rule` rows carry a SpEL
expression evaluated against the feature snapshot:

```
emi_delay > 30 and cash_flow_change < -0.40
```

Business teams edit rows; nobody deploys. SpEL rather than Drools — seven rules do
not justify a rules platform.

A trip-wire alert survives a model retrain. That is the entire point: every alert
stays auditable and explainable independently of whatever the model currently
believes.

---

## 6. Build status

| Area | State |
|---|---|
| Field officer console — 8 screens | ✅ Built |
| Enterprise PWA — 4 tabs, offline, Hindi-first | ✅ Built |
| Ledger + offline queue + sync | ✅ Built |
| Auth, JWT, two scoped surfaces | ✅ Built |
| Serving entities, repositories, services | ✅ Built |
| Master data, controllers, seeder | ◐ Phase A |
| Ingestion (Agmarknet, climate, scheduler) | ✅ Phase B — see §6.1 |
| Feature store + derivations | ✅ Phase C — see §6.2 |
| Risk engine + rule engine + admin API | ✅ Phase D — see §6.3 |
| Console wire-up: rules screen, market panel, freshness | ✅ Phase E — see §6.4 |
| LightGBM quantile heads + SHAP | ○ R1 milestone |

Phases B, C and D each terminate in a table you can inspect, so each is testable
without any of the machine learning existing.

### 6.1 What Phase B delivered

Package `com.lincfusion.udyamlinc.ingestion`.

| Piece | Notes |
|---|---|
| `DataGovClient` | Agmarknet via data.gov.in. Case-insensitive field reads, because the resource's own casing changes between versions. Returns `null` on a failed call and an empty list on a filter that matched nothing — an outage and a typo must not look alike. |
| `OpenMeteoClient` | Reanalysis archive for the district centroid. No key. |
| `MandiIngestionService` | Folds daily arrivals into one `market_signal` per (district, commodity, ISO week). Upserts, so re-running corrects a partial week. |
| `ClimateIngestionService` | Weekly `climate_signal` with the 30-year normal alongside, and `climate_normal` computed once per district from a single baseline request. |
| `IngestionScheduler` | The 02:00 cron. Off by default; one pass at a time; every outcome — including a skip — lands in `ingestion_run`. |
| `IngestionController` | `/admin/ingestion/{runs,calls,signals/market,signals/climate,freshness}`, plus `POST /run` and `/run/{source}`. Reads open to any officer, triggers supervisor-only under the existing `/admin/**` rule. |

Both signal tables carry `valid_for_date` + `observed_at`, and each repository
exposes a `findTrailing(...)` that filters on both — the point-in-time window
Phase C's derivations will read. `/admin/ingestion/freshness` is the staleness
number Phase D's trip-wires must consult before firing.

**Configuration.** `udyam.ingestion.enabled` is `false` by default so a demo boot
never calls out. `udyam.ingestion.datagov.api-key` (env `DATAGOV_API_KEY`) is
required for Agmarknet; without it the run records `SKIPPED` rather than failing.
Climate needs no key.

**Still deferred from §4.** NECC egg rates (no API — needs an HTML scrape), IMD
Agromet advisories, and the dairy federation rate (no API anywhere). Their
commodities are seeded and their `source` values reserved; nothing pulls them yet.

### 6.2 What Phase C delivered

Package `com.lincfusion.udyamlinc.features`, plus the two institutional tables of
§3 the derivations read from (`loan_record`, `upi_summary`).

| Piece | Notes |
|---|---|
| `feature_definition` | The catalogue: family, unit, source kind, direction, and `staleness_tolerance_days` — the column a trip-wire consults before it is allowed to fire. Seeded idempotently at boot. |
| `feature_value` | The tall store. A feature that cannot be computed is written as **null with a `missing_reason`**, never skipped — `NO_SIGNAL`, `NO_LEDGER`, `NO_CBS`, `NO_CONSENT`, `NO_FORECAST`, `INSUFFICIENT_HISTORY`. "We did not know" and "we never looked" are different facts. |
| `feature_snapshot` | The wide store, one per (enterprise, as-of date), carrying `feature_count` / `missing_count` / `max_staleness_days` so a snapshot mostly made of holes cannot read as confident. `asFeatureMap()` exposes the same values under their snake_case codes — the shape a SpEL rule is written against. |
| `FeatureEngineeringService` | All fourteen derivations. Every read is point-in-time on both `valid_for_date` and `observed_at`, so recomputing last month uses only what was knowable last month. |
| `FeatureController` | `/admin/features/{definitions,snapshot/{id},values/{id},coverage}` and `POST /compute`, `POST /compute/{id}`. `coverage` reports the populated share per feature — a feature at 4 % coverage is a column, not a feature. |

**Catalogue.** §4's eleven, plus three the seeded trip-wires already named:
`dscr_p10` (worst month of `min(p10 / obligation)` over the latest run),
`days_since_last_entry`, and `emi_to_inflow`. A rule referring to a feature
nobody computes is a rule that silently never fires. These three also give the
OBLIGATION_LOAD family the inputs it otherwise lacked.

**Demo inputs.** The seeder now writes one `loan_record` per enterprise with DPD
tracking the band, so `emi_delay` and `emi_to_inflow` are real without a CBS
feed. `upi_summary` has an entity and a point-in-time query but no source until
Account Aggregator registration — `upi_inflow_change` therefore resolves to null
with `NO_CONSENT`, which is the honest answer and exercises the degradation path.
Price and climate features stay null until Phase B has actually run.

### 6.3 What Phase D delivered

Packages `rules` and `risk`.

| Piece | Notes |
|---|---|
| `RuleEvaluator` | SpEL over the feature rows, in a `SimpleEvaluationContext` restricted to property access — these expressions are authored by business users, so no method calls or bean lookups. Compiled expressions cached by rule code **and version**, so an edit invalidates its own entry; the admin write endpoints also clear the cache. |
| `RuleOutcome` | Every rule returns FIRED / NOT_FIRED / SKIPPED_STALE / SKIPPED_MISSING / SKIPPED_SECTOR / ERROR. "No alert because the enterprise is fine" and "no alert because the feed has been down a fortnight" must never look the same. |
| `ScoringStrategy` | The seam the model arrives through. `udyam.risk.engine` binds the implementation; LightGBM later is that one line. Implementations read features and nothing else — no reaching back into the signal tables. |
| `RuleBasedScoringStrategy` | Five families onto 0–100, combined with the sector's basis-point weights. Each family takes its **worst** signal, not the mean — averaging lets a calm dimension mask an acute one, and the acute one is where the officer needs to go. |
| `RiskEngineService` | Compute features → score → evaluate trip-wires against the **same feature rows, never the score** → persist score, top-3 reason codes, alerts. |
| `RiskEngineController` | `POST /admin/scoring/run`, `/run/{id}`, `GET /admin/scoring/runs`, and `GET /admin/scoring/explain/{id}` — every rule's outcome with no writes. |
| `ScoringScheduler` | 03:00, an hour behind ingestion so it scores what that job wrote. Off by default. |

**The staleness gate.** Before any expression runs, each of a rule's
`requiredFeatures` must have a value and be within the **stricter** of the rule's
`maxStalenessDays` and the catalogue's per-feature tolerance — a supervisor
tightening a rule must not be overridden by a generous default. This is the
promise of §4's degradation rules, enforced rather than documented.

**Band versus score.** A fired trip-wire escalates the band; it never rewrites
the score. They answer different questions — "how stressed is this enterprise"
and "has a specific agreed line been crossed" — and an audit needs both.
A family with no usable feature falls back to `udyam.risk.family-prior` rather
than scoring zero, because zero reads as "safe here" when the truth is "unknown",
and each family on a prior costs confidence.

**What the seeded rules can actually do today.** TW-01 (`dscr_p10`) and TW-03
(`days_since_last_entry`) evaluate from seeded data. TW-04 (`emi_to_inflow`)
evaluates once loan records exist, which the seeder now writes. TW-02
(`upi_inflow_change`) and TW-05 (`price_z_score`, `heat_anomaly`) will report
SKIPPED_MISSING until Account Aggregator consent and Phase B ingestion
respectively — visibly skipped, which is the intended behaviour rather than a
silent no-op.

**Not built here.** The quantile forecast is still the seeded path; this engine
scores and alerts but does not produce `forecast_point` rows. `dscr_p10` reads
whatever the latest run stored. Generating forecasts is part of the R1 model
milestone.

### 6.4 What Phase E delivered

In `UdyamLinc-UI`, two console screens and one piece of chrome — all three about
provenance, because the failure mode that matters is silent. A score computed
from a three-week-old mandi price looks exactly like one computed this morning.

| Piece | Notes |
|---|---|
| `features/officer/DataFeedsPage.tsx` | `/officer/feeds` — freshness per district first, then the mandi panel, the rainfall/heat panel with the 30-year normal beside each observation, and the ingestion run log. A skipped run appears as a row, so "nothing ran last night" never reads as "it ran and found nothing". |
| `features/officer/RulesPage.tsx` | `/officer/rules` — every trip-wire with its expression and tolerance, its outcome for a chosen enterprise, and the feature snapshot underneath with each null's reason. A banner counts rules that could not run. |
| `FreshnessChip` in `OfficerShell` | Data age in the header on every screen, amber past ten days, linking to the feeds page. |
| `lib/demo/feeds.ts` | Demo fallback that applies the **same two guards in the same order** as `RuleEvaluator` — missing, then stale, then the expression — so the offline demo cannot claim a rule fired that the real engine would have skipped. |

Both screens follow the house pattern: `useQuery` against the real endpoint,
in-file demo fallback, and a "Demo data" pill when the backend is unreachable.
`npm run build` and `oxlint` are clean.

**Deliberately not added:** rule *editing* from the console. The backend accepts
`POST`/`PUT /admin/rules` and bumps the version on every edit, but a supervisor
changing a live threshold from a demo build with no audit trail in front of it is
a worse idea than the round trip through an API client.

---

## 7. Explicitly not being built yet

Kafka, Neo4j, Elasticsearch, Keycloak, MinIO, and separate deployables. All appear
on the submission's technology slide; none earn their keep at four districts.
Postgres plus a scheduled job carries the pilot comfortably. Kafka arrives when
ingestion volume or a second consumer demands it — not before.