feat: the UdyamLinc ML layer — scoring, explanation and quantile forecast

Brings the Python service under version control for the first time. It has
been running and serving the demo since 23 August; until now it existed only
on one developer's disk.

Contents:

  training/build_dataset.py   point-in-time join over the feature store, with
                              leakage assertions. dscr_p10 and dscr_p50 are
                              excluded deliberately — both are derived from
                              the forecast, which is derived from the
                              instalment, so the model would read its own
                              answer.
  training/train_stress.py    LightGBM vs XGBoost vs two single-feature
                              baselines, expanding-window validation over
                              as-of dates, publishes to the model_metric table
  training/train_forecast.py  three quantile heads (P10/P50/P90), horizon as a
                              feature rather than one model per horizon
  serving/explain.py          SHAP TreeExplainer folded into the five risk
                              families the console already speaks. The family
                              map mirrors RuleBasedScoringStrategy in the API
                              repo; where they disagree the rule engine wins.
  serving/app.py              /score, /score/batch, /forecast, /health, /model

The trained models, their metrics and the training dataset are committed
alongside the code. Without them the published AUC of 0.760 is a claim nobody
else can check; at ~2.2 MB together they are worth the space.

Measured, on 5,523 rows / 263 enterprises / 242 positives / 100% synthetic
labels: LightGBM AUC-ROC 0.7599, and the emi_to_inflow baseline 0.7685 on the
same folds. The baseline wins. That is recorded rather than tuned away — on
generated labels, beating it would mostly prove the model had learned the
generator.

The quantile forecast beats the deterministic strategy on all three quantiles
(pinball loss, 12-33% better) but flags every borrower as breaching, so the
deployment runs udyam.forecast.engine=DETERMINISTIC and this model is held
back pending calibration against real repayment outcomes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

serving/explain.py

@@ -0,0 +1,111 @@
+"""
+SHAP contributions, folded into the five families the console already speaks.
+
+The console, the reason sentences and the Hindi PWA are all written against
+``liquidity | behavioural | trendVolatility | obligationLoad | externalShock``.
+A model that explained itself in thirteen feature names would need every one of
+those surfaces rewritten, and would hand a field officer a vocabulary they have
+never used. So the SHAP values are aggregated to the families instead, and the
+feature that contributed most inside each family is named alongside it.
+
+The family map mirrors ``RuleBasedScoringStrategy`` exactly. Where the two
+disagree, the rule engine is right and this file is wrong — an officer must not
+see the same enterprise explained two different ways depending on which engine
+happened to score it.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+import pandas as pd
+import shap
+
+LIQUIDITY = "liquidity"
+BEHAVIOURAL = "behavioural"
+TREND_VOLATILITY = "trendVolatility"
+OBLIGATION_LOAD = "obligationLoad"
+EXTERNAL_SHOCK = "externalShock"
+
+FAMILIES = [LIQUIDITY, BEHAVIOURAL, TREND_VOLATILITY, OBLIGATION_LOAD, EXTERNAL_SHOCK]
+
+FAMILY_OF: dict[str, str] = {
+    # liquidity — is there cash in the business
+    "cash_flow_change": LIQUIDITY,
+    "inventory_days": LIQUIDITY,
+    "upi_inflow_change": LIQUIDITY,
+    # behavioural — how the borrower has been acting
+    "emi_delay": BEHAVIOURAL,
+    "days_since_last_entry": BEHAVIOURAL,
+    "dpd_at_t": BEHAVIOURAL,
+    # trend / volatility — how unstable the market around them is
+    "price_z_score": TREND_VOLATILITY,
+    # obligation load — how heavy the instalment is against what comes in
+    "emi_to_inflow": OBLIGATION_LOAD,
+    "emi_at_t": OBLIGATION_LOAD,
+    "outstanding_at_t": OBLIGATION_LOAD,
+    # external shock — what the weather and the mandi are doing to them
+    "margin_squeeze": EXTERNAL_SHOCK,
+    "feed_cost_change": EXTERNAL_SHOCK,
+    "output_price_change": EXTERNAL_SHOCK,
+    "rainfall_anomaly": EXTERNAL_SHOCK,
+    "heat_anomaly": EXTERNAL_SHOCK,
+}
+
+
+class Explainer:
+    """Wraps a TreeExplainer and does the family arithmetic."""
+
+    def __init__(self, model, features: list[str]):
+        self.features = features
+        self._shap = shap.TreeExplainer(model)
+
+    def explain(self, row: pd.DataFrame) -> tuple[dict[str, int], list[dict]]:
+        """
+        :return: (sub-scores 0–100 per family, drivers sorted by contribution)
+
+        Only contributions that push the score **up** are reported as drivers.
+        A negative SHAP value means "this is one of the reasons the score is not
+        worse", which is true and useless to an officer deciding whether to visit.
+        """
+        values = self._shap.shap_values(row[self.features])
+        if isinstance(values, list):  # older binary API returns one array per class
+            values = values[1]
+        contributions = np.asarray(values).reshape(-1)
+
+        by_family: dict[str, float] = {f: 0.0 for f in FAMILIES}
+        best_feature: dict[str, tuple[str, float]] = {}
+
+        for name, value in zip(self.features, contributions):
+            base = name.removesuffix("__missing")
+            family = FAMILY_OF.get(base)
+            if family is None:
+                continue
+            by_family[family] += float(value)
+            held = best_feature.get(family)
+            if held is None or abs(value) > abs(held[1]):
+                best_feature[family] = (base, float(value))
+
+        positive = {f: max(0.0, v) for f, v in by_family.items()}
+        total = sum(positive.values())
+
+        # No family pushed the score up — a genuinely calm enterprise. Spread the
+        # weight evenly rather than emitting five zeros, which the console would
+        # render as an empty explanation for a score that still exists.
+        if total <= 0:
+            sub_scores = {f: 20 for f in FAMILIES}
+        else:
+            sub_scores = {f: int(round(100 * v / total)) for f, v in positive.items()}
+
+        drivers = []
+        for family in FAMILIES:
+            feature, raw = best_feature.get(family, (None, 0.0))
+            drivers.append({
+                "family": family,
+                "weight": sub_scores[family],
+                "featureCode": feature,
+                "value": float(row[feature].iloc[0]) if feature in row and pd.notna(row[feature].iloc[0]) else None,
+                "onPrior": bool(feature is None or (feature in row and pd.isna(row[feature].iloc[0]))),
+                "shap": round(raw, 5),
+            })
+        drivers.sort(key=lambda d: d["weight"], reverse=True)
+        return sub_scores, drivers