LincFusion-Nabard-UdyamLinc/udyamlinc-ml
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: the UdyamLinc ML layer — scoring, explanation and quantile forecast

Browse Source 
Brings the Python service under version control for the first time. It has
been running and serving the demo since 23 August; until now it existed only
on one developer's disk.

Contents:

  training/build_dataset.py   point-in-time join over the feature store, with
                              leakage assertions. dscr_p10 and dscr_p50 are
                              excluded deliberately — both are derived from
                              the forecast, which is derived from the
                              instalment, so the model would read its own
                              answer.
  training/train_stress.py    LightGBM vs XGBoost vs two single-feature
                              baselines, expanding-window validation over
                              as-of dates, publishes to the model_metric table
  training/train_forecast.py  three quantile heads (P10/P50/P90), horizon as a
                              feature rather than one model per horizon
  serving/explain.py          SHAP TreeExplainer folded into the five risk
                              families the console already speaks. The family
                              map mirrors RuleBasedScoringStrategy in the API
                              repo; where they disagree the rule engine wins.
  serving/app.py              /score, /score/batch, /forecast, /health, /model

The trained models, their metrics and the training dataset are committed
alongside the code. Without them the published AUC of 0.760 is a claim nobody
else can check; at ~2.2 MB together they are worth the space.

Measured, on 5,523 rows / 263 enterprises / 242 positives / 100% synthetic
labels: LightGBM AUC-ROC 0.7599, and the emi_to_inflow baseline 0.7685 on the
same folds. The baseline wins. That is recorded rather than tuned away — on
generated labels, beating it would mostly prove the model had learned the
generator.

The quantile forecast beats the deterministic strategy on all three quantiles
(pinball loss, 12-33% better) but flags every borrower as breaching, so the
deployment runs udyam.forecast.engine=DETERMINISTIC and this model is held
back pending calibration against real repayment outcomes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
LincPay
2026-08-25 13:43:40 +05:30
co-authored by Claude Opus 5
commit 4ed61a2427
13 changed files with 1659 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
training/build_dataset.py
+288
View File
@@ -0,0 +1,288 @@
"""
Assemble the training set from the feature store, point-in-time correct.
Everything downstream is worthless if this file is wrong, and nothing downstream
can detect it — a leaked column produces excellent metrics and a useless model.
So this module does three things and states all of them in the manifest it
writes: what it selected, what it deliberately excluded, and how much of the
label is real.
Shape of a row
--------------
One row per (enterprise, as-of month end):
features at T -> did DPD exceed 30 by T + 3 months
The features come from ``feature_value``, which carries both ``valid_for_date``
(the date the fact describes) and ``observed_at`` (when we learned it). The label
comes from ``loan_record`` three months later. The two never touch the same
table, so the label cannot leak through the feature join.
What is deliberately excluded
-----------------------------
``dscr_p10`` and ``dscr_p50`` are dropped. Both are computed by
``FeatureEngineeringService.dscr()``, which reads ``forecast_point`` with **no
as-of filter** — so a backfilled row for March 2025 carries a forecast produced
in August 2026. That is the present leaking into the past. They are the two most
predictive-looking columns in the store and they would flatter every metric.
A caveat the manifest records
-----------------------------
The backfilled rows are reconstructed by *fact date*, not by *knowledge date*: a
mandi price for March was ingested in August, so a March row uses a number a live
March run would not yet have had. Not label leakage, but not a live recording
either, and anything trained on it should say so.
"""
from __future__ import annotations
import argparse
import json
from dataclasses import dataclass, asdict
from pathlib import Path
import pandas as pd
import psycopg
# The catalogue, minus the two forecast-derived columns. Order is fixed and
# written to the manifest: a model served its columns in a different order fails
# silently, which is the worst way for a model to fail.
FEATURES = [
"feed_cost_change",
"output_price_change",
"margin_squeeze",
"price_z_score",
"rainfall_anomaly",
"heat_anomaly",
"cash_flow_change",
"emi_delay",
"inventory_days",
"upi_inflow_change",
"feed_staleness_days",
"days_since_last_entry",
"emi_to_inflow",
]
EXCLUDED = {
"dscr_p10": "reads forecast_point with no as-of filter — present leaks into past",
"dscr_p50": "same",
}
# Months between the feature date and the outcome we are predicting.
HORIZON_MONTHS = 3
DPD_THRESHOLD = 30
DEFAULT_DSN = "postgresql://postgres:root@localhost:5432/UdyamLinc_DEMO"
# --------------------------------------------------------------------------- #
FEATURE_SQL = """
select v.enterprise_id,
v.valid_for_date as as_of_date,
v.feature_code,
v.value,
v.missing_reason,
v.staleness_days
from feature_value v
where v.valid_for_date between %(start)s and %(end)s
and v.feature_code = any(%(codes)s)
"""
# The label, and the enterprise attributes that are legitimately known at T.
# `dpd_at_t` is the borrower's state on the feature date, not the outcome — it is
# knowable then and is the single strongest honest predictor available.
LABEL_SQL = """
with lab as (
select enterprise_id, as_of_date, max(dpd) as dpd,
sum(outstanding_amount) as outstanding, sum(instalment_amount) as emi_amount
from loan_record
group by 1, 2
)
select f.enterprise_id,
f.as_of_date,
e.sector_id,
e.district_id,
now_.dpd as dpd_at_t,
now_.outstanding as outstanding_at_t,
now_.emi_amount as emi_at_t,
fut.dpd as dpd_at_horizon
from (select distinct enterprise_id, valid_for_date as as_of_date
from feature_value
where valid_for_date between %(start)s and %(end)s) f
join enterprise e on e.enterprise_id = f.enterprise_id
left join lab now_ on now_.enterprise_id = f.enterprise_id
and now_.as_of_date = f.as_of_date
join lab fut on fut.enterprise_id = f.enterprise_id
and fut.as_of_date = (
date_trunc('month', f.as_of_date + make_interval(months => %(horizon)s))
+ interval '1 month - 1 day')::date
"""
# Provenance of the label itself, reported rather than assumed.
PROVENANCE_SQL = """
select coalesce(source, 'UNKNOWN') as source, count(*) as n
from loan_record group by 1
"""
@dataclass
class Manifest:
rows: int
enterprises: int
as_of_dates: int
date_min: str
date_max: str
positives: int
positive_rate: float
features: list[str]
excluded: dict[str, str]
horizon_months: int
dpd_threshold: int
label_source_split: dict[str, int]
synthetic_label_share: float
feature_coverage: dict[str, float]
caveats: list[str]
# --------------------------------------------------------------------------- #
def build(dsn: str, start: str, end: str) -> tuple[pd.DataFrame, Manifest]:
with psycopg.connect(dsn) as conn:
params = {"start": start, "end": end, "codes": FEATURES, "horizon": HORIZON_MONTHS}
long_features = pd.read_sql(FEATURE_SQL, conn, params=params)
spine = pd.read_sql(LABEL_SQL, conn, params=params)
provenance = pd.read_sql(PROVENANCE_SQL, conn)
if spine.empty:
raise SystemExit(
"No rows: the feature store and the loan book do not overlap at this horizon.\n"
"Run POST /admin/features/backfill first."
)
# Wide feature matrix, plus a null indicator per column. `missing_reason` is
# itself signal — NO_CONSENT is a different fact from NO_SIGNAL — but the tree
# only needs to know the value was absent, and the reason is preserved in the
# store for a human to read.
wide = long_features.pivot_table(
index=["enterprise_id", "as_of_date"],
columns="feature_code",
values="value",
aggfunc="first",
).reset_index()
for code in FEATURES:
if code not in wide.columns:
wide[code] = pd.NA
wide[f"{code}__missing"] = wide[code].isna().astype("int8")
df = spine.merge(wide, on=["enterprise_id", "as_of_date"], how="inner")
df["label"] = (df["dpd_at_horizon"] > DPD_THRESHOLD).astype("int8")
df = df.drop(columns=["dpd_at_horizon"])
# Sector and district stay ordinal ints rather than pandas categoricals.
# A one-row request at serving time cannot reproduce a category dtype with the
# training levels, and LightGBM rejects the mismatch rather than warning. With
# five sectors and fourteen districts an ordinal split costs nothing.
for col in FEATURES + ["dpd_at_t", "outstanding_at_t", "emi_at_t"]:
df[col] = pd.to_numeric(df[col], errors="coerce")
df = df.sort_values(["as_of_date", "enterprise_id"]).reset_index(drop=True)
_assert_no_leakage(df, dsn, start, end)
source_split = dict(zip(provenance["source"], provenance["n"].astype(int)))
synthetic = sum(n for s, n in source_split.items() if s != "REAL")
total_labels = sum(source_split.values()) or 1
coverage = {c: round(float(df[c].notna().mean() * 100), 1) for c in FEATURES}
manifest = Manifest(
rows=len(df),
enterprises=int(df["enterprise_id"].nunique()),
as_of_dates=int(df["as_of_date"].nunique()),
date_min=str(df["as_of_date"].min()),
date_max=str(df["as_of_date"].max()),
positives=int(df["label"].sum()),
positive_rate=round(float(df["label"].mean()), 4),
features=FEATURES,
excluded=EXCLUDED,
horizon_months=HORIZON_MONTHS,
dpd_threshold=DPD_THRESHOLD,
label_source_split=source_split,
synthetic_label_share=round(synthetic / total_labels, 4),
feature_coverage=coverage,
caveats=[
"Feature rows are reconstructed by fact date, not knowledge date — the "
"archive was ingested in one pass, so a past row uses a value a live run "
"of that date would not yet have had.",
"Labels are generated DPD history (SEEDED_HISTORY). Metrics computed on "
"this dataset measure the generator as much as the borrowers, and must be "
"reported as computed on synthetic history.",
"dscr_p10 and dscr_p50 are excluded: both read the current forecast at "
"every as-of date.",
],
)
return df, manifest
def _assert_no_leakage(df: pd.DataFrame, dsn: str, start: str, end: str) -> None:
"""
Two checks, because the failure is silent.
The first is structural: no feature row may describe a date after the as-of
date it was filed under. The second is the excluded set — if a forecast-derived
column ever reappears in the matrix, the build must stop rather than train on it.
"""
leaked = [c for c in EXCLUDED if c in df.columns]
if leaked:
raise SystemExit(f"Excluded columns present in the matrix: {leaked}")
with psycopg.connect(dsn) as conn, conn.cursor() as cur:
cur.execute(
"""
select count(*) from feature_value
where valid_for_date between %s and %s
and staleness_days is not null and staleness_days < 0
""",
(start, end),
)
future = cur.fetchone()[0]
if future:
raise SystemExit(f"{future} feature rows carry negative staleness — data from after their as-of date")
def main() -> None:
ap = argparse.ArgumentParser(description="Build the UdyamLinc training set")
ap.add_argument("--dsn", default=DEFAULT_DSN)
ap.add_argument("--start", default="2024-08-01")
ap.add_argument("--end", default="2026-06-30")
ap.add_argument("--out", default="data")
args = ap.parse_args()
df, manifest = build(args.dsn, args.start, args.end)
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
df.to_parquet(out / "training.parquet", index=False)
(out / "manifest.json").write_text(json.dumps(asdict(manifest), indent=2), encoding="utf-8")
print(f"rows {manifest.rows:,}")
print(f"enterprises {manifest.enterprises}")
print(f"as-of dates {manifest.as_of_dates} ({manifest.date_min} .. {manifest.date_max})")
print(f"positives {manifest.positives} ({manifest.positive_rate:.1%})")
print(f"synthetic labels {manifest.synthetic_label_share:.0%}")
print(f"written {out / 'training.parquet'}")
if manifest.positives < 50:
raise SystemExit(
f"\nRefusing to call this trainable: {manifest.positives} positive cases.\n"
"A classifier fitted here would learn the majority class."
)
if __name__ == "__main__":
main()
training/train_forecast.py
+294
View File
@@ -0,0 +1,294 @@
"""
Quantile heads for the cash-flow forecast.
The problem statement is "AI-Driven **Cash Flow Prediction** & Risk Flagging".
The risk head is a model; the forecast, until now, was
``DeterministicForecastStrategy`` — a median of monthly ledger position with a
capped drift and a band widening as sqrt(h). Honest, explainable, and not AI.
This trains the replacement.
Shape
-----
One row per (enterprise, as-of month end, horizon 1..6):
features known at T -> net ledger position in month T + h
Three models, at alpha = 0.10 / 0.50 / 0.90. **Horizon is a feature, not a
separate model per step**: six models per quantile would be eighteen artifacts
that cannot share the signal they all depend on, and the h=6 model would train on
a sixth of the rows.
Two rules carried over from the deterministic strategy
-----------------------------------------------------
1. Quantile regression does not guarantee ordering — P10 can be predicted above
P50 on an individual row. The three predictions are sorted per row before
they leave here. A crossed band is worse than no band.
2. **Missing inputs widen the band; they never lower its centre.** A pessimistic
P50 built on ignorance manufactures breaches, which is how an early-warning
system loses an officer's trust in one season.
What the metrics mean
---------------------
Pinball loss, per quantile, against the deterministic strategy reproduced on the
same held-out months. Lower is better. This is the comparison that decides
whether the model earns the switch — and, as with the stress head, it is measured
against a generated ledger, so it measures the generator too.
"""
from __future__ import annotations
import argparse
import json
import pickle
from datetime import datetime
from pathlib import Path
import lightgbm as lgb
import numpy as np
import pandas as pd
import psycopg
DSN = "postgresql://postgres:root@localhost:5432/UdyamLinc_DEMO"
QUANTILES = {"p10": 0.10, "p50": 0.50, "p90": 0.90}
HORIZONS = 6
# Features known at T. The point-in-time discipline is the same as the stress
# head's: `valid_for_date <= T`. Forecast-derived features are excluded for the
# same reason they are there — dscr_* reads the current forecast at every date.
FEATURE_CODES = [
"feed_cost_change",
"output_price_change",
"margin_squeeze",
"price_z_score",
"rainfall_anomaly",
"heat_anomaly",
"cash_flow_change",
"emi_delay",
"inventory_days",
"feed_staleness_days",
"days_since_last_entry",
"emi_to_inflow",
]
# --------------------------------------------------------------------------- #
# Monthly net position, and the trailing statistics a forecaster would have had.
# Everything is computed from months strictly before the as-of month, so a row
# never sees the month it is filed under, let alone the one it predicts.
SPINE_SQL = """
with monthly as (
select enterprise_id,
date_trunc('month', entry_date)::date as mo,
sum(case when kind = 'IN' then amount when kind = 'OUT' then -amount else 0 end) as net,
sum(case when kind = 'IN' then amount else 0 end) as inflow
from ledger_entry
group by 1, 2
),
stats as (
select m.enterprise_id,
m.mo as as_of_mo,
avg(h.net) as trail_mean,
stddev_pop(h.net) as trail_sd,
min(h.net) as trail_min,
avg(h.inflow) as trail_inflow,
count(*) as trail_months
from monthly m
join monthly h
on h.enterprise_id = m.enterprise_id
and h.mo < m.mo
and h.mo >= m.mo - interval '6 month'
group by 1, 2
),
obligation as (
select enterprise_id, as_of_date, sum(instalment_amount) as emi
from loan_record group by 1, 2
)
select s.enterprise_id,
s.as_of_mo,
s.trail_mean, s.trail_sd, s.trail_min, s.trail_inflow, s.trail_months,
e.sector_id, e.district_id,
o.emi,
t.mo as target_mo,
t.net as target_net,
(extract(year from t.mo) * 12 + extract(month from t.mo))
- (extract(year from s.as_of_mo) * 12 + extract(month from s.as_of_mo)) as horizon
from stats s
join enterprise e on e.enterprise_id = s.enterprise_id
left join obligation o
on o.enterprise_id = s.enterprise_id
and o.as_of_date = (s.as_of_mo + interval '1 month - 1 day')::date
join monthly t
on t.enterprise_id = s.enterprise_id
and t.mo > s.as_of_mo
and t.mo <= s.as_of_mo + interval '6 month'
where s.trail_months >= 3
"""
FEATURES_SQL = """
select enterprise_id, valid_for_date, feature_code, value
from feature_value
where feature_code = any(%(codes)s)
"""
def build(dsn: str) -> pd.DataFrame:
with psycopg.connect(dsn) as conn:
spine = pd.read_sql(SPINE_SQL, conn)
feats = pd.read_sql(FEATURES_SQL, conn, params={"codes": FEATURE_CODES})
if spine.empty:
raise SystemExit("No rows. Has the ledger backfill run?")
# Features are filed at month end; the spine is keyed on month start.
feats["as_of_mo"] = pd.to_datetime(feats["valid_for_date"]).values.astype("datetime64[M]")
wide = feats.pivot_table(
index=["enterprise_id", "as_of_mo"], columns="feature_code", values="value", aggfunc="first"
).reset_index()
spine["as_of_mo"] = pd.to_datetime(spine["as_of_mo"]).values.astype("datetime64[M]")
df = spine.merge(wide, on=["enterprise_id", "as_of_mo"], how="left")
for code in FEATURE_CODES:
if code not in df.columns:
df[code] = np.nan
df[f"{code}__missing"] = df[code].isna().astype("int8")
numeric = FEATURE_CODES + [
"trail_mean", "trail_sd", "trail_min", "trail_inflow", "trail_months",
"emi", "horizon", "target_net", "sector_id", "district_id",
]
for col in numeric:
df[col] = pd.to_numeric(df[col], errors="coerce")
df = df.dropna(subset=["target_net", "horizon"])
return df.sort_values(["as_of_mo", "enterprise_id", "horizon"]).reset_index(drop=True)
def feature_columns(df: pd.DataFrame) -> list[str]:
cols = FEATURE_CODES + [f"{c}__missing" for c in FEATURE_CODES]
cols += ["trail_mean", "trail_sd", "trail_min", "trail_inflow", "trail_months",
"emi", "horizon", "sector_id", "district_id"]
return [c for c in cols if c in df.columns]
def pinball(y: np.ndarray, pred: np.ndarray, alpha: float) -> float:
"""The loss quantile regression is actually optimising. Lower is better."""
delta = y - pred
return float(np.mean(np.maximum(alpha * delta, (alpha - 1) * delta)))
def deterministic_baseline(frame: pd.DataFrame, alpha: float) -> np.ndarray:
"""
The strategy currently in production, reproduced so the comparison is fair.
Centre is the trailing mean; the band widens with sqrt(horizon) at the same
18% base spread the Java implementation uses. If the model cannot beat this,
the switch is not earned and the config value stays on DETERMINISTIC.
"""
centre = frame["trail_mean"].to_numpy()
spread = 0.18 * np.sqrt(frame["horizon"].to_numpy())
if alpha < 0.5:
return centre * (1 - spread)
if alpha > 0.5:
return centre * (1 + spread)
return centre
def expanding_folds(months: np.ndarray, n_folds: int = 3, min_train: int = 8):
uniq = np.array(sorted(pd.unique(months)))
step = max(1, (len(uniq) - min_train) // max(1, n_folds))
for k in range(n_folds):
cut = min_train + k * step
if cut >= len(uniq):
break
test = uniq[cut : cut + step] if k < n_folds - 1 else uniq[cut:]
yield uniq[:cut], test
def run(out_dir: Path, dsn: str = DSN) -> dict:
df = build(dsn)
cols = feature_columns(df)
months = df["as_of_mo"].to_numpy()
results: dict = {
"rows": int(len(df)),
"enterprises": int(df["enterprise_id"].nunique()),
"as_of_months": int(df["as_of_mo"].nunique()),
"horizons": int(df["horizon"].max()),
"validation": "expanding-window time-series split over as-of months",
"feature_order": cols,
"quantiles": {},
}
print(f"rows {len(df):,} · enterprises {results['enterprises']} · months {results['as_of_months']}")
for name, alpha in QUANTILES.items():
model_losses, base_losses = [], []
for train_months, test_months in expanding_folds(months):
tr = df[df["as_of_mo"].isin(train_months)]
te = df[df["as_of_mo"].isin(test_months)]
if te.empty or tr.empty:
continue
model = lgb.LGBMRegressor(
objective="quantile", alpha=alpha,
n_estimators=400, learning_rate=0.05, num_leaves=15,
min_child_samples=30, subsample=0.85, subsample_freq=1,
colsample_bytree=0.85, verbose=-1, random_state=42,
)
model.fit(tr[cols], tr["target_net"])
pred = model.predict(te[cols])
y = te["target_net"].to_numpy()
model_losses.append(pinball(y, pred, alpha))
base_losses.append(pinball(y, deterministic_baseline(te, alpha), alpha))
results["quantiles"][name] = {
"alpha": alpha,
"pinball_model": round(float(np.mean(model_losses)), 2),
"pinball_deterministic": round(float(np.mean(base_losses)), 2),
}
q = results["quantiles"][name]
better = q["pinball_model"] < q["pinball_deterministic"]
print(f"{name} pinball model {q['pinball_model']:>10,.0f} "
f"deterministic {q['pinball_deterministic']:>10,.0f} "
f"{'model wins' if better else 'DETERMINISTIC WINS'}")
# Refit each quantile on everything for serving.
final = {}
for name, alpha in QUANTILES.items():
model = lgb.LGBMRegressor(
objective="quantile", alpha=alpha,
n_estimators=400, learning_rate=0.05, num_leaves=15,
min_child_samples=30, subsample=0.85, subsample_freq=1,
colsample_bytree=0.85, verbose=-1, random_state=42,
)
model.fit(df[cols], df["target_net"])
final[name] = model
wins = sum(1 for q in results["quantiles"].values()
if q["pinball_model"] < q["pinball_deterministic"])
results["verdict"] = (
f"{wins}/3 quantiles beat the deterministic strategy on held-out months"
)
print(f"\n{results['verdict']}")
out_dir.mkdir(parents=True, exist_ok=True)
with open(out_dir / "forecast_model.pkl", "wb") as fh:
pickle.dump({"models": final, "features": cols, "horizons": HORIZONS,
"version": "lightgbm-forecast-v1"}, fh)
(out_dir / "forecast_metrics.json").write_text(
json.dumps(results | {"trainedAt": datetime.now().isoformat()}, indent=2), encoding="utf-8"
)
print(f"written: {out_dir / 'forecast_model.pkl'}")
return results
def main() -> None:
ap = argparse.ArgumentParser(description="Train the UdyamLinc quantile forecast heads")
ap.add_argument("--out", default="models")
ap.add_argument("--dsn", default=DSN)
args = ap.parse_args()
run(Path(args.out), args.dsn)
if __name__ == "__main__":
main()
training/train_stress.py
+317
View File
@@ -0,0 +1,317 @@
"""
Train the stress head, and measure it honestly.
Two algorithms are fitted on identical folds — LightGBM and XGBoost — so the
choice between them is a recorded number rather than a preference. Validation is
an expanding-window time-series split: fold *k* trains on every as-of date before
a cut and tests on the dates after it, which is the only split that means
anything for a model whose job is to see forward.
Nothing here is calibrated against real defaults. The labels are generated DPD
history, and every metric this writes is stamped with that fact.
Outputs
-------
``models/stress_<algo>.pkl`` the fitted pipeline plus its feature order
``models/metrics.json`` fold-by-fold and pooled metrics
"""
from __future__ import annotations
import argparse
import json
import pickle
from datetime import datetime
from pathlib import Path
import numpy as np
import pandas as pd
from sklearn.metrics import (
average_precision_score,
brier_score_loss,
precision_score,
recall_score,
roc_auc_score,
)
import lightgbm as lgb
import xgboost as xgb
# Mirrors udyam.risk.red-threshold / watch-threshold in application.properties.
# The console, the apex suppression rules and the Hindi PWA all speak these three
# bands; the model maps into them rather than inventing a fourth.
RED_THRESHOLD = 65
WATCH_THRESHOLD = 35
def load(data_dir: Path) -> tuple[pd.DataFrame, dict]:
df = pd.read_parquet(data_dir / "training.parquet")
manifest = json.loads((data_dir / "manifest.json").read_text(encoding="utf-8"))
return df, manifest
def feature_columns(df: pd.DataFrame, manifest: dict) -> list[str]:
"""
Feature order is fixed here and written into the artifact. A model served its
columns in a different order does not error — it returns confident nonsense.
"""
cols = list(manifest["features"])
cols += [f"{c}__missing" for c in manifest["features"]]
cols += ["dpd_at_t", "outstanding_at_t", "emi_at_t", "sector_id", "district_id"]
return [c for c in cols if c in df.columns]
def expanding_folds(dates: np.ndarray, n_folds: int = 4, min_train: int = 8):
"""
Expanding-window folds over the distinct as-of dates.
Never a random split. A random split over a panel puts March and June of the
same enterprise on opposite sides of the fence, and the model learns that
enterprise rather than the pattern — which reports as excellent accuracy and
reproduces as nothing.
"""
uniq = np.array(sorted(pd.unique(dates)))
if len(uniq) < min_train + n_folds:
n_folds = max(1, len(uniq) - min_train)
step = max(1, (len(uniq) - min_train) // max(1, n_folds))
for k in range(n_folds):
cut = min_train + k * step
if cut >= len(uniq):
break
yield uniq[:cut], uniq[cut : cut + step] if k < n_folds - 1 else uniq[cut:]
def fit_lightgbm(x: pd.DataFrame, y: np.ndarray, pos_weight: float):
model = lgb.LGBMClassifier(
objective="binary",
n_estimators=250,
learning_rate=0.04,
num_leaves=7,
min_child_samples=45,
subsample=0.85,
subsample_freq=1,
colsample_bytree=0.85,
reg_lambda=5.0,
scale_pos_weight=pos_weight,
verbose=-1,
random_state=42,
)
model.fit(x, y)
return model
def fit_xgboost(x: pd.DataFrame, y: np.ndarray, pos_weight: float):
model = xgb.XGBClassifier(
objective="binary:logistic",
n_estimators=400,
learning_rate=0.04,
max_depth=4,
min_child_weight=5,
subsample=0.85,
colsample_bytree=0.85,
reg_lambda=1.0,
scale_pos_weight=pos_weight,
tree_method="hist",
enable_categorical=True,
eval_metric="aucpr",
random_state=42,
)
model.fit(x, y)
return model
FITTERS = {"lightgbm": fit_lightgbm, "xgboost": fit_xgboost}
def score_to_band(score: int) -> str:
if score >= RED_THRESHOLD:
return "RED"
if score >= WATCH_THRESHOLD:
return "WATCH"
return "STANDARD"
def evaluate_fold(y_true: np.ndarray, proba: np.ndarray) -> dict:
"""
Precision at the Red band is the number that matters operationally: the cost
of a false positive there is one field visit, and an officer who is sent on
three wasted visits stops believing the fourth alert.
"""
scores = np.clip(np.round(proba * 100), 0, 100)
red = scores >= RED_THRESHOLD
flagged = red | (scores >= WATCH_THRESHOLD)
out = {
"n": int(len(y_true)),
"positives": int(y_true.sum()),
"auc_roc": float(roc_auc_score(y_true, proba)) if y_true.sum() else None,
"pr_auc": float(average_precision_score(y_true, proba)) if y_true.sum() else None,
"brier": float(brier_score_loss(y_true, proba)),
"red_flagged": int(red.sum()),
"precision_red": float(precision_score(y_true, red, zero_division=0)),
"recall_red": float(recall_score(y_true, red, zero_division=0)),
"recall_flagged": float(recall_score(y_true, flagged, zero_division=0)),
}
return out
def run(data_dir: Path, out_dir: Path) -> dict:
df, manifest = load(data_dir)
cols = feature_columns(df, manifest)
dates = df["as_of_date"].values
results: dict = {
"dataset": {
"rows": manifest["rows"],
"positives": manifest["positives"],
"positive_rate": manifest["positive_rate"],
"as_of_dates": manifest["as_of_dates"],
"synthetic_label_share": manifest["synthetic_label_share"],
},
"validation": "expanding-window time-series split over as-of dates",
"feature_order": cols,
"algorithms": {},
"caveats": manifest["caveats"],
}
for name, fitter in FITTERS.items():
folds = []
for train_dates, test_dates in expanding_folds(dates):
tr = df[df["as_of_date"].isin(train_dates)]
te = df[df["as_of_date"].isin(test_dates)]
if te.empty or tr["label"].sum() == 0:
continue
pos_weight = float((tr["label"] == 0).sum() / max(1, (tr["label"] == 1).sum()))
model = fitter(tr[cols], tr["label"].values, pos_weight)
proba = model.predict_proba(te[cols])[:, 1]
fold = evaluate_fold(te["label"].values, proba)
fold["train_to"] = str(max(train_dates))
fold["test_from"] = str(min(test_dates))
folds.append(fold)
pooled = {
k: round(float(np.mean([f[k] for f in folds if f[k] is not None])), 4)
for k in ("auc_roc", "pr_auc", "brier", "precision_red", "recall_red", "recall_flagged")
if any(f[k] is not None for f in folds)
}
results["algorithms"][name] = {"folds": folds, "pooled": pooled}
print(f"{name:10s} AUC {pooled.get('auc_roc', float('nan')):.3f} "
f"PR-AUC {pooled.get('pr_auc', float('nan')):.3f} "
f"precision@RED {pooled.get('precision_red', float('nan')):.3f}")
# Baselines, on the identical folds.
#
# Without these a metric is unreadable: AUC 0.74 is either a real result or a
# restatement of "a borrower who is late today is late in three months",
# and only the comparison tells you which. The Reports page charts the model
# against these rather than against an invented bureau score.
results["baselines"] = {}
for label, column in (("repayment_history_only", "dpd_at_t"),
("obligation_ratio_only", "emi_to_inflow")):
folds = []
for train_dates, test_dates in expanding_folds(dates):
te = df[df["as_of_date"].isin(test_dates)]
if te.empty or te["label"].sum() == 0:
continue
raw = te[column].fillna(te[column].median()).values.astype(float)
# Rank-scale to [0,1] so it is comparable to a probability.
proba = (pd.Series(raw).rank(pct=True)).values
folds.append(evaluate_fold(te["label"].values, proba))
if folds:
results["baselines"][label] = {
k: round(float(np.mean([f[k] for f in folds if f[k] is not None])), 4)
for k in ("auc_roc", "pr_auc", "precision_red")
if any(f[k] is not None for f in folds)
}
print(f"{label:24s} AUC {results['baselines'][label].get('auc_roc', float('nan')):.3f} "
f"PR-AUC {results['baselines'][label].get('pr_auc', float('nan')):.3f}")
# The winner is chosen by PR-AUC, not AUC-ROC. At a 4% positive rate ROC is
# flattered by the majority class; precision-recall is the honest read.
winner = max(results["algorithms"], key=lambda a: results["algorithms"][a]["pooled"].get("pr_auc", 0))
results["winner"] = winner
results["base_rate"] = round(float(df["label"].mean()), 4)
print(f"\nwinner: {winner} (by PR-AUC)")
# Refit the winner on everything for serving.
pos_weight = float((df["label"] == 0).sum() / max(1, (df["label"] == 1).sum()))
final = FITTERS[winner](df[cols], df["label"].values, pos_weight)
out_dir.mkdir(parents=True, exist_ok=True)
with open(out_dir / "stress_model.pkl", "wb") as fh:
pickle.dump({"model": final, "algorithm": winner, "features": cols,
"red_threshold": RED_THRESHOLD, "watch_threshold": WATCH_THRESHOLD}, fh)
(out_dir / "metrics.json").write_text(json.dumps(results, indent=2), encoding="utf-8")
print(f"written: {out_dir / 'stress_model.pkl'}")
return results
DSN = "postgresql://postgres:root@localhost:5432/UdyamLinc_DEMO"
METRICS = ("auc_roc", "pr_auc", "precision_red", "recall_red", "brier")
def publish(results: dict, dsn: str = DSN) -> int:
"""
Write the backtest to `model_metric`, where the console reads it.
The Reports page used to render 0.87 from a constant nobody computed. These
rows are the replacement, and each one carries the share of synthetic labels
it was measured against — so the accuracy cannot be quoted without the
caveat, because they are the same record.
Baselines are written as rows too. A model AUC means nothing on its own; the
single-feature arms are what say whether it is a result or a restatement.
"""
import psycopg
stamp = datetime.now()
ds = results["dataset"]
version = f"{results['winner']}-stress-v1"
rows = []
for arm, payload in results["algorithms"].items():
for metric, value in payload["pooled"].items():
if metric in METRICS:
rows.append((version, arm, False, metric, value))
for arm, payload in results.get("baselines", {}).items():
for metric, value in payload.items():
if metric in METRICS:
rows.append((version, arm, True, metric, value))
with psycopg.connect(dsn) as conn, conn.cursor() as cur:
for version_, arm, is_baseline, metric, value in rows:
cur.execute(
"""
insert into model_metric
(model_version, arm, baseline, metric, value, dataset_rows,
dataset_positives, synthetic_label_share, validation, computed_at)
values (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(version_, arm, is_baseline, metric, round(float(value), 5),
ds["rows"], ds["positives"], ds["synthetic_label_share"],
results["validation"][:120], stamp),
)
conn.commit()
print(f"published {len(rows)} metric rows at {stamp:%Y-%m-%d %H:%M}")
return len(rows)
def main() -> None:
ap = argparse.ArgumentParser(description="Train the UdyamLinc stress head")
ap.add_argument("--data", default="data")
ap.add_argument("--out", default="models")
ap.add_argument("--no-publish", action="store_true",
help="train and evaluate without writing to model_metric")
args = ap.parse_args()
results = run(Path(args.data), Path(args.out))
if not args.no_publish:
publish(results)
if __name__ == "__main__":
main()