Banking Intent Router — Ninditya Salma Nur Aini

01 · Business Impact

From $0.050 to $0.0056 per query.

All-LLM (Naive)

$0.050

Every query routed to GPT-4-class model regardless of complexity

→

4-Tier Router

$0.0056

↓ 88% projected reduction

Each query routed to cheapest tier that can answer it correctly

Banking queries follow a predictable distribution — 80% don't need GPT-4. Sending everything to the same LLM wastes $0.044 on every routine FAQ that a template could have answered in 8ms.

Routine FAQs (70–80%) — "lost card", "forgot PIN", "check balance" — perfectly handled by templates
Knowledge queries (15–25%) — "what's the limit for international wire?" — handled by RAG over a curated FAQ
Complex/edge cases (~5%) — multi-turn reasoning, unusual phrasing — actually need an LLM

⚠️ Projected savings based on assumed traffic distribution — not observed production data. Actual cost reduction depends on real traffic mix. A conservative mix with heavier LLM usage will produce lower savings.

02 · The Architecture

Route by confidence, not by hope.

Every query first hits a small SBERT (all-MiniLM-L6-v2) encoder, then a LinearSVC classifier predicts intent + confidence. The confidence score then routes the query through one of four tiers.

Banking Intent Router — 4-tier confidence routing pipeline: regex pre-filter, SBERT + LinearSVC classifier, template handler, RAG pipeline, and LLM escalation — System overview — 4-tier confidence routing, cost comparison, and key metrics.

# Tier 0: regex pre-filter (bypass model entirely)
if is_greeting_or_farewell(query): route → "conversational"    # $0.000, sub-ms

# Tiers 1–3: confidence-based routing
embedding = sbert_encoder.encode(query, normalize=True)
proba     = classifier.predict_proba(embedding)
confidence = proba.max()

if   confidence >= 0.55: route → "template_handler"   # $0.001, ~8ms
elif confidence >= 0.30: route → "rag_pipeline"       # $0.010, ~300ms
else:                  route → "llm_escalation"     # $0.050, ~2s

00

Conversational

bilingual regex match

Greetings & farewells in English + Indonesian ("hi", "halo", "selamat pagi", "terima kasih") bypass the ML model entirely.

Cost$0.000 Latency<1ms

01

Template Handler

confidence ≥ 0.55

High-confidence intents get a pre-written templated answer from the FAQ knowledge base. No generation needed.

Cost$0.001 Latency~8ms

02

RAG Pipeline

0.30 ≤ conf < 0.55

Medium-confidence: retrieve top-3 from FAISS-indexed 59-entry knowledge base, return contextual answer with citations.

Cost$0.010 Latency~300ms

03

LLM Escalation

confidence < 0.30

Low-confidence: hand off to OpenAI / Groq / Ollama / Mock with intent context. Reserved for the ambiguous ~5%.

Cost$0.050 Latency~2s

Why 0.55 and 0.30? — Threshold selection.

Thresholds were selected by evaluating the cost–quality tradeoff across a range. Lower template threshold → more queries escalated to LLM (higher cost, safer quality). Higher threshold → more queries handled by templates (lower cost, risk of over-confident routing on edge cases).

Template Threshold	% → Template	% → LLM	Est. cost/query	Verdict
0.40	~48%	~8%	$0.0082	Too conservative — misses savings
0.50	~56%	~6%	$0.0065	Acceptable
0.55 ✓	~60%	~5%	$0.0056	Chosen — maximizes cost-quality tradeoff
0.60	~64%	~3%	$0.0048	Quality risk on edge cases
0.65	~68%	~2%	$0.0044	Over-confident routing — not defensible

The clever bit: confidence-aware LLM hint

Even when escalating to the LLM, the classifier's output isn't thrown away. The predicted intent label is passed as a soft context hint to the LLM — but only if confidence ≥ 0.25. Below that threshold, the prediction is too noisy to be useful and would mislead the model, so the hint is dropped. Engineering decisions like this are what separate a notebook from a production service.

03 · Results

Benchmarked on Banking77.

13,083 customer queries across 77 banking intents — the standard benchmark for banking NLP. F1 Macro 0.9305 places SBERT + LinearSVC well above the TF-IDF baseline (0.8843) on the same dataset.

Metric	Score	Note
F1 Macro	0.9305	Across 77 intents · Banking77 benchmark
Accuracy	0.9285	Balanced eval set
Avg latency	~50ms	Weighted by tier distribution
Projected Cost Reduction	−88 to −96%	Based on assumed traffic mix — see math below

Model selection: why the method changed

Iteration	Method	F1 Macro	Decision
Baseline 1	TF-IDF + Logistic Regression	~0.86	First interpretable baseline
Baseline 2	TF-IDF + LinearSVC	0.8843	SVM margin outperforms LR on short intent text
Final	SBERT + LinearSVC	0.9305	Dense embeddings close the synonym gap — chosen for serving

RAG pipeline evaluation

The RAG tier uses FAISS IndexFlatIP over 59 banking FAQ entries, each embedded with the same SBERT encoder. Retrieval quality on the FAQ test set:

RAG Metric	Value	What it means
Recall@3	91%	Correct answer appears in top-3 results
Hit Rate@1	74%	Top-1 result matches ground truth
MRR	0.82	Mean Reciprocal Rank — answer ranks consistently high

Projected savings — how the math works

The 88–96% range is sensitive to traffic distribution. Representative banking traffic mix (10% conversational + 60% routine FAQs + 25% knowledge queries + 5% complex):

# All-LLM baseline (every query → GPT-4-class)
1.00 × $0.050 = $0.0500 / query

# 4-tier router
0.10 × $0.000 + 0.60 × $0.001 + 0.25 × $0.010 + 0.05 × $0.050
= $0.0000 + $0.0006 + $0.0025 + $0.0025
= $0.0056 / query

# Reduction: ($0.050 − $0.0056) / $0.050 = ~89%
# A very template-heavy mix approaches ~96%.
# A conservative mix (heavier LLM usage) sits closer to ~70%.

04 · Production Engineering

Not a notebook. A service.

Serving stack

FastAPI Uvicorn Docker HuggingFace Spaces

Single Docker image, served via FastAPI with Pydantic schemas. /predict endpoint returns intent_id, confidence, routing decision, latency, and the actual generated answer. Custom chat UI in static/index.html with per-tier routing badges.

ML stack

sentence-transformers scikit-learn FAISS OpenAI · Groq · Ollama · Mock

SBERT (all-MiniLM-L6-v2) for encoding; LinearSVC wrapped for predict_proba; FAISS for RAG retrieval; 4-way pluggable LLM client with automatic fallback to Mock when API keys are missing — service stays alive instead of crashing.

Observability

Prometheus Grafana Cloud

12 custom metrics exposed via /metrics: request count, latency histograms, routing distribution, confidence histogram, rolling low-confidence rate, cumulative estimated cost per tier. Async push to Grafana Cloud via Prometheus remote_write (manual protobuf + snappy compression).

Resilience

Auto-fallback Lifespan hooks

LLM call failures degrade gracefully to a support-message fallback. Health endpoint reports model + RAG + LLM provider status. Lifespan hooks ensure model load happens before serving traffic. Provider misconfiguration is logged, not raised.

Hugging Face Spaces live demo — chat UI showing routing tier badge, confidence score, and latency per query

Live demo · routing badge + confidence score visible per query

Grafana Cloud monitoring dashboard — request rate, routing distribution, latency histograms, and cost metrics

Grafana Cloud · 12 custom Prometheus metrics in production

05 · Production Decisions

The small choices that matter.

A notebook lives for 30 minutes. A service lives in production. These are the trade-offs that show up only when you actually ship something.

CPU-only PyTorch in the Dockerfile. ~200 MB instead of ~2 GB CUDA wheels. HF Spaces free tier has no GPU — bundling CUDA just bloats the image and slows cold starts.
Model artifacts in a separate repo. The Docker image pulls model.pkl + encoder_name.txt from a dedicated HF Hub model repo at build time. Image stays small. Model is versioned independently from serving code.
Multi-provider LLM with auto-fallback. OpenAI, Groq, Ollama, or Mock — switchable via LLM_PROVIDER env var. Missing API keys don't crash the service; they log a warning and fall back to Mock. Service availability over feature completeness.
Confidence-aware LLM prompting. The classifier's predicted intent is passed to the LLM as a soft hint — but only if confidence ≥ 0.25. Below that, the prediction is too noisy and gets dropped to avoid misleading the LLM.
Bilingual conversational pre-filter. The keyword set includes Indonesian greetings/farewells (halo, selamat pagi, terima kasih) — built for actual Indonesian users, not an English-only demo.
Pinned exact dependency versions. All 16 serving deps pinned to specific versions. No "works on my machine" surprises three months later.

06 · Methodology

TF-IDF alone was never enough. The question was always how to represent meaning.

This project didn't start from a banking brief. It started from a 2021 undergraduate thesis that ran into the exact same problem: raw word counts don't capture what a sentence means.

2021 thesis: the LSA insight.

In my thesis, I was building a question classifier for a university chatbot using KNN. The first approach — KNN on raw TF-IDF vectors — had a structural problem. A sparse vector treats "lost card" and "card stolen" as completely unrelated: they share zero terms, so their distance is maximal even though they mean the same thing. KNN on sparse, high-dimensional vectors is both slow to compute and semantically blind.

LSA fixed this via SVD: decompose the TF-IDF matrix, keep the top-k singular values, and project every question into a lower-dimensional latent space. In that space, words that appear in similar contexts cluster together — "lost" and "stolen" both appear near "card" in the corpus, so they end up geometrically close. Accuracy improved from 92.1% to 93.2%, compute time dropped 35%. The lesson: co-occurrence implies meaning, and you need a dense space to exploit it.

2025: SBERT solves the same problem at a different scale.

SBERT (Sentence-BERT) does exactly what LSA did — but instead of decomposing a matrix built from 1,410 local questions, a transformer learns dense representations from pre-training on over a billion sentence pairs. The 384-dimensional output vector encodes meaning through attention weights, not co-occurrence statistics.

"Lost my card" and "card stolen" map to nearby points in the SBERT embedding space not because they appeared in the same documents in a small corpus, but because the model learned their shared semantic role across billions of examples. The classifier (LinearSVC) then operates in that already-meaningful space — and confidence in that space becomes a reliable routing signal.

	LSA · 2021 thesis	SBERT · this project
Semantic knowledge	Co-occurrence in 1,410 questions	Pre-trained on 1B+ sentence pairs
How meaning is found	SVD on TF-IDF matrix (k = 661 dims)	Transformer attention → 384-dim vector
Synonym handling	Partial — only if both appear in corpus	Yes — learned from pre-training
Downstream classifier	KNN (Manhattan distance)	LinearSVC + calibrated probabilities
Core mechanism	Co-occurrence → matrix decomposition	Context → attention → embedding

"The insight was the same in 2021 and 2025: counting words isn't enough. You need to represent meaning. LSA used matrix algebra to find it in a small corpus. SBERT learned it from pre-training at scale." — the thread connecting thesis to production

Full breakdown of the 2021 LSA implementation: NLP KNN LSA & Chatbot — thesis case study →

07 · How It Was Built (hidden)

What I actually built, step by step.

This project was not just a model benchmark. I took Banking77 from raw exploration to a monitored FastAPI service: EDA, preprocessing, baseline modelling, SBERT upgrade, confidence routing, RAG answer layer, LLM fallback, deployment, and observability.

Method selection: why the model changed

I did not start with SBERT. The first modelling path was deliberately simple: TF-IDF features with Logistic Regression. After that, I compared LinearSVC on the same TF-IDF representation, because short intent texts often work well with linear margin-based classifiers. SVM won the sparse-feature baseline, then SBERT embeddings pushed the score higher again.

Iteration	Method	Decision
Baseline 1	TF-IDF + Logistic Regression	Used as the first interpretable baseline for multi-class intent classification.
Baseline 2	TF-IDF + LinearSVC	Performed better in tuning; became the classical ML baseline with F1 macro around 0.8843.
Final	SBERT + LinearSVC	Chosen for serving because dense sentence embeddings improved F1 macro to 0.9305 while staying fast enough for routing.

Step 01 · EDA

Read the Banking77 shape first.

I started by checking sample count, 77 intent labels, class balance, text length distribution, and examples per intent. The goal was to confirm this was an intent-routing problem, not a free-form generation problem.

Output: intent distribution
Output: text length distribution

Step 02 · Preprocessing

Turn notebook steps into a repeatable pipeline.

I moved manual cleaning into a reusable pipeline: lowercase text, remove noisy characters, encode labels, keep a stratified split, and save train/test artifacts so modelling and CI use the same data contract.

Input: Hugging Face PolyAI/banking77
Output: train/test artifacts and label encoder

Step 03 · Baseline

Start simple with TF-IDF + Logistic Regression.

I first used TF-IDF + Logistic Regression as a clean baseline, then tuned LinearSVC on the same TF-IDF features. LinearSVC produced the stronger classical result, reaching roughly 0.8843 F1 macro.

Feature: TF-IDF 1-2 gram, max 10,000 features
Models compared: Logistic Regression vs LinearSVC

Step 04 · SBERT Upgrade

Replace sparse features with dense sentence embeddings.

I used sentence-transformers/all-MiniLM-L6-v2 to encode each query into a 384-dimensional normalized embedding, then tested LinearSVC and Logistic Regression configs with calibrated probabilities.

Best: SBERT + LinearSVC, C=5
Result: F1 macro 0.9305, accuracy 0.9285

Step 05 · Confidence Routing

Use confidence as the production decision signal.

I converted classifier probability into a routing policy. High confidence goes to templates, medium confidence goes to RAG, and low confidence escalates to the LLM.

Template: confidence >= 0.55
RAG: 0.30 <= confidence < 0.55
LLM: confidence < 0.30

Step 06 · RAG Layer

Add retrieval for medium-confidence queries.

I built the RAG layer with FAISS IndexFlatIP. FAQ questions are embedded with the same SBERT encoder, normalized, and retrieved by inner product, which acts like cosine similarity for normalized vectors.

Knowledge base: banking FAQ entries
Template path retrieves top-1; RAG path retrieves top-3 with sources

Step 07 · Serving

Package the model behind a FastAPI contract.

I exposed predict, health, and metrics endpoints. The response includes intent id, confidence, routing tier, latency, answer, RAG sources, and LLM answer when escalation happens.

Schema: Pydantic request/response models
UI: static chat page with per-tier routing badges

Step 08 · LLM Fallback

Escalate only when cheaper layers are not enough.

I made the LLM provider configurable: OpenAI, Groq, Ollama, or Mock. If keys are missing or a call fails, the service falls back cleanly instead of crashing.

Provider is switchable by environment variable
Intent hint is only passed when confidence is still useful

Step 09 · Deployment

Make it runnable outside the notebook.

I prepared the serving package with pinned dependencies and a Dockerfile, then deployed it as a Hugging Face Space. Model artifacts are loaded at startup, and CPU-only dependencies keep the image smaller for free-tier hosting.

Runtime: FastAPI + Uvicorn + Docker
Demo: Hugging Face Spaces

Step 10 · Monitoring

Instrument the router like a service.

I added Prometheus metrics for request volume, latency, routing distribution, confidence drift, estimated cost, errors, active requests, and throughput. Grafana dashboards make it clear whether traffic is getting more expensive or less confident over time.

Endpoint: /metrics
Dashboard: Grafana Cloud compatible JSON

Steps 1-2 of 10

07 · Limitations

What this project doesn't answer yet.

Banking77 is a clean, balanced, English-only benchmark. Production banking chatbots face a different reality — and a senior interviewer will ask about this gap.

Banking77 ≠ production traffic

The dataset is pre-labeled, single-intent, and balanced across 77 classes. Real customer queries contain typos, slang, mixed languages (Indonesian-English code-switching), ambiguous phrasings, and multi-intent requests. F1 0.9305 on Banking77 does not directly translate to production performance.

Confidence calibration not validated

LinearSVC probabilities come from Platt scaling (CalibratedClassifierCV). Calibration quality — whether a confidence of 0.55 actually means 55% correct — was not separately validated with a reliability diagram or ECE metric. Threshold choices assume calibration is adequate.

Projected, not observed cost savings

The 88–96% cost reduction is a simulation based on an assumed traffic mix. No production traffic was routed through the system. Actual savings depend on real query distribution, which may differ significantly from the assumed 60%/25%/5% split.

RAG limited to 59 FAQ entries

The knowledge base contains 59 manually curated entries. This is sufficient as a prototype but not representative of a real banking FAQ corpus, which may contain hundreds of entries, conflicting answers, and outdated information that requires regular maintenance.

"Banking77 is a benchmark dataset and does not fully represent noisy production traffic. Future work includes evaluation on multilingual and real customer conversations." — honest limitation, not a failure

08 · What I Learned

Four lessons I'll carry forward.

1. Real ML engineering = knowing when NOT to use the big model.

The instinct to throw GPT-4 at every problem is the most expensive habit in modern ML. The router proves a small model + good architecture can beat a big model on cost-per-correct-answer.

2. Confidence is a first-class signal — use it.

Most classifiers throw away the confidence score. It's the most useful signal in production: it tells you when to escalate, when to ask for clarification, when to log for review.

3. Observability is not optional for ML services.

The Prometheus metrics aren't decoration. router_low_confidence_rate trending up = your distribution is drifting. router_estimated_cost_usd_total tells finance exactly what the chatbot costs them per day. Without these, you're blind.

4. Classical NLP clarifies what modern NLP is actually solving.

Implementing LSA from scratch in 2021 — manually computing SVD, projecting into the latent space, measuring cosine similarity — made it immediately obvious what SBERT was doing when I picked it up in 2025. The mechanism changed; the problem didn't. Understanding the classical version makes you a sharper judge of when the modern version is overkill, when it's necessary, and what its failure modes look like.

"Real engineering isn't about using the biggest model. It's about knowing when not to — and understanding the classical baseline is how you develop that judgment." — Banking Intent Router case study

09 · References

Sources I used to ground the technical choices.

The project decisions are tied to public datasets, papers, model cards, and official documentation rather than generic AI claims.

BANKING77 paper

Original paper behind the banking intent dataset: customer-service examples across fine-grained banking intents.

PolyAI/banking77 dataset card

Dataset source used by the preprocessing pipeline and train/test artifacts.

scikit-learn TfidfVectorizer

Reference for the sparse text baseline using TF-IDF weighted n-gram features.

scikit-learn LinearSVC

Reference for the SVM classifier chosen after tuning TF-IDF and SBERT feature representations.

Sentence-BERT paper

Foundation for using sentence embeddings as compact semantic features before a classifier.

all-MiniLM-L6-v2 model card

Model card for the SBERT encoder used in training, RAG indexing, and serving.

Retrieval-Augmented Generation paper

Reference for combining retrieval with answer generation or answer selection.

FastAPI documentation

Serving framework used for prediction, health, metrics, and Pydantic response schemas.

Prometheus metric types

Reference for Counter, Gauge, Histogram, and Summary metrics used in monitoring.

Grafana Cloud remote_write docs

Reference for pushing Prometheus-compatible metrics to Grafana Cloud.

RouteLLM — Ong et al. (2024)

Framework for learning cost-effective LLM routing between strong and weak models using preference data. Direct academic grounding for the 4-tier routing architecture.

FrugalGPT — Chen et al. (2023)

Cascade strategy that achieves GPT-4-level quality at 98% lower cost. Motivates the regex→template→RAG→LLM escalation pattern used in this project.

On Calibration of Modern Neural Networks — Guo et al. (2017)

Foundational study on confidence miscalibration in deep networks. Grounds the confidence threshold design for tier-escalation decisions.

BERT — Devlin et al. (2019)

Pre-training deep bidirectional transformer representations from unlabelled text. The architectural foundation that SBERT extends for sentence-level embeddings.

Cut banking LLM costs ~90%.