AT&T · DESP 2026 · Full-Stack Technical Report
Live ASL Interpretation & Correction Pipeline
Real-time American Sign Language recognition wrapped in an agentic correction-and-translation pipeline — deployed end-to-end on Azure Kubernetes Service.
The Team
System Architecture
Browser / Client
MediaPipe Holistic
27-keypoint subset · JWT
↓
HTTPS + JWT (13 KB JSON)
nginx Ingress
route · TLS · auth check
↓
authService
:8001
register · login · refresh
JWT · RBAC · bcrypt
inferenceService
:8002
/predict · /models · /stats
ST-GCN + Random Forest
agentService
:8003
/translate
Gloss Assembler · Translator
↓
Databricks
MLflow Registry · Delta Lake (events, feedback) · SQL Dashboard
Documentation Sections
🛠️
Technology Stack
Docker, AKS, Databricks, FastAPI, PyTorch, MLflow — with full design rationale
View section →
🏗️
Microservices & Architecture
Auth, Inference, and Agent services — 13 REST endpoints, K8s deployment
View section →
📊
Dataset & Data Pipeline
ASL Citizen (84k videos), ETL, Databricks Delta Lake, feature engineering
View section →
🤖
ML Models
ST-GCN recognizer + Random Forest baseline, MLflow lifecycle, logit masking
View section →
⚙️
Correction Pipeline
4-stage agentic pipeline: Gloss Assembler + Translator, BLEU/chrF ablation
View section →
🔐
REST APIs & Security
Full endpoint reference, JWT HS256, RBAC, rate limiting, error handling
View section →
📈
Dashboard & Monitoring
Databricks SQL real-time dashboard, latency percentiles, alert thresholds
View section →
🧠
AI-Assisted Development
Claude Code + GitHub Copilot: ~95% codebase AI-generated, hallucinations logged
View section →
💻
Frontend & UI
SignStream — vanilla JS single-page app, MediaPipe integration, user flow
View section →
Key Results
| Metric |
Value |
| Primary Model (ST-GCN) |
80–90%+ top-1 · 95%+ top-3 on 100-sign subset |
| Baseline Model (RF) |
40–55% top-1 · 80–90% top-3 (trained end-to-end) |
| Inference Latency p50 / p95 |
120 / 180 ms (ST-GCN) · 30 / 50 ms (RF) |
| ASL Vocabulary |
100 signs from ASL Citizen corpus |
| AI Code Generated |
~95% of codebase (Claude Code) |
| AI Tests Generated |
100% of test suite (~40 tests, Claude Code) |
| Net Productivity Gain |
4–6 hours saved daily |
✨ Core Innovation
A correction-and-translation pipeline that wraps a pretrained recognizer with a Gloss Assembler agent (contextual top-k correction) and a Translator agent (ASL gloss → fluent English). The agentic layer turns raw labels into usable captions and logs every correction.
🚀 Deployment
Azure Kubernetes Service (AKS) with Databricks backend. Browser-side landmark extraction via MediaPipe Holistic reduces server load to CPU-only inference — p95 around 165 ms per sign.
🎓 ML Lifecycle
Two models demonstrate the full lifecycle: pretrained ST-GCN for production (Microsoft Research, MIT licensed) + trained Random Forest baseline end-to-end (training, tuning, MLflow registration, deployment, monitoring).
Key Design Decisions
🌐 Browser-Side Landmarks
MediaPipe Holistic runs client-side, reducing server payload to 13 KB per sign and enabling CPU-only inference.
📐 Logit Masking
Mask before softmax (not after) to ensure a valid probability distribution over 100 in-vocabulary signs.
🤖 Two Models
Pretrained ST-GCN for production + trained Random Forest to satisfy the ML-lifecycle rubric.
🔐 Structured LLM Outputs
OpenAI SDK with Pydantic schemas eliminates JSON parsing errors; retry-once on parse failure.
💾 Databricks Free Edition
Unity Catalog Volumes (no DBFS), serverless compute, and hosted MLflow simplify ops.
🏗️ Vanilla JS Frontend
No build step, MediaPipe JS bundled directly, single HTML file served by nginx.
Quick Start
# Local development
cp .env.example .env
# Fill in JWT_SECRET_KEY: openssl rand -hex 32
docker compose up --build
Open http://localhost:8080 and watch three service health indicators turn green:
- ✓ authService (port 8001)
- ✓ inferenceService (port 8002)
- ✓ agentService (port 8003)
Last Updated: June 2026 · Team: 3 developers · Timeline: 4-week sprint · Repo: phanmtim/att-tourists