Correction Pipeline

The 4-stage pipeline that wraps the recognizer with contextual intelligence and grammatical reconstruction.

Overview

The Pipeline in Action

Live gpt-4o-mini runs on three reference items:

Stage 1: Recognizer with Top-3

Model: ST-GCN or Random Forest Input: Video landmarks (27 keypoints × 2 coords × T frames) Output: Ordered list of 3 candidates [(label, confidence), ...]

This is where the alternative candidates come from. Without top-3, the downstream agents have no flexibility to correct.

Stage 2: Gloss Assembler (LLM Agent)

Role: Filter low-confidence signs, apply contextual correction

Behavior:

• Drops signs with confidence < 0.40
• Flags signs with confidence in [0.40, 0.65)
• Contextual correction: if top-1 is implausible given surrounding signs, swap with an alternative and log why

System Prompt (abridged):

You are an ASL gloss correction expert. You receive a time-ordered sequence
of recognized signs. Each sign has top-1, top-2, top-3 candidates with confidence scores.

TASKS:
1. Drop signs with confidence < 0.40
2. Flag (keep, but mark) signs with confidence in [0.40, 0.65)
3. Contextual correction: if the top candidate looks wrong given the surrounding signs,
   swap it for a top-2 or top-3 candidate and explain the correction.

EXAMPLE:
Input: [
  {label: "HAPPY", confidence: 0.92, candidates: ["HAPPY", "LAUGH", "SMILE"]},
  {label: "VERY", confidence: 0.35, candidates: ["VERY", "SO", "MORE"]},
  {label: "YOU", confidence: 0.88, candidates: ["YOU", "THEM", "PERSON"]}
]

Output:
{
  "cleaned_gloss": ["HAPPY", "YOU"],
  "flags": [{index: 1, label: "VERY", confidence: 0.35, reason: "below 0.40 threshold"}],
  "corrections": []
}

Pydantic Schema (app/schemas.py):

class AssemblerOutput(BaseModel):
    cleaned_gloss: list[str]
    flags: list[dict]      # {index, label, confidence, reason}
    corrections: list[dict] # {original, chosen, reason}

Implementation (app/agents.py):

def run_gloss_assembler(sign_sequence: list[dict]) -> AssemblerOutput:
    response = client.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": ASSEMBLER_SYSTEM_PROMPT},
            {"role": "user", "content": json.dumps(sign_sequence, indent=2)}
        ],
        response_format=AssemblerOutput
    )
    return response.choices[0].message.parsed

Stage 3: Translator (LLM Agent)

Role: Convert ASL gloss (topic-comment order) to grammatical English (SVO order)

ASL Grammar Quirks:

• Topic-comment: [TOMORROW] [STORE] [I] [GO] means “Tomorrow, store, I go”
• Time markers at front: temporal expressions come first
• No articles: “STORE” not “the store”
• No copulas: “HAPPY” not “is happy”

System Prompt (abridged):

You are an ASL-to-English translator. ASL uses topic-comment structure and
omits copulas and articles. Your job is to reconstruct grammatical English.

RULES:
- Reorder from topic-comment to SVO
- Insert copulas where needed (is, are, was, were)
- Insert articles where needed (a, the)
- Expand single words to full phrases

EXAMPLES:
Input: ["TOMORROW", "STORE", "I", "GO"]
Output: "I'm going to the store tomorrow."

Input: ["RAIN", "OUTSIDE"]
Output: "It's raining outside."

Input: ["HAPPY", "I"]
Output: "I am happy."

Implementation:

def run_translator(cleaned_gloss: list[str]) -> TranslatorOutput:
    response = client.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": TRANSLATOR_SYSTEM_PROMPT},
            {"role": "user", "content": " ".join(cleaned_gloss)}
        ],
        response_format=TranslatorOutput
    )
    return response.choices[0].message.parsed

Both agents use gpt-4o-mini with Pydantic structured outputs, JSON-only, in-prompt examples, and retry-once on parse failure.

Stage 4: User Feedback Loop

UI: After each prediction, show:

• Thumbs up / thumbs down
• Dropdown with top-2 alternatives
• Correction reason (from assembler)

Endpoint: POST /feedback

{
  "sign": "HELLO",
  "rating": 1,
  "chosenAlternative": "WAVE"
}

Data written to asl.userFeedback Delta table:

{timestamp, userId, sign, rating, chosenAlternative}

Evaluation: Sentence-Level BLEU/chrF

We quantify the pipeline’s contribution via ablation on a reference corpus, evaluated by pipelines/evaluation/sentenceEval.py.

Variants:

1. Raw — top-1 glosses joined with spaces (no pipeline)
2. +Assembler — cleaned gloss joined (assembler applied, no translation)
3. +Translator — full pipeline (assembler + translator)

Metrics: BLEU (corpus-level, detokenized) + chrF (character F-score, robust on short sentences).

Ablation Results

Per-Item Results

Why This Matters

Without the pipeline: You get top-1 glosses joined with spaces. With the pipeline: You get grammatical English with contextual corrections and reasoning.

The contribution is not “we wrapped a pretrained model” — it is “we built a correction and reconstruction layer on top that produces usable captions.”