Frontend & User Interface

Single-page application for ASL sign recognition and translation — branded as SignStream.

The Interface

SignStream · live ASL recognition session showing recognized signs, agent corrections, and the final English translation

The single-page app shows everything in one view: live camera feed (center), service health dots and user session (top right), recognized signs panel with per-sign feedback chips (right), the final English translation, sliced clips from the current session, and the full 100-sign model vocabulary (bottom).

Architecture

Framework: Vanilla HTML/JS (no build step) MediaPipe: Holistic pose detection via JS API HTTP Client: Fetch API with JWT token handling Styling: Plain CSS (responsive design) Server: nginx serving static files + reverse proxy to services

User Flow

1. Page Load

1. User opens http://localhost:8080/
2. Browser loads index.html; JS modules load: mediapipe.js, api.js, keypointPreprocessing.js, ui.js
3. Three health checks fire in parallel:
   • GET /health → authService
   • GET /health → inferenceService
   • GET /health → agentService
4. Status indicators rendered: ● auth · ● infer · ● agent

2. Start Signing

1. User clicks Start signing → browser captures MediaPipe Holistic frames at ~30 fps
2. For each frame:
   • Run MediaPipe Holistic → full pose + hands + face
   • Run keypointPreprocessing.js:
     • Subset to 27 OpenHands keypoints
     • Drop z (depth) coordinates
     • Normalize by shoulder distance
     • Collect in array

// frontend/js/mediapipe.js (abridged)
const vision = await mediapipe.FilesetResolver.forVisionTasks(
  'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.0/wasm'
);

const poseDetector = await mediapipe.PoseLandmarker.createFromOptions(vision, {
  baseOptions: { modelAssetPath: '...' },
  runningMode: 'VIDEO',
  numPoses: 1
});

function processFrame(videoElement) {
  const result = poseDetector.detectForVideo(videoElement, performance.now());
  return result.landmarks[0];  // 33 landmarks (full pose)
}

3. Keypoint Preprocessing

Full 33 MediaPipe landmarks → 27 OpenHands subset → drop z → shoulder-normalize → collect.

// frontend/js/keypointPreprocessing.js (abridged)
function preprocessLandmarks(fullLandmarks) {
  const subset = OPENHANDS_INDICES.map(i => fullLandmarks[i]);

  const leftShoulder = subset[0];
  const rightShoulder = subset[1];
  const shoulderDist = distance(leftShoulder, rightShoulder);

  const center = [
    (leftShoulder.x + rightShoulder.x) / 2,
    (leftShoulder.y + rightShoulder.y) / 2
  ];

  return subset.map(kp => [
    (kp.x - center[0]) / shoulderDist,
    (kp.y - center[1]) / shoulderDist
  ]);
}

4. Send for Inference

On clip boundary (motion pause), POST to /predict:

async function predict(landmarks, modelName = 'stgcn') {
  const token = localStorage.getItem('accessToken');

  const response = await fetch('/api/predict', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${token}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ landmarks, modelName })
  });

  if (response.status === 401) {
    await refreshToken();
    return predict(landmarks, modelName);
  }

  return response.json();
}

Response: { sign, confidence, latencyMs, eventId, modelVersion, alternatives: [...] }

5. Display Recognized Signs

Each recognized sign appears in the Recognized Signs panel (right) with:

• Sign label + corrected/raw badge
• Confidence + model name
• Thumbs-up / thumbs-down chips
• Swap-alternative dropdown

In the screenshot above:

• THEY1 (corrected from a lower-confidence alternative)
• GO (98.5% confidence · ST-GCN)
• HOSPITAL1 (100.0% confidence · ST-GCN)
• PARTY1 (corrected from 4% alternative)

6. Translate

User clicks Translate → batched signs POST to /translate:

{
  "signs": [
    {"label": "THEY1", "confidence": 0.17},
    {"label": "GO", "confidence": 0.985},
    {"label": "HOSPITAL1", "confidence": 1.0},
    {"label": "PARTY1", "confidence": 0.04}
  ]
}

agentService runs the Gloss Assembler then the Translator (both gpt-4o-mini).

7. Final English

The English Translation panel renders the agent output:

They are going to the hospital for a party.

Plus a collapsible ▶ agent trace showing the corrections the assembler made and the grammar transformations the translator applied.

8. Feedback Loop

Every sign card has thumbs-up / thumbs-down and a swap dropdown. User actions write to asl.userFeedback:

{ "sign": "GO", "rating": 1, "chosenAlternative": null }

UI Components

Session Management

Login flow:

async function login(username, password) {
  const response = await fetch('/api/login', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ username, password })
  });

  const data = await response.json();
  localStorage.setItem('accessToken', data.accessToken);
  localStorage.setItem('refreshToken', data.refreshToken);
}

async function refreshToken() {
  const refreshToken = localStorage.getItem('refreshToken');
  const response = await fetch('/api/refresh', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ refreshToken })
  });

  const data = await response.json();
  localStorage.setItem('accessToken', data.accessToken);
}

Performance Considerations