Dataset & Data Pipeline — ASL Interpretation & Correction Pipeline

Dataset Origin

Name: ASL Citizen Source: Microsoft Research (Desai et al., 2023) License: MIT Repository: github.com/microsoft/ASL-citizen-code Published: June 2023

Citation:

Desai, S., Gouws, A., Knysh, I., Johnson, D., & Liang, P. (2023).
ASL Citizen: an open dataset for American Sign Language.
arXiv preprint arXiv:2306.16268.

Dataset Characteristics

Property	Value
Total videos	84,000
Total signs (vocabulary)	2,731
Sign distribution	Multiple takes per sign, from different signers
Video resolution	720×480 (landscape)
Frame rate	30 fps
Signer pool	Crowdsourced from deaf/HoH signers (paid)
Accessibility focus	Designed for underrepresented communities
Data split	80% train, 10% validation, 10% test

Subset for This Project

We restrict our working vocabulary to 100 signs chosen for visual distinctiveness, ensuring the recognizer task is tractable within the 4-week sprint timeline:

Subset: ["HELLO", "GOODBYE", "THANK YOU", "SORRY", "YES", "NO",
         "LOVE", "HAPPY", "SAD", "ANGRY", "TIRED", "HUNGRY",
         "WATER", "FOOD", "SLEEP", "WORK", "SCHOOL", "FRIEND",
         "DOG1", "BEE1", "CLOUD1", "RIVER1", "EAT1", "BREAKFAST1",
         "LUNCH1", "DINNER1", "LETTUCE1", "BITE1", "HOSPITAL1",
         ... (100 total, chosen to minimize visual confusion)]

Rationale for 100-sign subset:

Full 2,731-class vocabulary is research-scale; 100 signs allow end-to-end demo
Visual distinctiveness reduces inter-sign confusion during evaluation
Matches scoping decision in aslProjectPlan.md section 1.1

ETL Process

Stage 1: Download from GitHub Release

# In Databricks notebook (running on serverless compute)
import urllib.request
import tarfile

url = "https://github.com/microsoft/ASL-citizen-code/releases/download/checkpoints_v1/asl_citizen_videos.tar.gz"
urllib.request.urlretrieve(url, "asl_citizen_videos.tar.gz")

with tarfile.open("asl_citizen_videos.tar.gz") as tar:
    tar.extractall("/Volumes/workspace/default/asl_raw/")

Note: On Databricks Free Edition, direct internet downloads may hit an allowlist. Fallback: use a Colab session to download and push via the SDK.

Stage 2: Filter to 100-Sign Subset

import os
import shutil

SUBSET_SIGNS = ["HELLO", "GOODBYE", ..., ]  # 100 signs

src = "/Volumes/workspace/default/asl_raw/"
dst = "/Volumes/workspace/default/asl_subset_100/"

for sign_dir in os.listdir(src):
    if sign_dir in SUBSET_SIGNS:
        shutil.copytree(
            os.path.join(src, sign_dir),
            os.path.join(dst, sign_dir)
        )

Stage 3: Extract Landmarks (PySpark)

# Spark job: run MediaPipe Holistic on each video, extract 27-keypoint OpenHands subset
# Output: Parquet files with (video_id, landmarks_array)

import pyspark.sql.functions as F
from pyspark.sql.types import ArrayType, FloatType

videos_df = spark.read.format("image").load("/Volumes/.../asl_subset_100/")

# UDF: extract landmarks from video frame data
def extract_landmarks(video_bytes):
    # Use cv2 to decode, MediaPipe to process
    # Return (T, 27, 2) array as list
    return landmarks_array

extract_landmarks_udf = F.udf(extract_landmarks, ArrayType(ArrayType(FloatType())))

landmarks_df = videos_df.withColumn(
    "landmarks",
    extract_landmarks_udf(videos_df.data)
).select("path", "landmarks")

landmarks_df.write.mode("overwrite").parquet("/Volumes/.../asl_landmarks/")

Stage 4: Load to Delta Tables

# Convert Parquet to Delta Lake (immutable, versioned)
landmarks_df = spark.read.parquet("/Volumes/.../asl_landmarks/")

landmarks_df.write \
    .mode("overwrite") \
    .option("mergeSchema", "true") \
    .format("delta") \
    .saveAsTable("asl.landmarks")

spark.sql("ALTER TABLE asl.landmarks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)")

Data Storage

Databricks Unity Catalog Volumes

We use UC Volumes (not DBFS, which is unavailable on Free Edition):

/Volumes/workspace/default/
├── asl_raw/              # Original downloaded videos (intermediate)
├── asl_subset_100/       # Filtered to 100 signs
├── asl_landmarks/        # Extracted MediaPipe keypoints (Parquet)
└── model_artifacts/      # Pretrained ST-GCN .pt file

Delta Lake Tables

Table	Schema	Purpose
`asl.landmarks`	`{video_id: str, landmarks: array<array<float>>, sign: str, split: str}`	Training/eval features (immutable)
`asl.inferenceEvents`	`{timestamp, modelName, sign, confidence, latencyMs, eventId, modelVersion}`	Real-time predictions (streaming write from inferenceService)
`asl.userFeedback`	`{timestamp, userId, sign, rating, chosenAlternative}`	User-correction loop data (feedback)
`asl.evaluationResults`	`{runId, variant, bleu, chrF, correctionsCount}`	Pipeline ablation results

Feature Engineering

Per-Clip Aggregation (for Random Forest)

# Each video clip → single feature vector for ML training

def aggregate_clip_features(landmarks_array):
    """
    Input: (T, 27, 2) landmarks
    Output: feature vector

    Features:
    - Per-keypoint mean, std, min, max (27 * 4 = 108)
    - Velocity (frame-to-frame delta): mean, std (2 per keypoint = 54)
    - Trajectory curvature: mean (1 per keypoint = 27)
    - Handedness confidence (left/right hand present)
    - Total ~200 features per clip
    """

    import numpy as np

    features = []

    # Static features
    features.extend(np.mean(landmarks_array, axis=0).flatten())  # mean
    features.extend(np.std(landmarks_array, axis=0).flatten())   # std
    features.extend(np.min(landmarks_array, axis=0).flatten())   # min
    features.extend(np.max(landmarks_array, axis=0).flatten())   # max

    # Velocity
    if landmarks_array.shape[0] > 1:
        velocity = np.diff(landmarks_array, axis=0)
        features.extend(np.mean(velocity, axis=0).flatten())
        features.extend(np.std(velocity, axis=0).flatten())

    return np.array(features)

This per-clip feature vector is fed to the Random Forest model for training and inference.

Data Pipeline DAG

ASL Citizen Release

84k videos · 2,731 signs (MIT)

↓

download

asl_raw/

All videos in UC Volume

↓

filter to 100 signs

asl_subset_100/

Working subset

↓

MediaPipe extract

asl_landmarks/

27-keypoint Parquet

↓

to Delta

asl.landmarks

Versioned · ACID · queryable

↓

RF Training

→ asl.model_rf (MLflow)

Inference Events

→ asl.inferenceEvents (stream)

Constraints & Workarounds

Free Edition Limitations

Issue	Impact	Workaround
No DBFS	Cannot use legacy file storage	Use UC Volumes instead
No GPU	Cannot train ST-GCN from scratch	Use pretrained weights, CPU inference
256 MB model serialization cap (Spark Connect)	Spark ML RF exceeds cap	Fall back to sklearn RF
5 GB UI upload limit	Large datasets cannot be uploaded via UI	Use Databricks SDK / direct download
Daily compute quota	Long-running jobs eat quota	Schedule evaluations off-peak, tear down sessions
Serverless compute only	No custom cluster configs	Use Databricks-managed serverless (acceptable)

Data Privacy

ASL Citizen videos are published research data (MIT licensed)
No PII in landmark coordinates (just x, y positions of body joints)
User feedback (asl.userFeedback) is aggregated and anonymized before downstream analysis

Quality Assurance

Validation Checks

# After each stage, validate row counts and schema

def validate_landmarks_table():
    df = spark.read.table("asl.landmarks")

    # Check counts
    assert df.count() > 0, "landmarks table is empty"

    # Check schema
    assert "landmarks" in df.columns, "missing landmarks column"
    assert "sign" in df.columns, "missing sign column"

    # Check landmark shape
    def check_shape(arr):
        if arr:
            return len(arr) > 0 and len(arr[0]) == 2
        return False

    bad_rows = df.filter(~F.udf(check_shape)(F.col("landmarks"))).count()
    assert bad_rows == 0, f"{bad_rows} rows have malformed landmarks"

    print("✓ landmarks table validation passed")