AI-Assisted Development Analysis

Honest assessment of how GenAI tools impacted this project’s software development lifecycle.

Tools & Their Roles

The cross-provider split is intentional: dev tooling is Anthropic (Claude), runtime LLM is OpenAI. Each one earns its keep in a different layer.

Quantitative Impact

Where AI Tools Were Used

1. Service Scaffolding (Claude Code)

Claude generated FastAPI microservice boilerplate for authService, inferenceService, and agentService:

• app/main.py with router setup, health endpoint
• app/models.py with Pydantic schemas
• app/auth.py with JWT middleware
• conftest.py for test fixtures
• Dockerfile and requirements.txt

Quality: 7/10 on first pass. Boilerplate was correct; edge cases (JWT refresh logic, error handling) needed manual refinement.

2. Agent Prompt Engineering (Claude Code + iteration)

Claude drafted the system prompts for Gloss Assembler and Translator. Iteration loop:

1. Claude generated initial assembler prompt → LLM output was valid JSON but too verbose
2. Refined constraints (e.g., “Keep corrections list to max 5 items”)
3. Added in-prompt examples for topic-comment ASL grammar
4. Final iteration: added explicit instruction “Return ONLY valid JSON, no preamble”

3. Test Generation (Claude Code)

100% of the ~40-test suite was generated by Claude Code. Example:

@pytest.mark.asyncio
async def test_predict_returns_top3():
    response = await client.post("/predict", json={...})
    assert response.status_code == 200
    assert len(response.json()["predictions"]) == 3

Human refinement focused on mocking MLflow loads and adding auth-failure / malformed-input cases.

4. Deployment Debugging (GitHub Copilot in GitHub Actions)

The “explain with copilot” feature inside GitHub Actions made it dramatically faster to diagnose failing CI/CD runs. We’d hit a red workflow, click “Explain”, and get an actionable hypothesis instead of grepping logs.

5. Kubernetes Manifest Generation (Claude Code)

Generated auth-deployment.yaml, inference-deployment.yaml, agent-deployment.yaml, ingress.yaml. Quality: 6/10 — needed manual tuning:

• Replicas and resource requests were guessed
• JWT_SECRET_KEY was initially hardcoded → manually moved to Kubernetes Secrets
• Ingress TLS cert configuration was omitted → manually added cert-manager integration

Hallucinations & Corrections

Hallucination 1: SQL Table Column Names

Symptom: The streaming INSERT into the Databricks SQL Warehouse was failing.

Root cause: streaming.py (Claude-generated) dynamically built INSERT statements from the full event dictionary, which included fields like kind and alternatives that weren’t in the original asl.inferenceEvents table schema. This caused UNRESOLVED_COLUMN errors in Databricks.

Resolution:

• Immediate fix: Added the missing columns directly to the Delta tables via ALTER TABLE.
• Proper fix: Later stripped the unnecessary fields before building the INSERT, so the schema stays clean.

-- Immediate unblock
ALTER TABLE asl.inferenceEvents ADD COLUMN kind STRING;
ALTER TABLE asl.inferenceEvents ADD COLUMN alternatives ARRAY<STRING>;

# Proper fix in streaming.py: whitelist columns before INSERT
ALLOWED_COLS = {"timestamp", "modelName", "sign", "confidence",
                "latencyMs", "eventId", "modelVersion"}
clean_event = {k: v for k, v in event.items() if k in ALLOWED_COLS}

Hallucination 2: Nightly-Teardown Workflow

What happened: Claude generated a nightly-teardown.yml workflow intended to tear down the AKS cluster overnight (to save Azure spend) and rebuild it on the first push of the day.

What it actually did: Only deleted the resource group — wiping more than intended and leaving the cluster in an awkward state when it rebuilt.

Resolution: Deleted the workflow and now tear down manually when necessary. The lesson: don’t hand AI tools production resource-management responsibility without strict guardrails.

Agentic Workflow

The agentic layer uses two LLM agents in sequence to turn raw recognizer output into usable English:

Agent roles:

• Gloss Assembler — Quality filter + context-aware swap. Drops the low-confidence noise the recognizer can’t suppress.
• Translator — Grammar repair. Re-orders topic-comment ASL into SVO English, inserts copulas/articles.

Decision flow: Browser collects signs → on pause, batches them → assembler cleans → translator reconstructs → UI renders. The user feedback loop closes the cycle.

How it uses APIs/ML: The flow is /predict (inferenceService, ML model) → /translate (agentService, LLM agents). Both agents call OpenAI directly with Pydantic-typed response_format for guaranteed schema compliance.

Challenges & Lessons Learned

Tips for Future Teams

• ✅ Commit a verified skeleton before splitting work — everyone branches off a known-good base
• ✅ Drop a CLAUDE.md of conventions so sessions don’t drift on naming, structure, or style
• ✅ Document environment constraints (Databricks Free Edition caps, AKS quotas) so AI doesn’t hallucinate around them
• ✅ Always verify by hand: security code, infra code, and ML model-loading code — the AI gets these wrong most often

Overall Usefulness

Where AI Added Most Value

1. Service scaffolding — 90% correct on first try
2. Test generation — 100% from Claude, ~40 tests
3. Deployment debugging — “Explain with Copilot” in GitHub Actions
4. Documentation — first drafts of docstrings and READMEs

Where AI Added Least Value

1. Kubernetes manifests — heavy manual refinement
2. Security-critical code — needed careful review for edge cases
3. Agent prompts — required iterative refinement and validation
4. Resource-management workflows — the nightly-teardown hallucination cost real time

Time-Benefit Analysis

• Best case: 40%+ faster on boilerplate, tests, and docs
• Realistic average: 4–6 hours saved per developer per day
• Worst case: Hallucinations like the SQL schema and nightly teardown soaked ~5–6 hours of debugging

Recommendations

1. Use Claude Code early for architectural guidance and service scaffolding
2. Use Copilot’s “Explain” feature in GitHub Actions for CI debugging — much faster than grepping
3. Use LLM APIs (gpt-4o-mini) for domain-specific agents with structured outputs
4. Always verify security, infra, and ML-model-loading code by hand
5. Document constraints (Databricks Free, AKS quotas) so AI doesn’t hallucinate around them