Inputs
Versioned inputs
01LLM Evaluation Platform
LLM Reliability + EvalOps Platform
Full-stack LLM evaluation platform with versioned datasets, graders, quality gates, and cost-latency analysis.
- Python
- FastAPI
- PostgreSQL
- Next.js
- TypeScript
- Ownership
- Owned project
- Timeline
- August 2026
- Status
- Live reference implementation
LLM Reliability + EvalOps Platform
Synthetic/demo evaluation data shown in the public reference implementation.
Overview
A Next.js, FastAPI, and PostgreSQL reference implementation for measuring the quality, reliability, estimated cost, and latency of LLM-powered workflows across versioned datasets, prompts, models, and graders.
Problem
LLM changes can improve one metric while quietly degrading reliability, cost, or latency. Teams need reproducible evaluation runs, failed-example inspection, and explicit release gates rather than anecdotal prompt testing.
My role
Designed and built the full-stack reference platform, evaluation runner, grader system, analytics UI, persistence layer, and CI quality-gate workflow.
System architecture
API
FastAPI control plane
Runner
Evaluation runner
Graders
Grader layer
Store
PostgreSQL / Alembic
Analytics
Next.js analytics
Gate
CI quality gate
What I built
- Versioned datasets, prompt versions, and model configurations.
- Synchronous evaluation runs with result, estimated-cost, and latency tracking.
- Exact-match, JSON-schema, text-similarity, citation/grounding, and optional Gemini LLM-as-judge grading.
- Composite scoring, run history, breakdowns, and failed-example analysis.
- Cost-versus-quality and latency-versus-quality comparisons.
- A CI evaluation-gate CLI with pass-rate, score, cost, and p95 thresholds, stable exit codes, and JSON reports.
- Document-grounded RAG QA using supplied documents rather than a vector database.
- Next.js dashboard, FastAPI API, PostgreSQL persistence, and Alembic migrations.
- Vercel/Cloud Run packaging and GitHub Actions validation.
Technical decisions
- Version artifacts so comparisons can be reproduced.
- Combine deterministic graders with an optional model-based judge rather than making every evaluation provider-dependent.
- Give CI gates stable exit codes and JSON reports for automation.
- Keep document-grounded RAG QA tied to supplied documents rather than suggesting a general vector-search platform.
- Surface cost and latency next to quality so one-dimensional improvements are visible.
Testing and validation
- Fixed 20-case RAG QA dataset held constant across both recorded runs.
- Baseline run: 95.0% pass rate, 0.987 average score, 1 failed case, $0.02939670 estimated cost, 745.5 ms average latency, 867 ms p95.
- Intentionally degraded prompt: 85.0% pass rate, 0.953 average score, 3 failed cases, $0.06300110 estimated cost, 808.15 ms average latency, 908 ms p95.
- Both recorded runs reported zero provider errors.
- 221 backend tests and a successful frontend production build at the evidence snapshot.
Measured evidence
95.0% → 85.0%
Pass rate, baseline vs degraded prompt
Repository-recorded 20-case RAG regression documented on 2026-08-05.
Controlled comparison, not a customer or usage metric.
1 → 3
Failed cases across the same dataset
Same fixed 20-case dataset; only the prompt version changed.
$0.0294 → $0.0630
Estimated run cost
Estimated model cost recorded by the runner for each of the two runs.
Estimate produced by the platform, not a billed amount.
221
Backend tests
Latest successful main-branch Backend CI run 32046570725 (2026-08-17), rechecked 2026-08-18.
Point-in-time count, not a live counter.
A reproducible way to compare prompt and model versions on quality, cost, and latency together, with a CI gate that fails a change before it ships rather than after.
Limitations
- A reference implementation seeded with synthetic data, not a product.
- No general-purpose authentication or secure multi-tenant boundary is claimed.
- Public write endpoints would require authentication, authorization, rate limits, audit logging, and threat-model review.
- The runner is synchronous. One documented run of approximately 16 minutes outlasted its initiating browser request; production scale would require asynchronous workers and polling.
- No enterprise SaaS, production multi-tenancy, or large-scale usage is claimed.
Technology stack
- Next.js
- FastAPI
- PostgreSQL
- Python
- TypeScript
- Alembic
- Docker
- GitHub Actions
- Vercel
- Google Cloud Run
Data and privacy
Seeded with synthetic evaluation data. No customer content, credentials, or production traffic is present in the public deployment.