I'm working on evaluating AI systems and the data behind them: designing rubrics and evaluation sets, collecting and checking human labels, and comparing LLM autograders with human ratings to see where they can be relied on.
- 🎯 Focus: evaluating conversational AI, LLMs, and speech systems
- 💡 Principle: a metric is only useful if it predicts real-world quality and people trust it
- 🔁 Approach: measure → diagnose failures → fix → re-measure → gate the release
- ✍️ Writing: notes on evaluation and data quality on Substack
| Area | What it involves |
|---|---|
| 🤖 LLM autograders & rubrics | Rubric design, evaluation set development, and calibrating an LLM autograder against human ratings |
| 📊 Metrics that mean something | Cohen's kappa for human–LLM agreement (including the kappa paradox), error rates, and checking that findings hold on larger, harder samples |
| 📚 Grounded, factual responses | Groundedness, citation accuracy, and hallucination rate for retrieval-based (RAG) answers |
| 🎙️ Conversational & speech quality | Intent, transcription, and speaker annotation, plus speech recognition accuracy |
| ✅ Quality gates | Golden sets, sampling audits, annotator scorecards, and CI quality gates (GitHub Actions) that block regressions before release |
🔬 Featured work: annotation-eval-lab
An end-to-end lab for building trustworthy evaluation, from human annotation to automated grading and release gates.
| Layer | What it covers |
|---|---|
| Human annotation | Image, audio/voice, text, and video labeling with clear guidelines and QA checks |
| Speech evaluation | Comparing automatic transcripts with human ones (WER) |
| LLM evaluation | Rubric ratings, preference judgments, and a Claude autograder calibrated against human ratings with Cohen's kappa, then re-tested on harder prompts |
| RAG evaluation | Groundedness, citation accuracy, and hallucination rate |
| Translation quality | Human error annotation compared with automatic scores |
| Quality framework | Golden sets, sampling audits, annotator scorecards, and CI quality gates with GitHub Actions |
| Red teaming (separate repo) | Find failures, fix, re-scan, and lock in regression tests |
Built mainly with Python and Label Studio.