Evaluating an LLM Agent Before You Have Eval Infrastructure
Fourth and final post in this series about a WhatsApp lead-qualification agent we built at a CRM for small businesses. The engine (Postgres and Celery durable runs), the loop (debounce, polling, exits), and the LLM layer (structured extraction, cost metering, injection defenses) are the first three posts. This one answers the question every engineer asks about agent work, usually first: how do you eval it? The honest answer: our v0 is about 90 lines of framework, four scripted scenarios, a staff-only page, and a human doing the judging. No scoring pipeline, no LLM-as-judge, no eval database. This post is why that is a design and not a shortcut, and why we skipped the eval packages we had already shortlisted. ...