Ask an LLM the same question five times and you might get five different answers. Four sound plausible, one is actually correct. For thirty years we tested by asserting equality: one input, one expected output, pass or fail. AI quietly broke that contract, and a system can be confidently wrong while still passing every accuracy benchmark.

So how do you test something that never behaves the same way twice?

This session makes the shift from verifying correctness to evaluating behaviour. We look at why AI fails in patterns across layers, then get hands-on with the evaluation toolkit: reference-based metrics like BLEU and ROUGE, reference-free metrics like semantic similarity, grounding and faithfulness, and LLM as a judge.

The big takeaway is a layered approach to AI quality: just as failures happen layer by layer, evaluation has to be built the same way, from deterministic checks at the bottom to human judgement at the top.

You will leave with a clear mental model and a practical toolkit you can apply at work.