LLM Evaluation Techniques in Practice

This course explores benchmarking for open-ended generation tasks like summarization. You'll experiment with different prompting styles, compare models like GPT-3.5 and GPT-4, and evaluate results using both fuzzy string similarity and semantic similarity via embeddings.

In this course, you'll explore how to evaluate the fluency and likelihood of LLM outputs using internal scoring signals like log probabilities and perplexity. You'll work with OpenAI's completion models to analyze how models "think" under the hood. This course builds naturally on the first two by focusing on model-internal evaluation instead of external references.

In this course, you’ll experiment with deeper aspects of LLM evaluation: token usage efficiency, temperature sensitivity, model output consistency, and detecting hallucinations. Through lightweight API experiments, you’ll develop intuition for how models behave beyond accuracy scores.