Udemyexperten

RAG-LLM Evals & Test Automation for Beginners

Learn RAG (Retrieval-Augmented Generation) for LLMs! Build, evaluate & test AI systems from scratch using RAGAS, Python & Pytest.

4.4(605)
6,295students
8h 40m
Certificate
Rahul Shetty Academy
Rahul Shetty AcademyQA Instructor @ 1 Million Students| 25+ Best Selling Courses

Course Content

RAG-LLM Evals & Test Automation for Beginners is an advanced Udemy course taught by Rahul Shetty Academy. It runs 8h 40m, is taught in English, and includes a certificate of completion. Learners rate it 4.4 out of 5 from 605 reviews, with 6,295 students enrolled. It is a paid course, last updated September 2025.

Large Language Models (LLMs) are rapidly transforming business, with companies increasingly deploying custom AI-powered Retrieval Augmented Generation (RAG) LLMs to enhance customer service. But how do engineers ensure these systems are performing optimally? Unlike traditional software, evaluating AI systems demands a specialized approach. This course provides a comprehensive understanding of LLM testing methodologies.

We'll begin with a foundational overview of how LLMs function, exploring the underlying architecture. Then, we'll delve into the essential LLM evaluation metrics used to assess performance. You'll gain practical skills using the RAGAS framework, a powerful library for evaluating LLM metrics through scripted examples. Learn to leverage Pytest assertions to verify benchmark scores and build a robust LLM test and evaluation automation framework.

What you will learn:

  • High-level overview of Large Language Models (LLMs): Understand the core concepts and capabilities of LLMs.
  • Custom LLM Development with RAG: Learn how to build custom LLMs using the Retrieval Augmented Generation (RAG) architecture.
  • Common RAG LLM Evaluation Metrics: Explore the key benchmarks and metrics used to evaluate RAG-based LLMs.
  • RAGAS Evaluation Framework: Get an introduction to the RAGAS framework for evaluating and testing LLMs.
  • Automated Metric Assertions: Generate practical scripts to automate and assert LLM metric scores.
  • Automating Interaction Scenarios: Automate both single-turn and multi-turn interactions with LLMs using the RAGAS framework.
  • Test Data Generation: Learn to generate test data for evaluating LLM metrics using the RAGAS framework.

By the end of this course, you'll be equipped to create a RAGAS Pytest evaluation framework to confidently assert the metrics of your RAG-based (custom) LLMs.

Important Note: This course covers the Top 7 Metrics commonly used to evaluate and test LLMs. The same principles can be applied to evaluating other metrics.

Hands-on Experience:

The course provides hands-on practice with RAG-LLMs. For the scripting phase, a basic subscription to OpenAI is required (a minimum of $10 credit is sufficient) to access their APIs.

Course Prerequisites:

  • Python: Basic Python programming skills are required.
  • Pytest Basics: Familiarity with Pytest is essential for understanding the framework.
  • API Testing: Basic knowledge of API testing is beneficial.

We've included two dedicated sections at the end of the course to reinforce the Python and Pytest knowledge needed to succeed.

$ whoami — the devs behind tutorialsearch.io

Want AI in your product? Hire the team behind TutorialSearch.io

LLM features, RAG, agents and automation wired into a real product — built by the devs who shipped this site.

5.0
1h 13m
intermediate
pluralsight
pluralsight
5.0
2h
intermediate
pluralsight