Masterclass: EvalOps & AIOps in praksis - From LLM og RAG Evaluation to AI in operations

AI systems are moving from experiments to business-critical solutions – and with that comes a new responsibility: How do you ensure that models actually deliver stable value, quality, and compliance in operation? This gives rise to two new disciplines:

• EvalOps – systematic evaluation of AI and LLM solutions throughout their entire lifecycle.
• AIOps – intelligent automation and monitoring of AI system performance in production.

In this masterclass, you will learn how to operationalize EvalOps and AIOps in practice – with a focus on LLM and RAG evaluation, measuring business value, and continuous improvement through feedback loops, observability, and CI/CD integration. 

Learning points

By the end of the day, you will be able to:

• Establish an EvalOps framework that makes AI performance measurable and actionable.
• Implement LLM and RAG evaluation in development and operations—with metrics such as correctness, faithfulness, contextual relevance, and hallucination rate.
• Integrate AIOps principles such as observability, operational reliability, and incident prediction into AI workflows.
• Design evaluation pipelines that link technical metrics with business KPIs (e.g., efficiency, quality, user response, ROI).
• Create a closed feedback loop between development, users, and operations for continuous optimization.

We combine presentation + interactive workshop, Hands-on demonstrations with DeepEval and EvalOps pipelines. We will focus on usable frameworks rather than heavy coding. Participants receive EvalOps template and toolkit for AI evaluation.

Target audience:

You will need to have an understanding of AI and data architecture (no heavy coding required). An experience with ML/LLM, data engineering, or IT operations is an advantage.

• IT managers, 
• AI developers, 
• Data scientists, 
• Architects,
• Product managers 
• Compliance proffessionals and Auditors, who want to understand the technical aspects of AI risk measurement
• Other professions responsible for developing, evaluating, and operating AI/GenAI solutions.