Observing LLMs with FOSS - A Beginner's Guide

• Large Language Models (LLMs) are used widely in applications to generate text, answer questions & automate workflows, but they are non-deterministic [behave differently from traditional systems]

• These applications face challenges such as hallucinations, data leaks, spike cost and difficulty in understanding why a model misbehaved

• LLM Observability enables the ability to see what is happening inside the system - providing visibility into prompts, model calls, latency, costs using traces, metrics, costs & evaluations

• This talks introduces FOSS tools such as OpenTelemetry, OpenLLMetry, Langfuse, Opik can be used to observe and reason about LLM systems

• Based on my experience & exploration with these tools, I'll be showcasing how observability reveals the behaviour of LLMs using these FOSS tools

• By the end of the session, attendees will understand why LLM observability matters and how their workflows are simplified by using open-source tools

Why LLM Observability Matters:

• Understand why LLM systems fail silently and how observability helps make them debuggable and trustworthy

Core Observability Concepts:

• How traces, metrics, logs and evaluations mean for real-world LLM applications

FOSS Tools Landscape:

• Get a clear picture of how FOSS tools like OpenTelemetry, Langfuse, LangTrace & Opik fit into LLM workflows