Presentation: From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry

In an **InfoQ** presentation titled Presentation: From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry, **Ben O'Mahony** outlined an approach for constructing custom **AI-powered Language Server Protocols** (**LSPs**). The method shifts developer tooling beyond standard rule-based checkers by capturing production telemetry to inform model training.

The technical mechanics center on natively instrumenting **AI agents** using **OpenTelemetry** (**OTEL**). System events track specific user behaviors, including accepting, dismissing, or regenerating code fixes, and convert these actions into implicit labels. This feedback stream forms a continuous data flywheel designed to capture model behavior during active development.

For software engineers and system architects, this design decouples model optimization from manual data labeling. By leveraging implicit feedback loops from developer interactions, teams can extract and distill **frontier model** performance into specialized, cost-effective **small language models** (**SLMs**).

In terms of competitive architecture, this approach replaces static analysis paradigms with dynamic observability pipelines. Using **OpenTelemetry** inside the language server enables organizations to decrease reliance on costly **frontier models**, reducing runtime cost and latency by shifting workload to localized or smaller models.

The primary practical takeaway is to instrument developer interaction points to capture implicit signals like acceptances, dismissals, and regenerations via **OpenTelemetry**. System builders should watch how effectively these production telemetry pipelines convert real-time feedback into datasets for training smaller downstream models.