OTel Collector can redact, route, and count auth-failure signals in real time. Learn production-ready Go and Collector setup with verification. Read now.
What Security Observability Means in OpenTelemetry
Security observability is the practice of treating auth failures, policy denials, and related control-plane signals as first-class telemetry—not as buried log lines you grep after an incident. OpenTelemetry gives you a shared pipeline for traces, metrics, and logs; the Collector sits in the middle of that pipeline so you can shape sensitive data before it leaves your network. For auth failures, the useful outcome is simple: you keep enough context to detect abuse and debug access problems, while stripping secrets, tokens, and PII that should never land in a backend.
A production-ready setup starts in your Go services and finishes in the Collector. Applications emit structured events when login, token validation, or API-key checks fail. The Collector then redacts high-risk attributes, routes security signals to a dedicated destination, and counts failures so operators can alert on rate and pattern—not on every raw log line.
Instrument Auth Failures in Go
Emit auth-failure signals at the decision point, not deep inside framework middleware you do not control. Prefer a small set of stable attributes: result (denied), reason category (invalid credential, expired token, missing scope), principal type (user, service, anonymous), and resource or route class. Avoid putting raw passwords, full bearer tokens, session cookies, or full request bodies into span or log attributes. If you need correlation, use a short-lived request or span ID you already generate elsewhere.
Use the OpenTelemetry Go SDK to create either a span event on the auth path or a structured log record tied to the active context. Pair that with a counter metric for auth failures labeled by reason category and service. Metrics give you cheap real-time rates; logs and spans give you enough detail for investigation. Keep label cardinality low—never put user IDs or full email addresses on metric labels—so the series stay usable under load.
Redact, Route, and Count in the Collector
Configure the Collector as the enforcement point between apps and storage. Redaction processors should drop or hash attributes that match secret patterns (authorization headers, cookie values, API keys) and truncate free-text fields that might hold credentials. Routing exporters (or processor-based pipelines) should send security-tagged telemetry to a hardened pipeline—stricter retention, tighter access—while general app telemetry continues on the default path. Counting can happen via metrics already emitted by the app, or by transforming filtered log/span signals into metric points in the Collector so dashboards and alerts stay consistent even if log volume is throttled.
- Redact: remove or mask tokens, passwords, and PII before export.
- Route: isolate auth-failure streams from general observability backends.
- Count: expose failure rates by reason and service for real-time alerting.
Keep pipelines explicit: one receiver path, processors in a fixed order (filter → transform/redact → batch), then separate exporters. Order matters; redaction must run before any exporter that leaves the trust boundary.
Verify the Pipeline Before You Rely on It
Verification is part of the tutorial outcome, not an afterthought. Trigger controlled auth failures in a non-production environment: wrong password, expired token, missing role. Confirm the Go service emits the expected metric increment and a redacted log or span event. Confirm the Collector drops forbidden attributes, forwards only the security-tagged stream to the intended exporter, and that your counter (or derived metric) moves in real time. Check negative cases too: a successful login should not inflate the failure counter, and a deliberate secret in an attribute should never appear downstream after redaction.
Document the attribute contract and the Collector config as code so reviews catch regressions. When the pipeline is verified end to end, you have a practical security observability path: auth failures are visible, countable, and safe enough for production backends—without treating every log dump as an investigation surface.