It is easy to instrument what a framework exposes and difficult to decide what an operator will need during an unfamiliar failure. The first approach creates dashboards. The second creates understanding.
Begin with a user outcome
Ask whether a user can complete the important transaction: authenticate, submit an order, retrieve a record, or receive a result. Then work backward through the dependencies that make the outcome possible.
This produces signals with context. Request success, latency, queue age, and dependency errors say more than a wall of component utilization. Resource metrics still matter, but they explain constraints after the service-level symptom establishes why anyone should care.
Preserve useful dimensions
Telemetry should let an operator separate behavior by region, version, dependency, operation, and error class without exposing sensitive data or creating unbounded cardinality. Choose those dimensions deliberately; adding them during an incident is too late.
Instrumentation earns its cost when it helps reject a hypothesis, not merely when it proves the system emitted data.
Connect the evidence
Metrics identify shape and timing. Traces show the path. Logs preserve detailed events. Consistent service names, timestamps, deployment markers, and correlation context let responders move between them without rebuilding the incident timeline by hand.
Also capture absence. A missing heartbeat, an expected event that never appears, or a queue that stops draining may be the clearest signal in the system.
Remove what nobody uses
Observability has an operating cost: storage, query performance, alert fatigue, and cognitive load. Review dashboards and alerts after incidents. Keep the signals that changed a decision. Improve those that almost helped. Retire those that consistently produce no action.
Use design reviews to ask the future incident questions early. How will we distinguish a bad dependency from a bad release? How will we find one affected tenant without exposing tenant data? What evidence survives if the application is unavailable? The answers should shape instrumentation before launch.
A good observability practice is a loop between questions and evidence. The system changes, the questions get sharper, and the telemetry follows.