If we're asking AppSRE to provide a modern observability suite, we should be poised to make use of it. To that end, we should make sure that:
- Distributed traces are propagated correctly.
- Trace contexts/baggage are not leaked to external services.
- Trace spans are used throughout, ideally for both runtime and distributed traces.
Additionally, we should:
- Migrate direct usage of prometheus to otel/metric.
- Investigate using Sentry for trace exporting.
- Investigate enabling a flight recorder.