How to Trace a Slow API Request Across Services

An order status endpoint takes two seconds, but the application log reports only 40 milliseconds of handler work. Before increasing timeouts or adding capacity, follow one slow request across the services it actually used.

Start with the request the user sees

Record end-to-end latency by route and outcome. Compare a normal request with one from the slow tail, such as the 95th percentile. Keep failed requests separate so that fast failures do not make successful requests look faster. Google’s SRE guide to monitoring explains why latency, traffic, errors and saturation provide complementary views of a system.

Find the wait in the trace

In OpenTelemetry, a trace follows a request through a set of spans. Instrument the gateway, handler and outbound calls first. If the handler takes two seconds but the database query takes 30 milliseconds, add a span around connection acquisition to distinguish waiting for the pool from executing the query. Compare parent and child timings on the timeline. Parallel calls can overlap, so summing every span can overstate the time the user waited.

Use route templates and operation names to group traces. Keep customer identifiers, authorization headers and request bodies out of telemetry unless their collection is explicitly governed. A route and a trace ID are usually enough to start the investigation.

Verify a change with comparable traffic

Write down the hypothesis before changing configuration. If pool wait dominates, check connection usage and database saturation before raising the pool limit. A bigger pool can move the queue to the database. Reproduce the same traffic shape after the change and compare end-to-end latency, error rates and saturation. Keep the change only if the user-facing metric improves without creating a new bottleneck.

Start with one route, one slow trace and one measurable hypothesis. That gives the team a test it can repeat when the next latency regression appears.