When your backend was still a simple monolith, debugging felt peaceful. A slow request? Just peek at the server log. Or attach a local profiler. Everything ran in one process, one memory space, and one tidy log file.
Once the architecture splits into a dozen microservices, the situation changes drastically. There is an Auth Service. There is an Order Service. A Payment Gateway. Then a Notification Service and an Inventory Service. One click of the "Checkout" button can trigger dozens of asynchronous network calls between services.
So when a user complains that checkout takes 8 seconds and then times out with a 500, how do you know which service is slow? A database query in the Inventory Service? Or a queue backlog in the payment worker? 🤔
This is where Distributed Tracing arrives as the rescue.
Key takeaways
- Distributed tracing is the key to mapping a request's path across services, something traditional centralized logging cannot crack on its own.
- Its basic structure is just two things: a trace representing the whole journey of a request, and a span recording a specific unit of work in each service or database.
- Context propagation through the W3C trace context, the
traceparentheader, keeps the correlation ID connected between services. - OpenTelemetry is the vendor-agnostic standard for backend instrumentation, easy to wire into visualizers like Jaeger or Grafana Tempo.
Why Is Centralized Logging Not Enough?
Many developers think a centralized log aggregator is enough. Something like Elasticsearch, Loki, or Datadog. That is not enough for solving problems in microservices.
When hundreds of thousands of requests arrive per second, logs from different services mix into a giant sea of text. Finding the link between a log in Service A and a log in Service D becomes a nightmare. There is no common thread tying them together.
Centralized logging tells you what happened inside one service. Distributed tracing tells you how a request flowed across the whole ecosystem. Those are two very different things.
The Anatomy of Distributed Tracing: Trace vs Span
To understand distributed tracing, there are two fundamental concepts to master.
A trace is the entire journey of one transaction or request, from start to finish. Think of a trace as a passenger's travel ticket, from the departure station to the final destination. Every trace has a globally unique ID called a Trace ID.
A span is a single segment, or unit of work, inside a trace. One trace consists of one or many spans. Together they form a tree structure (directed acyclic graph). Each span records:
- The operation name (for example
POST /orders,SELECT * FROM users, orPublish to RabbitMQ) - The start time and execution duration
- Extra metadata in tags/attributes (for example
http.status_code: 200,db.system: postgresql,user.id: 1042) - Event logs (events/logs) plus the error status
With span visualization, you can see which service took longest. That is the critical path. You also learn immediately which point the first error erupted at.
How Context Propagation Works
The big question: how does Service B know that the request it received continues from Service A?
The answer is context propagation. When Service A calls Service B over HTTP, Service A slips the Trace ID and Parent Span ID into the request header. The same happens over a message queue.
The most widely used modern industry standard is W3C Trace Context. The header it sends looks like traceparent:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
This header has 4 parts:
version: the W3C spec version (currently00).trace-id: a unique 16-byte ID for the whole trace (4bf92f3577b34da6a3ce929d0e0e4736).parent-id/span-id: a unique 8-byte ID for the calling span (00f067aa0ba902b7).trace-flags: the flag option. The value01marks this span as recorded, or sampled.
The moment Service B receives the request, its tracing library reads that header. It extracts the Trace ID, then creates a new child span under the matching Parent ID. The chain keeps connecting.
Modern Implementation With OpenTelemetry
In the past we were spoiled for choice. There was OpenTracing, OpenCensus, the Jaeger client, even the Zipkin client. Fortunately the industry agreed on one open standard. It is called OpenTelemetry (OTel). This open source project sits under the Cloud Native Computing Foundation (CNCF).
OpenTelemetry provides SDKs for many languages. There is Python, Go, Node.js, and Java. There is also a central Collector for processing and forwarding telemetry data. That data then goes to visualization backends like Jaeger, Grafana Tempo, or Datadog.
A simple example of automatic instrumentation in FastAPI with Python:
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# 1. Set up the Provider & Exporter to the OpenTelemetry Collector / Jaeger
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
# 2. Initialize FastAPI
app = FastAPI(title="Order Service")
# 3. Automatic FastAPI instrumentation
FastAPIInstrumentor.instrument_app(app)
@app.get("/checkout")
async def checkout():
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("process_payment_logic") as span:
span.set_attribute("payment.method", "credit_card")
# Payment processing logic...
return {"status": "success", "message": "Order processed"}
With a few lines of configuration, every HTTP request to the /checkout endpoint automatically produces a span. The status code is captured, latency is measured, and the context is passed downstream. No need to poke at each piece by hand. 😎
When Does Your System Genuinely Need This?
If your backend is still a monolith and everything runs in one process, tracing can be excess weight. But once the number of independent services passes two, you start to need it. Especially with many asynchronous background workers.
Building microservices without distributed tracing is like driving a foggy highway with no headlights. You know the car is moving. You do not know when you are going to hit a pothole. 😅
Start with instrumenting the HTTP gateway first. Wire it to a local Jaeger through Docker. Enjoy finding the root of a performance problem in seconds.
Happy observability, and may your traces always connect end to end. 👋