Deep Dive into Distributed Tracing in ASP.NET Core: OpenTelemetry in Practice
Track requests across services, APIs, databases, and queues using OpenTelemetry and distributed tracing
As applications become more distributed, understanding what happens during a single user request becomes increasingly difficult. A request may travel through multiple APIs, databases, caches, message queues, and background services before completing. Distributed tracing solves this challenge by providing end-to-end visibility into request flows.
In this guide, you’ll learn how distributed tracing works, why OpenTelemetry has become the industry standard, and how to implement practical tracing solutions in ASP.NET Core applications.
Why Traditional Logging Is No Longer Enough
Years ago, many applications were relatively simple.
A user made a request.
The request hit a web server.
The application queried a database.
A response was returned.
If something went wrong, logs were usually enough.
Modern systems are very different.
A single request may now travel through:
API Gateway
ASP.NET Core Web API
Authentication Service
Product Service
Redis Cache
SQL Database
Message Queue
Background Worker
Each component may have its own logs.
When failures occur, finding the root cause becomes difficult.
This is where distributed tracing becomes essential.
What Is Distributed Tracing?
Distributed tracing allows developers to follow a request as it moves through multiple systems.
Think of it as attaching a GPS tracker to every request.
Instead of viewing isolated logs, you can see:
Where the request started
Which services it visited
How long each step took
Where failures occurred
Which dependencies caused delays
Distributed tracing provides a complete picture of request execution.
Understanding Traces, Spans, and Context
Distributed tracing revolves around three key concepts.
Trace
A trace represents the complete journey of a request.
For example:
User Request
├─ API Gateway
├─ Product Service
├─ SQL Database
├─ Redis Cache
└─ ResponseEverything belongs to a single trace.
Span
A span represents an individual operation.
Examples include:
Calling a database
Querying Redis
Sending a message
Invoking another API
Each trace contains multiple spans.
Context
Context links spans together.
Without context propagation, traces break apart and become useless.
OpenTelemetry automatically handles much of this process.
Why OpenTelemetry Has Become the Standard
Before OpenTelemetry, observability tools often used proprietary approaches.
Organizations frequently became locked into specific vendors.
OpenTelemetry was created to solve this problem.
It provides:
Vendor-neutral instrumentation
Standardized telemetry collection
Cross-platform support
Broad ecosystem integration
OpenTelemetry is now supported by major platforms including:
Microsoft Azure
AWS
Google Cloud
Grafana
Datadog
New Relic
Official project:
https://opentelemetry.io
Observability vs Monitoring
These terms are often confused.
Monitoring focuses on answering known questions.
Examples:
Is the service running?
How many requests failed?
Is CPU usage high?
Observability helps answer unknown questions.
Examples:
Why is checkout suddenly slow?
Which dependency caused latency spikes?
Which service introduced failures?
Distributed tracing is a core pillar of observability.
The Three Pillars of Observability
Modern observability typically includes:
Logs
Metrics
Traces
Logs explain what happened.
Metrics show how often something happens.
Traces reveal where it happened.
Together they provide a complete operational picture.
Why Tracing Matters in ASP.NET Core
Consider an e-commerce application.
A customer clicks “Place Order.”
The request triggers:
Authentication
Inventory validation
Payment processing
Shipping creation
Notification service
A failure occurs.
Which step caused the issue?
Without tracing:
Search logs manually
Correlate timestamps
Guess relationships
With tracing:
View the entire workflow instantly
This becomes especially valuable in architectures we’ve discussed previously:
ASP.NET Core and Azure Service Bus
Saga Patterns
Fault-Tolerant Systems
Advanced Retry Strategies
Distributed tracing ties these systems together.
Installing OpenTelemetry
Start by adding required packages.
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.SqlClientThese packages provide automatic instrumentation.
Configuring OpenTelemetry
In Program.cs:
builder.Services.AddOpenTelemetry()
.WithTracing(tracing =>
{
tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddSqlClientInstrumentation();
});This immediately starts collecting trace information.
Automatic Instrumentation
One of OpenTelemetry’s greatest strengths is automatic instrumentation.
Without changing business logic, you can collect traces from:
Incoming HTTP requests
Outgoing HTTP calls
SQL queries
Entity Framework operations
gRPC services
This dramatically reduces implementation effort.
Viewing Traces Locally
During development, traces can be exported to the console.
builder.Services.AddOpenTelemetry()
.WithTracing(builder =>
{
builder
.AddAspNetCoreInstrumentation()
.AddConsoleExporter();
});This helps developers understand how traces are structured.
Understanding Trace Hierarchies
Imagine this workflow:
Order Request
├─ Validate User
├─ Check Inventory
│ ├─ Redis Cache
│ └─ SQL Query
├─ Process Payment
└─ Send ConfirmationEach operation becomes a span.
The complete tree becomes a trace.
This hierarchy makes bottlenecks easy to identify.
Adding Custom Spans
Automatic instrumentation is powerful.
But custom spans provide additional business insight.
Example:
private static readonly ActivitySource ActivitySource =
new("OrderService");
public async Task ProcessOrderAsync()
{
using var activity =
ActivitySource.StartActivity("ProcessOrder");
await Task.Delay(100);
}This creates a custom span visible in trace visualizations.
Capturing Business Operations
Not everything revolves around technical dependencies.
Sometimes you want visibility into business workflows.
Examples:
Calculate Discount
Validate Coupon
Generate Invoice
Process Refund
Custom spans allow these operations to appear in traces.
Context Propagation
Tracing only works if context moves between services.
Consider:
API A
↓
API B
↓
API CIf trace context is lost between services:
Visibility breaks
Relationships disappear
OpenTelemetry automatically propagates context through standard HTTP headers.
This allows traces to remain connected.
Tracing HTTP Requests
Outgoing HTTP requests are automatically captured.
Example:
var response =
await _httpClient.GetAsync("/products");OpenTelemetry records:
URL
Duration
Status code
Parent trace
This creates a complete dependency map.
Tracing Database Calls
Database performance often becomes a bottleneck.
OpenTelemetry automatically traces:
await _dbContext.Products
.Where(p => p.IsActive)
.ToListAsync();Useful information includes:
Query duration
Database dependency
Success or failure
This greatly simplifies performance investigations.
Tracing Message-Based Architectures
Many modern ASP.NET Core systems rely on messaging.
Examples include:
Azure Service Bus
RabbitMQ
Kafka
Distributed tracing becomes even more important because workflows become asynchronous.
Tracing helps developers understand:
Message publication
Message consumption
Processing delays
Queue bottlenecks
OpenTelemetry and ASP.NET Core APIs
Minimal APIs work seamlessly.
app.MapGet("/products", async () =>
{
return Results.Ok();
});Each request automatically generates trace information.
No additional code required.
Sampling Strategies
Not every trace needs to be collected.
High-volume systems can generate millions of traces daily.
Sampling helps control costs.
Common approaches:
Always sample
Never sample
Percentage-based sampling
Adaptive sampling
Example:
.SetSampler(new TraceIdRatioBasedSampler(0.1))This collects approximately 10% of traces.
Exporting Trace Data
OpenTelemetry separates collection from storage.
Popular destinations include:
Azure Monitor
Jaeger
Zipkin
Grafana Tempo
Datadog
This flexibility prevents vendor lock-in.
Visualizing Traces with Jaeger
Jaeger is one of the most popular tracing platforms.
It provides:
Trace timelines
Dependency graphs
Span details
Performance analysis
Developers can visually inspect entire request journeys.
Diagnosing Performance Problems
Suppose a page suddenly becomes slow.
Tracing reveals:
Request Duration: 3.8s
Authentication: 40ms
Product Service: 120ms
Redis Cache: 5ms
SQL Query: 3.4sThe bottleneck becomes obvious immediately.
Without tracing, finding this issue could take hours.
Tracing and Distributed Caching
Distributed caching often improves performance dramatically.
But cache misses can still cause problems.
Tracing helps identify:
Cache hit rates
Cache miss paths
Database fallbacks
This pairs naturally with distributed caching architectures.
Tracing Fault-Tolerant Systems
In our previous article on fault tolerance, we discussed:
Retries
Circuit breakers
Bulkheads
Graceful degradation
Tracing makes these patterns visible.
You can observe:
Retry attempts
Circuit breaker activations
Fallback executions
This turns resilience from theory into measurable behavior.
Tracing Saga Workflows
Saga patterns coordinate long-running distributed transactions.
Tracing allows developers to follow:
Order creation
Inventory reservation
Payment processing
Shipment generation
Across multiple services.
This provides enormous operational value.
Security Considerations
Be careful with trace data.
Avoid recording:
Passwords
Credit card numbers
Personal information
Authentication tokens
Telemetry should support troubleshooting without exposing sensitive data.
Common Mistakes
A frequent mistake is collecting traces without using them.
Another is tracing everything indiscriminately.
This creates:
Storage costs
Noise
Analysis difficulties
Focus on meaningful visibility.
When Distributed Tracing Is Most Valuable
Distributed tracing provides the greatest value when systems include:
Multiple services
External APIs
Message queues
Background processing
Cloud infrastructure
Simple applications may not require extensive tracing.
Complex systems almost always benefit.
Real-World Example
Imagine a food delivery platform.
A customer places an order.
The workflow touches:
User Service
Restaurant Service
Payment Service
Delivery Service
Notification Service
A trace shows every step.
If delays occur, engineers can pinpoint the exact dependency causing problems.
This is the power of distributed tracing.
How This Fits Your ASP.NET Core Journey
So far, we’ve explored:
Azure Service Bus
Saga Patterns
Distributed Caching
Retry Strategies
Fault-Tolerant Systems
Distributed tracing now provides the visibility layer that ties all these architectural patterns together.
Building distributed systems is only half the challenge.
Understanding them in production is the other half.
Closing Thoughts
Modern ASP.NET Core systems are increasingly distributed.
As complexity grows, traditional logging alone becomes insufficient.
OpenTelemetry provides a standardized, vendor-neutral approach to distributed tracing that helps developers understand exactly how requests move through their systems.
By implementing tracing early, teams gain:
Faster troubleshooting
Better performance visibility
Improved reliability
Stronger operational insight
As your applications scale, distributed tracing becomes less of a luxury and more of a necessity.
Join The Community
Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core architecture guides, observability strategies, and real-world engineering patterns. Join the Substack Chat and connect with developers building modern cloud-native applications.