In this guide, we’ll explore how to build an Internal Developer Platform using ASP.NET Core and create a foundation that allows development teams to focus on writing business features instead of managing infrastructure.

Why Development Slows as Teams Grow

A small startup might have:

• Five developers

• One application

• One deployment pipeline

Everyone knows where everything is.

Need a database?

Someone creates one.

Need a new API?

Someone builds it.

Need production access?

Just ask.

Everything works.

Until it doesn’t.

As organizations grow, so do requests.

Developers begin asking questions like:

• Can someone create a Kubernetes namespace?

• Can I get a SQL database?

• Who creates Azure resources?

• Where do I request secrets?

• How do I deploy to staging?

• Which logging standard should I follow?

Eventually engineers spend more time waiting than building.

What Is an Internal Developer Platform?

An Internal Developer Platform is software built for developers inside an organization.

Instead of manually requesting resources, developers use a platform to provision them themselves.

Think of it as an internal app store.

Instead of installing games, developers request:

• New projects

• Databases

• Storage accounts

• API keys

• Kubernetes environments

• Deployment pipelines

Everything follows company standards automatically.

Why Organizations Build Internal Platforms

The biggest goal is reducing developer friction.

Without an IDP:

Developer writes ticket.

Operations creates infrastructure.

Security reviews access.

Platform team configures deployment.

Developer waits.

With an IDP:

Developer clicks a button.

Platform performs everything automatically.

The developer continues working.

Platform Engineering vs DevOps

These terms are closely related.

DevOps focuses on collaboration between development and operations.

Platform Engineering creates products that help developers succeed.

Instead of manually helping every team, platform engineers build reusable tools.

Those tools become the Internal Developer Platform.

Understanding Self-Service Development

Imagine buying groceries.

Years ago, every item required asking a shop assistant.

Today we simply use self-checkout.

Internal platforms work the same way.

Instead of requesting resources manually, developers provision them independently.

The platform performs all the complicated work behind the scenes.

What Can an Internal Developer Platform Provide?

A mature platform might offer:

• Project templates

• API creation

• Infrastructure provisioning

• Secret management

• Logging configuration

• Monitoring dashboards

• CI/CD pipelines

• Deployment approvals

• Security scanning

• Documentation

Developers interact with one consistent interface.

Why ASP.NET Core Is a Great Choice

ASP.NET Core excels at building internal tools.

It offers:

• High performance

• Cross-platform support

• Excellent security

• REST API development

• Background services

• Identity integration

• Cloud-native capabilities

These features make it ideal for building platform APIs.

Designing the Platform

Think of the platform as multiple services working together.

Developer Portal
        │
        ▼
Platform API
        │
 ┌──────┼────────┐
 ▼      ▼        ▼
Templates Deployments Infrastructure
        │
        ▼
Cloud Resources

Each component has a specific responsibility.

This separation keeps the platform maintainable.

Building the Platform API

Everything begins with an API.

Developers request services through HTTP endpoints.

Example:

POST /projects

Payload:

{
  "name": "InventoryService",
  "template": "webapi"
}

The platform performs the remaining work automatically.

Creating the API

Example:

app.MapPost("/projects",
(ProjectRequest request) =>
{
    return Results.Ok(
        $"Creating {request.Name}");
});

Simple APIs often orchestrate much larger workflows.

Automating Project Creation

Suppose every microservice requires:

• Standard folder structure

• Logging

• Health checks

• Docker support

• OpenTelemetry

• Authentication

Instead of manually configuring each project, developers choose a template.

The platform generates everything consistently.

Project Templates

Templates eliminate repetitive work.

A template might include:

• Authentication configured

• Swagger enabled

• Logging configured

• Dockerfile included

• CI/CD pipeline ready

• Unit tests added

Every new service begins with the same high-quality foundation.

Managing Templates

Templates evolve over time.

For example:

Version 1:

• .NET 8

• Basic logging

Version 2:

• OpenTelemetry

• Health checks

• Distributed tracing

• Rate limiting

Developers automatically benefit from platform improvements.

Infrastructure as Code

Infrastructure should never be created manually.

Instead use:

• Bicep

• Terraform

• ARM Templates

The platform invokes these tools automatically.

Developers simply request resources.

Provisioning Cloud Resources

Imagine a developer needs:

• Azure SQL

• Azure Storage

• Service Bus

• Key Vault

Instead of opening multiple Azure portals, they submit one request.

The platform provisions everything.

Background Processing

Some operations take several minutes.

Examples:

• Creating Kubernetes clusters

• Deploying infrastructure

• Provisioning databases

ASP.NET Core Background Services handle these long-running jobs efficiently.

Example:

public class ProvisioningWorker
    : BackgroundService
{
    protected override async Task ExecuteAsync(
        CancellationToken token)
    {
        while (!token.IsCancellationRequested)
        {
            await Task.Delay(5000, token);
        }
    }
}

Background workers keep APIs responsive.

Building a Developer Portal

The portal becomes the front door of the platform.

Developers can:

• Create projects

• Request environments

• View deployments

• Browse documentation

• Monitor applications

Everything is available from one location.

Authentication

Only authorized developers should access platform resources.

ASP.NET Core Identity integrates easily with:

• Microsoft Entra ID

• Azure AD

• OAuth

• OpenID Connect

Authentication ensures platform requests remain secure.

Authorization

Different developers require different permissions.

Examples:

Developers

• Create projects

Team Leads

• Approve production deployments

Platform Engineers

• Manage templates

Administrators

• Configure infrastructure

Policy-based authorization keeps permissions manageable.

This builds directly on concepts from our previous article on advanced authorization.

API Versioning

Internal platforms evolve continuously.

Versioning prevents breaking existing developer workflows.

Example:

/api/v1/projects

/api/v2/projects

Existing automation continues working while new features are introduced.

Logging Platform Activity

Everything the platform does should be logged.

Examples:

• Project created

• Deployment started

• Database provisioned

• Secret generated

Audit logging becomes especially valuable for compliance and troubleshooting.

As discussed in our previous article, audit logs provide accountability across complex systems.

Monitoring Platform Health

An Internal Developer Platform quickly becomes business-critical.

Monitor:

• API availability

• Provisioning success rate

• Deployment duration

• Queue lengths

• Failed workflows

Health checks and OpenTelemetry provide excellent visibility.

This connects naturally with our recent observability articles.

Building for Reuse

One of the biggest goals of an IDP is eliminating duplicated effort.

Instead of every team solving the same problems independently, the platform provides reusable solutions that everyone benefits from.

This creates consistency across the entire organization.

Coming in Part 2

Our Internal Developer Platform can now provide developers with self-service tools and standardized project creation, but there’s still more work to do before it becomes a truly enterprise-ready platform.

In Part 2, we’ll automate CI/CD pipelines, provision development and production environments on demand, manage secrets securely, implement deployment approvals and governance, and explore how to scale the platform across multiple teams. We’ll finish by building a complete real-world Internal Developer Platform that helps developers deliver software faster while maintaining security, consistency, and operational excellence.

Closing Thoughts

An Internal Developer Platform is much more than a collection of internal tools. It becomes the foundation that enables development teams to work faster, deliver more consistently, and spend less time on repetitive operational tasks. By providing self-service capabilities, standardized templates, automated infrastructure provisioning, and reusable APIs, organizations can reduce bottlenecks while improving security, reliability, and developer experience. As your platform evolves, you can expand it with automated deployment pipelines, environment provisioning, secret management, governance controls, and observability, transforming it into a central hub that supports every stage of the software development lifecycle.

Subscribe Now

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core tutorials, architecture deep dives, and real-world engineering guides. Join our Substack Chat to connect with fellow developers, ask questions, and stay up to date with the latest ASP.NET Core best practices.

Subscribe now

In this guide, you’ll learn how to build audit logging into ASP.NET Core applications, automatically capture meaningful events, support regulatory compliance, and create audit trails that remain useful as your applications grow.

Why Audit Logging Matters

Imagine a customer calls support and says:

“I never changed my shipping address.”

A week later another customer reports:

“My account was deleted.”

An administrator notices that sensitive information has disappeared from the database.

Without an audit log, you only know that something happened.

You don’t know:

• Who made the change

• When it happened

• What the previous value was

• What the new value became

• Which application made the change

Audit logging fills in these missing details.

It creates a permanent history of important events that helps developers troubleshoot problems, helps administrators investigate incidents, and helps organizations satisfy compliance requirements.

Application Logs vs Audit Logs

Many developers assume ordinary application logs are enough.

They are not.

Application logs record how the application behaves.

Examples include:

• Exceptions

• Startup events

• Performance metrics

• Debug information

Audit logs answer business questions instead.

Examples include:

• Who updated this customer?

• Who approved this payment?

• Who downloaded this report?

• Who deleted this document?

The two logging systems complement each other.

They should not replace one another.

What Should Be Audited?

Not every action deserves an audit record.

Focus on events that affect business operations or security.

Examples include:

• User logins

• Failed login attempts

• Password changes

• Profile updates

• Record creation

• Record modification

• Record deletion

• Permission changes

• Role assignments

• Administrative actions

• Data exports

• Financial approvals

Auditing every tiny event often creates unnecessary noise.

Choose events that would matter during an investigation.

Designing an Audit Record

Every audit entry should answer five simple questions.

Who?

What?

When?

Where?

Why?

A typical audit record might contain:

• User ID

• Username

• Action performed

• Entity affected

• Previous values

• New values

• Timestamp

• IP address

• Browser or device

• Correlation ID

The more context captured, the more useful the audit log becomes.

A Simple Audit Model

Example:

public class AuditLog
{
    public int Id { get; set; }

    public string UserId { get; set; }

    public string Action { get; set; }

    public string Entity { get; set; }

    public string OldValues { get; set; }

    public string NewValues { get; set; }

    public DateTime Timestamp { get; set; }

    public string IpAddress { get; set; }
}

This model provides a solid starting point.

Real-world systems often add more metadata.

Capturing User Identity

Audit logs become much more valuable when every action is linked to a specific user.

ASP.NET Core makes this easy.

Example:

var username = User.Identity?.Name;

For applications using JWT authentication:

var userId =
    User.FindFirst("sub")?.Value;

Capturing user identity should be standard practice.

Recording Request Information

Sometimes the request itself provides useful evidence.

Consider recording:

• IP address

• HTTP method

• URL

• User agent

• Correlation ID

Example:

var ip =
    HttpContext.Connection
        .RemoteIpAddress?
        .ToString();

These details help reconstruct what happened during investigations.

Automatically Auditing Entity Framework Core

One of the easiest ways to implement audit logging is by intercepting Entity Framework Core changes.

Every tracked entity already knows whether it has been:

• Added

• Modified

• Deleted

This makes EF Core an excellent source of audit events.

Detecting Changes

Example:

foreach (var entry in
    ChangeTracker.Entries())
{
    if (entry.State ==
        EntityState.Modified)
    {
        // Create audit record
    }
}

Instead of manually writing audit code throughout the application, you centralize it in one place.

This greatly simplifies maintenance.

Overriding SaveChanges

A common pattern is overriding SaveChangesAsync.

Example:

public override async Task<int>
SaveChangesAsync(
CancellationToken cancellationToken)
{
    CreateAuditEntries();

    return await
        base.SaveChangesAsync(
            cancellationToken);
}

Every database change automatically produces an audit record.

Developers no longer need to remember to create one manually.

Capturing Old and New Values

Suppose a customer changes their email address.

Instead of recording only:

Customer updated

Capture:

Old:
john@example.com

New:
john.smith@example.com

This information becomes invaluable during troubleshooting.

Recording Authentication Events

Authentication activity should also be audited.

Examples include:

• Successful logins

• Failed logins

• Account lockouts

• Password resets

• Multi-factor authentication failures

Security teams frequently investigate these events.

Recording Authorization Events

Authorization changes often have significant business impact.

Examples include:

• Administrator granted

• Permission revoked

• Role changed

• Access denied

These records help identify accidental or unauthorized permission changes.

Soft Deletes vs Hard Deletes

Deleting data permanently removes valuable evidence.

Many organizations prefer soft deletes.

Instead of removing a record:

IsDeleted = true

The audit log records:

• Who deleted it

• When

• Why

The record remains recoverable if needed.

Tamper Resistance

Audit logs should not be easy to modify.

Otherwise attackers can simply erase their tracks.

Common approaches include:

• Write-once storage

• Append-only databases

• Restricted permissions

• Digital signatures

An audit log should be treated as evidence.

Separating Audit Logs from Application Logs

Do not mix audit logs with application diagnostics.

Instead:

• Application logs → troubleshooting

• Audit logs → accountability

Keeping them separate improves security and simplifies compliance reporting.

Correlation IDs

Modern applications often span multiple services.

A single customer request might involve:

• API Gateway

• Orders Service

• Inventory Service

• Payment Service

Using a Correlation ID allows every audit record to be connected to the same business operation.

Example:

var correlationId =
    HttpContext.TraceIdentifier;

This becomes especially useful in distributed systems.

Audit Logging in Microservices

Microservices introduce new challenges.

Each service produces its own audit events.

Questions include:

• Should every service keep its own audit log?

• Should logs be centralized?

• How are timestamps synchronized?

Many organizations collect audit events into a centralized platform for easier analysis.

Asynchronous Audit Logging

Writing audit records synchronously can slow applications.

Instead consider:

• Background queues

• Azure Service Bus

• RabbitMQ

Applications remain responsive while audit records are written in the background.

This connects nicely with our earlier article on messaging systems.

Protecting Audit Data

Audit logs themselves often contain sensitive information.

Protect them using:

• Encryption

• Restricted access

• Backup policies

• Data retention rules

Never assume audit data is harmless.

Compliance Requirements

Many regulations require organizations to maintain audit trails.

Examples include:

GDPR

Requires accountability for personal data access.

HIPAA

Requires auditing of healthcare information.

PCI DSS

Requires tracking access to payment information.

SOC 2

Emphasizes monitoring and accountability.

Audit logging helps organizations demonstrate compliance.

What Audit Logs Should Not Contain

Avoid recording:

• Passwords

• Encryption keys

• Credit card numbers

• Authentication tokens

• Sensitive secrets

Instead record enough information to investigate activity without exposing confidential data.

Common Mistakes

Several mistakes appear repeatedly.

Logging Too Much

Huge audit tables become difficult to search.

Logging Too Little

Critical investigations become impossible.

Missing User Information

An action without a user has little value.

Ignoring Failed Attempts

Unauthorized access attempts often matter as much as successful ones.

Real-World Example

Imagine a banking application.

A customer transfers money.

The audit log records:

• User identity

• Source account

• Destination account

• Amount

• Timestamp

• IP address

• Approval status

• Correlation ID

Months later an investigation occurs.

The organization can reconstruct the entire transaction history within seconds.

How Audit Logging Supports Earlier Topics

This article connects naturally with several recent posts.

Zero Trust verifies identities.

Authorization controls permissions.

Data Protection secures information.

Audit Logging records every important action.

Distributed Tracing follows requests across services.

Together these practices create applications that are secure, observable, and accountable.

Best Practices Checklist

When designing an audit logging system:

• Audit important business events rather than every operation.

• Capture user identity and request context.

• Record previous and new values where appropriate.

• Keep audit logs separate from application logs.

• Protect audit records from unauthorized modification.

• Encrypt sensitive audit data.

• Use asynchronous processing for high-volume systems.

• Define retention policies that satisfy compliance requirements.

• Regularly review audit logs for suspicious activity.

Following these practices helps build audit systems that remain valuable as applications grow.

Closing Thoughts

Building secure applications involves much more than preventing unauthorized access.

Organizations also need visibility into what happens after users sign in.

Audit logging provides that visibility by recording important events in a consistent, reliable, and searchable way.

ASP.NET Core offers all the building blocks needed to create robust audit logging systems, especially when combined with Entity Framework Core, authentication, authorization, and distributed tracing.

As your applications grow in complexity, audit logging becomes one of the most valuable tools for troubleshooting, security investigations, and regulatory compliance.

Knowing what happened yesterday often determines how quickly you can solve tomorrow’s problem.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core architecture, security, and enterprise development guides. Join the Substack Chat to discuss modern ASP.NET Core development with developers building secure, scalable applications.

Subscribe now

In this guide, you’ll learn how advanced authorization works, when to use policy-based and attribute-based approaches, and how to build scalable authorization rules for real-world applications.

Authentication vs Authorization

These two concepts are closely related but solve different problems.

Authentication answers:

Who are you?

Authorization answers:

What are you allowed to do?

Imagine entering an airport.

Showing your passport proves your identity.

That is authentication.

Your boarding pass determines which gate you may enter and whether you can access the airline lounge.

That is authorization.

ASP.NET Core separates these responsibilities, making applications easier to secure and maintain.

Why Authorization Becomes Difficult

Small applications often begin with simple roles.

Examples include:

• Administrator

• Manager

• Employee

• Customer

Initially this works well.

As applications grow, new questions appear.

Can managers approve expenses over $10,000?

Can support engineers edit customer information but not delete it?

Can finance users export reports only during business hours?

Can users manage only projects belonging to their own department?

Roles alone cannot answer these questions cleanly.

Understanding Role-Based Authorization

ASP.NET Core supports role-based authorization out of the box.

Example:

[Authorize(Roles = "Administrator")]
public IActionResult DeleteUser()
{
    return Ok();
}

Only users belonging to the Administrator role can access this action.

Simple.

Easy to understand.

Perfect for straightforward scenarios.

However, real businesses usually require more flexibility.

The Limitations of Roles

Imagine an organization with:

• Sales Manager

• Regional Sales Manager

• Global Sales Manager

• Finance Manager

• HR Manager

• Support Manager

• Operations Manager
  

Soon the number of roles begins to grow rapidly.

Eventually you encounter:

• Duplicate permissions

• Conflicting responsibilities

• Difficult maintenance

• Authorization bugs
  

This is often called role explosion.

Policies solve this problem.

What Is Policy-Based Authorization?

A policy defines rules instead of roles.

Instead of asking:

“Is this user an Administrator?”

You ask:

“Does this user satisfy these requirements?”

A policy can evaluate:

• Claims

• Roles

• Permissions

• Custom business rules

• Time of day

• Department

• Resource ownership
  

Policies provide much greater flexibility.

Creating a Policy

Example:

builder.Services.AddAuthorization(options =>
{
    options.AddPolicy(
        "CanApproveOrders",
        policy =>
        {
            policy.RequireClaim(
                "permission",
                "orders.approve");
        });
});

The policy now checks whether the user has the required permission claim.

Applying a Policy

[Authorize(Policy = "CanApproveOrders")]
public IActionResult Approve()
{
    return Ok();
}

Notice how the controller contains no business logic.

The authorization system handles everything.

Understanding Claims

Claims describe information about a user.

Examples include:

• Name

• Department

• Country

• Permission

• Subscription Level

• Employee Number
  

Unlike roles, claims provide detailed information that policies can evaluate.

Claims-Based Authorization

Suppose users receive:

Department = Finance

Permission = ApproveInvoices

Your policy can require either or both.

This creates much more granular authorization.

Combining Multiple Requirements

Policies may require multiple conditions.

Example:

options.AddPolicy(
    "SeniorFinance",
    policy =>
    {
        policy.RequireRole("Manager");

        policy.RequireClaim(
            "Department",
            "Finance");
    });

Only finance managers satisfy the policy.

Custom Authorization Requirements

Sometimes business rules become very specific.

Example:

A project manager may edit only projects belonging to their own region.

This requires custom authorization.

Create a requirement:

public class RegionRequirement
    : IAuthorizationRequirement
{
}

Then implement an authorization handler.

The handler contains your business logic.

This keeps controllers clean and reusable.

Resource-Based Authorization

Sometimes authorization depends on the resource itself.

Example:

A customer may edit only their own profile.

Not someone else’s.

Instead of checking only user claims, ASP.NET Core evaluates:

• User

• Resource

• Policy
  

Together.

This is called resource-based authorization.

Attribute-Based Access Control (ABAC)

Attribute-Based Access Control goes even further.

Instead of checking only roles or permissions, it evaluates attributes.

Examples include:

User attributes:

• Department

• Clearance level

• Country
  

Resource attributes:

• Owner

• Classification

• Status
  

Environment attributes:

• Time

• Device

• Network location
  

Actions:

• Read

• Update

• Delete
  

All these factors influence authorization decisions.

Example Scenario

Suppose an employee wants to download payroll information.

Rules:

• Must belong to HR

• Must use a company device

• Must be inside the corporate network

• Must access during working hours
  

A single role cannot express this.

Attribute-based authorization can.

Authorization Handlers

Authorization handlers evaluate custom requirements.

Example:

public class MinimumAgeHandler
    : AuthorizationHandler<
        MinimumAgeRequirement>
{
    protected override Task HandleRequirementAsync(
        AuthorizationHandlerContext context,
        MinimumAgeRequirement requirement)
    {
        // Validation logic

        return Task.CompletedTask;
    }
}

Handlers keep authorization logic centralized.

Dynamic Policies

Large enterprise systems often load permissions from databases.

Instead of hardcoding every policy, applications generate them dynamically.

Benefits include:

• Easier administration

• Less code duplication

• Business-managed permissions
  

This approach is common in SaaS platforms.

Authorization in Minimal APIs

Policies also work with Minimal APIs.

Example:

app.MapGet("/reports", () =>
{
    return Results.Ok();
})
.RequireAuthorization("CanViewReports");

Security remains consistent across application styles.

Authorization and Microservices

In a microservices architecture, authorization becomes distributed.

Each service should verify:

• Identity

• Permissions

• Claims
  

Never assume another service has already performed sufficient authorization.

This aligns with the Zero Trust principles discussed in our previous article.

Combining Authentication and Authorization

Authentication produces identity.

Authorization evaluates identity.

Neither replaces the other.

Together they provide layered security.

Common Authorization Mistakes

Avoid:

• Hardcoding permissions

• Mixing authorization into controllers

• Relying only on roles

• Trusting client-side validation

• Forgetting API authorization
  

Centralized authorization policies improve consistency.

Testing Authorization

Authorization should be tested just like business logic.

Verify:

• Authorized users succeed.

• Unauthorized users receive 403 Forbidden.

• Anonymous users receive 401 Unauthorized when appropriate.
  

Automated tests help prevent accidental permission changes.

Real-World Example

Imagine a project management platform.

Users include:

• Developers

• Team Leads

• Project Managers

• Clients
  

Developers may edit only assigned tasks.

Project Managers approve budgets.

Clients view only their own projects.

Finance exports reports.

Instead of creating dozens of overlapping roles, policies evaluate permissions, ownership, and department.

The authorization model remains clean even as the application grows.

Authorization and Compliance

Strong authorization also supports compliance requirements.

Frameworks such as:

• GDPR

• HIPAA

• SOC 2

• PCI DSS
  

Require organizations to restrict access to sensitive information.

Policy-based authorization helps enforce these restrictions consistently.

How This Fits Your ASP.NET Core Journey

Over the past several articles we’ve explored:

• Zero Trust Architecture

• Data Protection

• Encryption

• Key Management

• Secure Microservices
  

Advanced authorization builds directly on those concepts.

Authentication proves identity.

Encryption protects data.

Authorization determines who can use that data.

Together they form the foundation of secure ASP.NET Core applications.

Closing Thoughts

Authorization is much more than checking whether a user is an administrator.

Modern applications require flexible, maintainable access control that adapts to changing business requirements.

ASP.NET Core’s policy-based and attribute-based authorization features provide a scalable way to secure APIs, web applications, and distributed systems without filling controllers with complex permission logic.

As your applications grow, investing in a strong authorization strategy pays dividends in security, maintainability, and compliance.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core architecture, security, and cloud-native development guides. Join the Substack Chat to discuss modern ASP.NET Core development with developers from around the world.

Subscribe now

In this guide, we’ll explore how ASP.NET Core supports data protection through encryption, secure key management, and practical compliance strategies that help keep sensitive information safe.

Why Data Protection Matters

Every application stores data.

Examples include:

• Customer profiles

• Login credentials

• Payment information

• Business records

• Application secrets

• Session tokens

If this information becomes exposed, organizations can face:

• Financial losses

• Reputational damage

• Regulatory penalties

• Customer distrust

Data protection is no longer optional.

It is a fundamental requirement of modern software development.

The Expanding Risk Landscape

Modern applications rarely exist on a single server.

Today’s ASP.NET Core applications often interact with:

• Cloud databases

• Microservices

• Message queues

• Third-party APIs

• Mobile applications

• Browser clients

Each connection introduces new opportunities for data exposure.

As systems become more distributed, protecting data becomes more challenging.

This builds naturally on concepts we explored in:

• Securing Microservices in ASP.NET Core

• Distributed Tracing with OpenTelemetry

• Fault-Tolerant Systems in ASP.NET Core

Security must exist throughout the entire application lifecycle.

Understanding Data Protection

Data protection focuses on ensuring information remains:

• Confidential

• Accurate

• Available

These goals align closely with the CIA Triad:

Confidentiality

Only authorized users can access information.

Integrity

Data cannot be altered without authorization.

Availability

Authorized users can access data when needed.

Strong protection strategies balance all three requirements.

Encryption: The Foundation of Data Protection

Encryption converts readable information into an unreadable format.

Without the correct key, encrypted data becomes useless to attackers.

For example:

Original:
CustomerPassword123

Encrypted:
Qm1kT3dQVnNQaW5GcXhaZw==

While this example is simplified, the principle remains the same.

Encryption protects sensitive information even if storage systems become compromised.

Data at Rest vs Data in Transit

When discussing encryption, two scenarios matter most.

Data at Rest

Data stored in:

• Databases

• Files

• Backups

• Cloud storage
  

Examples:

• SQL Server records

• Azure Blob Storage

• Backup archives
  

Data in Transit

Data moving between systems.

Examples:

• Browser to API

• API to database

• Service to service communication

• Message queue traffic
  

Both require protection.

Securing one without the other leaves gaps.

Encrypting Data in Transit

HTTPS should be mandatory for every ASP.NET Core application.

HTTPS uses TLS (Transport Layer Security) to encrypt communication.

Benefits include:

• Preventing eavesdropping

• Preventing data tampering

• Verifying server identity
  

ASP.NET Core makes HTTPS straightforward.

app.UseHttpsRedirection();

Microsoft documentation:

https://learn.microsoft.com/aspnet/core/security/enforcing-ssl

HTTP Strict Transport Security (HSTS)

HSTS tells browsers to always use HTTPS.

if (!app.Environment.IsDevelopment())
{
    app.UseHsts();
}

This prevents downgrade attacks where attackers attempt to force insecure connections.

Encrypting Data at Rest

Data at rest remains vulnerable if storage systems are compromised.

Common approaches include:

• Database encryption

• Disk encryption

• File encryption

• Application-level encryption
  

Each protects different layers of the system.

SQL Server Encryption

SQL Server supports:

• Transparent Data Encryption (TDE)

• Column encryption

• Always Encrypted
  

Official documentation:

TDE encrypts database files automatically.

Applications require minimal changes.

When Application-Level Encryption Is Necessary

Sometimes specific fields require stronger protection.

Examples include:

• National identification numbers

• Payment data

• Medical information

• API credentials
  

In these cases, encrypting sensitive values before storage provides additional security.

Example: AES Encryption

using System.Security.Cryptography;
using System.Text;

public static string Encrypt(
    string plaintext,
    byte[] key,
    byte[] iv)
{
    using var aes = Aes.Create();

    aes.Key = key;
    aes.IV = iv;

    using var encryptor =
        aes.CreateEncryptor();

    using var ms = new MemoryStream();

    using var cs = new CryptoStream(
        ms,
        encryptor,
        CryptoStreamMode.Write);

    using var sw = new StreamWriter(cs);

    sw.Write(plaintext);

    return Convert.ToBase64String(
        ms.ToArray());
}

This demonstrates the concept of application-level encryption.

Production implementations require additional safeguards and validation.

Introducing ASP.NET Core Data Protection

ASP.NET Core includes a built-in Data Protection system.

The framework uses it internally for:

• Authentication cookies

• Session data

• CSRF protection

• Temporary tokens
  

Official documentation:

This system simplifies secure encryption and key management.

Configuring Data Protection

Basic setup:

builder.Services
    .AddDataProtection();

The framework automatically generates and manages encryption keys.

Protecting Sensitive Values

Example:

public class SecretService
{
    private readonly IDataProtector _protector;

    public SecretService(
        IDataProtectionProvider provider)
    {
        _protector =
            provider.CreateProtector(
                "CustomerData");
    }

    public string Protect(string value)
    {
        return _protector.Protect(value);
    }

    public string Unprotect(string value)
    {
        return _protector.Unprotect(value);
    }
}

This approach avoids many common encryption implementation mistakes.

Why Key Management Matters

Encryption is only as strong as its keys.

Many breaches occur because keys are poorly managed.

Common mistakes include:

• Storing keys in source code

• Sharing keys between environments

• Never rotating keys

• Exposing keys in configuration files
  

Protecting encryption keys is often more important than protecting the encrypted data itself.

Understanding Key Rotation

Keys should not remain active forever.

Key rotation involves:

• Creating new keys

• Retiring old keys

• Re-encrypting data when appropriate
  

Benefits include:

• Reduced exposure

• Limited attack windows

• Improved compliance
  

ASP.NET Core Data Protection supports automatic key rotation.

Storing Keys Securely

Development environments often store keys locally.

Production environments require stronger solutions.

Options include:

• Azure Key Vault

• AWS KMS

• Hardware Security Modules (HSMs)

• Dedicated secret stores
  

Using Azure Key Vault

Azure Key Vault provides:

• Secure secret storage

• Key management

• Certificate management

• Access control
  

Official documentation:

Key Vault integrates well with ASP.NET Core applications.

Configuring Azure Key Vault

Example:

builder.Configuration
    .AddAzureKeyVault(
        new Uri(vaultUrl),
        new DefaultAzureCredential());

This removes sensitive secrets from application configuration files.

Protecting Configuration Data

Configuration often contains:

• API keys

• Database connection strings

• Service credentials
  

Never commit these values to source control.

Instead use:

• Secret managers

• Environment variables

• Key Vault solutions
  

ASP.NET Core Secret Manager

Data Protection in Distributed Systems

Modern applications frequently operate across multiple services.

Examples include:

• Microservices

• Background workers

• Message processors
  

These systems often need access to shared encryption keys.

Careful planning becomes essential.

Improper key distribution can create significant security risks.

Compliance Requirements

Many organizations must satisfy regulatory requirements.

Common examples include:

GDPR

Protects personal data for EU residents.

HIPAA

Protects healthcare information in the United States.

PCI DSS

Protects payment card information.

SOC 2

Focuses on security and operational controls.

Compliance requirements often influence data protection strategies.

Encryption Alone Does Not Guarantee Compliance

A common misconception is:

“We encrypted the data, so we’re compliant.”

Compliance involves much more.

Organizations must also consider:

• Access controls

• Audit logging

• Data retention

• Incident response

• Monitoring
  

Encryption is only one piece of the puzzle.

Auditing and Monitoring

Data access should be traceable.

Important events include:

• Login attempts

• Data modifications

• Key access

• Permission changes
  

Audit trails support:

• Security investigations

• Compliance reporting

• Incident response
  

Least Privilege Access

Not every user should access every piece of data.

Apply least privilege principles:

• Grant only required permissions

• Remove unnecessary access

• Review permissions regularly
  

This aligns closely with the Zero Trust concepts discussed in our previous article.

Protecting Backups

Backups often contain the same sensitive data as production systems.

Organizations sometimes secure production databases while neglecting backups.

Backups should be:

• Encrypted

• Access controlled

• Monitored

• Retained appropriately
  

A compromised backup can be just as damaging as a compromised database.

Data Retention and Deletion

Keeping data forever creates unnecessary risk.

Questions to consider:

• How long should data be retained?

• When should records be deleted?

• Are retention requirements documented?
  

Compliance frameworks frequently require documented retention policies.

Real-World Example: Customer Management Platform

Imagine a SaaS application storing:

• Customer profiles

• Billing information

• Authentication data
  

A strong protection strategy might include:

• HTTPS everywhere

• Database encryption

• Azure Key Vault

• ASP.NET Core Data Protection

• Audit logging

• Automated key rotation
  

If a database backup were stolen, encryption would help prevent attackers from accessing sensitive information.

Common Data Protection Mistakes

Several mistakes appear repeatedly.

Hardcoded Secrets

Secrets should never exist in source code.

Weak Key Management

Protecting data while exposing keys defeats the purpose.

Lack of Rotation

Long-lived keys increase risk.

Unencrypted Backups

Backup systems require the same protections as production.

Excessive Access

Too many users often have unnecessary permissions.

Avoiding these mistakes significantly improves security.

How This Fits Your ASP.NET Core Journey

So far, we’ve explored:

• Microservice security

• Zero Trust Architecture

• Distributed tracing

• Chaos engineering

• Fault tolerance
  

Data protection builds on all of these topics.

Security focuses on who can access systems.

Data protection focuses on safeguarding the information those systems manage.

Together they form a critical foundation for modern cloud-native applications.

Closing Thoughts

Protecting data is one of the most important responsibilities developers have.

ASP.NET Core provides powerful tools for:

• Encryption

• Key management

• Secret storage

• Data protection
  

Combined with proper compliance practices, monitoring, and access controls, these capabilities help organizations secure sensitive information and reduce risk.

Security threats will continue to evolve.

Strong data protection practices ensure your applications remain prepared.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core architecture guides, security best practices, and real-world development strategies. Join the Substack Chat and connect with developers building secure and reliable modern applications.

Subscribe now

In this guide, we’ll explore how to secure ASP.NET Core microservices using Zero Trust principles and practical security best practices.

Why Microservices Change the Security Landscape

In a traditional monolithic application, security boundaries are relatively simple.

Users interact with a single application. That application communicates with a single database. Most communication happens internally.

Microservices introduce a very different environment.

A single user request may pass through:

• API Gateway

• Authentication Service

• Product Service

• Inventory Service

• Payment Service

• Notification Service

• Message Queue

Each service creates a potential attack surface. Every API endpoint becomes a possible entry point. Every service-to-service call becomes a security concern.

This is why securing microservices requires a different mindset.

The Problem with Traditional Network Trust

Historically, many organizations relied on perimeter-based security.

The model was simple:

• Protect the network boundary

• Trust everything inside

This approach worked reasonably well when systems lived inside a single corporate network.

Modern cloud-native systems rarely operate this way.

Today we have:

• Cloud infrastructure

• Hybrid environments

• Remote workforces

• Third-party integrations

• Distributed services

The network perimeter has effectively disappeared.

Trusting everything inside the network is no longer sufficient.

What Is Zero Trust Architecture?

Zero Trust follows a simple principle:

Never trust. Always verify.

Every request must be validated regardless of where it originates.

This applies to:

• Users

• Services

• Devices

• Applications

• APIs

Being inside the network does not automatically grant trust.

Every interaction must continuously prove its legitimacy.

Microsoft’s Zero Trust guidance

Core Principles of Zero Trust

Zero Trust is built around three core ideas.

Verify Explicitly

Always authenticate and authorize every request.

Do not rely on network location.

Do not assume trust.

Use Least Privilege Access

Grant only the permissions required.

Nothing more.

This limits the damage caused by compromised accounts or services.

Assume Breach

Design systems as though attackers are already inside.

This encourages stronger monitoring, segmentation, and verification.

Why ASP.NET Core Fits Zero Trust Well

ASP.NET Core provides several built-in security capabilities:

• Authentication middleware

• Authorization policies

• Identity integration

• JWT support

• OAuth integration

• OpenID Connect support

These features make ASP.NET Core an excellent platform for implementing Zero Trust principles.

Official documentation

Authentication vs Authorization

These concepts are often confused.

Authentication answers:

Who are you?

Authorization answers:

What are you allowed to do?

Both are essential.

A user may be authenticated successfully but still lack permission to perform specific actions.

JWT Authentication for Microservices

JSON Web Tokens (JWTs) are commonly used in microservices environments.

A user authenticates once.

The identity provider issues a token. The token accompanies subsequent requests.

Example:

Authorization: Bearer eyJhbGciOi...

Services validate the token before processing requests. This eliminates the need for centralized session storage.

Configuring JWT Authentication

ASP.NET Core provides built-in support.

builder.Services.AddAuthentication("Bearer")
    .AddJwtBearer(options =>
    {
        options.Authority =
            "https://identity.example.com";

        options.Audience = "orders-api";
    });

This validates incoming access tokens automatically.

Securing APIs with Authorization Policies

Role-based authorization works well for many scenarios.

Example:

[Authorize(Roles = "Administrator")]
public IActionResult DeleteOrder()
{
    return Ok();
}

However, microservices often require more granular controls.

Policy-based authorization provides greater flexibility.

Policy-Based Authorization

Example:

builder.Services.AddAuthorization(options =>
{
    options.AddPolicy(
        "CanProcessPayments",
        policy =>
        {
            policy.RequireClaim(
                "permission",
                "payments.process");
        });
});

Controller:

[Authorize(Policy = "CanProcessPayments")]
public IActionResult ProcessPayment()
{
    return Ok();
}

This approach scales well across complex systems.

Service-to-Service Authentication

Zero Trust applies to services too.

One common mistake is assuming internal services can trust each other automatically.

Example:

API Gateway
     ↓
Orders Service
     ↓
Inventory Service
     ↓
Payment Service

Every service should verify the identity of the caller.

This prevents attackers from moving laterally through compromised systems.

Machine-to-Machine Authentication

Service communication often uses OAuth 2.0 client credentials flow.

Instead of users authenticating:

• Services authenticate themselves
  

Example workflow:

Inventory Service
     ↓
Identity Provider
     ↓
Access Token
     ↓
Payment Service

This creates secure service-to-service communication.

OAuth documentation

Mutual TLS (mTLS)

Mutual TLS strengthens service authentication.

Normally:

• Client verifies server
  

With mTLS:

• Client verifies server

• Server verifies client

Both parties prove their identities.

This significantly improves security for internal service communication.

API Gateways and Security

Most microservices architectures include an API gateway.

The gateway becomes a central security layer.

Responsibilities often include:

• Authentication

• Authorization

• Rate limiting

• Request validation

• Threat detection

Popular gateways include:

• YARP

• Kong

• Azure API Management

YARP documentation

Rate Limiting for Protection

Rate limiting protects APIs from abuse.

ASP.NET Core includes built-in rate limiting.

Example:

builder.Services.AddRateLimiter(options =>
{
    options.AddFixedWindowLimiter(
        "api",
        opt =>
        {
            opt.PermitLimit = 100;
            opt.Window = TimeSpan.FromMinutes(1);
        });
});

This reduces:

• Brute-force attacks

• Abuse

• Resource exhaustion
  

Protecting Secrets

Secrets are often one of the weakest links.

Common mistakes include:

• Hardcoded passwords

• Secrets in source control

• Shared credentials

Instead use:

• Azure Key Vault

• AWS Secrets Manager

• Environment variables
  

Azure Key Vault

Secret Rotation

Secrets should not live forever.

Implement regular rotation policies for:

• API keys

• Database credentials

• Certificates

• Service accounts
  

Automated rotation reduces risk significantly.

Securing Message-Based Systems

Many ASP.NET Core applications use:

• Azure Service Bus

• RabbitMQ

• Kafka

Security must extend to messaging systems.

Consider:

• Authentication

• Authorization

• Encryption

• Auditing

Message queues should never be treated as trusted by default.

This connects directly with our earlier article on Azure Service Bus.

Encrypt Data in Transit

All service communication should use TLS.

Without encryption:

• Credentials can be intercepted

• Data can be modified

• Sensitive information becomes exposed

HTTPS should be mandatory for every service endpoint.

Encrypt Sensitive Data at Rest

Data stored in:

• Databases

• Message queues

• Object storage

Should be encrypted.

This limits exposure if storage systems become compromised.

Logging Security Events

Security incidents rarely announce themselves.

Logs should capture:

• Authentication failures

• Authorization failures

• Suspicious activity

• Policy violations
  

These events become critical during investigations.

Distributed Security Monitoring

As discussed in our article on OpenTelemetry and distributed tracing:

https://www.asptoday.com/p/distributed-tracing-opentelemetry-aspnet-core

Observability plays a major role in security.

Tracing can reveal:

• Unexpected service calls

• Unauthorized access attempts

• Abnormal traffic patterns
  

Security and observability increasingly overlap.

The Principle of Least Privilege

Least privilege remains one of the most effective security controls.

Avoid:

• Administrator everywhere

• Shared service accounts

• Broad permissions
  

Instead grant only the permissions required for a specific task.

This limits the impact of compromised credentials.

Network Segmentation

Not every service should communicate with every other service.

Segmenting networks reduces attack paths.

Example:

Public APIs
     ↓
Business Services
     ↓
Database Layer

Restricted communication paths improve security.

Common Security Mistakes

Several mistakes appear repeatedly in microservices environments.

Blind Trust Between Services

Just because a request originates internally does not mean it is safe.

Shared Credentials

Shared service accounts make auditing and access control difficult.

Excessive Permissions

Services often receive more access than they need.

Missing Input Validation

Never trust incoming data.

Validate everything.

Hardcoded Secrets

Secrets should never be stored in source code.

Real-World Example: E-Commerce Platform

Imagine an online store.

Components include:

• API Gateway

• Orders Service

• Inventory Service

• Payments Service

• Notification Service

Zero Trust implementation would require:

• JWT authentication

• Service-to-service identity

• Authorization policies

• TLS encryption

• Secret management

• Distributed monitoring

Each interaction must prove legitimacy.

No component receives automatic trust.

Security and Resilience Work Together

This article connects closely with several recent topics:

• Fault-Tolerant Systems

• Advanced Retry Strategies

• Chaos Engineering

• Distributed Tracing

A resilient system that is insecure remains vulnerable.

A secure system that cannot survive failures remains unreliable.

Modern architecture requires both.

Preparing for Compliance Requirements

Many organizations must satisfy regulations such as:

• GDPR

• HIPAA

• PCI DSS

• SOC 2

Zero Trust principles help support compliance efforts by enforcing stronger access controls and auditability.

How This Fits Your ASP.NET Core Journey

So far, we’ve explored:

• Distributed messaging

• Sagas

• Retry strategies

• Fault tolerance

• Observability

• Chaos engineering

Security is the next critical layer.

Once systems become distributed, visibility alone is not enough.

Every service must verify identities, enforce permissions, and protect sensitive data continuously.

Closing Thoughts

Microservices create tremendous flexibility and scalability.

They also introduce significant security challenges.

Zero Trust Architecture provides a practical framework for addressing those challenges by eliminating assumptions of trust and requiring continuous verification.

By combining:

• Authentication

• Authorization

• Least privilege

• Service identity

• Encryption

• Monitoring

You can build ASP.NET Core microservices that remain secure even as systems grow increasingly distributed and complex.

In modern architecture, trust is not granted.

Trust is earned continuously.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core architecture guides, security strategies, and real-world engineering practices. Join the Substack Chat and connect with developers building secure and resilient cloud-native applications.

Subscribe now

In this guide, we’ll explore chaos engineering in ASP.NET Core, how it works, when to use it, and how to safely test failure scenarios that improve reliability and resilience.

Why Testing Success Is Not Enough

Most application testing focuses on successful outcomes.

We verify that:

• APIs return expected responses

• Database operations complete successfully

• Authentication works correctly

• Business workflows function as expected

These tests are important.

But production environments rarely behave perfectly.

Real systems experience:

• Network interruptions

• Database outages

• Message queue failures

• Cloud service disruptions

• Slow dependencies

• Infrastructure failures

The challenge is that many of these situations are difficult to reproduce during normal testing.

Chaos engineering helps solve this problem.

What Is Chaos Engineering?

Chaos engineering is the practice of intentionally introducing failures into a system to observe how it responds.

The goal is not destruction.

The goal is learning.

By safely creating controlled failures, teams can identify weaknesses before customers experience them.

The concept was popularized by Netflix, which developed Chaos Monkey to randomly terminate production instances and verify that services remained available.

Official Chaos Monkey project:

Why Chaos Engineering Matters

Imagine your application depends on:

• A SQL database

• Redis cache

• Payment provider

• Email service

• Azure Service Bus

Everything works perfectly during development.

Then one day:

• Redis becomes unavailable

• Payment API starts timing out

• Network latency increases dramatically

What happens?

Many teams discover the answer only after customers start reporting issues.

Chaos engineering helps uncover these weaknesses before they become incidents.

Chaos Engineering Is Not Random Destruction

A common misconception is that chaos engineering means breaking things randomly.

Effective chaos engineering is controlled and scientific.

Every experiment begins with a hypothesis.

For example:

If Redis becomes unavailable, product pages should still load using database fallbacks.

Then you test the hypothesis.

If reality differs from expectations, you’ve found an improvement opportunity.

The Scientific Method for Reliability

Chaos engineering follows a structured process:

1. Define steady-state behavior

2. Create a hypothesis

3. Introduce controlled failure

4. Observe results

5. Improve the system

This makes chaos engineering an engineering discipline rather than a guessing exercise.

Understanding Steady State

Before introducing failures, you need to understand normal behavior.

Examples include:

• Average response times

• Error rates

• Throughput

• Queue depth

• Resource utilization

Without a baseline, it’s impossible to evaluate the impact of failure scenarios.

This is one reason observability is so important.

As discussed in our previous article on distributed tracing, visibility is critical when investigating system behavior.

Chaos Engineering and Distributed Tracing

Distributed tracing and chaos engineering work exceptionally well together.

Tracing helps answer:

• Which services were affected?

• Where did failures originate?

• How far did failures spread?

• Which dependencies became bottlenecks?

Using OpenTelemetry, engineers can visualize the impact of chaos experiments across an entire distributed system.

Common Failure Scenarios

Chaos engineering experiments often focus on realistic production failures.

Examples include:

• Service outages

• Network latency

• Packet loss

• Dependency failures

• Database connection exhaustion

• High CPU usage

• Memory pressure

• Message queue delays

These are failures that eventually happen in real systems.

The question is whether your application handles them gracefully.

Simulating API Failures

Suppose your application calls a payment provider.

Normally:

var response = await _paymentClient.ProcessAsync(payment);

What happens if:

• The API returns HTTP 500?

• Requests timeout?

• The service becomes unavailable?
  

Chaos testing allows you to simulate these scenarios safely.

Testing Timeouts

Timeouts are one of the most common production issues.

A dependency may not fail completely.

Instead, it becomes extremely slow.

Example:

await Task.Delay(TimeSpan.FromSeconds(30));

How does your application react?

Do users receive helpful feedback?

Or does everything become stuck waiting indefinitely?

Validating Retry Policies

Our previous article explored:

• Exponential backoff

• Jitter

• Idempotency
  

Chaos engineering helps verify those patterns actually work.

For example:

• Simulate API failures

• Observe retries

• Verify recovery
  

Many teams discover retry configurations are too aggressive or too conservative.

Testing reveals these weaknesses.

Circuit Breakers Under Stress

Circuit breakers are designed to prevent failing dependencies from overwhelming a system.

Example using Polly:

var circuitBreaker = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        5,
        TimeSpan.FromSeconds(30));

Chaos testing verifies:

• Does the breaker open correctly?

• Does traffic stop flowing?

• Does recovery happen automatically?
  

Without testing, assumptions remain unverified.

Database Failure Experiments

Databases are among the most critical dependencies.

Experiments may include:

• Connection failures

• High latency

• Deadlocks

• Resource exhaustion
  

Questions to ask:

• Does the application fail gracefully?

• Are users informed properly?

• Do background processes recover?
  

These are valuable insights before production incidents occur.

Testing Distributed Caching Failures

Many ASP.NET Core systems rely on Redis.

What happens if Redis becomes unavailable?

A well-designed system should:

• Continue operating

• Fall back to database queries

• Maintain acceptable performance
  

Chaos experiments validate these assumptions.

Message Queue Failures

Applications using Azure Service Bus or RabbitMQ should test scenarios such as:

• Delayed message delivery

• Queue unavailability

• Poison messages

• High backlog conditions
  

Questions include:

• Are messages retried correctly?

• Are dead-letter queues used properly?

• Does the system recover automatically?

Chaos Engineering and Saga Patterns

Sagas coordinate distributed transactions.

Failures can occur during:

• Inventory reservation

• Payment processing

• Shipment creation
  

Chaos testing helps verify:

• Compensation actions execute correctly

• Eventual consistency is maintained

• Workflows recover safely
  

This is especially valuable in complex business processes.

Infrastructure-Level Experiments

Not all chaos experiments target application code.

Infrastructure testing can include:

• Container restarts

• VM shutdowns

• Kubernetes pod failures

• DNS issues

• Network partitions
  

Modern cloud-native systems should tolerate these conditions.

Latency Injection

Sometimes dependencies do not fail.

They simply become slow.

Latency injection simulates this behavior.

Example:

app.Use(async (context, next) =>
{
    await Task.Delay(2000);
    await next();
});

This helps reveal:

• Timeout issues

• User experience problems

• Resource bottlenecks

Fault Injection Middleware

ASP.NET Core makes it easy to inject failures.

Example:

app.Use(async (context, next) =>
{
    if (Random.Shared.Next(100) < 10)
    {
        context.Response.StatusCode = 500;
        return;
    }

    await next();
});

This introduces controlled failures into requests.

Such experiments should only be used in non-production environments unless carefully managed.

Monitoring During Chaos Experiments

Observability is essential.

Monitor:

• Error rates

• Response times

• Queue depth

• Memory consumption

• CPU utilization

• Retry activity
  

Without visibility, chaos experiments provide little value.

Defining Blast Radius

One of the most important concepts in chaos engineering is blast radius.

Blast radius refers to the scope of impact.

Start small.

Instead of testing the entire platform:

• Test one service

• Test one dependency

• Test one workflow

Expand gradually as confidence increases.

Running Experiments Safely

Every chaos experiment should include:

• Clear objectives

• Success criteria

• Monitoring

• Rollback plans

Safety must always come first.

The goal is learning, not causing outages.

Common Mistakes

One mistake is introducing failures without clear hypotheses.

Another is performing experiments without sufficient observability.

Also avoid:

• Testing too much at once

• Running experiments without rollback procedures

• Ignoring lessons learned
  

The experiment is only valuable if it produces actionable insights.

Real-World Example: E-Commerce Platform

Imagine an online store.

Chaos experiments might simulate:

• Redis outage

• Payment provider latency

• Inventory service failure

Expected behavior:

• Cached data falls back to database

• Payments retry automatically

• Inventory failures trigger compensating actions
  

If the platform remains operational, confidence increases significantly.

The Relationship Between Chaos and Reliability

Chaos engineering is not about proving systems are perfect.

It is about discovering where they are fragile.

Every weakness uncovered is an opportunity to improve resilience.

Over time, systems become stronger because failures are explored proactively rather than reactively.

When NOT to Use Chaos Engineering

Small internal applications may not need extensive chaos testing.

Likewise, teams lacking:

• Monitoring

• Alerting

• Operational maturity
  

Should establish those foundations first.

Chaos engineering works best when observability already exists.

How This Fits Your ASP.NET Core Journey

So far, we’ve explored:

• Distributed messaging

• Saga patterns

• Retry strategies

• Fault-tolerant systems

• OpenTelemetry and distributed tracing
  

Chaos engineering brings these concepts together.

It validates whether resilience patterns actually work under realistic failure conditions.

This is where architecture moves from theory into real-world operational confidence.

Closing Thoughts

Failures are inevitable.

The most resilient systems are not those that avoid failure entirely.

They are the systems that have already practiced failure.

Chaos engineering provides a structured way to uncover weaknesses, validate assumptions, and strengthen ASP.NET Core applications before production incidents occur.

By combining:

• Observability

  Distributed tracing

• Retries

• Circuit breakers

• Sagas

• Fault injection
  

Teams can build systems that remain reliable even when the unexpected happens.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core architecture guides, resilience strategies, and real-world engineering practices. Join the Substack Chat and connect with developers building modern cloud-native applications.

Subscribe now

In this guide, you’ll learn how distributed tracing works, why OpenTelemetry has become the industry standard, and how to implement practical tracing solutions in ASP.NET Core applications.

Why Traditional Logging Is No Longer Enough

Years ago, many applications were relatively simple.

A user made a request.

The request hit a web server.

The application queried a database.

A response was returned.

If something went wrong, logs were usually enough.

Modern systems are very different.

A single request may now travel through:

• API Gateway

• ASP.NET Core Web API

• Authentication Service

• Product Service

• Redis Cache

• SQL Database

• Message Queue

• Background Worker

Each component may have its own logs.

When failures occur, finding the root cause becomes difficult.

This is where distributed tracing becomes essential.

What Is Distributed Tracing?

Distributed tracing allows developers to follow a request as it moves through multiple systems.

Think of it as attaching a GPS tracker to every request.

Instead of viewing isolated logs, you can see:

• Where the request started

• Which services it visited

• How long each step took

• Where failures occurred

• Which dependencies caused delays

Distributed tracing provides a complete picture of request execution.

Understanding Traces, Spans, and Context

Distributed tracing revolves around three key concepts.

Trace

A trace represents the complete journey of a request.

For example:

User Request
 ├─ API Gateway
 ├─ Product Service
 ├─ SQL Database
 ├─ Redis Cache
 └─ Response

Everything belongs to a single trace.

Span

A span represents an individual operation.

Examples include:

• Calling a database

• Querying Redis

• Sending a message

• Invoking another API

Each trace contains multiple spans.

Context

Context links spans together.

Without context propagation, traces break apart and become useless.

OpenTelemetry automatically handles much of this process.

Why OpenTelemetry Has Become the Standard

Before OpenTelemetry, observability tools often used proprietary approaches.

Organizations frequently became locked into specific vendors.

OpenTelemetry was created to solve this problem.

It provides:

• Vendor-neutral instrumentation

• Standardized telemetry collection

• Cross-platform support

• Broad ecosystem integration

OpenTelemetry is now supported by major platforms including:

• Microsoft Azure

• AWS

• Google Cloud

• Grafana

• Datadog

• New Relic

Official project:

https://opentelemetry.io

Observability vs Monitoring

These terms are often confused.

Monitoring focuses on answering known questions.

Examples:

• Is the service running?

• How many requests failed?

• Is CPU usage high?

Observability helps answer unknown questions.

Examples:

• Why is checkout suddenly slow?

• Which dependency caused latency spikes?

• Which service introduced failures?

Distributed tracing is a core pillar of observability.

The Three Pillars of Observability

Modern observability typically includes:

• Logs

• Metrics

• Traces
  

Logs explain what happened.

Metrics show how often something happens.

Traces reveal where it happened.

Together they provide a complete operational picture.

Why Tracing Matters in ASP.NET Core

Consider an e-commerce application.

A customer clicks “Place Order.”

The request triggers:

1. Authentication

2. Inventory validation

3. Payment processing

4. Shipping creation

5. Notification service
   

A failure occurs.

Which step caused the issue?

Without tracing:

• Search logs manually

• Correlate timestamps

• Guess relationships
  

With tracing:

• View the entire workflow instantly
  

This becomes especially valuable in architectures we’ve discussed previously:

• ASP.NET Core and Azure Service Bus

• Saga Patterns

• Fault-Tolerant Systems

• Advanced Retry Strategies

Distributed tracing ties these systems together.

Installing OpenTelemetry

Start by adding required packages.

dotnet add package OpenTelemetry.Extensions.Hosting

dotnet add package OpenTelemetry.Instrumentation.AspNetCore

dotnet add package OpenTelemetry.Instrumentation.Http

dotnet add package OpenTelemetry.Instrumentation.SqlClient

These packages provide automatic instrumentation.

Configuring OpenTelemetry

In Program.cs:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation();
    });

This immediately starts collecting trace information.

Automatic Instrumentation

One of OpenTelemetry’s greatest strengths is automatic instrumentation.

Without changing business logic, you can collect traces from:

• Incoming HTTP requests

• Outgoing HTTP calls

• SQL queries

• Entity Framework operations

• gRPC services
  

This dramatically reduces implementation effort.

Viewing Traces Locally

During development, traces can be exported to the console.

builder.Services.AddOpenTelemetry()
    .WithTracing(builder =>
    {
        builder
            .AddAspNetCoreInstrumentation()
            .AddConsoleExporter();
    });

This helps developers understand how traces are structured.

Understanding Trace Hierarchies

Imagine this workflow:

Order Request
 ├─ Validate User
 ├─ Check Inventory
 │   ├─ Redis Cache
 │   └─ SQL Query
 ├─ Process Payment
 └─ Send Confirmation

Each operation becomes a span.

The complete tree becomes a trace.

This hierarchy makes bottlenecks easy to identify.

Adding Custom Spans

Automatic instrumentation is powerful.

But custom spans provide additional business insight.

Example:

private static readonly ActivitySource ActivitySource =
    new("OrderService");

public async Task ProcessOrderAsync()
{
    using var activity =
        ActivitySource.StartActivity("ProcessOrder");

    await Task.Delay(100);
}

This creates a custom span visible in trace visualizations.

Capturing Business Operations

Not everything revolves around technical dependencies.

Sometimes you want visibility into business workflows.

Examples:

• Calculate Discount

  Validate Coupon

• Generate Invoice

• Process Refund
  

Custom spans allow these operations to appear in traces.

Context Propagation

Tracing only works if context moves between services.

Consider:

API A
 ↓
API B
 ↓
API C

If trace context is lost between services:

• Visibility breaks

• Relationships disappear
  

OpenTelemetry automatically propagates context through standard HTTP headers.

This allows traces to remain connected.

Tracing HTTP Requests

Outgoing HTTP requests are automatically captured.

Example:

var response =
    await _httpClient.GetAsync("/products");

OpenTelemetry records:

• URL

• Duration

• Status code

• Parent trace

This creates a complete dependency map.

Tracing Database Calls

Database performance often becomes a bottleneck.

OpenTelemetry automatically traces:

await _dbContext.Products
    .Where(p => p.IsActive)
    .ToListAsync();

Useful information includes:

• Query duration

• Database dependency

• Success or failure

This greatly simplifies performance investigations.

Tracing Message-Based Architectures

Many modern ASP.NET Core systems rely on messaging.

Examples include:

• Azure Service Bus

• RabbitMQ

• Kafka

Distributed tracing becomes even more important because workflows become asynchronous.

Tracing helps developers understand:

• Message publication

• Message consumption

• Processing delays

• Queue bottlenecks
  

OpenTelemetry and ASP.NET Core APIs

Minimal APIs work seamlessly.

app.MapGet("/products", async () =>
{
    return Results.Ok();
});

Each request automatically generates trace information.

No additional code required.

Sampling Strategies

Not every trace needs to be collected.

High-volume systems can generate millions of traces daily.

Sampling helps control costs.

Common approaches:

• Always sample

• Never sample

• Percentage-based sampling

• Adaptive sampling
  

Example:

.SetSampler(new TraceIdRatioBasedSampler(0.1))

This collects approximately 10% of traces.

Exporting Trace Data

OpenTelemetry separates collection from storage.

Popular destinations include:

• Azure Monitor

• Jaeger

• Zipkin

• Grafana Tempo

• Datadog
  

This flexibility prevents vendor lock-in.

Visualizing Traces with Jaeger

Jaeger is one of the most popular tracing platforms.

Official website

It provides:

• Trace timelines

• Dependency graphs

• Span details

• Performance analysis
  

Developers can visually inspect entire request journeys.

Diagnosing Performance Problems

Suppose a page suddenly becomes slow.

Tracing reveals:

Request Duration: 3.8s

Authentication: 40ms
Product Service: 120ms
Redis Cache: 5ms
SQL Query: 3.4s

The bottleneck becomes obvious immediately.

Without tracing, finding this issue could take hours.

Tracing and Distributed Caching

Distributed caching often improves performance dramatically.

But cache misses can still cause problems.

Tracing helps identify:

• Cache hit rates

• Cache miss paths

• Database fallbacks
  

This pairs naturally with distributed caching architectures.

Tracing Fault-Tolerant Systems

In our previous article on fault tolerance, we discussed:

• Retries

• Circuit breakers

• Bulkheads

• Graceful degradation

Tracing makes these patterns visible.

You can observe:

• Retry attempts

• Circuit breaker activations

• Fallback executions
  

This turns resilience from theory into measurable behavior.

Tracing Saga Workflows

Saga patterns coordinate long-running distributed transactions.

Tracing allows developers to follow:

• Order creation

• Inventory reservation

• Payment processing

• Shipment generation
  

Across multiple services.

This provides enormous operational value.

Security Considerations

Be careful with trace data.

Avoid recording:

• Passwords

• Credit card numbers

• Personal information

• Authentication tokens
  

Telemetry should support troubleshooting without exposing sensitive data.

Microsoft guidance:

Common Mistakes

A frequent mistake is collecting traces without using them.

Another is tracing everything indiscriminately.

This creates:

• Storage costs

• Noise

• Analysis difficulties

Focus on meaningful visibility.

When Distributed Tracing Is Most Valuable

Distributed tracing provides the greatest value when systems include:

• Multiple services

• External APIs

• Message queues

• Background processing

• Cloud infrastructure
  

Simple applications may not require extensive tracing.

Complex systems almost always benefit.

Real-World Example

Imagine a food delivery platform.

A customer places an order.

The workflow touches:

• User Service

• Restaurant Service

• Payment Service

• Delivery Service

• Notification Service
  

A trace shows every step.

If delays occur, engineers can pinpoint the exact dependency causing problems.

This is the power of distributed tracing.

How This Fits Your ASP.NET Core Journey

So far, we’ve explored:

• Azure Service Bus

• Saga Patterns

• Distributed Caching

• Retry Strategies

• Fault-Tolerant Systems
  

Distributed tracing now provides the visibility layer that ties all these architectural patterns together.

Building distributed systems is only half the challenge.

Understanding them in production is the other half.

Closing Thoughts

Modern ASP.NET Core systems are increasingly distributed.

As complexity grows, traditional logging alone becomes insufficient.

OpenTelemetry provides a standardized, vendor-neutral approach to distributed tracing that helps developers understand exactly how requests move through their systems.

By implementing tracing early, teams gain:

• Faster troubleshooting

• Better performance visibility

• Improved reliability

• Stronger operational insight
  

As your applications scale, distributed tracing becomes less of a luxury and more of a necessity.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core architecture guides, observability strategies, and real-world engineering patterns. Join the Substack Chat and connect with developers building modern cloud-native applications.

Subscribe now

In this guide, we’ll explore the key patterns, architectural decisions, and trade-offs involved in building resilient fault-tolerant systems with ASP.NET Core.

Why Fault Tolerance Matters

In small applications, failures are often manageable.

A user refreshes the page.
An administrator restarts the service.
The problem disappears.

But modern systems are different.

Applications now operate across:

• Multiple APIs

• Distributed databases

• Cloud services

• Message brokers

• External providers

• Containerized infrastructure

This means failures become inevitable.

At scale, the question changes from:

“Will failures happen?”

To:

“How will the system behave when they do?”

That shift is the foundation of fault-tolerant design.

What Is a Fault-Tolerant System?

A fault-tolerant system continues functioning even when parts of the system fail.

This does not mean:

• zero downtime

• perfect operation

• infinite reliability

Instead, fault-tolerant systems:

• degrade gracefully

• recover automatically

• isolate failures

• minimize user impact

The user experience remains stable even during internal problems.

Understanding Failure in Distributed Systems

Failures in distributed systems are normal.

Examples include:

• network interruptions

• slow APIs

• overloaded databases

• dropped messages

• DNS failures

• container crashes

• cloud outages

This connects directly with concepts from:

• Advanced Retry Strategies in ASP.NET Core

• Implementing Saga Patterns in ASP.NET Core

• ASP.NET Core and Azure Service Bus

Distributed systems must expect these conditions continuously.

Designing for Failure from the Beginning

One of the biggest mistakes developers make is treating resilience as something added later.

Fault tolerance must influence:

• architecture

• communication patterns

• deployment strategy

• state management

• monitoring

Systems designed for failure behave very differently from systems that merely react to failure.

The Fallacies of Distributed Computing

Many reliability problems come from false assumptions.

Developers often assume:

• networks are reliable

• latency is zero

• bandwidth is infinite

• services are always available

These assumptions break quickly in production environments.

Recognizing this reality is the first step toward resilient architecture.

Fault Tolerance vs High Availability

These terms are related but different.

High Availability

Focuses on:

• minimizing downtime

• ensuring services remain accessible

Fault Tolerance

Focuses on:

• surviving failures gracefully

• maintaining functionality during problems

You can have high availability without strong fault tolerance.

And vice versa.

Core Principles of Fault-Tolerant Design

Fault-tolerant systems typically rely on several key principles:

• redundancy

• isolation

• retries

• graceful degradation

• observability

• recovery automation

Each introduces trade-offs.

Retries as a Foundation

Retries are one of the simplest resilience patterns.

Transient failures often resolve automatically.

Retrying can recover from:

• temporary network issues

• overloaded services

• cloud throttling

This was covered deeply in Advanced Retry Strategies in ASP.NET Core

But retries alone are not enough.

The Danger of Aggressive Retries

Poorly designed retries can amplify failures.

If thousands of services retry simultaneously:

• traffic spikes

• failing systems collapse further

• cascading outages occur

This is why:

• exponential backoff

• jitter

• retry limits

Are essential in production systems.

Circuit Breaker Pattern

Circuit breakers prevent repeated calls to failing dependencies.

Without circuit breakers:

• services continuously hammer failing systems

With circuit breakers:

• failing dependencies are temporarily isolated

This allows recovery.

Implementing Circuit Breakers with Polly

var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        5,
        TimeSpan.FromSeconds(30));

After repeated failures:

• the circuit opens

• requests stop temporarily
  

Official Polly documentation.

Bulkhead Isolation Pattern

Bulkheads isolate failures between components.

The term comes from ships:

• compartments prevent flooding from sinking the entire vessel
  

In software:

• resource pools become isolated
  

Example:

• one failing service cannot exhaust all threads or database connections
  

Example Bulkhead Policy

var bulkheadPolicy = Policy.BulkheadAsync(
    maxParallelization: 10,
    maxQueuingActions: 20);

This limits concurrent resource usage.

Graceful Degradation

Fault-tolerant systems do not always fail completely.

Sometimes reduced functionality is acceptable.

Example:

• recommendation engine unavailable

• checkout still works
  

Users may lose non-critical features while core operations remain operational.

This dramatically improves perceived reliability.

Real-World Example: Streaming Platforms

Streaming services often degrade gracefully.

If recommendation systems fail:

• videos still stream
  

If subtitles fail:

• playback continues
  

This separation prevents small failures from becoming catastrophic outages.

Timeouts Prevent Resource Exhaustion

One dangerous issue in distributed systems is waiting forever.

Without timeouts:

• threads become blocked

• connections accumulate

• cascading failures spread
  

Timeouts protect system resources.

Example Timeout Policy

var timeoutPolicy = Policy.TimeoutAsync(
    TimeSpan.FromSeconds(5));

After 5 seconds:

• operation cancels automatically
  

Combining Resilience Policies

Modern systems usually combine:

• retries

• timeouts

• circuit breakers

• bulkheads
  

Example:

var policyWrap = Policy.WrapAsync(
    retryPolicy,
    circuitBreakerPolicy,
    timeoutPolicy);

This creates layered resilience behavior.

Idempotency and Safe Recovery

Retries introduce duplication risks.

A request may succeed internally while the client never receives the response.

If retried blindly:

• duplicate payments

• duplicate orders

• repeated operations
  

May occur.

This is why idempotency matters.

Designing Idempotent APIs

Clients send unique request identifiers:

Idempotency-Key: xyz789

Server stores processed keys.

Repeated requests:

• return existing results

• avoid duplicate processing
  

This pattern becomes essential in fault-tolerant workflows.

Event-Driven Resilience

Message queues improve fault tolerance significantly.

Instead of synchronous dependencies:

• messages become buffered

• services process asynchronously
  

This reduces direct coupling between systems.

Queue-Based Recovery

If a service becomes unavailable:

• messages remain queued

• processing resumes later
  

This prevents immediate system-wide failures.

Fault Tolerance and Saga Patterns

Distributed transactions create additional reliability challenges.

Saga patterns solve this through:

• compensating actions

• eventual consistency

• recovery workflows
  

This was explored in Implementing Saga Patterns in ASP.NET Core.

Fault-tolerant systems often rely heavily on Saga coordination.

Redundancy Improves Reliability

Redundancy means having backup resources.

Examples:

• multiple application instances

• replicated databases

• failover regions

• backup message brokers
  

Redundancy improves availability but increases:

• cost

• synchronization complexity

• operational overhead
  

Horizontal Scaling and Resilience

Fault-tolerant systems avoid single points of failure.

Horizontal scaling helps distribute load across:

• containers

• servers

• cloud instances
  

If one instance fails:

• others continue operating
  

This improves survivability dramatically.

Health Checks in ASP.NET Core

ASP.NET Core includes built-in health checks.

builder.Services.AddHealthChecks();

Expose endpoint:

app.MapHealthChecks("/health");

These help orchestrators:

• detect unhealthy services

• restart failed instances

• reroute traffic automatically
  

Official documentation

Observability Is Critical

Fault tolerance without observability is dangerous.

You must monitor:

• failures

• latency

• retries

• queue depth

• circuit breaker activity

• resource exhaustion
  

Otherwise:

• problems remain invisible
  

Logging and Distributed Tracing

Modern systems require:

• centralized logging

• correlation IDs

• distributed tracing
  

These tools help trace failures across services.

Platforms commonly used:

• OpenTelemetry

• Application Insights

• Seq

• Grafana

• Prometheus
  

Cascading Failures

One failing dependency can trigger widespread outages.

Example:

• database slows down

• API threads become blocked

• retries increase traffic

• queue processing stalls

• system collapses
  

Fault-tolerant systems isolate failures early.

Backpressure and Load Shedding

Sometimes systems must reject traffic intentionally.

This sounds counterintuitive, but controlled rejection is often safer than overload collapse.

Techniques include:

• rate limiting

• queue limits

• request throttling

ASP.NET Core supports built-in rate limiting

CAP Theorem Trade-Offs

Distributed systems cannot simultaneously guarantee:

• consistency

• availability

• partition tolerance
  

Trade-offs become necessary.

Fault-tolerant systems often prioritize:

• availability

• partition tolerance
  

While accepting eventual consistency.

Trade-Offs of Fault-Tolerant Design

Resilience introduces complexity.

More resilience means:

• more infrastructure

• more monitoring

• more operational overhead

• more architecture decisions
  

Simple systems are easier to understand.

Highly fault-tolerant systems are harder to build and maintain.

When Simplicity Is Better

Not every application requires advanced fault tolerance.

Internal tools with low traffic may not need:

• distributed queues

• regional failover

• complex resilience policies
  

Overengineering creates unnecessary operational burden.

Architecture should match business requirements.

Real-World Example: E-Commerce Checkout

Consider checkout processing.

A fault-tolerant flow might:

• queue orders asynchronously

• retry payment calls

• use idempotency keys

• coordinate inventory via Sagas

• isolate recommendation systems
  

Even during failures:

• customers still complete purchases
  

This is resilience in practice.

Fault Injection and Chaos Testing

Resilient systems should be tested under failure conditions.

Chaos engineering introduces:

• random outages

• latency spikes

• dependency failures
  

This validates recovery behavior before production incidents occur.

Netflix popularized this approach through Chaos Monkey.

The Human Side of Reliability

Fault tolerance is not only technical.

Teams also need:

• incident response plans

• operational runbooks

• monitoring alerts

• rollback procedures
  

Human processes are part of resilience engineering too.

Reliability Is an Ongoing Process

Fault tolerance is never “finished.”

As systems evolve:

• traffic grows

• dependencies change

• new failure modes emerge
  

Resilience requires continuous refinement.

How This Fits Your ASP.NET Core Journey

So far, you’ve explored:

• distributed messaging

• retries

• Sagas

• modular architecture

• distributed caching

• API evolution
  

This blog now ties those concepts together into a broader resilience strategy.

This is where architecture becomes production-grade engineering.

Final Thoughts

Failures are unavoidable in distributed systems.

The goal is not creating systems that never fail.

The goal is creating systems that:

• recover gracefully

• isolate failures

• maintain critical functionality

• minimize user impact
  

By combining:

• retries

• circuit breakers

• timeouts

• bulkheads

• graceful degradation

• observability
  

You can build ASP.NET Core systems that remain stable even under difficult real-world conditions.

True reliability comes from designing with failure in mind from the very beginning.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core architecture guides, distributed systems strategies, and real-world resilience engineering insights. Join the Substack Chat and connect with developers building reliable modern systems.

Subscribe now

In this guide, we’ll explore advanced retry strategies in ASP.NET Core using exponential backoff, jitter, and idempotent design to build systems that remain reliable under real-world conditions.

Why Retries Matter in Modern Applications

In simple applications running on one machine, failures are often straightforward.

A request succeeds or fails.

But modern ASP.NET Core applications are different.

They communicate with:

• External APIs

• Payment providers

• Message brokers

• Distributed databases

• Cloud services

• Microservices

And every network call introduces uncertainty.

Sometimes failures are temporary:

• A service is overloaded

• A network packet is lost

• A database connection pool is exhausted

• A container restarts

Retrying the operation a few moments later may succeed perfectly.

This is why retry strategies are critical in resilient architecture.

Understanding Transient Failures

Not all failures are permanent.

A transient failure is temporary and often resolves itself automatically.

Examples include:

• HTTP 503 Service Unavailable

• Network timeout

• Temporary DNS resolution issues

• Cloud throttling

• Deadlocks in databases

These failures are ideal candidates for retries.

Permanent Failures Should Not Be Retried

Retries are not always appropriate.

Some failures are permanent:

• Invalid credentials

• Bad requests

• Validation failures

• Missing resources

Retrying these repeatedly only wastes resources.

Good retry systems distinguish between:

• Transient failures

• Permanent failures

This is one reason resilience engineering requires careful design.

The Simplest Retry Strategy

The most basic retry logic looks like this:

for (int i = 0; i < 3; i++)
{
    try
    {
        await CallApiAsync();
        break;
    }
    catch
    {
        await Task.Delay(1000);
    }
}

This works, but it has serious problems.

Why Simple Retries Become Dangerous

Imagine thousands of clients retrying simultaneously.

If a service becomes overloaded:

• Every client retries immediately

• Traffic spikes even more

• The failing service collapses further
  

This creates a retry storm.

Instead of helping recovery, retries make things worse.

Introducing Exponential Backoff

Exponential backoff solves this problem.

Instead of retrying at a fixed interval, delays increase gradually.

Example:

• Retry 1 → wait 1 second
  

• Retry 2 → wait 2 seconds
  

• Retry 3 → wait 4 seconds
  

• Retry 4 → wait 8 seconds
  

This gives struggling systems time to recover.

Why Exponential Backoff Works

Exponential backoff:

• Reduces pressure on failing systems

• Spreads retry attempts over time

• Prevents traffic spikes

• Improves overall stability
  

Cloud providers strongly recommend this strategy.

Microsoft guidance

Implementing Exponential Backoff in ASP.NET Core

The most common solution is Polly.

Official site

Install:

dotnet add package Polly

Basic Polly Retry Policy

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(3, retryAttempt =>
        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

This creates:

• 2s delay

• 4s delay

• 8s delay
  

Using Polly with HttpClientFactory

ASP.NET Core integrates beautifully with Polly.

builder.Services.AddHttpClient("ApiClient")
    .AddPolicyHandler(retryPolicy);

This automatically applies retry behavior to outgoing HTTP requests.

The Problem with Predictable Retries

Even exponential backoff has a hidden issue.

If thousands of clients fail at the same moment:

• They all retry at identical intervals
  

Example:

• Everyone retries at 2 seconds

• Then everyone retries at 4 seconds
  

This creates synchronized retry waves.

Enter Jitter

Jitter introduces randomness into retry delays.

Instead of:

• Exactly 2 seconds
  

Clients wait:

• 1.7 seconds

• 2.4 seconds

• 2.1 seconds
  

This spreads traffic naturally.

Why Jitter Is Important

Jitter:

• Prevents synchronized retry spikes

• Smooths traffic patterns

• Reduces cascading failures

• Improves system recovery
  

In large distributed systems, jitter becomes extremely important.

Implementing Jitter with Polly

var random = new Random();

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(5, retryAttempt =>
    {
        var exponentialDelay = Math.Pow(2, retryAttempt);

        var jitter = random.NextDouble();

        return TimeSpan.FromSeconds(exponentialDelay + jitter);
    });

Now retries become staggered naturally.

Retry Strategies and Distributed Systems

Retries are foundational in distributed systems.

They connect directly to:

• Service communication

• Messaging reliability

• Distributed transactions
  

This builds naturally on concepts from:

• Implementing Saga Patterns in ASP.NET Core

• ASP.NET Core and Azure Service Bus

• Building Modular Monoliths in ASP.NET Core
  

Distributed systems assume failure will happen.

Retries help systems recover gracefully.

Understanding Idempotency

Retries introduce another major challenge.

What happens if the first request actually succeeded, but the response was lost?

The client retries.

Now the operation executes twice.

This can cause serious issues.

Example:

• Double payments

• Duplicate orders

• Repeated emails
  

This is why idempotency matters.

What Is Idempotency?

An operation is idempotent if:

• Repeating it produces the same result
  

Examples:

• Updating a profile

• Setting a status value
  

Not naturally idempotent:

• Charging a credit card

• Creating a new order
  

Designing Idempotent APIs

One common approach uses idempotency keys.

Client sends:

Idempotency-Key: abc123

Server stores processed keys.

If the same request arrives again:

• Return previous result

• Do not process twice
  

Example Idempotency Middleware

public class IdempotencyMiddleware
{
    private readonly RequestDelegate _next;

    public IdempotencyMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task Invoke(HttpContext context)
    {
        var key = context.Request.Headers["Idempotency-Key"];

        if (string.IsNullOrEmpty(key))
        {
            await _next(context);
            return;
        }

        // Check if request already processed

        await _next(context);
    }
}

This pattern becomes critical in payment systems.

Retry Policies for Different Scenarios

Different systems require different retry behavior.

API Calls

Usually:

• Short retries

• Exponential backoff

• Jitter

Database Operations

Often:

• Fewer retries

• Shorter delays

• Deadlock handling

Messaging Systems

Can tolerate:

• Longer retries

• Queue persistence

• Delayed processing

Retrying Database Transactions

Entity Framework Core supports retry behavior.

builder.Services.AddDbContext<AppDbContext>(options =>
    options.UseSqlServer(connectionString,
        sqlOptions =>
        {
            sqlOptions.EnableRetryOnFailure();
        }));

This helps recover from transient SQL failures automatically.

Avoid Infinite Retries

Retries must always have limits.

Otherwise:

• Systems become stuck

• Resources get exhausted

• Failure cascades worsen
  

Always define:

• Maximum attempts

• Maximum delay

• Timeout limits

Combining Retries with Circuit Breakers

Retries and circuit breakers often work together.

Retries:

• Handle temporary failures
  

Circuit breakers:

• Stop repeated calls to failing services
  

This prevents overloaded systems from collapsing further.

Example Circuit Breaker

var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30));

After repeated failures:

• Requests stop temporarily

• System gets time to recover
  

Retry Storms in Cloud Systems

Large cloud systems are especially vulnerable to retry storms.

Imagine:

• One dependency fails

• Thousands of services retry simultaneously
  

Traffic multiplies instantly.

This is why:

• Backoff

• Jitter

• Circuit breakers

Are essential together.

Real-World Example: Payment Processing

A payment API times out.

Did payment fail?

Or did payment succeed while the response was lost?

Without idempotency:

• Retry may double-charge customers
  

A resilient payment system:

• Uses retries

• Uses idempotency keys

• Stores transaction states carefully

This combination prevents catastrophic duplication.

Observability and Retry Monitoring

Retries should never become invisible.

You must monitor:

• Retry counts

• Failure frequency

• Circuit breaker activity

• Recovery times

Good observability helps identify unhealthy dependencies early.

Logging Retry Attempts

.WaitAndRetryAsync(3,
    retryAttempt => TimeSpan.FromSeconds(2),
    onRetry: (exception, timeSpan, retryCount, context) =>
    {
        logger.LogWarning(
            "Retry {RetryCount} after {Delay}",
            retryCount,
            timeSpan);
    });

This provides operational visibility.

Common Mistakes to Avoid

One major mistake is retrying everything blindly.

Another is retrying too aggressively.

Also avoid:

• Ignoring idempotency

• Infinite retries

• Missing timeout policies

• Combining retries with long-running synchronous operations
  

Retries must be intentional.

When Retries Should NOT Be Used

Retries are inappropriate when:

• Validation fails

• Authentication fails

• Business rules fail

• Data is invalid

These are not transient problems.

Retrying them only wastes resources.

Resilience Is About Accepting Failure

One of the biggest mindset shifts in distributed architecture is realizing:

Failures are normal.

Resilient systems:

• Expect failures

• Prepare for failures

• Recover gracefully from failures
  

This is the foundation of modern cloud-native architecture.

How This Fits Your Architecture Journey

So far, you’ve explored:

• Messaging systems

• Distributed transactions

• Saga coordination

• API evolution

• Modular architectures

Retry strategies now add:

Reliability and fault tolerance

This is where systems stop being merely functional and start becoming production-ready.

Closing Thoughts

Retries seem simple at first.

But in distributed systems, retry behavior directly impacts:

• Stability

• Scalability

• Reliability

• User experience
  

By combining:

• Exponential backoff

• Jitter

• Idempotency

• Circuit breakers
  

You can build ASP.NET Core applications that remain resilient even under failure conditions.

Failures will always happen.

The goal is not eliminating them.

The goal is surviving them gracefully.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core insights, distributed systems patterns, and real-world architecture guidance. Join the Substack Chat and connect with developers building resilient modern systems.

Subscribe now

In this guide, we’ll explore how Saga patterns help ASP.NET Core applications manage distributed transactions safely, reliably, and at scale.

Why Distributed Transactions Become Difficult

In a traditional monolithic application, transactions are usually straightforward.

You wrap operations in a database transaction:

using var transaction = await _dbContext.Database.BeginTransactionAsync();

try
{
    await _dbContext.SaveChangesAsync();

    await transaction.CommitAsync();
}
catch
{
    await transaction.RollbackAsync();
}

Everything succeeds or everything rolls back.

Simple.

But modern systems rarely live inside a single database anymore.

Today’s applications often involve:

• Multiple services

• Independent databases

• Message queues

• External APIs
  

Now imagine this workflow:

1. Create an order

2. Reserve inventory

3. Process payment

4. Send confirmation

What happens if payment fails after inventory was reserved?

You can’t simply roll back everything with one database transaction anymore.

This is where Saga patterns become essential.

What Is a Saga Pattern?

A Saga is a way of managing distributed transactions across multiple services or modules.

Instead of one large transaction:

• Each step completes independently

• If something fails, compensating actions undo previous work
  

Think of it like a chain of small transactions working together.

Understanding the Core Idea

Imagine booking a vacation:

• Reserve flights

• Book hotel

• Rent a car
  

If hotel booking fails:

• Cancel the flight

• Cancel the car reservation

You don’t rewind time. You perform compensating actions.

That’s exactly how a Saga works.

Why Traditional Distributed Transactions Don’t Scale Well

Older distributed transaction systems relied on protocols like Two-Phase Commit (2PC).

While powerful, they introduce:

• Tight coupling

• Performance overhead

• Availability problems

Modern cloud-native systems prefer eventual consistency instead.

This aligns with many of the distributed architecture concepts we’ve explored earlier:

• ASP.NET Core Azure Service Bus

• Designing Data Pipelines in ASP.NET Core

• Building Modular Monoliths in ASP.NET Core

Saga patterns fit naturally into these systems.

Two Types of Saga Patterns

There are two common approaches.

Choreography-Based Sagas

In choreography:

• Services react to events

• No central coordinator exists

Each service listens for events and performs actions independently.

Example Flow

1. Order service publishes OrderCreated

2. Inventory service reserves stock

3. Payment service processes payment

4. Shipping service prepares shipment

Each service reacts automatically.

Benefits

• Loosely coupled

• Easy to extend

• Natural event-driven design
  

Challenges

• Harder to debug

• Workflow becomes distributed

• Complex flows become difficult to track
  

Orchestration-Based Sagas

In orchestration:

• A central coordinator controls the workflow
  

The orchestrator decides:

• What step runs next

• What happens on failure

• Which compensation actions execute
  

Example Flow

Saga Orchestrator
    ↓
Create Order
    ↓
Reserve Inventory
    ↓
Process Payment
    ↓
Ship Product

If payment fails:

Cancel Inventory Reservation
Cancel Order

Which Approach Should You Choose?

Smaller systems often start with choreography.

As workflows grow more complex, orchestration usually becomes easier to manage.

Most enterprise systems eventually lean toward orchestration because visibility and control become important.

Implementing Saga Patterns in ASP.NET Core

Let’s build a simplified orchestration example.

Step 1: Create Saga State

public class OrderSagaState
{
    public Guid OrderId { get; set; }

    public bool InventoryReserved { get; set; }

    public bool PaymentProcessed { get; set; }

    public bool ShipmentCreated { get; set; }
}

The Saga tracks progress across the workflow.

Step 2: Create the Orchestrator

public class OrderSaga
{
    private readonly InventoryService _inventoryService;
    private readonly PaymentService _paymentService;

    public async Task ExecuteAsync(OrderSagaState state)
    {
        try
        {
            await _inventoryService.ReserveAsync(state.OrderId);
            state.InventoryReserved = true;

            await _paymentService.ProcessAsync(state.OrderId);
            state.PaymentProcessed = true;
        }
        catch
        {
            await CompensateAsync(state);
        }
    }
}

Step 3: Compensation Logic

private async Task CompensateAsync(OrderSagaState state)
{
    if (state.PaymentProcessed)
    {
        await _paymentService.RefundAsync(state.OrderId);
    }

    if (state.InventoryReserved)
    {
        await _inventoryService.ReleaseAsync(state.OrderId);
    }
}

Compensation restores consistency.

Using Messaging Systems with Sagas

Most Saga implementations rely heavily on messaging systems.

This is where tools like:

• Azure Service Bus

• RabbitMQ

• MassTransit

become important.

Messaging enables:

• Reliable communication

• Asynchronous workflows

• Loose coupling

Saga Patterns with MassTransit

MassTransit provides strong Saga support for ASP.NET Core.

Official documentation

Install:

dotnet add package MassTransit

Configure MassTransit

builder.Services.AddMassTransit(x =>
{
    x.AddSagaStateMachine<OrderStateMachine, OrderState>();

    x.UsingInMemory((context, cfg) =>
    {
        cfg.ConfigureEndpoints(context);
    });
});

This simplifies orchestration significantly.

Long-Running Transactions

One major advantage of Sagas is handling long-running workflows.

Traditional transactions:

• Hold database locks

• Must complete quickly
  

Sagas:

• Run asynchronously

• Can span minutes or hours

This is critical in real-world distributed systems.

Eventual Consistency

Saga patterns embrace eventual consistency.

Instead of requiring every service to update instantly:

• Systems become consistent over time
  

This is one of the biggest mindset shifts when building distributed systems.

Failure Handling in Sagas

Failures are normal in distributed systems.

Networks fail.

Services go down.

Messages arrive late.

Saga patterns are designed with this reality in mind.

Retry Strategies

Retries are commonly used before compensation.

Policy
    .Handle<Exception>()
    .RetryAsync(3);

Libraries like Polly help implement resilience.

This connects naturally to upcoming reliability topics in the series.

Idempotency Matters

Messages may be delivered more than once.

Your handlers must safely handle duplicate processing.

Example:

if (order.Status == "Processed")
{
    return;
}

Without idempotency, duplicate events can create serious issues.

Real-World Example: E-Commerce Checkout

A checkout workflow may involve:

• Orders service

• Inventory service

• Payments service

• Shipping service
  

If shipping fails after payment succeeds:

• Refund payment

• Release inventory

• Cancel order

The Saga coordinates recovery automatically.

Sagas and Modular Monoliths

Saga patterns are not only for microservices.

They also work well inside modular monoliths.

Modules can:

• Publish events internally

• Coordinate workflows

• Remain loosely coupled

This creates clean internal architecture.

Monitoring and Observability

Distributed workflows are harder to debug.

Tracking:

• Which step failed

• Which compensation executed

• Current workflow state

Becomes essential.

Good logging and tracing are critical.

This prepares us perfectly for the upcoming observability topics later in the series.

Common Mistakes to Avoid

One common mistake is trying to make distributed systems behave like a single database transaction.

Another is forgetting compensation logic entirely.

Also avoid:

• Tight coupling between services

• Shared databases across services

• Ignoring idempotency

When NOT to Use Sagas

Not every workflow needs a Saga.

Simple applications with:

• One database

• One service

• Simple transactions

May not benefit from the added complexity.

Use Sagas when workflows become distributed and failure recovery matters.

How Saga Patterns Fit Your Architecture Journey

So far, you’ve learned:

• How systems communicate

• How APIs evolve

• How modular architectures work

• How distributed messaging operates
  

Saga patterns now connect all those pieces into coordinated workflows.

This is a major step toward designing resilient distributed systems.

Closing Thoughts

Distributed systems are powerful, but they introduce new challenges around consistency and failure handling.

Saga patterns provide a practical way to manage these challenges without relying on heavy distributed transaction protocols.

By combining:

• Messaging

• Compensation logic

• Event-driven workflows

• Eventual consistency
  

You can build scalable ASP.NET Core systems that remain reliable even when failures occur. Start simple, model workflows carefully, and design for failure from the beginning.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core insights and real-world architecture patterns. Join the Substack Chat and connect with developers building modern systems.

Subscribe now

In this guide, we’ll explore how to design modular monoliths in ASP.NET Core with clean boundaries, maintainable structure, and long-term scalability.

Why Modular Monoliths Matter

When building modern applications, many teams jump straight into microservices.

It sounds like the right move:

• Independent services

• Scalability

• Flexibility

But in reality, microservices introduce:

• Network complexity

• Deployment overhead

• Debugging challenges

• Distributed data problems

For many applications, this is too much, too soon.

A modular monolith provides a better starting point.

You still:

• Keep everything in one application

• Avoid network overhead

• Simplify deployment

But you also:

• Organize your system into modules

• Define clear boundaries

• Prepare for future scaling

This approach fits perfectly with everything we’ve built so far in the series, from data pipelines to API design:

Designing Data Pipelines

Advanced API Design

What Is a Modular Monolith?

A modular monolith is a single application that is internally divided into independent modules.

Each module:

• Has its own responsibility

• Owns its own logic

• Avoids direct coupling with others

Think of it like a house.

It’s one building, but:

• The kitchen has its own purpose

• The bedroom has its own purpose

• The bathroom has its own purpose

You don’t mix everything into one room.

The Problem with Traditional Monoliths

A traditional monolith often becomes messy over time.

You see:

• Shared models everywhere

• Tight coupling between features

• Hard-to-maintain code

Everything depends on everything else.

This makes:

• Changes risky

• Debugging difficult

• Scaling painful

A modular monolith solves this by enforcing structure.

Core Principles of a Modular Monolith

A good modular monolith follows a few simple rules.

Each module should:

• Own its own data and logic

• Expose only what’s necessary

• Avoid direct access to other modules’ internals

Communication happens through clear interfaces.

This idea connects with distributed data ownership concepts we explored earlier — even inside a single application.

Structuring a Modular Monolith in ASP.NET Core

Let’s look at a practical structure.

/Modules
    /Orders
        Application
        Domain
        Infrastructure
    /Customers
        Application
        Domain
        Infrastructure
    /Products
        Application
        Domain
        Infrastructure

/API
    Controllers

/Shared
    Common utilities

Each module is isolated.

Inside a Module

Each module typically has:

• Domain → business logic

• Application → use cases

• Infrastructure → database, external services

This structure keeps responsibilities clear.

Enforcing Boundaries

Structure alone is not enough.

You must enforce boundaries.

Avoid This

var customer = _dbContext.Customers.First();

Accessing another module’s data directly breaks boundaries.

Do This Instead

var customer = await _customerService.GetCustomer(id);

Always go through interfaces.

Communication Between Modules

Modules should not depend directly on each other’s internals.

Instead, use:

• Interfaces

• Events

• Application services

This is similar to how services communicate in distributed systems, but without network overhead.

Using MediatR for Internal Communication

MediatR is commonly used for decoupled communication.

dotnet add package MediatR

Example

public class CreateOrderCommand : IRequest
{
    public int CustomerId { get; set; }
} 

public class CreateOrderHandler : IRequestHandler<CreateOrderCommand>
{
    public async Task<Unit> Handle(CreateOrderCommand request, CancellationToken cancellationToken)
    {
        // Handle logic here
        return Unit.Value;
    }
}

This keeps modules loosely coupled.

Data Ownership Inside a Monolith

Each module should manage its own data.

Even if using the same database, logically separate it.

Example

• Orders module → Orders table

• Customers module → Customers table

Avoid sharing tables across modules.

This aligns with distributed data ownership concepts:

https://www.asptoday.com/p/designing-distributed-data-ownership-with-sqlite-databases

Scaling a Modular Monolith

One of the biggest advantages is scalability.

You can scale:

• Code complexity

• Team collaboration

• Features

Without immediately needing microservices.

When to Split into Microservices

You should consider splitting when:

• A module becomes too large

• It requires independent scaling

• Teams need full autonomy

Because your system is already modular, extraction becomes easier.

Testing in Modular Monoliths

Testing becomes simpler with clear boundaries.

You can:

• Test modules independently

• Mock interfaces

• Avoid system-wide dependencies

Performance Benefits

Because everything runs in the same process:

• No network latency

• Faster execution

• Easier debugging
  

This is a major advantage over microservices.

Real-World Example

Imagine an e-commerce system.

Modules:

• Orders

• Customers

• Products

• Payments
  

Each module:

• Has its own logic

• Communicates through interfaces

• Remains independent

You can later extract Payments into a microservice without rewriting everything.

Common Mistakes to Avoid

One common mistake is calling it modular but not enforcing boundaries.

Another is sharing everything through a common layer.

This defeats the purpose.

Also avoid over-engineering. Keep modules simple.

How This Fits Your Architecture Journey

So far, you’ve learned:

• How data flows (pipelines)

• How APIs evolve

• How systems communicate

Now, modular monoliths give you:

👉 A clean internal structure

Before jumping into:

👉 Distributed systems
👉 Microservices

Final Thoughts

A modular monolith is not a compromise.

It’s a strategic choice.

It allows you to:

• Build clean systems

• Scale gradually

• Avoid unnecessary complexity

Start modular, stay structured, and evolve when needed.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core insights and real-world architecture patterns. Join the Substack Chat and connect with developers building modern systems.

Subscribe now

In this guide, we’ll explore how to design APIs in ASP.NET Core that evolve safely using versioning, HATEOAS, and practical long-term design strategies.

Why API Design Matters More Than You Think

At the start, APIs feel simple. You expose endpoints, return data, and everything works.

But once your API is used by:

• Mobile apps

• Frontend applications

• Third-party integrations

You lose the freedom to change things freely.

Even small changes can break clients.

This is where API design becomes critical. You are no longer just building endpoints. You are designing a contract.

And once that contract is used, it becomes very difficult to change.

The Problem with “Simple” APIs

Most APIs start like this:

[HttpGet(”{id}”)]
public async Task<IActionResult> Get(int id)
{
    var product = await _service.GetProduct(id);
    return Ok(product);
}

This works fine initially.

But over time:

• New fields are added

• Old fields need to be removed

• Data shapes change

• New rules apply

If your API isn’t designed to evolve, you’ll either:

• Break existing clients

• Or accumulate messy workarounds

API Versioning: The Foundation of Change

Versioning is the first step in building APIs that can evolve.

Instead of changing an API directly, you introduce versions.

Microsoft provides official guidance here:
https://learn.microsoft.com/aspnet/core/web-api/advanced/advanced-versioning

Types of Versioning

There are a few common approaches.

URL Versioning

/api/v1/products  
/api/v2/products  

Simple and easy to understand.

Header Versioning

api-version: 1.0

Cleaner URLs, but harder to test manually.

Query String Versioning

/api/products?version=1

Simple, but less commonly used in modern APIs.

Implementing Versioning in ASP.NET Core

Install:

dotnet add package Microsoft.AspNetCore.Mvc.Versioning

Configure Versioning

builder.Services.AddApiVersioning(options =>
{
    options.AssumeDefaultVersionWhenUnspecified = true;
    options.DefaultApiVersion = new ApiVersion(1, 0);
    options.ReportApiVersions = true;
});

Use Versioned Controllers

[ApiVersion(”1.0”)]
[Route(”api/v{version:apiVersion}/products”)]
public class ProductsController : ControllerBase
{
}

When Should You Create a New Version?

Not every change requires versioning.

Safe changes:

• Adding new fields

• Adding optional parameters

Breaking changes:

• Removing fields

• Changing response structure

• Changing behavior

When in doubt, version.

Designing APIs for Evolution

Versioning is only part of the story.

Good API design reduces the need for versioning.

Principles for Long-Term APIs

Avoid exposing internal models directly.

Use DTOs instead.

public class ProductResponse
{
    public string Name { get; set; }
    public decimal Price { get; set; }
}

This gives you flexibility to evolve your domain without breaking APIs.

Keep Responses Stable

Clients depend on consistency.

Even small changes can cause issues.

Instead of removing fields, consider deprecating them.

Introducing HATEOAS

HATEOAS stands for:

👉 Hypermedia As The Engine Of Application State

It sounds complex, but the idea is simple.

Instead of just returning data, your API also tells clients what they can do next.

Basic Example

Without HATEOAS:

{
  “id”: 1,
  “name”: “Laptop”
}

With HATEOAS:

{
  “id”: 1,
  “name”: “Laptop”,
  “links”: [
    { “rel”: “self”, “href”: “/api/products/1” },
    { “rel”: “update”, “href”: “/api/products/1” },
    { “rel”: “delete”, “href”: “/api/products/1” }
  ]
}

Now the API is guiding the client.

Why HATEOAS Matters

HATEOAS reduces tight coupling.

Clients don’t need to hardcode routes.

They discover actions dynamically.

This becomes useful in systems where APIs evolve frequently.

Implementing HATEOAS in ASP.NET Core

You can build link objects manually.

public class Link
{
    public string Rel { get; set; }
    public string Href { get; set; }
}

Add Links to Responses

var response = new ProductResponse
{
    Name = product.Name,
    Price = product.Price,
    Links = new List<Link>
    {
        new Link { Rel = “self”, Href = $”/api/products/{product.Id}” }
    }
};

API Evolution Strategies

As your API grows, managing change becomes critical.

Backward Compatibility

Always aim to keep older clients working.

Avoid breaking changes unless absolutely necessary.

Deprecation

Mark old endpoints as deprecated before removing them.

[Obsolete(”This endpoint will be removed in v2”)]

Version Lifecycle

Plan your API lifecycle:

• Introduce

• Maintain

• Deprecate

• Remove

API Design and Distributed Systems

In modern architectures, APIs are everywhere.

They connect:

• Microservices

• External systems

• Frontend clients

This connects with previous topics like messaging and pipelines.

APIs are often the entry point into these systems.

Performance Considerations

API design impacts performance.

Avoid:

• Over-fetching data

• Returning large payloads

• Unnecessary nested objects
  

Consider pagination:

/api/products?page=1&pageSize=10

Real-World Example: Evolving a Product API

Version 1:

{
  “name”: “Laptop”,
  “price”: 1200
}

Version 2 adds:

{
  “name”: “Laptop”,
  “price”: 1200,
  “currency”: “USD”
}

Instead of breaking v1, both versions coexist.

Common Mistakes to Avoid

One common mistake is skipping versioning entirely.

Another is over-versioning for minor changes.

Also, avoid tightly coupling APIs to database structures.

Final Thoughts

APIs are long-term contracts.

Designing them properly from the beginning saves time, reduces bugs, and avoids breaking clients.

By combining:

• Versioning

• HATEOAS

• Thoughtful evolution strategies

You can build APIs that grow with your application instead of holding it back.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core insights and real-world architecture patterns. Join the Substack Chat and connect with developers building modern systems.

Subscribe now

In this guide, we’ll explore how to implement caching with Redis, understand cache invalidation, and design strategies that keep your applications fast and scalable.

Why Caching Becomes Essential

In the early stages of building an application, performance often feels “good enough.” A request comes in, the application queries the database, and a response is returned.

But as usage increases, cracks start to appear.

You begin to notice:

• Slow response times

• Increased database load

• Repeated queries for the same data

The problem is simple. Your system is doing the same work over and over again.

Caching solves this by remembering results so they don’t have to be recomputed.

Instead of asking the database every time, your application can return a cached result instantly.

This becomes especially important in systems that already process large volumes of data, such as the pipelines we explored previously.

What Is Distributed Caching?

Caching can be done in two ways.

In-memory caching stores data inside a single application instance. It’s fast but limited. Once the app restarts, the cache is gone, and it doesn’t work well across multiple servers.

Distributed caching, on the other hand, stores cached data in a shared external system.

This means:

• Multiple application instances share the same cache

• Data persists beyond a single process

• The system scales more easily

Microsoft provides official guidance on distributed caching in ASP.NET Core:
https://learn.microsoft.com/aspnet/core/performance/caching/distributed

Why Redis Is the Go-To Choice

Redis is one of the most popular distributed caching systems.

Official documentation:
https://redis.io/docs/

It’s:

• Extremely fast (in-memory)

• Simple to use

• Designed for high throughput

• Widely supported

Redis stores data as key-value pairs, making it perfect for caching.

Setting Up Redis in ASP.NET Core

To use Redis, install the package:

dotnet add package Microsoft.Extensions.Caching.StackExchangeRedis

Configure Redis

In Program.cs:

builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = “localhost:6379”;
    options.InstanceName = “MyAppCache”;
});

Using the Cache

public class ProductService
{
    private readonly IDistributedCache _cache;

    public ProductService(IDistributedCache cache)
    {
        _cache = cache;
    }

    public async Task<string> GetProductAsync(string id)
    {
        var cacheKey = $”product:{id}”;

        var cached = await _cache.GetStringAsync(cacheKey);

        if (cached != null)
        {
            return cached;
        }

        var product = await FetchFromDatabase(id);

        await _cache.SetStringAsync(cacheKey, product,
            new DistributedCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10)
            });

        return product;
    }
}

This pattern is simple and effective.

Understanding Cache Invalidation

Caching is powerful, but it introduces a new problem.

What happens when data changes?

If your cache still holds old data, users will see outdated information.

This is called cache invalidation, and it’s one of the hardest problems in software.

Common Invalidation Strategies

There are a few common approaches.

Time-Based Expiration

AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10)

The cache expires after a fixed time.

Simple, but not always accurate.

Sliding Expiration

The cache resets its lifetime every time it’s accessed.

Good for frequently used data.

Manual Invalidation

await _cache.RemoveAsync(cacheKey);

You explicitly remove cache when data changes.

This is more accurate but requires discipline.

Cache Strategies That Actually Work

Caching is not just about storing data. It’s about how and when you use it.

Cache-Aside (Lazy Loading)

This is the most common pattern.

• Check cache first

• If not found, fetch from database

• Store in cache

This is what we used earlier.

Write-Through

Data is written to both the database and cache at the same time.

Ensures consistency but adds complexity.

Write-Behind

Data is written to cache first, then persisted later.

Improves performance but introduces risk.

Choosing the Right Strategy

Most ASP.NET Core applications start with cache-aside because it’s simple and reliable.

As systems grow, more advanced strategies can be introduced.

Distributed Caching in Real Systems

In real-world applications, caching is used everywhere.

Example: Product Catalog

Instead of querying the database for every request:

• Cache product details

• Cache category lists

• Cache frequently viewed items

This reduces database load significantly.

Example: API Responses

Entire API responses can be cached.

This is especially useful for read-heavy endpoints.

Caching and Data Pipelines

Caching plays a critical role in data pipelines.

Instead of recalculating results repeatedly, intermediate results can be cached.

This improves performance and reduces system load.

If you recall the pipeline concepts from the previous blog, caching acts as a shortcut in the flow.

Caching and NoSQL Systems

Caching works well alongside NoSQL databases.

Since NoSQL systems often handle large volumes of data, caching frequently accessed data reduces pressure on the database.

This ties into our earlier NoSQL discussion:

https://www.asptoday.com/p/working-with-nosql-databases-in-aspnet-core

Performance Considerations

Caching improves performance, but misuse can cause issues.

Be careful of:

• Over-caching large objects

• Storing unnecessary data

• Using long expiration times

Also, serialization matters. Storing large JSON objects can impact performance.

Performance tuning remains important:

Handling Cache Failures

Your system should not depend entirely on the cache.

If Redis goes down:

• The application should still work

• It should fall back to the database

Caching is an optimization, not a requirement.

Real-World Example: Order Dashboard

Imagine a dashboard showing recent orders.

Without caching:

• Every request hits the database

With caching:

• Results are stored for a short time

• Multiple users see fast responses
  

This significantly improves scalability.

Common Mistakes to Avoid

One common mistake is caching everything.

Not all data benefits from caching.

Another mistake is ignoring invalidation.

Stale data can cause serious issues.

Finally, avoid tightly coupling cache logic to business logic. Keep caching as a separate concern.

Final Thoughts

Distributed caching is one of the simplest and most effective ways to improve application performance.

By using Redis and applying the right strategies, you can reduce load, improve response times, and scale your application more efficiently.

Start with simple patterns like cache-aside, understand invalidation, and evolve your approach as your system grows.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core insights and real-world architecture patterns. Join the Substack Chat and connect with developers building modern systems.

Subscribe now

In this guide, we’ll explore how to design data pipelines in ASP.NET Core using ETL, streaming, and real-time processing so your systems can handle data efficiently at scale.

Why Data Pipelines Matter More Than Ever

In earlier stages of building applications, data feels simple. A user submits a form, you save it to a database, and maybe return a response.

But as systems grow, that model breaks down.

Now your application needs to:

• Process large batches of data

• React to events in real time

• Move data between services

• Transform raw input into useful output

Instead of simple requests and responses, your system becomes a flow of data moving continuously.

This is where data pipelines come in.

A data pipeline is simply a structured way of moving data from one place to another while processing it along the way.

If you’ve worked through earlier topics in this series, you’ve already built pieces of this idea:

• File uploads bring data into your system

• Bulk processing handles large workloads

• Messaging systems move data between services

• Data export sends processed data out

A data pipeline connects all of these into one continuous system.

Understanding the Three Core Pipeline Types

Data pipelines generally fall into three categories.

ETL pipelines handle structured, scheduled data processing. Streaming pipelines process data as it arrives. Real-time pipelines react instantly to events.

Each serves a different purpose, and in most applications, you’ll use a combination of all three.

ETL Pipelines in ASP.NET Core

ETL stands for Extract, Transform, Load.

It’s one of the oldest and most widely used data processing patterns.

You:

1. Extract data from a source

2. Transform it into the required format

3. Load it into a destination

A Simple ETL Example

Imagine you receive a CSV file with sales data.

You:

• Read the file

• Clean and transform the data

• Store it in your database

public async Task ProcessCsvAsync(Stream fileStream)
{
    using var reader = new StreamReader(fileStream);

    while (!reader.EndOfStream)
    {
        var line = await reader.ReadLineAsync();
        var values = line.Split(’,’);

        var record = new SalesRecord
        {
            Product = values[0],
            Amount = decimal.Parse(values[1])
        };

        await _repository.SaveAsync(record);
    }
}

This is a basic ETL pipeline.

When to Use ETL

ETL works best when:

• Data is processed in batches

• Timing is not critical

• You’re transforming structured datasets

In real systems, ETL often runs in the background using scheduled jobs or workers.

Streaming Pipelines

Streaming pipelines process data continuously as it flows through the system.

Instead of waiting for a batch, data is handled as soon as it arrives.

This is common in:

• Event-driven systems

• Messaging platforms

• Real-time dashboards

Streaming with Messaging Systems

In ASP.NET Core, streaming pipelines often rely on message brokers like Azure Service Bus.

public async Task HandleMessageAsync(ServiceBusReceivedMessage message)
{
    var data = JsonSerializer.Deserialize<OrderEvent>(message.Body);

    await _processor.ProcessAsync(data);
}

Each message becomes part of a continuous data stream.

Microsoft provides detailed guidance on messaging and event processing:

Why Streaming Matters

Streaming pipelines:

• Reduce latency

• Enable real-time processing

• Scale better with distributed systems

They are especially useful when systems must react immediately.

Real-Time Processing Pipelines

Real-time pipelines are about immediate reactions.

As soon as an event occurs, something happens.

Examples:

• Fraud detection

• Notifications

• Live analytics

Real-Time Example

public async Task ProcessOrderAsync(Order order)
{
    if (order.Amount > 10000)
    {
        await _alertService.TriggerHighValueAlert(order);
    }
}

This logic runs instantly as data enters the system.

Designing a Pipeline in ASP.NET Core

A well-designed pipeline has clear stages.

Think of it like a production line.

1. Input (data enters)

2. Processing (transform, validate)

3. Output (store, send, or act)

Clean Pipeline Structure

public async Task HandleAsync(InputData input)
{
    var validated = _validator.Validate(input);

    var transformed = _transformer.Transform(validated);

    await _repository.SaveAsync(transformed);

    await _eventPublisher.PublishAsync(transformed);
}

Each step has a clear responsibility.

This keeps the system maintainable.

Handling Large Data Efficiently

Pipelines often deal with large datasets.

Loading everything into memory is a common mistake.

Instead:

• Process data in chunks

• Use streaming where possible

• Avoid blocking operations

This ties closely to patterns discussed in bulk processing.

Pipeline + NoSQL Integration

Document databases work well with pipelines because they handle flexible data structures.

For example, raw events can be stored directly without rigid schemas.

This connects with our NoSQL discussion.

Error Handling in Pipelines

Pipelines must handle failure gracefully.

A failed step should not break the entire system.

Common strategies include:

• Retry logic

• Dead-letter queues

• Logging and monitoring

This is where reliability patterns become important, which we’ll explore deeper in upcoming blogs.

Performance Considerations

Data pipelines can become bottlenecks if not optimized.

Focus on:

• Asynchronous processing

• Parallel execution

• Efficient I/O operations

Performance tuning techniques are critical here.

Real-World Example: Order Processing Pipeline

Let’s bring everything together.

An order pipeline might look like this:

• User places an order

• API validates input

• Event is published

• Worker processes payment

• Data is stored

• Notification is sent

Each step is part of a pipeline.

Instead of one large process, the system flows step by step.

How Pipelines Connect Everything

At this point in your journey, pipelines tie everything together.

They connect:

• Input systems

• Processing systems

• Storage systems

• Output systems

Without pipelines, systems become disconnected.

With pipelines, everything flows smoothly.

Final Thoughts

Data pipelines are not just a feature. They are the backbone of modern applications.

As your systems grow, handling data as a continuous flow becomes essential.

ASP.NET Core provides the flexibility to build ETL, streaming, and real-time pipelines that scale with your needs.

Start simple, structure your pipeline clearly, and evolve it as your application grows.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core insights and real-world architecture patterns. Join the Substack Chat and connect with developers building modern systems.

Subscribe now

In this guide, we’ll walk through how to structure validation and business rules in ASP.NET Core using FluentValidation and the Specification Pattern so your code stays clean as your application grows.

Why Validation Gets Messy So Quickly

At the beginning of any project, validation feels simple.

You check if fields are required. You validate formats. Maybe you enforce a range or two. Everything lives neatly inside your models or controllers.

Then the application grows.

Now validation depends on:

• Other fields

• Database state

• User roles

• Time or external conditions

Suddenly, your controller is doing too much. Your services start duplicating logic. And your validation rules are scattered across multiple layers.

This is especially common in systems that already handle complex workflows like:

• Batch processing

• Messaging systems

• Data exports

If you’ve worked through earlier topics like bulk processing or Service Bus integration, you’ve already seen how quickly complexity can build.

Validation needs structure just as much as data flow and processing do.

The Problem with Data Annotations

ASP.NET Core gives you built-in validation using Data Annotations.

[Required]
[EmailAddress]
public string Email { get; set; }

This works well for basic scenarios, but it starts to fall apart when things get more complex.

Data Annotations:

• Are tightly coupled to models

• Don’t scale well for complex rules

• Are hard to reuse across contexts

• Don’t handle conditional logic cleanly

Microsoft’s guidance also suggests using custom validation approaches when complexity increases:
https://learn.microsoft.com/aspnet/core/mvc/models/validation

This is where FluentValidation comes in.

Introducing FluentValidation

FluentValidation is a widely used library that allows you to define validation rules in a clean, readable, and testable way.

Official docs:
https://docs.fluentvalidation.net

Install it:

dotnet add package FluentValidation.AspNetCore

A Better Way to Write Validation

Instead of cluttering your models, you define validators separately.

public class CreateOrderValidator : AbstractValidator<CreateOrderRequest>
{
    public CreateOrderValidator()
    {
        RuleFor(x => x.CustomerId)
            .NotEmpty();

        RuleFor(x => x.Amount)
            .GreaterThan(0);
    }
}

This approach:

• Keeps models clean

• Makes validation reusable

• Improves readability

• Makes testing easier

Plugging FluentValidation into ASP.NET Core

Register validators:

builder.Services.AddValidatorsFromAssemblyContaining<CreateOrderValidator>();

Once registered, FluentValidation integrates directly into the request pipeline.

No extra controller logic needed.

Real-World Validation Scenarios

Basic validation is only the beginning.

In real systems, validation becomes dynamic.

Conditional Validation

RuleFor(x => x.Discount)
    .GreaterThan(0)
    .When(x => x.HasDiscount);

Cross-Field Validation

RuleFor(x => x.EndDate)
    .GreaterThan(x => x.StartDate);

Custom Rules

RuleFor(x => x.Email)
    .Must(email => email.EndsWith(”@company.com”))
    .WithMessage(”Only company emails are allowed.”);

Validation vs Business Rules (Critical Distinction)

This is where many systems go wrong.

Validation answers:

👉 “Is the data valid?”

Business rules answer:

👉 “Is this action allowed?”

Example:

• “Amount must be greater than 0” ———→ validation

• “Customer must have sufficient balance” ———→ business rule
  

Mixing these leads to confusion and tightly coupled code.

Introducing the Specification Pattern

The Specification Pattern is used to define business rules in a reusable and composable way.

Instead of writing logic inline, you encapsulate it.

Basic Specification

public interface ISpecification<T>
{
    bool IsSatisfiedBy(T entity);
}

Example: Active Customer Rule

public class ActiveCustomerSpecification : ISpecification<Customer>
{
    public bool IsSatisfiedBy(Customer customer)
    {
        return customer.IsActive;
    }
}

Using It

if (!activeCustomerSpec.IsSatisfiedBy(customer))
{
throw new Exception(”Customer is not active.”);
}

Now your business rules are:

• Reusable

• Testable

• Decoupled

Combining FluentValidation and Specification

This is where things get powerful.

FluentValidation:

• Handles input validation
  

Specification Pattern:

• Handles business rules
  

Together, they create a clean separation of concerns.

Example Flow

public async Task Handle(CreateOrderRequest request)
{
    var customer = await _repository.GetCustomer(request.CustomerId);

    if (!_customerSpecification.IsSatisfiedBy(customer))
    {
        throw new Exception(”Customer does not meet requirements.”);
    }

    // Continue processing
}

Scaling Validation in Real Systems

In modern applications, validation doesn’t just happen at the API level.

It happens across:

• Background jobs

• Message handlers

• Batch processors
  

For example, if you’re exporting data or processing large batches, validation ensures data integrity before processing.

https://www.asptoday.com/p/building-data-export-and-reporting-systems-in-aspnet-core

Avoiding Common Pitfalls

One of the most common mistakes is putting business logic inside validators.

Validators should remain lightweight.

Another issue is duplicating rules across layers.

Instead, centralize business rules using specifications.

Also, avoid overcomplicating things too early. Start simple and evolve your architecture as your application grows.

Performance Considerations

Validation can become a bottleneck if done incorrectly.

Avoid:

• Heavy database calls inside validators

• Blocking operations

• Repeated validations
  

Keep validation fast and delegate heavier logic to specifications or services.

If your system handles high throughput, performance tuning becomes important.

Real-World Example: Order Processing System

Let’s bring this together.

Validation:

• CustomerId must exist

• Amount must be greater than zero
  

Business rules:

• Customer must be active

• Customer must have sufficient balance
  

FluentValidation ensures the request is valid.

Specification ensures the operation is allowed.

This separation keeps your system clean and scalable.

Why This Approach Matters Long-Term

As your system evolves:

• Rules change

• New conditions are added

• Complexity increases
  

Without structure, validation becomes fragile.

With FluentValidation and Specification:

• Rules stay organized

• Code stays readable

• Changes are easier to manage
  

This becomes critical in systems that already involve:

• Distributed messaging

• Data processing pipelines

• Cloud-native architectures

Final Thoughts

Validation is often underestimated, but it plays a critical role in system stability.

FluentValidation gives you a clean way to manage input validation, while the Specification Pattern helps structure business rules properly.

Together, they allow you to build systems that scale without turning into a maintenance nightmare.

If you invest in this structure early, your future self will thank you.

Join The Community

Enjoyed this post? Subscribe to ASP Today for practical ASP.NET Core insights and real-world architecture patterns. Join the Substack Chat and connect with developers building modern systems every day.

Subscribe now

As applications grow, traditional relational databases don’t always fit every use case. They work well for structured data and strict relationships, but modern systems often deal with evolving schemas, distributed workloads, and large volumes of semi-structured data.

That’s where NoSQL databases come in.

NoSQL databases, especially document stores, allow you to store data in flexible formats like JSON. This makes it easier to evolve your application over time without constantly redesigning your schema.

In ASP.NET Core applications, NoSQL databases are often used alongside relational databases, not as replacements, but as complementary tools.

In this article, we’ll explore how document databases work, how to integrate MongoDB and Azure Cosmos DB into ASP.NET Core, and how to design systems that take advantage of their strengths.

What Are NoSQL and Document Databases?

NoSQL databases are designed to handle data that doesn’t fit neatly into tables.

Instead of rows and columns, document databases store data as documents, typically in JSON format.

A simple example:

{
  “id”: “1”,
  “name”: “Laptop”,
  “price”: 1200,
  “tags”: [”electronics”, “computers”]
}

This structure is flexible. You can add fields without changing a schema.

Microsoft explains NoSQL databases as systems optimized for scalability, performance, and flexible data models:

https://learn.microsoft.com/en-us/azure/cosmos-db/overview

This flexibility is especially useful in applications where data changes frequently or varies between records.

Why Use NoSQL in ASP.NET Core Applications

NoSQL databases solve several common problems in modern applications.

They allow developers to move faster because there’s no rigid schema to manage. They scale horizontally, making them ideal for cloud environments. And they map naturally to object-oriented code since documents resemble C# objects.

For example, if you’re working with event-driven systems or messaging, storing JSON payloads directly can simplify your design. This aligns closely with patterns we discussed in our Azure Service Bus guide.

Similarly, when handling large datasets or batch workflows, document databases can reduce complexity compared to relational joins. This ties into bulk processing strategies we covered earlier.

Getting Started with MongoDB in ASP.NET Core

MongoDB is one of the most popular document databases. It stores data as BSON (binary JSON) and provides powerful querying capabilities.

Official documentation can be found here - https://www.mongodb.com/docs/

Install the MongoDB driver:

dotnet add package MongoDB.Driver

Configuring MongoDB

In appsettings.json:

{
  “MongoDbSettings”: {
    “ConnectionString”: “mongodb://localhost:27017”,
    “DatabaseName”: “MyAppDb”
  }
}

Create a configuration class:

public class MongoDbSettings
{
    public string ConnectionString { get; set; }
    public string DatabaseName { get; set; }
}

Register it:

builder.Services.Configure<MongoDbSettings>(
    builder.Configuration.GetSection(”MongoDbSettings”));

Creating a MongoDB Service

public class ProductService
{
    private readonly IMongoCollection<Product> _collection;

    public ProductService(IOptions<MongoDbSettings> settings)
    {
        var client = new MongoClient(settings.Value.ConnectionString);
        var database = client.GetDatabase(settings.Value.DatabaseName);

        _collection = database.GetCollection<Product>(”Products”);
    }

    public async Task<List<Product>> GetAsync() =>
        await _collection.Find(_ => true).ToListAsync();

    public async Task CreateAsync(Product product) =>
        await _collection.InsertOneAsync(product);
}

This structure integrates cleanly with ASP.NET Core dependency injection.

Querying Data in MongoDB

MongoDB uses expressive filters for querying.

var products = await _collection
    .Find(p => p.Price > 1000)
    .ToListAsync();

You can also build more complex queries using builders.

Document databases shine here because queries map naturally to object structures.

Using Azure Cosmos DB with ASP.NET Core

Azure Cosmos DB is Microsoft’s globally distributed NoSQL database.

It supports multiple APIs, including a MongoDB-compatible API and a native NoSQL API.

Official documentation:

https://learn.microsoft.com/azure/cosmos-db/

Install the SDK:

dotnet add package Microsoft.Azure.Cosmos

Connecting to Cosmos DB

var client = new CosmosClient(connectionString);
var container = client.GetContainer(”MyDatabase”, “Products”);

Inserting Data

await container.CreateItemAsync(product, new PartitionKey(product.Id));

Querying Data

var query = container.GetItemQueryIterator<Product>(
    “SELECT * FROM c WHERE c.price > 1000”);

var results = new List<Product>();

while (query.HasMoreResults)
{
    var response = await query.ReadNextAsync();
    results.AddRange(response);
}

Cosmos DB is designed for global scale and low latency, making it ideal for distributed systems.

Designing with Document Databases

When using document databases, data modeling is different.

Instead of normalizing data across tables, you often embed related data in a single document.

For example:

{
  “orderId”: “123”,
  “customer”: {
    “name”: “John”,
    “email”: “john@example.com”
  },
  “items”: [
    { “product”: “Laptop”, “price”: 1200 }
  ]
}

This reduces the need for joins and improves read performance.

When to Use MongoDB vs Cosmos DB

MongoDB is a great choice when:

• You want full control over your database

• You’re running on your own infrastructure

• You need flexible querying

Cosmos DB is ideal when:

• You want a fully managed cloud service

• You need global distribution

• You require automatic scaling

In many ASP.NET Core applications, Cosmos DB fits naturally with Azure-based architectures.

Performance Considerations

NoSQL databases are fast, but they still require proper design.

Indexing is critical. Without indexes, queries can become slow.

MongoDB example:

await _collection.Indexes.CreateOneAsync(
    new CreateIndexModel<Product>(
        Builders<Product>.IndexKeys.Ascending(p => p.Price)));

Cosmos DB automatically indexes data, but you can customize indexing policies for better performance.

If you’re working with high-throughput systems, performance tuning becomes essential. For deeper insights, see our performance tuning guide.

Handling Large Data Workloads

Document databases work well with large datasets, but you still need to manage memory and processing carefully.

Use pagination instead of loading all records at once.

Process data in batches when needed. This aligns with export and reporting workflows discussed here.

Real-World Example: Event Storage

Imagine an application that processes user events.

Each event is stored as a document:

{
  “eventType”: “OrderPlaced”,
  “timestamp”: “2026-01-01T10:00:00Z”,
  “data”: {
    “orderId”: “123”,
    “amount”: 500
  }
}

This structure is flexible and works well with messaging systems.

It also integrates naturally with file-based workflows like uploads and processing.

Common Mistakes to Avoid

One common mistake is treating NoSQL databases like relational databases.

Trying to normalize everything defeats the purpose of document storage.

Another mistake is ignoring indexing. Without indexes, performance will degrade quickly.

Finally, avoid storing excessively large documents. While document databases allow flexibility, extremely large records can impact performance.

Closing Thoughts

NoSQL databases are not a replacement for relational databases, but they are an essential tool for modern applications.

MongoDB and Azure Cosmos DB provide flexible, scalable options for storing and querying data in ASP.NET Core applications.

By understanding how document databases work and designing your data models appropriately, you can build systems that are easier to maintain and scale.

As your application grows, combining NoSQL with messaging, batch processing, and export systems creates a strong foundation for handling real-world workloads.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core insights and real-world development patterns. Join our Substack Chat to connect with developers building modern applications.

Subscribe now

Most applications eventually need reporting. Users want to download data, finance teams want structured reports, and operations teams need exports they can analyze offline.

At first, it seems simple. Just take some data and convert it into a file. But in production systems, exporting data introduces real challenges. Large datasets can impact performance. Formatting matters. File generation can block requests. And different formats require different handling.

ASP.NET Core gives you the flexibility to build clean, scalable export systems. When combined with the right libraries, you can generate professional reports without adding unnecessary complexity.

In this article, we’ll focus on three common formats: Excel, PDF, and CSV. We’ll look at how to generate each, when to use them, and how to structure your code so it scales as your application grows.

Why Data Export Matters

Data export is more than just a feature. It’s part of how users interact with your system.

Business users often rely on exports to:

• Analyze data in Excel

• Share reports with stakeholders

• Archive records

• Integrate with other systems

In many cases, exporting data becomes just as important as displaying it in the UI.

If you’re already working with large datasets or background processing, this ties closely to how bulk data workflows are handled. We explored similar patterns in our guide on batch processing in ASP.NET Core.

Designing a Simple Export Architecture

Before jumping into libraries, it helps to structure your export logic properly.

A clean approach is:

• Controller handles the request

• Service prepares the data

• Export service generates the file

This keeps your application maintainable.

Example:

[HttpGet(”export”)]
public async Task<IActionResult> Export()
{
    var data = await _reportService.GetDataAsync();

    var fileBytes = _exportService.GenerateCsv(data);

    return File(fileBytes, “text/csv”, “report.csv”);
}

This pattern works for all formats.

Generating CSV Files with CsvHelper

CSV is the simplest format and often the most efficient.

It’s lightweight, fast, and easy to generate. It’s also widely supported.

We’ll use CsvHelper, a popular library for handling CSV data in .NET.

Install:

dotnet add package CsvHelper

Example:

public byte[] GenerateCsv<T>(IEnumerable<T> data)
{
    using var memoryStream = new MemoryStream();
    using var writer = new StreamWriter(memoryStream);
    using var csv = new CsvWriter(writer, CultureInfo.InvariantCulture);

    csv.WriteRecords(data);
    writer.Flush();

    return memoryStream.ToArray();
}

This generates a CSV file directly in memory.

CSV works best when:

• You need raw data export

• Formatting is not important

• Performance matters
  

Generating Excel Files with EPPlus

Excel is more powerful than CSV because it supports formatting, formulas, and multiple sheets.

We’ll use EPPlus.

Install:

dotnet add package EPPlus

Example:

public byte[] GenerateExcel(List<Product> products)
{
    using var package = new ExcelPackage();
    var worksheet = package.Workbook.Worksheets.Add(”Products”);

    worksheet.Cells[1, 1].Value = “Id”;
    worksheet.Cells[1, 2].Value = “Name”;
    worksheet.Cells[1, 3].Value = “Price”;

    for (int i = 0; i < products.Count; i++)
    {
        worksheet.Cells[i + 2, 1].Value = products[i].Id;
        worksheet.Cells[i + 2, 2].Value = products[i].Name;
        worksheet.Cells[i + 2, 3].Value = products[i].Price;
    }

    return package.GetAsByteArray();
}

Excel is ideal when:

• Users need structured reports

• Formatting matters

• Data needs to be analyzed
  

For large exports, you should avoid loading everything into memory at once. This is where streaming or batching becomes important, similar to techniques discussed in our file upload and processing guide.

Generating PDFs with QuestPDF

PDFs are best for presentation-ready reports.

We’ll use QuestPDF, which is simple and modern.

Install:

dotnet add package QuestPDF

Example:

public byte[] GeneratePdf(List<Product> products)
{
    return Document.Create(container =>
    {
        container.Page(page =>
        {
            page.Content().Column(column =>
            {
                column.Item().Text(”Product Report”).FontSize(20);

                foreach (var product in products)
                {
                    column.Item().Text($”{product.Name} - {product.Price}”);
                }
            });
        });
    }).GeneratePdf();
}

PDF is best when:

• You need formatted reports

• Documents are shared externally

• Layout matters
  

Handling Large Exports

Large exports can slow down your application if handled incorrectly.

Instead of generating files during the request, you can:

• Queue the export job

• Process it in the background

• Notify the user when it’s ready
  

This is especially important for large datasets.

You can use background services or queues, similar to patterns used in Azure Service Bus messaging systems.

Streaming File Downloads

For large files, avoid loading everything into memory.

Instead, stream the response:

return File(stream, “application/octet-stream”, “report.xlsx”);

Streaming improves performance and reduces memory usage.

Choosing the Right Format

Each format serves a different purpose.

CSV is best for raw data.

Excel is best for analysis.

PDF is best for presentation.

In many applications, you’ll support all three.

Real-World Example: Sales Report Export

Imagine a dashboard where users can export sales data.

The workflow might look like this:

The user selects a date range.

The system retrieves data.

The user chooses a format.

The system generates the file.

For small datasets, this can happen instantly.

For large datasets, the export runs in the background and the user downloads it later.

This keeps the application responsive.

Performance Considerations

Exporting data impacts performance.

Keep in mind:

• Avoid large in-memory operations

• Use async calls

• Consider background processing

• Monitor export usage
  

Performance tuning plays a big role here. If you want to go deeper, see our performance tuning guide.

Closing Thoughts

Data export systems are a core part of modern applications. They allow users to work with data outside your system and provide real value.

ASP.NET Core makes it easy to build these systems, but the key is structuring them properly. By separating concerns, using the right libraries, and handling large datasets carefully, you can build export features that scale.

Start simple with CSV, expand into Excel for richer reports, and use PDF when presentation matters.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical ASP.NET Core insights. Join our Substack Chat to connect with developers building real-world applications.

Subscribe now

As applications grow, direct communication between components becomes harder to manage. One service calling another synchronously might work early on, but it quickly introduces problems. If one service is slow or unavailable, everything depending on it starts to fail. That’s where messaging comes in.

Messaging allows systems to communicate without being directly connected. Instead of waiting for a response, one part of your application sends a message, and another part processes it when it’s ready. This decoupling is one of the key building blocks of scalable systems.

Azure Service Bus is Microsoft’s fully managed message broker designed for enterprise-grade messaging. It provides reliable message delivery, supports complex communication patterns, and integrates naturally with ASP.NET Core applications.

In this article, we’ll walk through how Azure Service Bus works, how to integrate it into your ASP.NET Core applications, and how to design messaging patterns that scale in real-world systems.

Why Messaging Matters in Modern Applications

In traditional applications, components often communicate directly using HTTP. While simple, this approach creates tight coupling. If one service is unavailable, the entire workflow can break.

Messaging solves this problem by introducing an intermediary layer.

Instead of Service A calling Service B directly, Service A sends a message to a queue. Service B processes that message independently. This means Service A doesn’t need to wait, retry, or even know whether Service B is currently online.

This approach improves:

Reliability, because messages are stored until processed
Scalability, because processing can happen in parallel
Resilience, because failures don’t cascade across services

Azure Service Bus is built specifically to handle these scenarios at scale.

According to Microsoft’s official documentation, it supports reliable message delivery, ordering, transactions, and advanced routing patterns
https://learn.microsoft.com/azure/service-bus-messaging/service-bus-messaging-overview

Understanding Azure Service Bus Concepts

Before diving into implementation, it’s important to understand the core building blocks.

Queues

Queues follow a simple pattern. A sender places a message into a queue, and a receiver processes it.

Each message is processed by a single consumer. This is useful for background jobs, task processing, and workloads that should only run once.

Think of queues as a to-do list. Each task is picked up and completed by one worker.

Topics and Subscriptions

Topics extend the queue model by allowing multiple consumers to receive the same message.

Instead of one receiver, messages published to a topic can be delivered to multiple subscriptions. Each subscription acts like its own queue.

This pattern is called publish-subscribe.

For example, when an order is placed:

One service processes payment
Another updates inventory
Another sends notifications

All of them receive the same message independently.

Dead-Letter Queues

Sometimes messages fail to process. Instead of losing them, Azure Service Bus moves them to a dead-letter queue.

This allows developers to inspect failed messages, debug issues, and retry processing.

Setting Up Azure Service Bus

To get started, you’ll need an Azure Service Bus namespace.

You can create one in the Azure Portal and then create:

• A queue for simple messaging

• A topic for publish-subscribe scenarios

Once created, you’ll get a connection string, which your ASP.NET Core application will use.

Install the official SDK:

dotnet add package Azure.Messaging.ServiceBus

Microsoft’s SDK documentation provides details on installation and usage
https://learn.microsoft.com/dotnet/api/overview/azure/messaging.servicebus-readme

Sending Messages from ASP.NET Core

Let’s start with sending messages to a queue.

var client = new ServiceBusClient(connectionString);
var sender = client.CreateSender(”orders-queue”);

var message = new ServiceBusMessage(”New order received”);

await sender.SendMessageAsync(message);

This code creates a message and sends it to a queue.

In a real application, the message would likely be serialized JSON representing a business event.

Receiving Messages in ASP.NET Core

Receiving messages typically happens in a background service.

If you’re new to background processing patterns, you may also find our guide on batch processing in ASP.NET Core helpful:
https://www.asptoday.com/p/implementing-bulk-operations-and-batch-processing-in-aspnet-core-applications

public class OrderProcessor : BackgroundService
{
    private readonly ServiceBusProcessor _processor;

    public OrderProcessor(ServiceBusClient client)
    {
        _processor = client.CreateProcessor(”orders-queue”);
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _processor.ProcessMessageAsync += ProcessMessage;
        _processor.ProcessErrorAsync += ErrorHandler;

        await _processor.StartProcessingAsync(stoppingToken);
    }

    private async Task ProcessMessage(ProcessMessageEventArgs args)
    {
        var body = args.Message.Body.ToString();

        // Process message here

        await args.CompleteMessageAsync(args.Message);
    }

    private Task ErrorHandler(ProcessErrorEventArgs args)
    {
        // Log error
        return Task.CompletedTask;
    }
}

This pattern allows your application to process messages continuously in the background.

Using Topics and Subscriptions

Publishing to a topic is similar to sending to a queue.

var sender = client.CreateSender(”orders-topic”);

await sender.SendMessageAsync(new ServiceBusMessage(”Order placed”));

Each subscription receives its own copy of the message.

Subscribers process messages independently, which allows multiple services to react to the same event.

Message Serialization Best Practices

Messages should be structured and versioned carefully.

A common approach is JSON serialization:

var payload = JsonSerializer.Serialize(order);

var message = new ServiceBusMessage(payload);

Avoid tightly coupling message formats to internal models. Instead, use contracts that can evolve over time.

Reliability and Message Delivery

Azure Service Bus guarantees at-least-once delivery. This means messages may be delivered more than once.

Your application should handle duplicate messages safely.

This is known as idempotency.

For example, if processing an order, ensure that processing the same message twice does not create duplicate records.

Handling Failures and Retries

Failures are inevitable in distributed systems.

Azure Service Bus includes built-in retry mechanisms. If processing fails, messages can be retried automatically.

If retries exceed a limit, messages move to the dead-letter queue.

You can then inspect and reprocess them.

Scaling Message Processing

One of the biggest advantages of messaging is scalability.

You can run multiple instances of your worker service, and Azure Service Bus will distribute messages across them.

This allows your system to handle increased workloads without changing your core logic.

Integrating with ASP.NET Core Dependency Injection

You should register Service Bus clients using dependency injection.

builder.Services.AddSingleton(_ =>
    new ServiceBusClient(connectionString));

This ensures efficient reuse of connections and improves performance.

Monitoring and Observability

Monitoring messaging systems is critical.

You should track:

• message throughput

• processing time

• failure rates

• dead-letter queue size

Azure Monitor and Application Insights provide visibility into Service Bus operations
https://learn.microsoft.com/azure/azure-monitor

Observability also ties closely with application performance. For a deeper dive into improving system performance, see our guide on performance tuning in ASP.NET Core:
https://www.asptoday.com/p/performance-tuning-aspnet-core-applications

Real-World Example: Order Processing System

Imagine an e-commerce platform.

When a user places an order, the system publishes a message to a topic.

Different services react:

The payment service processes payment

The inventory service updates stock

The notification service sends confirmation emails

Each service operates independently.

If one service fails, others continue working.

This design improves resilience and scalability.

If you're building foundational knowledge, you can also start with our introduction to ASP.NET Core:
https://www.asptoday.com/p/introduction-to-aspnet-core-whats-new-and-why-it-matters

When to Use Queues vs Topics

Queues are best when:

A task should be handled by only one consumer

Topics are best when:

Multiple services need to react to the same event

In most modern systems, both are used together.

Closing Thoughts

Azure Service Bus provides a powerful foundation for building reliable, scalable messaging systems in ASP.NET Core applications.

By using queues, topics, and subscriptions, you can decouple services, improve resilience, and scale your application more effectively.

Messaging isn’t just a technical feature. It’s an architectural shift that allows your system to grow without becoming fragile.

As your applications evolve, adopting messaging patterns early can save significant time and complexity later.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical insights into ASP.NET Core and modern cloud architecture. Join our Substack Chat to connect with developers building scalable systems and share your experiences.

Subscribe now

Modern web applications often deal with large datasets. Whether you’re importing thousands of customer records, updating product catalogs, processing financial transactions, or synchronizing data between services, handling records individually quickly becomes inefficient.

Processing data one record at a time might work during early development, but as applications grow, it introduces performance bottlenecks, increased database load, and longer processing times. That’s where bulk operations and batch processing become critical architectural patterns.

ASP.NET Core provides the flexibility to implement these techniques using a combination of efficient database access patterns, background services, and scalable cloud infrastructure. When implemented correctly, batch processing can significantly reduce server load while improving throughput and responsiveness.

In this guide, we’ll explore practical strategies for implementing bulk operations and batch workflows in ASP.NET Core applications. We’ll look at how to optimize database operations, design batch pipelines, and ensure that large data workloads remain reliable and scalable.

Why Bulk Operations Matter in Modern Applications

Bulk operations allow multiple records to be processed in a single operation instead of executing repeated commands. This approach reduces the number of database round trips, lowers network overhead, and improves overall system performance.

For example, imagine an application that needs to update 10,000 records. If the system updates each record individually, it might generate 10,000 separate database calls. Each call introduces latency, database locking, and transaction overhead.

A bulk update can perform the same task using a single optimized query.

This difference becomes particularly important in enterprise systems that deal with high data volumes.

Microsoft’s guidance on performance optimization for ASP.NET Core applications emphasizes minimizing unnecessary database calls and batching operations whenever possible:
https://learn.microsoft.com/aspnet/core/performance

Batch processing is commonly used for:

• Importing large CSV or Excel datasets

• Synchronizing records with external APIs

• Running scheduled data cleanup tasks

• Processing financial or transactional workloads

• Sending large volumes of notifications

Instead of running these tasks during user requests, they are usually processed asynchronously in controlled batches.

Understanding Batch Processing

Batch processing refers to executing groups of tasks together as a single job. These jobs typically run in the background and handle large sets of records in stages.

A batch workflow often looks like this:

Data is collected or queued, grouped into manageable chunks, processed sequentially or in parallel, and then stored or transmitted.

This approach prevents large workloads from overwhelming application servers.

For example, instead of processing 100,000 records at once, a system might process batches of 1,000 records at a time.

This provides several advantages.

It limits memory consumption, reduces database locking conflicts, and allows the system to recover gracefully if a failure occurs.

Designing Bulk Operations in ASP.NET Core

Implementing bulk operations in ASP.NET Core often begins with optimizing how data is sent to the database.

Developers frequently rely on Object-Relational Mapping frameworks like Entity Framework Core, but naive implementations can introduce performance problems when handling large datasets.

Consider a typical example where records are inserted one at a time.

foreach (var product in products)
{
    _context.Products.Add(product);
    await _context.SaveChangesAsync();
}

While simple, this approach generates a database transaction for each record.

A better approach is batching the inserts and committing them together.

_context.Products.AddRange(products);
await _context.SaveChangesAsync();

This significantly reduces database overhead.

The official documentation for Entity Framework Core explains how change tracking and batching improve performance when handling multiple entities:

https://learn.microsoft.com/ef/core/performance/

However, when datasets grow very large, even AddRange() can become inefficient.

That’s where specialized bulk libraries can help.

Using Bulk Libraries for High-Volume Operations

Libraries such as EFCore.BulkExtensions enable high-performance bulk inserts, updates, and deletes.

These libraries bypass parts of Entity Framework’s change tracking system and communicate more directly with the database.

Example:

await _context.BulkInsertAsync(products);

Bulk operations like this can improve performance dramatically when importing or updating large datasets.

This is particularly useful when importing data from external sources such as CSV files or third-party APIs.

Bulk processing libraries internally leverage database capabilities like SQL Server’s BULK INSERT command, which is designed for high-throughput data ingestion.

Microsoft provides additional guidance on bulk importing techniques for SQL Server here:

https://learn.microsoft.com/sql/relational-databases/import-export/bulk-import-and-export-of-data

Processing Data in Batches

Even when using bulk operations, it’s important to avoid loading massive datasets entirely into memory.

A better approach is to process records in smaller batches.

For example:

const int batchSize = 1000;

for (int i = 0; i < records.Count; i += batchSize)
{
    var batch = records.Skip(i).Take(batchSize).ToList();

    await _context.BulkInsertAsync(batch);
}

This strategy ensures that the application uses memory efficiently and avoids large transaction sizes.

Batch size should be tuned based on:

• database capacity

• network latency

• memory usage

• workload complexity

In many systems, batch sizes between 500 and 5,000 records offer a good balance between efficiency and stability.

Background Batch Processing in ASP.NET Core

Long-running bulk operations should not block HTTP requests.

Instead, they should run in background services.

ASP.NET Core provides the BackgroundService class to support this model.

public class BatchProcessingService : BackgroundService
{
    private readonly IServiceProvider _services;

    public BatchProcessingService(IServiceProvider services)
    {
        _services = services;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            using var scope = _services.CreateScope();

            var processor = scope.ServiceProvider.GetRequiredService<IRecordProcessor>();

            await processor.ProcessBatchAsync();

            await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
        }
    }
}

This allows applications to process workloads independently of user requests.

Background services are particularly useful for scheduled jobs such as nightly imports or cleanup operations.

The official ASP.NET Core documentation explains hosted services and background tasks in detail:

https://learn.microsoft.com/aspnet/core/fundamentals/host/hosted-services

Queue-Based Batch Processing

In large systems, batch processing is often driven by message queues.

Instead of running periodic jobs, tasks are added to a queue and processed asynchronously by worker services.

Popular queue technologies include:

• Azure Service Bus

• RabbitMQ

• Apache Kafka

A queue-based design provides several benefits.

It allows workloads to scale horizontally, ensures that tasks are processed reliably, and decouples processing logic from web APIs.

When a user uploads data or triggers an operation, the application can enqueue a processing task and return a response immediately.

Workers then process tasks in the background.

Microsoft’s Azure Service Bus documentation explains this asynchronous messaging model:

https://learn.microsoft.com/azure/service-bus-messaging

Handling Large Data Imports

A common scenario for batch processing is importing external data.

For example, a system might allow administrators to upload CSV files containing thousands of records.

Instead of inserting records directly during the upload request, a better workflow might be:

1. Upload the file

2. Store the file temporarily

3. Add a processing job to a queue

4. Process records in batches

5. Record import results

This design ensures that large imports do not slow down the main application.

It also allows administrators to track progress and review errors.

Error Handling and Retry Strategies

Batch processing systems must handle failures gracefully.

If a batch fails, the system should log the error, retry the operation when appropriate, and avoid losing data.

Retry strategies can include exponential backoff or retry queues.

For example, when processing records from an external API, network failures may occur temporarily.

Instead of failing permanently, the system should retry the operation after a short delay.

This pattern improves reliability and resilience.

Monitoring Batch Processing Systems

Visibility is essential for large-scale data processing systems.

You should monitor:

• batch execution time

• failure rates

• queue backlog

• processing throughput

Azure Application Insights provides telemetry capabilities that allow developers to track background job performance.

https://learn.microsoft.com/azure/azure-monitor/app/app-insights-overview

Monitoring helps teams detect issues early and scale processing infrastructure when workloads increase.

Performance Considerations

Several best practices can help ensure that bulk operations remain efficient.

Avoid loading massive datasets entirely into memory. Use streaming or paging to process records gradually.

Use asynchronous database calls to prevent thread blocking.

Optimize database indexes so batch operations execute quickly.

Finally, test batch workloads under realistic conditions to ensure that the system behaves predictably under heavy load.

Real-World Example: Product Catalog Synchronization

Imagine an e-commerce platform that synchronizes product catalogs with an external supplier.

The supplier provides a dataset containing 50,000 products each night.

Instead of processing all records during a single request, the application performs the following workflow.

First, the dataset is downloaded and stored temporarily.

Next, a background job begins processing products in batches of 1,000 records.

Bulk operations update or insert records efficiently.

Errors are logged for review.

Finally, the system updates inventory and search indexes.

This workflow allows the platform to synchronize large datasets reliably without affecting customer-facing performance.

When to Use Bulk Operations vs Batch Processing

Bulk operations focus on optimizing database operations.

Batch processing focuses on orchestrating large workloads over time.

In many applications, both techniques are used together.

Bulk operations improve the efficiency of each database operation, while batch processing ensures that large workloads are distributed safely.

Together, they form the foundation of scalable data processing systems.

Final Thoughts

Bulk operations and batch processing are powerful techniques for building scalable ASP.NET Core applications. As systems grow and data volumes increase, these patterns become essential for maintaining performance and reliability.

By grouping database operations, processing records in controlled batches, and running heavy workloads in background services, developers can ensure that their applications handle large datasets efficiently.

ASP.NET Core’s flexibility, combined with modern database capabilities and cloud infrastructure, makes it possible to design processing pipelines that scale with your application’s growth.

When implemented thoughtfully, batch processing systems allow teams to process massive workloads while keeping user-facing APIs fast and responsive.

Join The Community

Enjoyed this article? Subscribe to ASP Today for practical insights and real-world guidance on ASP.NET Core development. Join our growing community on Substack Chat to connect with fellow developers, share ideas, and stay updated on modern .NET practices.

Subscribe now

File uploads seem simple at first. A user selects a file, clicks upload, and your application stores it somewhere. But in production systems, file handling quickly becomes one of the most sensitive and performance-heavy parts of your application.

Large files can consume memory. Malicious files can introduce security risks. Poor validation can open the door to exploitation. And local disk storage often doesn’t scale in cloud environments.

ASP.NET Core provides strong primitives for handling file uploads safely and efficiently. Combined with cloud storage providers like Azure Blob Storage or Amazon S3, you can build upload systems that are secure, scalable, and reliable.

In this guide, we’ll cover:

• Streaming vs buffering

• Secure file validation

• Safe file storage patterns

• Processing uploads

• Cloud storage integration

• Performance and scalability considerations

Let’s start with the fundamentals.

Understanding How File Uploads Work in ASP.NET Core

When a browser uploads a file, it sends it as part of a multipart/form-data request. ASP.NET Core processes that request and exposes uploaded files through the IFormFile interface.

According to the official Microsoft documentation on file uploads in ASP.NET Core, there are two primary approaches: buffered uploads and streaming uploads
https://learn.microsoft.com/aspnet/core/mvc/models/file-uploads

Buffered uploads load the entire file into memory or temporary storage before your code handles it. Streaming uploads process the file in chunks as it arrives.

For small files, buffering is acceptable. For large files or high-traffic systems, streaming is the safer and more scalable option.

Basic File Upload Example

Let’s begin with a simple controller-based example.

[HttpPost]
public async Task<IActionResult> Upload(IFormFile file)
{
    if (file == null || file.Length == 0)
        return BadRequest(”No file selected.”);

    var filePath = Path.Combine(”uploads”, Path.GetFileName(file.FileName));

    using (var stream = new FileStream(filePath, FileMode.Create))
    {
        await file.CopyToAsync(stream);
    }

    return Ok(”File uploaded successfully.”);
}

This works, but it’s not production-ready.

It trusts the file name. It doesn’t validate file type. It writes directly to disk. And it buffers the file in memory.

We need to improve this.

Streaming File Uploads for Large Files

When handling large files, you should avoid buffering them entirely in memory.

ASP.NET Core allows streaming using Request.Body or the MultipartReader class.

Here’s a simplified streaming approach:

[HttpPost]
[DisableFormValueModelBinding]
public async Task<IActionResult> UploadLargeFile()
{
    var boundary = MultipartRequestHelper.GetBoundary(
        MediaTypeHeaderValue.Parse(Request.ContentType),
        70_000);

    var reader = new MultipartReader(boundary, Request.Body);
    var section = await reader.ReadNextSectionAsync();

    while (section != null)
    {
        if (ContentDispositionHeaderValue
            .TryParse(section.ContentDisposition, out var contentDisposition))
        {
            if (MultipartRequestHelper.HasFileContentDisposition(contentDisposition))
            {
                var filePath = Path.Combine(”uploads”, Guid.NewGuid().ToString());

                using var targetStream = System.IO.File.Create(filePath);
                await section.Body.CopyToAsync(targetStream);
            }
        }

        section = await reader.ReadNextSectionAsync();
    }

    return Ok();
}

Streaming keeps memory usage low, which is critical for high-load APIs.

Microsoft’s Kestrel server, designed for high-performance workloads, handles streaming efficiently

https://learn.microsoft.com/aspnet/core/fundamentals/servers/kestrel

If your application allows users to upload large videos or documents, streaming is not optional. It’s necessary.

Validating Uploaded Files Securely

Security is the most overlooked part of file uploads.

Never trust:

• File names

• File extensions

• Content type headers

Attackers can rename malicious files to bypass simple validation checks.

A safer approach includes:

1. Limiting file size

2. Checking allowed extensions

3. Verifying content signatures

4. Generating safe server-side file names

Limiting File Size

You can configure limits globally:

builder.Services.Configure<FormOptions>(options =>
{
    options.MultipartBodyLengthLimit = 50 * 1024 * 1024; // 50 MB
});

Or per action:

[RequestSizeLimit(50 * 1024 * 1024)]

This prevents memory exhaustion attacks.

Validating File Extensions

var permittedExtensions = new[] { “.jpg”, “.png”, “.pdf” };
var extension = Path.GetExtension(file.FileName).ToLowerInvariant();

if (!permittedExtensions.Contains(extension))
{
    return BadRequest(”Invalid file type.”);
}

But extension checking alone is insufficient.

Validating File Signatures

For stronger protection, inspect file signatures (magic numbers).

For example, PDF files begin with %PDF.

byte[] header = new byte[4];
await file.OpenReadStream().ReadAsync(header, 0, 4);

if (header[0] != 0x25 || header[1] != 0x50)
{
    return BadRequest(”Invalid PDF file.”);
}

For enterprise systems, consider antivirus scanning before final storage.

Microsoft recommends scanning uploaded files before making them available

https://learn.microsoft.com/aspnet/core/security/anti-request-forgery

Generating Safe File Names

Never store files using user-provided names.

Instead:

var safeFileName = $”{Guid.NewGuid()}{extension}”;

This avoids path traversal vulnerabilities and collisions.

You can still store the original name in a database for display purposes.

Processing Files After Upload

Often, uploading is just the first step.

Common processing tasks include:

• Image resizing

• PDF parsing

• Virus scanning

• Metadata extraction

• Video transcoding

These operations should not block the HTTP request.

Instead, use background processing.

ASP.NET Core supports hosted background services:

public class FileProcessingService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            // Process queued files
            await Task.Delay(1000, stoppingToken);
        }
    }
}

Register it:

builder.Services.AddHostedService<FileProcessingService>();

For larger systems, use message queues like Azure Service Bus or RabbitMQ.

Storing Files in Cloud Storage

Local disk storage works in development, but cloud-native applications should use object storage.

Azure Blob Storage is a natural fit for ASP.NET Core applications

https://learn.microsoft.com/azure/storage/blobs/storage-blobs-introduction

Install the package:

dotnet add package Azure.Storage.Blobs

Example upload:

var blobClient = new BlobClient(connectionString, containerName, safeFileName);

await blobClient.UploadAsync(file.OpenReadStream());

This provides:

• Automatic scalability

• Redundancy

• CDN integration

• Global availability

Amazon S3 offers similar capabilities

https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

Direct-to-Cloud Uploads

For very large files, you can generate pre-signed URLs and allow clients to upload directly to cloud storage.

This avoids routing large files through your server entirely.

Azure supports SAS tokens

https://learn.microsoft.com/azure/storage/common/storage-sas-overview

This pattern reduces:

• Server load

• Bandwidth costs

• Scaling complexity

Performance Considerations

File uploads impact:

• Memory usage

• Thread pool utilization

• Disk I/O

• Network throughput

Always:

• Use async I/O

• Avoid loading full files into memory

• Configure request limits

• Monitor upload endpoints separately

For high-scale APIs, consider separating upload services into their own microservice.

Handling Concurrent Uploads

High concurrency requires careful planning.

Monitor:

• CPU

• Memory

• Disk

• Network

Implement rate limiting using ASP.NET Core middleware:

builder.Services.AddRateLimiter(options =>
{
    options.AddFixedWindowLimiter(”upload”, limiterOptions =>
    {
        limiterOptions.PermitLimit = 10;
        limiterOptions.Window = TimeSpan.FromMinutes(1);
    });
});

This prevents abuse and protects resources.

Logging and Monitoring

Track:

• Upload failures

• Validation errors

• File size metrics

• Processing duration

Use Application Insights for telemetry

https://learn.microsoft.com/azure/azure-monitor/app/app-insights-overview

Monitoring upload endpoints separately helps identify bottlenecks early.

Common Mistakes to Avoid

Saving files with original names

Storing files inside wwwroot

Trusting Content-Type headers

Blocking requests during processing

Ignoring size limits

Skipping antivirus scanning

Small mistakes in upload systems can become serious vulnerabilities.

Real-World Example: Profile Image Upload

A simple pattern:

1. Validate extension and size

2. Stream file

3. Store in Blob Storage

4. Save metadata in database

5. Return public URL

This keeps your application clean and scalable.

When to Use Streaming vs Buffered Uploads

Buffered uploads are acceptable when:

• Files are small

• Traffic is low

• Simplicity is preferred

Streaming is preferred when:

• Files exceed a few MB

• Concurrent uploads are expected

• Cloud-native scaling matters

Final Thoughts

File upload handling is one of those features that seems simple until it isn’t.

Done correctly, it improves scalability and security. Done carelessly, it becomes a performance bottleneck or security risk.

ASP.NET Core gives you the tools. Cloud storage platforms give you scalability. Combining both thoughtfully is what makes production-ready systems possible.

Join The Community

If you found this guide helpful, subscribe to ASP Today on Substack for practical, real-world ASP.NET Core insights. Join us in Substack Chat to connect with other developers building modern .NET applications.

Subscribe now