Building Search Functionality with Elasticsearch and ASP.NET Core

Master full-text search and real-time analytics in your .NET applications

Search functionality has become the backbone of modern web applications, transforming how users interact with data and discover content.

Whether you’re building an e-commerce platform where customers need to find products instantly, a content management system requiring sophisticated document search, or a data analytics dashboard demanding real-time insights, implementing robust search capabilities can make or break your application’s user experience. Elasticsearch, combined with ASP.NET Core, provides a powerful solution that goes far beyond simple database queries, offering lightning-fast full-text search, complex aggregations, and the ability to handle massive amounts of data with ease.

Understanding Elasticsearch in the .NET Ecosystem

Elasticsearch represents a paradigm shift in how we approach search and analytics in modern applications. At its core, Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene, but calling it just a search engine undersells its capabilities. It’s a complete data platform that excels at making sense of structured and unstructured data alike, providing near real-time search results even across petabytes of information.

The integration between Elasticsearch and ASP.NET Core creates a particularly compelling combination for .NET developers. ASP.NET Core’s high-performance, cross-platform nature aligns perfectly with Elasticsearch’s distributed architecture, allowing you to build search solutions that scale horizontally across multiple servers while maintaining sub-second response times. This partnership becomes especially powerful when you consider that both technologies are designed with cloud-native deployments in mind, making them ideal for containerized environments and microservices architectures.

What sets Elasticsearch apart from traditional database search capabilities is its inverted index structure, which fundamentally changes how data is stored and retrieved. Instead of scanning through rows of data like a relational database would, Elasticsearch creates an index that maps terms to the documents containing them. This approach means that searching for a specific word across millions of documents happens almost instantaneously, rather than requiring expensive table scans that grow slower as your data grows larger.

Setting Up Your Development Environment

Getting started with Elasticsearch and ASP.NET Core requires some initial setup, but the process has become significantly streamlined in recent years. The first step involves installing Elasticsearch itself, which can be done through various methods depending on your development environment and deployment targets. For local development, Docker provides the fastest path to getting Elasticsearch running. Here’s a simple docker-compose configuration to get you started:

version: '3.8'
services:
  elasticsearch:
    image: elasticsearch:8.11.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - "9200:9200"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data

volumes:
  elasticsearch-data:

Once you have Elasticsearch running, you’ll need to integrate it with your ASP.NET Core application using the official Elasticsearch .NET client, also known as NEST. Install the required NuGet packages in your project:

dotnet add package NEST
dotnet add package Elasticsearch.Net

Configuration in ASP.NET Core follows the standard patterns you’re already familiar with. First, add your Elasticsearch settings to appsettings.json:

{
  "Elasticsearch": {
    "Uri": "http://localhost:9200",
    "DefaultIndex": "products",
    "Username": "",
    "Password": ""
  }
}

Then configure the Elasticsearch client in your Program.cs file:

using Nest;

var builder = WebApplication.CreateBuilder(args);

// Configure Elasticsearch
builder.Services.AddSingleton<IElasticClient>(provider =>
{
    var configuration = provider.GetRequiredService<IConfiguration>();
    var settings = configuration.GetSection("Elasticsearch");
    var uri = new Uri(settings["Uri"]);
    
    var connectionSettings = new ConnectionSettings(uri)
        .DefaultIndex(settings["DefaultIndex"])
        .EnableDebugMode()
        .PrettyJson()
        .RequestTimeout(TimeSpan.FromSeconds(30));
    
    // Add authentication if needed
    var username = settings["Username"];
    var password = settings["Password"];
    if (!string.IsNullOrEmpty(username) && !string.IsNullOrEmpty(password))
    {
        connectionSettings.BasicAuthentication(username, password);
    }
    
    return new ElasticClient(connectionSettings);
});

builder.Services.AddControllers();
var app = builder.Build();

// Ensure indices are created on startup
using (var scope = app.Services.CreateScope())
{
    var elasticClient = scope.ServiceProvider.GetRequiredService<IElasticClient>();
    var indexExists = await elasticClient.Indices.ExistsAsync("products");
    if (!indexExists.Exists)
    {
        await elasticClient.Indices.CreateAsync("products");
    }
}

app.MapControllers();
app.Run();

The beauty of this setup lies in how it integrates seamlessly with ASP.NET Core’s existing patterns. Your controllers and services can request the Elasticsearch client through constructor injection, just like any other dependency. This consistency means that developers familiar with ASP.NET Core can start working with Elasticsearch without learning entirely new architectural patterns.

Designing Your Search Schema

Creating an effective search schema requires thinking differently about your data compared to traditional relational database design. Let’s define a product model and configure how it should be indexed in Elasticsearch:

public class Product
{
    public string Id { get; set; }
    public string Name { get; set; }
    public string Description { get; set; }
    public string Category { get; set; }
    public string Brand { get; set; }
    public decimal Price { get; set; }
    public double Rating { get; set; }
    public int ReviewCount { get; set; }
    public List<string> Tags { get; set; }
    public DateTime CreatedAt { get; set; }
    public GeoLocation Location { get; set; }
    public Dictionary<string, string> Specifications { get; set; }
}

Now, let’s create a service that defines the mapping for this model:

public class ElasticsearchIndexService
{
    private readonly IElasticClient _elasticClient;
    private readonly ILogger<ElasticsearchIndexService> _logger;
    
    public ElasticsearchIndexService(
        IElasticClient elasticClient,
        ILogger<ElasticsearchIndexService> logger)
    {
        _elasticClient = elasticClient;
        _logger = logger;
    }
    
    public async Task CreateProductIndexAsync()
    {
        var indexName = "products";
        
        // Delete index if it exists (for development only!)
        var deleteResponse = await _elasticClient.Indices.DeleteAsync(indexName);
        
        // Create index with custom settings and mappings
        var createIndexResponse = await _elasticClient.Indices.CreateAsync(indexName, c => c
            .Settings(s => s
                .Analysis(a => a
                    .Analyzers(an => an
                        .Custom("product_analyzer", ca => ca
                            .Tokenizer("standard")
                            .Filters("lowercase", "stop", "synonyms", "stemmer")
                        )
                    )
                    .TokenFilters(tf => tf
                        .Synonym("synonyms", sf => sf
                            .Synonyms(
                                "laptop,notebook,computer",
                                "phone,mobile,cellphone",
                                "tv,television"
                            )
                        )
                        .Stemmer("stemmer", st => st
                            .Language(Language.English)
                        )
                    )
                )
                .NumberOfShards(2)
                .NumberOfReplicas(1)
            )
            .Map<Product>(m => m
                .Properties(p => p
                    .Keyword(k => k
                        .Name(n => n.Id)
                    )
                    .Text(t => t
                        .Name(n => n.Name)
                        .Analyzer("product_analyzer")
                        .Fields(f => f
                            .Keyword(k => k
                                .Name("raw")
                                .IgnoreAbove(256)
                            )
                        )
                    )
                    .Text(t => t
                        .Name(n => n.Description)
                        .Analyzer("product_analyzer")
                    )
                    .Keyword(k => k
                        .Name(n => n.Category)
                    )
                    .Keyword(k => k
                        .Name(n => n.Brand)
                    )
                    .Number(n => n
                        .Name(nn => nn.Price)
                        .Type(NumberType.Double)
                    )
                    .Number(n => n
                        .Name(nn => nn.Rating)
                        .Type(NumberType.Double)
                    )
                    .Number(n => n
                        .Name(nn => nn.ReviewCount)
                        .Type(NumberType.Integer)
                    )
                    .Keyword(k => k
                        .Name(n => n.Tags)
                    )
                    .Date(d => d
                        .Name(n => n.CreatedAt)
                    )
                    .GeoPoint(g => g
                        .Name(n => n.Location)
                    )
                    .Object<Dictionary<string, string>>(o => o
                        .Name(n => n.Specifications)
                        .Enabled(false)
                    )
                )
            )
        );
        
        if (createIndexResponse.IsValid)
        {
            _logger.LogInformation("Product index created successfully");
        }
        else
        {
            _logger.LogError("Failed to create product index: {Error}", 
                createIndexResponse.DebugInformation);
        }
    }
}

The mapping process in Elasticsearch determines how your documents and their fields are indexed and stored. This configuration demonstrates several important concepts: multi-field mappings that allow the same field to be analyzed differently for different use cases, custom analyzers that handle synonyms and stemming, and specialized field types like geo-point for location-based searches.

Implementing Basic Search Operations

With your schema in place, let’s implement a comprehensive search service that handles various search scenarios:

public class ProductSearchService
{
    private readonly IElasticClient _elasticClient;
    private readonly ILogger<ProductSearchService> _logger;
    
    public ProductSearchService(
        IElasticClient elasticClient, 
        ILogger<ProductSearchService> logger)
    {
        _elasticClient = elasticClient;
        _logger = logger;
    }
    
    // Basic text search
    public async Task<ISearchResponse<Product>> SearchProductsAsync(
        string searchTerm, 
        int page = 1, 
        int pageSize = 10)
    {
        var from = (page - 1) * pageSize;
        
        var searchResponse = await _elasticClient.SearchAsync<Product>(s => s
            .From(from)
            .Size(pageSize)
            .Query(q => q
                .MultiMatch(m => m
                    .Query(searchTerm)
                    .Fields(f => f
                        .Field(p => p.Name, boost: 3.0)
                        .Field(p => p.Brand, boost: 2.0)
                        .Field(p => p.Description, boost: 1.0)
                        .Field(p => p.Tags, boost: 1.5)
                    )
                    .Type(TextQueryType.BestFields)
                    .Fuzziness(Fuzziness.Auto)
                )
            )
            .Highlight(h => h
                .Fields(
                    f => f.Field(p => p.Name),
                    f => f.Field(p => p.Description)
                )
                .PreTags("<mark>")
                .PostTags("</mark>")
            )
        );
        
        LogSearchMetrics(searchResponse, searchTerm);
        return searchResponse;
    }
    
    // Advanced search with filters
    public async Task<ISearchResponse<Product>> AdvancedSearchAsync(
        ProductSearchRequest request)
    {
        var searchDescriptor = new SearchDescriptor<Product>()
            .From(request.From)
            .Size(request.Size);
        
        // Build the query
        var queries = new List<QueryContainer>();
        
        // Text search if provided
        if (!string.IsNullOrWhiteSpace(request.SearchTerm))
        {
            queries.Add(Query<Product>.MultiMatch(m => m
                .Query(request.SearchTerm)
                .Fields(f => f
                    .Field(p => p.Name, boost: 3.0)
                    .Field(p => p.Description)
                )
                .Fuzziness(Fuzziness.Auto)
            ));
        }
        
        // Category filter
        if (!string.IsNullOrWhiteSpace(request.Category))
        {
            queries.Add(Query<Product>.Term(t => t
                .Field(f => f.Category)
                .Value(request.Category)
            ));
        }
        
        // Brand filter
        if (request.Brands?.Any() == true)
        {
            queries.Add(Query<Product>.Terms(t => t
                .Field(f => f.Brand)
                .Terms(request.Brands)
            ));
        }
        
        // Price range filter
        if (request.MinPrice.HasValue || request.MaxPrice.HasValue)
        {
            queries.Add(Query<Product>.Range(r => r
                .Field(f => f.Price)
                .GreaterThanOrEquals(request.MinPrice ?? 0)
                .LessThanOrEquals(request.MaxPrice ?? double.MaxValue)
            ));
        }
        
        // Rating filter
        if (request.MinRating.HasValue)
        {
            queries.Add(Query<Product>.Range(r => r
                .Field(f => f.Rating)
                .GreaterThanOrEquals(request.MinRating.Value)
            ));
        }
        
        // Combine all queries
        searchDescriptor.Query(q => q
            .Bool(b => b
                .Must(queries.ToArray())
            )
        );
        
        // Add sorting
        if (!string.IsNullOrWhiteSpace(request.SortBy))
        {
            searchDescriptor.Sort(s => GetSortDescriptor(s, request.SortBy, request.SortOrder));
        }
        
        // Add aggregations for faceted search
        searchDescriptor.Aggregations(a => a
            .Terms("categories", t => t
                .Field(f => f.Category)
                .Size(20)
            )
            .Terms("brands", t => t
                .Field(f => f.Brand)
                .Size(50)
            )
            .Range("price_ranges", r => r
                .Field(f => f.Price)
                .Ranges(
                    rs => rs.To(50).Key("Under $50"),
                    rs => rs.From(50).To(100).Key("$50-$100"),
                    rs => rs.From(100).To(200).Key("$100-$200"),
                    rs => rs.From(200).Key("Over $200")
                )
            )
            .Histogram("rating_distribution", h => h
                .Field(f => f.Rating)
                .Interval(1)
                .MinimumDocumentCount(1)
            )
        );
        
        var response = await _elasticClient.SearchAsync<Product>(searchDescriptor);
        return response;
    }
    
    private IPromise<IList<ISort>> GetSortDescriptor(
        SortDescriptor<Product> sortDescriptor, 
        string sortBy, 
        string sortOrder)
    {
        var ascending = sortOrder?.ToLower() != "desc";
        
        return sortBy?.ToLower() switch
        {
            "price" => sortDescriptor.Field(f => f.Price, ascending ? SortOrder.Ascending : SortOrder.Descending),
            "rating" => sortDescriptor.Field(f => f.Rating, ascending ? SortOrder.Ascending : SortOrder.Descending),
            "reviews" => sortDescriptor.Field(f => f.ReviewCount, ascending ? SortOrder.Ascending : SortOrder.Descending),
            "date" => sortDescriptor.Field(f => f.CreatedAt, ascending ? SortOrder.Ascending : SortOrder.Descending),
            _ => sortDescriptor.Score(SortOrder.Descending)
        };
    }
    
    private void LogSearchMetrics(ISearchResponse<Product> response, string searchTerm)
    {
        _logger.LogInformation(
            "Search completed for '{SearchTerm}': {Hits} hits in {Took}ms",
            searchTerm,
            response.Total,
            response.Took
        );
    }
}

public class ProductSearchRequest
{
    public string SearchTerm { get; set; }
    public string Category { get; set; }
    public List<string> Brands { get; set; }
    public decimal? MinPrice { get; set; }
    public decimal? MaxPrice { get; set; }
    public double? MinRating { get; set; }
    public string SortBy { get; set; }
    public string SortOrder { get; set; }
    public int From { get; set; } = 0;
    public int Size { get; set; } = 10;
}

Now let’s create a controller that exposes these search capabilities through a RESTful API:

[ApiController]
[Route("api/[controller]")]
public class SearchController : ControllerBase
{
    private readonly ProductSearchService _searchService;
    private readonly ILogger<SearchController> _logger;
    
    public SearchController(
        ProductSearchService searchService, 
        ILogger<SearchController> logger)
    {
        _searchService = searchService;
        _logger = logger;
    }
    
    [HttpGet]
    public async Task<IActionResult> Search([FromQuery] string q, int page = 1, int pageSize = 10)
    {
        if (string.IsNullOrWhiteSpace(q))
        {
            return BadRequest("Search term is required");
        }
        
        var response = await _searchService.SearchProductsAsync(q, page, pageSize);
        
        if (!response.IsValid)
        {
            _logger.LogError("Search failed: {Error}", response.DebugInformation);
            return StatusCode(500, "Search failed");
        }
        
        var result = new SearchResultDto
        {
            Total = response.Total,
            Page = page,
            PageSize = pageSize,
            TookMs = response.Took,
            Items = response.Documents.Select(doc => new ProductDto
            {
                Id = doc.Id,
                Name = doc.Name,
                Description = doc.Description,
                Category = doc.Category,
                Brand = doc.Brand,
                Price = doc.Price,
                Rating = doc.Rating,
                ReviewCount = doc.ReviewCount,
                Highlights = response.Hits
                    .FirstOrDefault(h => h.Source.Id == doc.Id)?
                    .Highlight
                    .ToDictionary(
                        kvp => kvp.Key, 
                        kvp => string.Join(" ... ", kvp.Value)
                    )
            }).ToList()
        };
        
        return Ok(result);
    }
    
    [HttpPost("advanced")]
    public async Task<IActionResult> AdvancedSearch([FromBody] ProductSearchRequest request)
    {
        var response = await _searchService.AdvancedSearchAsync(request);
        
        if (!response.IsValid)
        {
            return StatusCode(500, "Search failed");
        }
        
        var facets = new Dictionary<string, object>();
        
        // Extract category facets
        var categoryAgg = response.Aggregations.Terms("categories");
        if (categoryAgg != null)
        {
            facets["categories"] = categoryAgg.Buckets.Select(b => new
            {
                Value = b.Key,
                Count = b.DocCount
            });
        }
        
        // Extract brand facets
        var brandAgg = response.Aggregations.Terms("brands");
        if (brandAgg != null)
        {
            facets["brands"] = brandAgg.Buckets.Select(b => new
            {
                Value = b.Key,
                Count = b.DocCount
            });
        }
        
        // Extract price range facets
        var priceAgg = response.Aggregations.Range("price_ranges");
        if (priceAgg != null)
        {
            facets["priceRanges"] = priceAgg.Buckets.Select(b => new
            {
                Label = b.Key,
                Count = b.DocCount
            });
        }
        
        var result = new AdvancedSearchResultDto
        {
            Total = response.Total,
            TookMs = response.Took,
            Items = response.Documents.Select(MapToDto).ToList(),
            Facets = facets
        };
        
        return Ok(result);
    }
    
    private ProductDto MapToDto(Product product)
    {
        return new ProductDto
        {
            Id = product.Id,
            Name = product.Name,
            Description = product.Description,
            Category = product.Category,
            Brand = product.Brand,
            Price = product.Price,
            Rating = product.Rating,
            ReviewCount = product.ReviewCount,
            Tags = product.Tags
        };
    }
}

Advanced Search Features and Aggregations

Moving beyond basic text search, let’s implement more sophisticated search features including autocomplete, fuzzy matching, and complex aggregations:

public class AdvancedSearchService
{
    private readonly IElasticClient _elasticClient;
    
    public AdvancedSearchService(IElasticClient elasticClient)
    {
        _elasticClient = elasticClient;
    }
    
    // Autocomplete/Suggest functionality
    public async Task<List<string>> GetSuggestionsAsync(string prefix)
    {
        var searchResponse = await _elasticClient.SearchAsync<Product>(s => s
            .Size(0)
            .Suggest(su => su
                .Completion("product-suggestions", c => c
                    .Field(f => f.Name)
                    .Prefix(prefix)
                    .Fuzzy(f => f
                        .Fuzziness(Fuzziness.Auto)
                    )
                    .Size(10)
                    .SkipDuplicates()
                )
            )
        );
        
        var suggestions = searchResponse.Suggest["product-suggestions"]
            .SelectMany(s => s.Options)
            .Select(o => o.Text)
            .Distinct()
            .ToList();
        
        return suggestions;
    }
    
    // Phrase search with slop for proximity
    public async Task<ISearchResponse<Product>> PhraseSearchAsync(
        string phrase, 
        int slop = 2)
    {
        return await _elasticClient.SearchAsync<Product>(s => s
            .Query(q => q
                .MatchPhrase(m => m
                    .Field(f => f.Description)
                    .Query(phrase)
                    .Slop(slop)
                )
            )
        );
    }
    
    // More Like This - find similar products
    public async Task<ISearchResponse<Product>> FindSimilarProductsAsync(string productId)
    {
        return await _elasticClient.SearchAsync<Product>(s => s
            .Query(q => q
                .MoreLikeThis(mlt => mlt
                    .Fields(f => f
                        .Field(ff => ff.Name)
                        .Field(ff => ff.Description)
                        .Field(ff => ff.Tags)
                    )
                    .Like(l => l
                        .Document(d => d
                            .Id(productId)
                            .Index("products")
                        )
                    )
                    .MinTermFrequency(1)
                    .MinDocumentFrequency(2)
                    .MaxQueryTerms(25)
                )
            )
            .Size(10)
        );
    }
    
    // Complex aggregations for analytics
    public async Task<Dictionary<string, object>> GetSearchAnalyticsAsync()
    {
        var response = await _elasticClient.SearchAsync<Product>(s => s
            .Size(0)
            .Aggregations(a => a
                // Top categories by average rating
                .Terms("top_categories", t => t
                    .Field(f => f.Category)
                    .Size(10)
                    .Order(o => o.Descending("avg_rating"))
                    .Aggregations(aa => aa
                        .Average("avg_rating", avg => avg
                            .Field(f => f.Rating)
                        )
                        .Sum("total_reviews", sum => sum
                            .Field(f => f.ReviewCount)
                        )
                    )
                )
                // Price statistics by brand
                .Terms("brand_price_stats", t => t
                    .Field(f => f.Brand)
                    .Size(20)
                    .Aggregations(aa => aa
                        .Stats("price_stats", st => st
                            .Field(f => f.Price)
                        )
                    )
                )
                // Date histogram for product creation trends
                .DateHistogram("creation_trends", dh => dh
                    .Field(f => f.CreatedAt)
                    .CalendarInterval(DateInterval.Month)
                    .Format("yyyy-MM")
                    .MinimumDocumentCount(1)
                )
                // Percentiles for price distribution
                .Percentiles("price_percentiles", p => p
                    .Field(f => f.Price)
                    .Percents(25, 50, 75, 90, 95, 99)
                )
            )
        );
        
        var analytics = new Dictionary<string, object>();
        
        // Process top categories
        var topCategories = response.Aggregations.Terms("top_categories");
        analytics["topCategories"] = topCategories.Buckets.Select(b => new
        {
            Category = b.Key,
            Count = b.DocCount,
            AverageRating = b.Average("avg_rating").Value,
            TotalReviews = b.Sum("total_reviews").Value
        });
        
        // Process brand price statistics
        var brandStats = response.Aggregations.Terms("brand_price_stats");
        analytics["brandPriceStats"] = brandStats.Buckets.Select(b =>
        {
            var stats = b.Stats("price_stats");
            return new
            {
                Brand = b.Key,
                Count = b.DocCount,
                MinPrice = stats.Min,
                MaxPrice = stats.Max,
                AvgPrice = stats.Average,
                StdDeviation = stats.StandardDeviation
            };
        });
        
        // Process creation trends
        var trends = response.Aggregations.DateHistogram("creation_trends");
        analytics["creationTrends"] = trends.Buckets.Select(b => new
        {
            Date = b.KeyAsString,
            Count = b.DocCount
        });
        
        // Process price percentiles
        var percentiles = response.Aggregations.Percentiles("price_percentiles");
        analytics["pricePercentiles"] = percentiles.Items.ToDictionary(
            i => $"p{i.Percentile}", 
            i => i.Value
        );
        
        return analytics;
    }
    
    // Geo-distance search for location-based queries
    public async Task<ISearchResponse<Product>> SearchNearbyProductsAsync(
        double latitude, 
        double longitude, 
        string distance = "10km")
    {
        return await _elasticClient.SearchAsync<Product>(s => s
            .Query(q => q
                .Bool(b => b
                    .Filter(f => f
                        .GeoDistance(g => g
                            .Field(ff => ff.Location)
                            .Location(latitude, longitude)
                            .Distance(distance)
                        )
                    )
                )
            )
            .Sort(so => so
                .GeoDistance(g => g
                    .Field(f => f.Location)
                    .PinLocation(latitude, longitude)
                    .Order(SortOrder.Ascending)
                    .Unit(DistanceUnit.Kilometers)
                )
            )
        );
    }
}

Optimizing Search Performance

Performance optimization in Elasticsearch requires understanding both how the engine works internally and how your specific search patterns impact resource usage. Let’s implement some performance optimization strategies:

public class OptimizedSearchService
{
    private readonly IElasticClient _elasticClient;
    private readonly IMemoryCache _cache;
    private readonly ILogger<OptimizedSearchService> _logger;
    
    public OptimizedSearchService(
        IElasticClient elasticClient,
        IMemoryCache cache,
        ILogger<OptimizedSearchService> logger)
    {
        _elasticClient = elasticClient;
        _cache = cache;
        _logger = logger;
    }
    
    // Implement search with caching
    public async Task<SearchResult<Product>> CachedSearchAsync(
        string searchTerm, 
        int page = 1, 
        int pageSize = 10)
    {
        var cacheKey = $"search:{searchTerm}:{page}:{pageSize}";
        
        if (_cache.TryGetValue<SearchResult<Product>>(cacheKey, out var cachedResult))
        {
            _logger.LogDebug("Cache hit for search: {SearchTerm}", searchTerm);
            return cachedResult;
        }
        
        var response = await _elasticClient.SearchAsync<Product>(s => s
            .From((page - 1) * pageSize)
            .Size(pageSize)
            .Query(q => q
                .Bool(b => b
                    .Should(sh => sh
                        .Match(m => m
                            .Field(f => f.Name)
                            .Query(searchTerm)
                            .Boost(2.0)
                        )
                    )
                    .Should(sh => sh
                        .Match(m => m
                            .Field(f => f.Description)
                            .Query(searchTerm)
                        )
                    )
                    .MinimumShouldMatch(1)
                )
            )
            // Use filter context for non-scoring queries
            .PostFilter(pf => pf
                .Range(r => r
                    .Field(f => f.Price)
                    .GreaterThan(0)
                )
            )
            // Limit fields returned to reduce payload
            .Source(src => src
                .Includes(i => i
                    .Fields(
                        f => f.Id,
                        f => f.Name,
                        f => f.Price,
                        f => f.Rating,
                        f => f.Category
                    )
                )
            )
            // Request only necessary aggregations
            .Aggregations(a => a
                .Terms("top_categories", t => t
                    .Field(f => f.Category)
                    .Size(5)
                )
            )
        );
        
        var result = new SearchResult<Product>
        {
            Items = response.Documents,
            Total = response.Total,
            TookMs = response.Took,
            Categories = response.Aggregations.Terms("top_categories")?
                .Buckets.Select(b => b.Key).ToList()
        };
        
        // Cache for 5 minutes
        _cache.Set(cacheKey, result, TimeSpan.FromMinutes(5));
        
        return result;
    }
    
    // Implement scroll API for large result sets
    public async Task<List<Product>> ScrollAllProductsAsync(string category)
    {
        var allProducts = new List<Product>();
        var scrollTimeout = "1m";
        
        // Initial search request
        var initialResponse = await _elasticClient.SearchAsync<Product>(s => s
            .Size(1000)
            .Scroll(scrollTimeout)
            .Query(q => q
                .Term(t => t
                    .Field(f => f.Category)
                    .Value(category)
                )
            )
        );
        
        allProducts.AddRange(initialResponse.Documents);
        var scrollId = initialResponse.ScrollId;
        
        // Continue scrolling until no more results
        while (initialResponse.Documents.Any())
        {
            var scrollResponse = await _elasticClient.ScrollAsync<Product>(scrollTimeout, scrollId);
            if (!scrollResponse.Documents.Any())
                break;
                
            allProducts.AddRange(scrollResponse.Documents);
            scrollId = scrollResponse.ScrollId;
        }
        
        // Clear scroll context
        await _elasticClient.ClearScrollAsync(c => c.ScrollId(scrollId));
        
        return allProducts;
    }
    
    // Bulk indexing for better performance
    public async Task<bool> BulkIndexProductsAsync(List<Product> products)
    {
        var bulkDescriptor = new BulkDescriptor();
        
        foreach (var product in products)
        {
            bulkDescriptor.Index<Product>(i => i
                .Document(product)
                .Id(product.Id)
            );
        }
        
        var bulkResponse = await _elasticClient.BulkAsync(bulkDescriptor);
        
        if (!bulkResponse.IsValid)
        {
            _logger.LogError("Bulk indexing failed: {Errors}", 
                string.Join(", ", bulkResponse.ItemsWithErrors.Select(i => i.Error)));
            return false;
        }
        
        _logger.LogInformation("Bulk indexed {Count} products in {Took}ms", 
            products.Count, bulkResponse.Took);
        
        return true;
    }
    
    // Use search_after for deep pagination
    public async Task<ISearchResponse<Product>> SearchAfterAsync(
        string searchTerm, 
        object[] searchAfterValues = null)
    {
        var searchDescriptor = new SearchDescriptor<Product>()
            .Size(20)
            .Query(q => q
                .Match(m => m
                    .Field(f => f.Name)
                    .Query(searchTerm)
                )
            )
            .Sort(s => s
                .Field(f => f.Price, SortOrder.Ascending)
                .Field(f => f.Id, SortOrder.Ascending)
            );
        
        if (searchAfterValues != null)
        {
            searchDescriptor.SearchAfter(searchAfterValues);
        }
        
        return await _elasticClient.SearchAsync<Product>(searchDescriptor);
    }
}

public class SearchResult<T>
{
    public List<T> Items { get; set; }
    public long Total { get; set; }
    public long TookMs { get; set; }
    public List<string> Categories { get; set; }
}

Handling Real-world Search Scenarios

Production search implementations must handle various real-world challenges. Let’s implement solutions for common scenarios:

public class RealWorldSearchService
{
    private readonly IElasticClient _elasticClient;
    private readonly IConfiguration _configuration;
    
    public RealWorldSearchService(
        IElasticClient elasticClient, 
        IConfiguration configuration)
    {
        _elasticClient = elasticClient;
        _configuration = configuration;
    }
    
    // Multi-language search support
    public async Task<ISearchResponse<Product>> MultilingualSearchAsync(
        string searchTerm, 
        string language = "en")
    {
        var analyzer = language switch
        {
            "es" => "spanish",
            "fr" => "french",
            "de" => "german",
            "zh" => "chinese",
            _ => "english"
        };
        
        return await _elasticClient.SearchAsync<Product>(s => s
            .Query(q => q
                .MultiMatch(m => m
                    .Query(searchTerm)
                    .Fields(f => f
                        .Field($"name.{language}", boost: 2.0)
                        .Field($"description.{language}")
                    )
                    .Analyzer(analyzer)
                )
            )
        );
    }
    
    // Implement search with user context and personalization
    public async Task<ISearchResponse<Product>> PersonalizedSearchAsync(
        string searchTerm, 
        UserContext userContext)
    {
        var userPreferredBrands = userContext.PreferredBrands ?? new List<string>();
        var userPriceRange = userContext.TypicalPriceRange;
        
        return await _elasticClient.SearchAsync<Product>(s => s
            .Query(q => q
                .Bool(b => b
                    .Must(m => m
                        .MultiMatch(mm => mm
                            .Query(searchTerm)
                            .Fields(f => f
                                .Field(p => p.Name)
                                .Field(p => p.Description)
                            )
                        )
                    )
                    .Should(
                        // Boost products from user's preferred brands
                        sh => sh.Terms(t => t
                            .Field(f => f.Brand)
                            .Terms(userPreferredBrands)
                            .Boost(1.5)
                        ),
                        // Boost products in user's typical price range
                        sh => sh.Range(r => r
                            .Field(f => f.Price)
                            .GreaterThanOrEquals(userPriceRange.Min)
                            .LessThanOrEquals(userPriceRange.Max)
                            .Boost(1.2)
                        )
                    )
                )
            )
            // Re-score top results based on user's past behavior
            .Rescore(r => r
                .Rescore(rr => rr
                    .WindowSize(50)
                    .Query(rq => rq
                        .ScoreMode(ScoreMode.Multiply)
                        .Query(q => q
                            .FunctionScore(fs => fs
                                .Functions(fn => fn
                                    .FieldValueFactor(fvf => fvf
                                        .Field(f => f.Rating)
                                        .Factor(1.2)
                                        .Modifier(FieldValueFactorModifier.Log1P)
                                    )
                                )
                            )
                        )
                    )
                )
            )
        );
    }
    
    // Implement typo-tolerant search with multiple strategies
    public async Task<ISearchResponse<Product>> TypoTolerantSearchAsync(string searchTerm)
    {
        // First try exact match
        var exactMatch = await _elasticClient.SearchAsync<Product>(s => s
            .Size(5)
            .Query(q => q
                .Match(m => m
                    .Field(f => f.Name)
                    .Query(searchTerm)
                    .Operator(Operator.And)
                )
            )
        );
        
        if (exactMatch.Total > 0)
        {
            return exactMatch;
        }
        
        // Fallback to fuzzy search
        return await _elasticClient.SearchAsync<Product>(s => s
            .Query(q => q
                .Bool(b => b
                    .Should(
                        sh => sh.Match(m => m
                            .Field(f => f.Name)
                            .Query(searchTerm)
                            .Fuzziness(Fuzziness.Auto)
                            .PrefixLength(2)
                        ),
                        sh => sh.MatchPhrasePrefix(mpp => mpp
                            .Field(f => f.Name)
                            .Query(searchTerm)
                            .MaxExpansions(10)
                        ),
                        sh => sh.Wildcard(w => w
                            .Field(f => f.Name.Suffix("raw"))
                            .Value($"*{searchTerm}*")
                            .Boost(0.5)
                        )
                    )
                    .MinimumShouldMatch(1)
                )
            )
        );
    }
    
    // Security-filtered search based on user permissions
    public async Task<ISearchResponse<Product>> SecureSearchAsync(
        string searchTerm, 
        ClaimsPrincipal user)
    {
        var userRoles = user.Claims
            .Where(c => c.Type == ClaimTypes.Role)
            .Select(c => c.Value)
            .ToList();
        
        var allowedCategories = GetAllowedCategoriesForRoles(userRoles);
        
        return await _elasticClient.SearchAsync<Product>(s => s
            .Query(q => q
                .Bool(b => b
                    .Must(m => m
                        .MultiMatch(mm => mm
                            .Query(searchTerm)
                            .Fields(f => f
                                .Field(p => p.Name)
                                .Field(p => p.Description)
                            )
                        )
                    )
                    .Filter(f => f
                        .Terms(t => t
                            .Field(ff => ff.Category)
                            .Terms(allowedCategories)
                        )
                    )
                )
            )
        );
    }
    
    private List<string> GetAllowedCategoriesForRoles(List<string> roles)
    {
        // Implementation would check role-based permissions
        var categories = new List<string> { "Electronics", "Books", "Clothing" };
        
        if (roles.Contains("Premium"))
        {
            categories.Add("Exclusive");
        }
        
        if (roles.Contains("B2B"))
        {
            categories.Add("Wholesale");
        }
        
        return categories;
    }
}

public class UserContext
{
    public List<string> PreferredBrands { get; set; }
    public PriceRange TypicalPriceRange { get; set; }
    public List<string> RecentSearches { get; set; }
    public Dictionary<string, int> CategoryPreferences { get; set; }
}

public class PriceRange
{
    public decimal Min { get; set; }
    public decimal Max { get; set; }
}

Monitoring and Maintaining Your Search Infrastructure

A successful Elasticsearch deployment requires comprehensive monitoring and maintenance. Let’s implement health checks and monitoring services:

public class ElasticsearchHealthCheck : IHealthCheck
{
    private readonly IElasticClient _elasticClient;
    private readonly ILogger<ElasticsearchHealthCheck> _logger;
    
    public ElasticsearchHealthCheck(
        IElasticClient elasticClient,
        ILogger<ElasticsearchHealthCheck> logger)
    {
        _elasticClient = elasticClient;
        _logger = logger;
    }
    
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            var pingResponse = await _elasticClient.PingAsync(cancellationToken: cancellationToken);
            
            if (!pingResponse.IsValid)
            {
                return HealthCheckResult.Unhealthy(
                    "Elasticsearch cluster is not responding",
                    pingResponse.OriginalException);
            }
            
            var clusterHealth = await _elasticClient.Cluster
                .HealthAsync(cancellationToken: cancellationToken);
            
            if (!clusterHealth.IsValid)
            {
                return HealthCheckResult.Unhealthy(
                    "Could not get cluster health",
                    clusterHealth.OriginalException);
            }
            
            var data = new Dictionary<string, object>
            {
                ["cluster_name"] = clusterHealth.ClusterName,
                ["status"] = clusterHealth.Status.ToString(),
                ["number_of_nodes"] = clusterHealth.NumberOfNodes,
                ["active_shards"] = clusterHealth.ActiveShards,
                ["relocating_shards"] = clusterHealth.RelocatingShards,
                ["unassigned_shards"] = clusterHealth.UnassignedShards
            };
            
            return clusterHealth.Status switch
            {
                Health.Green => HealthCheckResult.Healthy("Elasticsearch cluster is healthy", data),
                Health.Yellow => HealthCheckResult.Degraded("Elasticsearch cluster is degraded", null, data),
                _ => HealthCheckResult.Unhealthy("Elasticsearch cluster is unhealthy", null, data)
            };
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Health check failed");
            return HealthCheckResult.Unhealthy("Health check exception", ex);
        }
    }
}

// Register health checks in Program.cs
builder.Services
    .AddHealthChecks()
    .AddTypeActivatedCheck<ElasticsearchHealthCheck>(
        "elasticsearch",
        failureStatus: HealthStatus.Unhealthy,
        tags: new[] { "ready", "live" });

// Add monitoring service
public class ElasticsearchMonitoringService : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<ElasticsearchMonitoringService> _logger;
    private readonly Timer _timer;
    
    public ElasticsearchMonitoringService(
        IServiceProvider serviceProvider,
        ILogger<ElasticsearchMonitoringService> logger)
    {
        _serviceProvider = serviceProvider;
        _logger = logger;
    }
    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await MonitorClusterAsync(stoppingToken);
            await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
        }
    }
    
    private async Task MonitorClusterAsync(CancellationToken cancellationToken)
    {
        using var scope = _serviceProvider.CreateScope();
        var elasticClient = scope.ServiceProvider.GetRequiredService<IElasticClient>();
        
        try
        {
            // Monitor cluster stats
            var statsResponse = await elasticClient.Cluster
                .StatsAsync(cancellationToken: cancellationToken);
            
            if (statsResponse.IsValid)
            {
                _logger.LogInformation(
                    "Cluster stats - Docs: {DocCount}, Size: {SizeInBytes}MB, Queries: {QueryTotal}",
                    statsResponse.Indices.Documents.Count,
                    statsResponse.Indices.Store.SizeInBytes / (1024 * 1024),
                    statsResponse.Indices.Search.QueryTotal
                );
            }
            
            // Monitor index stats
            var indexStats = await elasticClient.Indices
                .StatsAsync("products", cancellationToken: cancellationToken);
            
            if (indexStats.IsValid)
            {
                var stats = indexStats.Indices["products"];
                _logger.LogInformation(
                    "Product index - Docs: {DocCount}, Size: {SizeInBytes}MB",
                    stats.Primaries.Documents.Count,
                    stats.Primaries.Store.SizeInBytes / (1024 * 1024)
                );
                
                // Check if index needs optimization
                if (stats.Primaries.Documents.Deleted > stats.Primaries.Documents.Count * 0.1)
                {
                    _logger.LogWarning(
                        "Product index has {DeletedRatio:P} deleted documents. Consider force merge.",
                        (double)stats.Primaries.Documents.Deleted / stats.Primaries.Documents.Count
                    );
                }
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to monitor Elasticsearch cluster");
        }
    }
}

Testing Your Search Implementation

Testing search functionality requires a comprehensive approach. Let’s implement various testing strategies:

[TestFixture]
public class SearchServiceTests
{
    private IElasticClient _elasticClient;
    private ProductSearchService _searchService;
    private string _testIndexName = "products_test";
    
    [OneTimeSetUp]
    public async Task Setup()
    {
        var settings = new ConnectionSettings(new Uri("http://localhost:9200"))
            .DefaultIndex(_testIndexName)
            .EnableDebugMode();
            
        _elasticClient = new ElasticClient(settings);
        _searchService = new ProductSearchService(
            _elasticClient, 
            new NullLogger<ProductSearchService>()
        );
        
        // Create test index with mapping
        await CreateTestIndexAsync();
        await SeedTestDataAsync();
    }
    
    private async Task CreateTestIndexAsync()
    {
        await _elasticClient.Indices.DeleteAsync(_testIndexName);
        
        await _elasticClient.Indices.CreateAsync(_testIndexName, c => c
            .Map<Product>(m => m
                .AutoMap()
            )
        );
    }
    
    private async Task SeedTestDataAsync()
    {
        var products = new List<Product>
        {
            new Product
            {
                Id = "1",
                Name = "iPhone 15 Pro",
                Description = "Latest Apple smartphone with titanium design",
                Category = "Electronics",
                Brand = "Apple",
                Price = 999,
                Rating = 4.5,
                Tags = new List<string> { "smartphone", "5g", "camera" }
            },
            new Product
            {
                Id = "2",
                Name = "Samsung Galaxy S24",
                Description = "Android flagship with AI features",
                Category = "Electronics",
                Brand = "Samsung",
                Price = 899,
                Rating = 4.3,
                Tags = new List<string> { "smartphone", "android", "ai" }
            }
        };
        
        var bulkResponse = await _elasticClient.BulkAsync(b => b
            .Index(_testIndexName)
            .IndexMany(products)
        );
        
        await _elasticClient.Indices.RefreshAsync(_testIndexName);
    }
    
    [Test]
    public async Task SearchProducts_WithValidTerm_ReturnsResults()
    {
        // Act
        var response = await _searchService.SearchProductsAsync("iPhone");
        
        // Assert
        Assert.That(response.IsValid, Is.True);
        Assert.That(response.Total, Is.GreaterThan(0));
        Assert.That(response.Documents.First().Name, Contains.Substring("iPhone"));
    }
    
    [Test]
    public async Task SearchProducts_WithFuzzyMatch_HandlesTypos()
    {
        // Act
        var response = await _searchService.SearchProductsAsync("ipone");
        
        // Assert
        Assert.That(response.IsValid, Is.True);
        Assert.That(response.Total, Is.GreaterThan(0));
    }
    
    [Test]
    public async Task AdvancedSearch_WithPriceFilter_ReturnsFilteredResults()
    {
        // Arrange
        var request = new ProductSearchRequest
        {
            MinPrice = 900,
            MaxPrice = 1000
        };
        
        // Act
        var response = await _searchService.AdvancedSearchAsync(request);
        
        // Assert
        Assert.That(response.IsValid, Is.True);
        Assert.That(response.Documents.All(d => d.Price >= 900 && d.Price <= 1000), Is.True);
    }
    
    [OneTimeTearDown]
    public async Task Cleanup()
    {
        await _elasticClient.Indices.DeleteAsync(_testIndexName);
    }
}

// Integration tests using WebApplicationFactory
public class SearchIntegrationTests : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly WebApplicationFactory<Program> _factory;
    private readonly HttpClient _client;
    
    public SearchIntegrationTests(WebApplicationFactory<Program> factory)
    {
        _factory = factory.WithWebHostBuilder(builder =>
        {
            builder.ConfigureServices(services =>
            {
                // Override Elasticsearch configuration for testing
                services.AddSingleton<IElasticClient>(provider =>
                {
                    var settings = new ConnectionSettings(new Uri("http://localhost:9200"))
                        .DefaultIndex("products_integration_test");
                    return new ElasticClient(settings);
                });
            });
        });
        
        _client = _factory.CreateClient();
    }
    
    [Fact]
    public async Task SearchEndpoint_ReturnsSuccessStatusCode()
    {
        // Act
        var response = await _client.GetAsync("/api/search?q=test");
        
        // Assert
        response.EnsureSuccessStatusCode();
        var content = await response.Content.ReadAsStringAsync();
        Assert.NotNull(content);
    }
    
    [Fact]
    public async Task HealthCheck_ReturnsHealthyStatus()
    {
        // Act
        var response = await _client.GetAsync("/health");
        
        // Assert
        Assert.Equal(HttpStatusCode.OK, response.StatusCode);
    }
}

The integration between Elasticsearch and ASP.NET Core opens up a world of possibilities for building sophisticated search experiences. From basic text matching to complex aggregations and machine learning-powered semantic search, the combination provides all the tools needed to create search functionality that delights users and scales with your application’s growth. The key to success lies in understanding both technologies deeply, designing your schema thoughtfully, and continuously monitoring and optimizing based on real-world usage patterns.

Join the Community

Ready to supercharge your ASP.NET Core applications with powerful search capabilities? Subscribe to ASP Today for weekly insights, tutorials, and best practices that will level up your development skills. Join our vibrant community on Substack Chat where developers share experiences, solve challenges together, and stay ahead of the latest trends in .NET development.