ASP.NET中怎么利用Elasticsearch记录API请求响应日志

在现代Web应用开发中，日志记录是系统监控、故障排查和性能分析的重要环节。本文将详细介绍如何在ASP.NET应用中集成Elasticsearch来记录API请求和响应日志，构建一个高效的日志管理系统。

一、为什么选择Elasticsearch记录日志

1.1 传统日志记录的局限性

传统的文本文件或数据库日志存在以下问题： - 难以快速搜索和分析 - 缺乏可视化能力 - 日志量大会导致性能下降 - 难以实现分布式日志聚合

1.2 Elasticsearch的优势

• 分布式架构，可水平扩展
• 近实时搜索能力
• 强大的全文检索功能
• 可与Kibana配合实现可视化
• 支持结构化日志存储

二、系统架构设计

2.1 整体架构

[ASP.NET应用] → [日志中间件] → [Elasticsearch] ← [Kibana可视化]

2.2 日志数据结构设计

{
  "timestamp": "2023-01-01T12:00:00Z",
  "level": "Information",
  "service": "OrderService",
  "request": {
    "method": "POST",
    "path": "/api/orders",
    "query": "?page=1",
    "headers": {},
    "body": "{...}"
  },
  "response": {
    "statusCode": 200,
    "headers": {},
    "body": "{...}",
    "duration": 125
  },
  "clientIp": "192.168.1.1",
  "correlationId": "abc123"
}

三、实现步骤

3.1 安装必要的NuGet包

Install-Package NLog
Install-Package NLog.Targets.ElasticSearch
Install-Package Microsoft.AspNetCore.Http.Extensions

3.2 配置NLog.config

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true">
  
  <extensions>
    <add assembly="NLog.Targets.ElasticSearch"/>
  </extensions>

  <targets>
    <target name="elastic" 
            xsi:type="ElasticSearch"
            uri="http://localhost:9200"
            index="webapi-logs-${date:format=yyyy.MM.dd}"
            documentType="logevent"
            includeAllProperties="true">
      <field name="timestamp" layout="${date:format=yyyy-MM-ddTHH\:mm\:ss.fffZ}" />
      <field name="level" layout="${level}" />
      <field name="message" layout="${message}" />
    </target>
  </targets>

  <rules>
    <logger name="*" minlevel="Info" writeTo="elastic" />
  </rules>
</nlog>

3.3 创建日志中间件

public class ApiLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<ApiLoggingMiddleware> _logger;

    public ApiLoggingMiddleware(RequestDelegate next, ILogger<ApiLoggingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task Invoke(HttpContext context)
    {
        var stopwatch = Stopwatch.StartNew();
        var correlationId = Guid.NewGuid().ToString();
        
        // 记录请求信息
        var request = await FormatRequest(context.Request);
        
        // 复制原始响应流以便记录
        var originalBodyStream = context.Response.Body;
        using var responseBody = new MemoryStream();
        context.Response.Body = responseBody;

        try
        {
            await _next(context);
        }
        finally
        {
            stopwatch.Stop();
            
            // 记录响应信息
            var response = await FormatResponse(context.Response);
            
            var logEntry = new
            {
                Timestamp = DateTime.UtcNow,
                Level = "Information",
                Service = context.Request.Path.Value?.Split('/')[1] ?? "Unknown",
                Request = new
                {
                    context.Request.Method,
                    Path = context.Request.Path,
                    Query = context.Request.QueryString.Value,
                    Headers = GetHeaders(context.Request.Headers),
                    Body = request
                },
                Response = new
                {
                    context.Response.StatusCode,
                    Headers = GetHeaders(context.Response.Headers),
                    Body = response,
                    Duration = stopwatch.ElapsedMilliseconds
                },
                ClientIp = context.Connection.RemoteIpAddress?.ToString(),
                CorrelationId = correlationId
            };

            _logger.LogInformation("API Request/Response: {@LogEntry}", logEntry);
            
            // 恢复响应流
            await responseBody.CopyToAsync(originalBodyStream);
        }
    }

    private async Task<string> FormatRequest(HttpRequest request)
    {
        request.EnableBuffering();
        var body = await new StreamReader(request.Body).ReadToEndAsync();
        request.Body.Position = 0;
        return body;
    }

    private async Task<string> FormatResponse(HttpResponse response)
    {
        response.Body.Seek(0, SeekOrigin.Begin);
        var body = await new StreamReader(response.Body).ReadToEndAsync();
        response.Body.Seek(0, SeekOrigin.Begin);
        return body;
    }

    private Dictionary<string, string> GetHeaders(IHeaderDictionary headers)
    {
        return headers.ToDictionary(h => h.Key, h => h.Value.ToString());
    }
}

3.4 注册中间件

在Program.cs中：

var builder = WebApplication.CreateBuilder(args);

// 添加NLog
builder.Logging.ClearProviders();
builder.Host.UseNLog();

var app = builder.Build();

// 使用中间件
app.UseMiddleware<ApiLoggingMiddleware>();

app.Run();

四、高级配置与优化

4.1 敏感信息过滤

private string FilterSensitiveData(string body)
{
    try
    {
        var json = JObject.Parse(body);
        if (json["password"] != null)
            json["password"] = "***REDACTED***";
        return json.ToString();
    }
    catch
    {
        return body;
    }
}

4.2 批量写入优化

修改NLog配置：

<target name="elastic" 
        xsi:type="BufferingWrapper"
        bufferSize="100"
        flushTimeout="5000">
  <target xsi:type="ElasticSearch" ... />
</target>

4.3 索引生命周期管理

在Elasticsearch中设置ILM策略：

PUT _ilm/policy/webapi-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

五、日志查询与分析

5.1 基本查询示例

GET webapi-logs-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "Request.Path": "/api/orders" }},
        { "range": { "timestamp": { "gte": "now-1d/d" }}}
      ]
    }
  },
  "sort": [ { "timestamp": "desc" } ]
}

5.2 Kibana可视化

1. 创建索引模式：webapi-logs-*
2. 制作仪表板：
   • 请求量时间序列图
   • 响应时间百分位图
   • 状态码分布饼图
   • 慢请求列表

六、性能与安全考虑

6.1 性能优化

• 异步写入日志
• 批量提交日志
• 避免记录过大响应体
• 设置合理的日志级别

6.2 安全措施

• 加密Elasticsearch通信
• 设置访问控制
• 过滤敏感数据
• 定期归档和清理旧日志

七、总结

通过本文介绍的方法，我们可以在ASP.NET应用中高效地记录API请求和响应日志到Elasticsearch，并利用其强大的搜索和分析能力来监控应用运行状况。这种方案特别适合微服务架构和云原生应用，能够显著提升系统的可观测性和故障排查效率。

实际实施时，建议根据具体业务需求调整日志格式、采样率和保留策略，在信息丰富度和系统性能之间取得平衡。