Capture response details for non-2xx HTTP status codes

[nimishjs17]

• enhancement

Release notes

Added support for capturing HTTP status codes and error response bodies in the HTTP filter plugin. New configuration options target_status_code, include_body_on_failure, and include_status_code enable better error handling and debugging capabilities for failed HTTP requests.

What does this PR do?

This PR enhances the HTTP filter plugin to capture and expose HTTP status codes and response bodies even for non-2xx responses, significantly improving error handling and debugging capabilities.

Changes implemented:

1. Added target_status_code configuration: A configurable field to store HTTP status codes for all responses (successful and failed)
2. Added include_body_on_failure option: Controls whether to capture response body for failed requests (default: true)
3. Added include_status_code option: Controls whether to always include the status code (default: true)
4. Enhanced error handling:
   • Status codes are captured for all responses (2xx, 4xx, 5xx)
   • Connection/client errors set status code to 0 for easy identification
   • Response bodies are preserved for error responses when enabled
   • Structured error messages for connection failures

Technical implementation:

• Modified the filter method to capture status codes and response bodies for all scenarios
• Updated process_response method to handle error responses without losing data
• Added ECS-compatible default field paths for status codes
• Maintained full backward compatibility with existing configuration

Why is it important/What is the impact to the user?

Current limitations:

Previously, the HTTP filter only added failure tags (_httprequestfailure) for non-2xx responses without exposing:

• The actual HTTP status code
• The error response body (which often contains valuable error details)

This made debugging API integrations difficult and prevented proper error handling in pipelines.

User benefits:

1. Better debugging: Access to actual error messages from APIs instead of just failure tags
2. Improved monitoring: Status codes available for metrics and alerting
3. Conditional processing: Ability to handle different error types (4xx vs 5xx) differently
4. Connection error detection: Status code 0 clearly identifies network/connection issues
5. No breaking changes: Existing pipelines continue working without modification

Example use case:

filter {
  http {
    url => "https://api.example.com/endpoint"
    target_body => "[api_response]"
    target_status_code => "[api_status]"
    include_body_on_failure => true
  }
  
  # Now users can handle errors properly
  if [api_status] >= 400 and [api_status] < 500 {
    # Handle client errors with actual error message
    mutate { add_field => { "error" => "%{[api_response][message]}" } }
  }
}

Checklist

• [ ✅ ] My code follows the style guidelines of this project
• [ ✅ ] I have commented my code, particularly in hard-to-understand areas
• [ ✅ ] I have made corresponding changes to the documentation
• [ ✅ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• [ ✅ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Scenario 1: API Rate Limiting
Given an API that returns 429 status with rate limit information
When the HTTP filter makes a request that triggers rate limiting
Then the status code (429) and response body (containing retry-after info) should be accessible

Scenario 2: API Validation Errors
Given an API that returns 400 with validation error details
When the HTTP filter sends invalid data
Then the validation errors in the response body should be accessible for logging/processing

Scenario 3: Service Unavailable
Given an API that is temporarily down (503)
When the HTTP filter attempts to connect
Then the 503 status should be captured for monitoring/alerting

Scenario 4: Network Issues
Given a network timeout or connection refused scenario
When the HTTP filter cannot establish connection
Then status code should be 0 with descriptive error message

Screenshots

Logs

[cla-checker-service]

💚 CLA has been signed