Failure injector plugin implementation.

[mashhurs]

Release notes

What does this PR do?

Introduces the failure_injector integration plugin to test the Logstash health. I don't see the input plugin type need, so filter and output plugin types are considered.
Usage: plugin has degrade_at (array) and crash_at (string) configurations define the phases failure_injector needs to simulate degrade or crash behavior.

Example config:

filter {
    failure_injector {
        #degrade_at => [register, filter, close] # receives one or any of [register, filter, close] 
        crash_at => close # [register, filter, close]
    }
}
output {
    failure_injector {
        degrade_at => [receive] # accepts one or any of [register, receive, close]
        #crash_at => receive # accepts one of [register, receive, close]
    }
}

Why is it important/What is the impact to the user?

N.A

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• Check the unknown status when plugin crashes at register or close phase (filter or receive phase causes TERMINATED which is our expectation)

How to test this PR locally

• This PR needs to be tested with health api: expose GET /_health_report with pipelines/*/up probe #16398
• Install the plugin with bin/logstash-plugin install qa/support/logstash-integration-failure_injector/logstash-integration-failure_injector-0.0.1.gem command
• Use following config and hit the GET /_health_report endpoint

# pipelines.yml
- pipeline.id: slow-start-pp
  config.string: |
    input { heartbeat {} }
    filter { failure_injector { degrade_at => [register] } }
    output { stdout {} }
- pipeline.id: filter-crashed-pp
  config.string: |
    input { generator { count => 10 } }
    filter { failure_injector { crash_at => register } }
    output { stdout {} }
- pipeline.id: normal-heavy-finish-pp
  config.string: |
    input { generator { count => 10 } }
    filter { failure_injector { degrade_at => filter } }
    output { failure_injector { degrade_at => receive } }
- pipeline.id: output-crashed-pp
  config.string: |
    input { generator { count => 10 } }
    output { failure_injector { crash_at => receive } }
- pipeline.id: pp-to-pp-upstream
  config.string: input { generator {} } output { pipeline { send_to => [myVirtualAddress] } }
- pipeline.id: pp-to-pp-downstream
  config.string: input { pipeline { address => myVirtualAddress } } output { failure_injector { crash_at => receive } }

• See the log section for the result

Related issues

Use cases

Screenshots

Logs

# Output
{
    "status": "red",
    "symptom": "1 indicator is unhealthy (`pipelines`)",
    "indicators": {
        "pipelines": {
            "status": "red",
            "symptom": "2 indicators are unhealthy (`pp-to-pp-downstream`, `output-crashed-pp`), 1 indicator is degraded or at risk of becoming unhealthy (`normal-heavy-finish-pp`), 2 indicators are healthy (`slow-start-pp`, `pp-to-pp-upstream`), and 1 indicator is unknown (`filter-crashed-pp`)",
            "indicators": {
                "filter-crashed-pp": {
                    "status": "unknown",
                    "symptom": "The pipeline is unknown; 1 area is impacted and 1 diagnosis is available.",
                    "diagnosis": [
                        {
                            "cause": "pipeline is not known; it may have been recently deleted",
                            "action": "check logs",
                            "help_url": null
                        }
                    ],
                    "impacts": [
                        {
                            "severity": 0,
                            "description": "pipeline is not known",
                            "impactAreas": [
                                "pipeline_execution"
                            ]
                        }
                    ],
                    "details": {
                        "run_state": "UNKNOWN"
                    }
                },
                "pp-to-pp-downstream": {
                    "status": "red",
                    "symptom": "The pipeline is unhealthy; 1 area is impacted and 1 diagnosis is available.",
                    "diagnosis": [
                        {
                            "cause": "pipeline is not running, likely because it has encountered an error",
                            "action": "check logs",
                            "help_url": null
                        }
                    ],
                    "impacts": [
                        {
                            "severity": 0,
                            "description": "pipeline is not running",
                            "impactAreas": [
                                "pipeline_execution"
                            ]
                        }
                    ],
                    "details": {
                        "run_state": "TERMINATED"
                    }
                },
                "slow-start-pp": {
                    "status": "green",
                    "symptom": "The pipeline is healthy; 0 areas are impacted and 0 diagnoses are available.",
                    "details": {
                        "run_state": "RUNNING"
                    }
                },
                "output-crashed-pp": {
                    "status": "red",
                    "symptom": "The pipeline is unhealthy; 1 area is impacted and 1 diagnosis is available.",
                    "diagnosis": [
                        {
                            "cause": "pipeline is not running, likely because it has encountered an error",
                            "action": "check logs",
                            "help_url": null
                        }
                    ],
                    "impacts": [
                        {
                            "severity": 0,
                            "description": "pipeline is not running",
                            "impactAreas": [
                                "pipeline_execution"
                            ]
                        }
                    ],
                    "details": {
                        "run_state": "TERMINATED"
                    }
                },
                "normal-heavy-finish-pp": {
                    "status": "yellow",
                    "symptom": "The pipeline is degraded or at risk of becoming unhealthy; 1 area is impacted and 1 diagnosis is available.",
                    "diagnosis": [
                        {
                            "cause": "pipeline is loading",
                            "action": "check logs",
                            "help_url": null
                        }
                    ],
                    "impacts": [
                        {
                            "severity": 1,
                            "description": "pipeline is loading",
                            "impactAreas": [
                                "pipeline_execution"
                            ]
                        }
                    ],
                    "details": {
                        "run_state": "LOADING"
                    }
                },
                "pp-to-pp-upstream": {
                    "status": "green",
                    "symptom": "The pipeline is healthy; 0 areas are impacted and 0 diagnoses are available.",
                    "details": {
                        "run_state": "RUNNING"
                    }
                }
            }
        }
    }
}

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: afc127d

Failed CI Steps

• 🥼 IT Persistent Queues / part 2