From cf974356895a59a5181c699ee62f763fa9c5beca Mon Sep 17 00:00:00 2001 From: "mergify[bot]" <37929162+mergify[bot]@users.noreply.github.com> Date: Mon, 22 Jan 2024 10:30:44 -0500 Subject: [PATCH] [8.12] Regex support of add_resource_metadata (backport #834) (#835) * Regex support of add_resource_metadata (#834) * adding page for adavnce agent with k8s * adding kubernetes provider example * fxing typos and indentation * fxing typos and indentation * Update docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * Update docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * Update docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * Update docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * Update docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * Update docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * Update docs/en/ingest-management/elastic-agent/advanced-kubernetes-managed-by-fleet.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * udpating code yaml box and adding parameter of manifest * updating number in list * updating number in list and iamge for kibana UI * adding initial doc for metadata enrichment * adding initial doc for metadata enrichment * adding initial doc for metadata enrichment * adding examples * Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * Update docs/en/ingest-management/processors/processor-add_kubernetes_metadata.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * fixing conflicts * fixing part of add_resource * Fix kubernetes provider documentation (#817) * fix namespace fields; add information about the deployment and cronjob settings in add_resource_metadata Signed-off-by: Tetiana Kravchenko * Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> * Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc Co-authored-by: Andrew Gizas * Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc Co-authored-by: Andrew Gizas --------- Signed-off-by: Tetiana Kravchenko Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> Co-authored-by: Andrew Gizas * squashing * fixing newline --------- Signed-off-by: Tetiana Kravchenko Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> Co-authored-by: Tetiana Kravchenko (cherry picked from commit 64bad765d4ef4d5c2f280995d27891c77c8622b2) # Conflicts: # docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc * Fix merge conflict --------- Co-authored-by: Andrew Gizas Co-authored-by: David Kilfoyle --- .../providers/kubernetes-provider.asciidoc | 24 ++++++++++++++----- ...processor-add_kubernetes_metadata.asciidoc | 8 ++++++- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc index cb1f4dd38..8d862a1eb 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes-provider.asciidoc @@ -68,21 +68,33 @@ will be excluded from the event. By default it is `true`. `annotations.dedot`:: (Optional) If set to be `true` in the provider config, then `.` in annotations will be replaced with `_`. By default it is `true`. -`add_resource_metadata`:: (Optional) Specify labels and annotations filters for the extra metadata coming from node and namespace. - `add_resource_metadata` can be done for `node` or `namespace`. By default all labels are included -while annotations are not. These settings are useful when storing labels and annotations -that require special handling to avoid overloading the storage output. The enrichment of `node` or `namespace` metadata -can be individually disabled by setting `enabled: false`. -Example: +`add_resource_metadata`:: (Optional) Specify filters and configration for the extra metadata, that will be added to the event. +Configuration parameters: + * `node` or `namespace`: Specify labels and annotations filters for the extra metadata coming from node and namespace. By default + all labels are included while annotations are not. To change the default behaviour `include_labels`, `exclude_labels` and `include_annotations` + can be defined. These settings are useful when storing labels and annotations that require special handling to avoid overloading the storage output. + The enrichment of `node` or `namespace` metadata can be individually disabled by setting `enabled: false`. + Wildcards are supported in these settings by using `use_regex_include: true` in combination with `include_labels`, and respectively by setting `use_regex_exclude: true` in combination with `exclude_labels`. + * `deployment`: If resource is `pod` and it is created from a `deployment`, by default the deployment name isn't added, this can be enabled by setting `deployment: true`. + * `cronjob`: If resource is `pod` and it is created from a `cronjob`, by default the cronjob name isn't added, this can be enabled by setting `cronjob: true`. + Example: ["source","yaml",subs="attributes"] ------------------------------------------------------------------------------------- add_resource_metadata: namespace: + #use_regex_include: false include_labels: ["namespacelabel1"] + #use_regex_exclude: false + #exclude_labels: ["namespacelabel2"] node: + #use_regex_include: false include_labels: ["nodelabel2"] include_annotations: ["nodeannotation1"] + #use_regex_exclude: false + #exclude_labels: ["nodelabel3"] + #deployment: false + #cronjob: false ------------------------------------------------------------------------------------- diff --git a/docs/en/ingest-management/processors/processor-add_kubernetes_metadata.asciidoc b/docs/en/ingest-management/processors/processor-add_kubernetes_metadata.asciidoc index 699605327..b5d2a6f43 100644 --- a/docs/en/ingest-management/processors/processor-add_kubernetes_metadata.asciidoc +++ b/docs/en/ingest-management/processors/processor-add_kubernetes_metadata.asciidoc @@ -137,7 +137,7 @@ a| Filters and configuration for adding extra metadata to the event. This settin By default all labels are included, but annotations are not. To change the default behavior, you can set `include_labels`, `exclude_labels`, and `include_annotations`. These settings are useful when storing labels and annotations that require special handling to avoid overloading the storage output. -Note that wildcards are not supported in these settings. +Wildcards are supported in these settings by using `use_regex_include: true` in combination with `include_labels`, and respectively by setting `use_regex_exclude: true` in combination with `exclude_labels`. To turn off enrichment of `node` or `namespace` metadata individually, set `enabled: false`. * `deployment`: If the resource is `pod` and it is created from a `deployment`, the deployment name is not added by default. To enable this behavior, set `deployment: true`. * `cronjob`: If the resource is `pod` and it is created from a `cronjob`, the cronjob name is not added by default. To enable this behavior, set `cronjob: true`. @@ -151,11 +151,17 @@ To turn off enrichment of `node` or `namespace` metadata individually, set `enab add_resource_metadata: namespace: include_labels: ["namespacelabel1"] + # use_regex_include: false + # use_regex_exclude: false + # exclude_labels: ["namespacelabel2"] #labels.dedot: true #annotations.dedot: true node: + # use_regex_include: false include_labels: ["nodelabel2"] include_annotations: ["nodeannotation1"] + # use_regex_exclude: false + # exclude_annotations: ["nodeannotation2"] #labels.dedot: true #annotations.dedot: true deployment: true