From a93c80da4bce61ec08e0d016ba7c8cc5024cb0d0 Mon Sep 17 00:00:00 2001 From: Tetiana Kravchenko Date: Mon, 8 Jan 2024 17:29:44 +0100 Subject: [PATCH 1/5] add missing kubernetes fields; update istio doc to use kubernetes.pod.ip instead of host Signed-off-by: Tetiana Kravchenko --- libbeat/docs/shared-autodiscover.asciidoc | 10 +++++++--- .../metricbeat/module/istio/proxy/_meta/docs.asciidoc | 2 +- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/libbeat/docs/shared-autodiscover.asciidoc b/libbeat/docs/shared-autodiscover.asciidoc index ed116fd6595..72735e8b206 100644 --- a/libbeat/docs/shared-autodiscover.asciidoc +++ b/libbeat/docs/shared-autodiscover.asciidoc @@ -203,10 +203,10 @@ Example: Different Beats that refer to the same leader lease will be competitors in holding the lease and only one will be elected as leader each time. -The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can -contain variables from the autodiscover event. They can be accessed under data namespace. +Configuration templates can contain variables from the autodiscover event. These variables can be accessed under the `data` +namespace, e.g. to access Pod IP: `${data.kubernetes.pod.ip}`. -These are the fields available within config templating. The `kubernetes.*` fields will be available on each emitted event. +These are the fields available within config templating. The `kubernetes.*` fields will be available on each emitted event: [float] ====== Generic fields: @@ -221,18 +221,22 @@ These are the fields available within config templating. The `kubernetes.*` fiel * kubernetes.container.image * kubernetes.container.name * kubernetes.namespace + * kubernetes.namespace_uid * kubernetes.node.name * kubernetes.pod.name * kubernetes.pod.uid + * kubernetes.pod.ip [float] ====== Node specific: * kubernetes.node.name * kubernetes.node.uid + * kubernetes.node.labels [float] ====== Service specific: * kubernetes.namespace + * kubernetes.namespace_uid * kubernetes.service.name * kubernetes.service.uid * kubernetes.annotations diff --git a/x-pack/metricbeat/module/istio/proxy/_meta/docs.asciidoc b/x-pack/metricbeat/module/istio/proxy/_meta/docs.asciidoc index b3c5f3a8807..06ae69d97e0 100644 --- a/x-pack/metricbeat/module/istio/proxy/_meta/docs.asciidoc +++ b/x-pack/metricbeat/module/istio/proxy/_meta/docs.asciidoc @@ -29,5 +29,5 @@ metricbeat.autodiscover: config: - module: istio metricsets: ["proxy"] - hosts: "${data.host}:15090" + hosts: "${data.kubernetes.pod.ip}:15090" -------------------------------------------- From 00cb4502b48fdaddc1a5ef0d259c7a49595ef42e Mon Sep 17 00:00:00 2001 From: Tetiana Kravchenko Date: Wed, 10 Jan 2024 18:13:07 +0100 Subject: [PATCH 2/5] extend documentation with full list of fields; fix add_resource_metadata.deployment and add_resource_metadata.cronjob as the default value is false Signed-off-by: Tetiana Kravchenko --- libbeat/docs/shared-autodiscover.asciidoc | 176 +++++++++++++++++++--- 1 file changed, 153 insertions(+), 23 deletions(-) diff --git a/libbeat/docs/shared-autodiscover.asciidoc b/libbeat/docs/shared-autodiscover.asciidoc index 72735e8b206..e59b3d716d0 100644 --- a/libbeat/docs/shared-autodiscover.asciidoc +++ b/libbeat/docs/shared-autodiscover.asciidoc @@ -178,8 +178,8 @@ Configuration parameters: * `node` or `namespace`: Specify labels and annotations filters for the extra metadata coming from node and namespace. By default all labels are included while annotations are not. To change default behaviour `include_labels`, `exclude_labels` and `include_annotations` can be defined. Those settings are useful when storing labels and annotations that require special handling to avoid overloading the storage output. Note: wildcards are not supported for those settings. The enrichment of `node` or `namespace` metadata can be individually disabled by setting `enabled: false`. - * `deployment`: If resource is `pod` and it is created from a `deployment`, by default the deployment name is added, this can be disabled by setting `deployment: false`. - * `cronjob`: If resource is `pod` and it is created from a `cronjob`, by default the cronjob name is added, this can be disabled by setting `cronjob: false`. + * `deployment`: If resource is `pod` and it is created from a `deployment`, by default the deployment name isn't added, this can be enabled by setting `deployment: true`. + * `cronjob`: If resource is `pod` and it is created from a `cronjob`, by default the cronjob name isn't added, this can be enabled by setting `cronjob: true`. + Example: ["source","yaml",subs="attributes"] @@ -190,8 +190,8 @@ Example: node: include_labels: ["nodelabel2"] include_annotations: ["nodeannotation1"] - deployment: false - cronjob: false + deployment: true + cronjob: true ------------------------------------------------------------------------------------- `unique`:: (Optional) Defaults to `false`. Marking an autodiscover provider as unique results into @@ -212,34 +212,164 @@ These are the fields available within config templating. The `kubernetes.*` fiel ====== Generic fields: * host * port (if exposed) - * kubernetes.labels - * kubernetes.annotations [float] ====== Pod specific: - * kubernetes.container.id - * kubernetes.container.image - * kubernetes.container.name - * kubernetes.namespace - * kubernetes.namespace_uid - * kubernetes.node.name - * kubernetes.pod.name - * kubernetes.pod.uid - * kubernetes.pod.ip +|=== +|Key |Type |Description + +|`kubernetes.namespace` +|`string` +|Namespace, where the Pod is running + +|`kubernetes.namespace_uuid` +|`string` +|UUID of the Namespace, where the Pod is running + +|`kubernetes.namespace_labels.*` +|`object` +|Labels of the Namespace, where the Pod is running. All namespace labels are available by default, see `add_resource_metadata` configuration setting above. + +|`kubernetes.namespace_annotations.*` +|`object` +|Annotations of the Namespace, where the Pod is running. Namespace annotations are not available by default, see `add_resource_metadata` configuration setting above. + +|`kubernetes.pod.name` +|`string` +|Name of the Pod + +|`kubernetes.pod.uuid` +|`string` +|UUID of the Pod + +|`kubernetes.pod.ip` +|`string` +|IP of the Pod + +|`kubernetes.labels.*` +|`object` +|Object of the Pod labels. All labels are available by default. + +|`kubernetes.annotations.*` +|`object` +|Object of the Pod annotations. Annotations are not available by default. + +|`kubernetes.container.name` +|`string` +|Name of the container + +|`kubernetes.container.runtime` +|`string` +|Runtime of the container + +|`kubernetes.container.id` +|`string` +|ID of the container + +|`kubernetes.container.image` +|`string` +|Image of the container + +|`kubernetes.container.port` +|`string` +|Port of the container (if defined) + +|`kubernetes.container.port_name` +|`string` +|Port's name for the container (if defined) + +|`kubernetes.node.name` +|`string` +|Name of the Node + +|`kubernetes.node.uid` +|`string` +|UID of the Node + +|`kubernetes.node.hostname` +|`string` +|Hostname of the Node + +|`kubernetes.node.labels.*` +|`string` +|Labels of the Node. All node labels are available by default, see `add_resource_metadata` configuration setting above + +|`kubernetes.node.annotations.*` +|`string` +|Annotations of the Node. Node annotations are not available by default, see `add_resource_metadata` configuration setting above + +|`kubernetes.deployment.name.*` +|`string` +|Deployment name of the Pod (if exists). Deployment name is not available by default, see `add_resource_metadata` configuration setting above +|=== [float] ====== Node specific: - * kubernetes.node.name - * kubernetes.node.uid - * kubernetes.node.labels +|=== +|Key |Type |Description + +|`kubernetes.labels.*` +|`object` +|Object of labels of the Node. All labels are available by default + +|`kubernetes.annotations.*` +|`object` +|Object of labels of the Node. Annotations are not available by default + +|`kubernetes.node.name` +|`string` +|Name of the Node + +|`kubernetes.node.uid` +|`string` +|UID of the Node + +|`kubernetes.node.hostname` +|`string` +|Hostname of the Node +|=== [float] ====== Service specific: - * kubernetes.namespace - * kubernetes.namespace_uid - * kubernetes.service.name - * kubernetes.service.uid - * kubernetes.annotations +|=== +|Key |Type |Description + +|`kubernetes.namespace` +|`string` +|Namespace of the Service + +|`kubernetes.namespace_uuid` +|`string` +|UUID of the Namespace of the Service + +|`kubernetes.namespace_labels.*` +|`object` +|Labels of the Namespace of the Service. All namespace labels are available by default, see `add_resource_metadata` configuration setting above + +|`kubernetes.namespace_annotations.*` +|`object` +|Annotations of the Namespace of the Service. Namespace annotations are not available by default, see `add_resource_metadata` configuration setting above + +|`kubernetes.labels.*` +|`object` +|Object of the Service labels. All labels are available by default + +|`kubernetes.annotations.*` +|`object` +|Object of the Service annotations. Annotations are not available by default + +|`kubernetes.service.name` +|`string` +|Name of the Service + +|`kubernetes.service.uid` +|`string` +|UID of the Service + +|`kubernetes.selectors.*` +|`string` +|Kubernetes selectors +|=== If the `include_annotations` config is added to the provider config, then the list of annotations present in the config are added to the event. From 2fbf5457f2a3855a08a3d817cafa0b97d7cb9603 Mon Sep 17 00:00:00 2001 From: Tetiana Kravchenko Date: Mon, 15 Jan 2024 19:05:15 +0100 Subject: [PATCH 3/5] test all variables and adjust documentation accordingly Signed-off-by: Tetiana Kravchenko --- libbeat/docs/shared-autodiscover.asciidoc | 61 +++++++---------------- 1 file changed, 18 insertions(+), 43 deletions(-) diff --git a/libbeat/docs/shared-autodiscover.asciidoc b/libbeat/docs/shared-autodiscover.asciidoc index e59b3d716d0..9c4ddede5b0 100644 --- a/libbeat/docs/shared-autodiscover.asciidoc +++ b/libbeat/docs/shared-autodiscover.asciidoc @@ -211,13 +211,16 @@ These are the fields available within config templating. The `kubernetes.*` fiel [float] ====== Generic fields: * host - * port (if exposed) [float] ====== Pod specific: |=== |Key |Type |Description +|`port` +|`string` +|Pod port. If pod has multiple ports exposed should be used `ports.` instead + |`kubernetes.namespace` |`string` |Namespace, where the Pod is running @@ -226,21 +229,17 @@ These are the fields available within config templating. The `kubernetes.*` fiel |`string` |UUID of the Namespace, where the Pod is running -|`kubernetes.namespace_labels.*` -|`object` -|Labels of the Namespace, where the Pod is running. All namespace labels are available by default, see `add_resource_metadata` configuration setting above. - |`kubernetes.namespace_annotations.*` |`object` -|Annotations of the Namespace, where the Pod is running. Namespace annotations are not available by default, see `add_resource_metadata` configuration setting above. +|Annotations of the Namespace, where the Pod is running. Annotations should be used in not dedoted format, e.g. `kubernetes.namespace_annotations.app.kubernetes.io/name` |`kubernetes.pod.name` |`string` |Name of the Pod -|`kubernetes.pod.uuid` +|`kubernetes.pod.uid` |`string` -|UUID of the Pod +|UID of the Pod |`kubernetes.pod.ip` |`string` @@ -248,11 +247,11 @@ These are the fields available within config templating. The `kubernetes.*` fiel |`kubernetes.labels.*` |`object` -|Object of the Pod labels. All labels are available by default. +|Object of the Pod labels. Labels should be used in not dedoted format, e.g. `kubernetes.labels.app.kubernetes.io/name` |`kubernetes.annotations.*` |`object` -|Object of the Pod annotations. Annotations are not available by default. +|Object of the Pod annotations. Annotations should be used in not dedoted format, e.g. `kubernetes.annotations.test.io/test` |`kubernetes.container.name` |`string` @@ -270,14 +269,6 @@ These are the fields available within config templating. The `kubernetes.*` fiel |`string` |Image of the container -|`kubernetes.container.port` -|`string` -|Port of the container (if defined) - -|`kubernetes.container.port_name` -|`string` -|Port's name for the container (if defined) - |`kubernetes.node.name` |`string` |Name of the Node @@ -289,18 +280,6 @@ These are the fields available within config templating. The `kubernetes.*` fiel |`kubernetes.node.hostname` |`string` |Hostname of the Node - -|`kubernetes.node.labels.*` -|`string` -|Labels of the Node. All node labels are available by default, see `add_resource_metadata` configuration setting above - -|`kubernetes.node.annotations.*` -|`string` -|Annotations of the Node. Node annotations are not available by default, see `add_resource_metadata` configuration setting above - -|`kubernetes.deployment.name.*` -|`string` -|Deployment name of the Pod (if exists). Deployment name is not available by default, see `add_resource_metadata` configuration setting above |=== [float] @@ -310,11 +289,11 @@ These are the fields available within config templating. The `kubernetes.*` fiel |`kubernetes.labels.*` |`object` -|Object of labels of the Node. All labels are available by default +|Object of labels of the Node |`kubernetes.annotations.*` |`object` -|Object of labels of the Node. Annotations are not available by default +|Object of annotations of the Node |`kubernetes.node.name` |`string` @@ -334,6 +313,10 @@ These are the fields available within config templating. The `kubernetes.*` fiel |=== |Key |Type |Description +|`port` +|`string` +|Service port + |`kubernetes.namespace` |`string` |Namespace of the Service @@ -342,21 +325,17 @@ These are the fields available within config templating. The `kubernetes.*` fiel |`string` |UUID of the Namespace of the Service -|`kubernetes.namespace_labels.*` -|`object` -|Labels of the Namespace of the Service. All namespace labels are available by default, see `add_resource_metadata` configuration setting above - |`kubernetes.namespace_annotations.*` |`object` -|Annotations of the Namespace of the Service. Namespace annotations are not available by default, see `add_resource_metadata` configuration setting above +|Annotations of the Namespace of the Service. Annotations should be used in not dedoted format, e.g. `kubernetes.namespace_annotations.app.kubernetes.io/name` |`kubernetes.labels.*` |`object` -|Object of the Service labels. All labels are available by default +|Object of the Service labels |`kubernetes.annotations.*` |`object` -|Object of the Service annotations. Annotations are not available by default +|Object of the Service annotations |`kubernetes.service.name` |`string` @@ -365,10 +344,6 @@ These are the fields available within config templating. The `kubernetes.*` fiel |`kubernetes.service.uid` |`string` |UID of the Service - -|`kubernetes.selectors.*` -|`string` -|Kubernetes selectors |=== If the `include_annotations` config is added to the provider config, then the list of annotations present in the config From 6041d3f1038ad1b4d5d26988067b51fc643336cd Mon Sep 17 00:00:00 2001 From: Tetiana Kravchenko Date: Thu, 18 Jan 2024 13:00:56 +0100 Subject: [PATCH 4/5] Update libbeat/docs/shared-autodiscover.asciidoc Co-authored-by: Andrew Gizas --- libbeat/docs/shared-autodiscover.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libbeat/docs/shared-autodiscover.asciidoc b/libbeat/docs/shared-autodiscover.asciidoc index 9c4ddede5b0..a5d12f6227b 100644 --- a/libbeat/docs/shared-autodiscover.asciidoc +++ b/libbeat/docs/shared-autodiscover.asciidoc @@ -190,7 +190,7 @@ Example: node: include_labels: ["nodelabel2"] include_annotations: ["nodeannotation1"] - deployment: true + # deployment: false cronjob: true ------------------------------------------------------------------------------------- From e9371aa5af7b74b2c5279928cf0054ca10566e50 Mon Sep 17 00:00:00 2001 From: Tetiana Kravchenko Date: Thu, 18 Jan 2024 13:01:07 +0100 Subject: [PATCH 5/5] Update libbeat/docs/shared-autodiscover.asciidoc Co-authored-by: Andrew Gizas --- libbeat/docs/shared-autodiscover.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libbeat/docs/shared-autodiscover.asciidoc b/libbeat/docs/shared-autodiscover.asciidoc index a5d12f6227b..e27c055cc95 100644 --- a/libbeat/docs/shared-autodiscover.asciidoc +++ b/libbeat/docs/shared-autodiscover.asciidoc @@ -191,7 +191,7 @@ Example: include_labels: ["nodelabel2"] include_annotations: ["nodeannotation1"] # deployment: false - cronjob: true + # cronjob: false ------------------------------------------------------------------------------------- `unique`:: (Optional) Defaults to `false`. Marking an autodiscover provider as unique results into