Configure kubernetes metadata enrichment

[MichaelKatsoulis]

This PR enhances elastic agent managed by fleet documentation with information and configuration for kubernetes metadata enrichment.

Part of elastic/beats#37858

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[mergify]

This pull request does not have a backport label. Could you fix it @MichaelKatsoulis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-/d./d is the label to automatically backport to the /d./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[gizas]

Add a single line what kubernetes metadata enrichment is

Maybe a link here https://www.elastic.co/guide/en/observability/current/monitor-kubernetes.html#beats-metadata

[gizas]

Suggested change

	environment the kubernetes metadata are collected by the https://www.elastic.co/guide/en/beats/metricbeat/current/add-kubernetes-metadata.html[add_kubernetes_metadata processor]. The processor is configurable when {agent} is managed by {fleet}.
	environment the kubernetes metadata are collected by the <<add_kubernetes_metadata-processor>>. The processor is configurable when {agent} is managed by {fleet}.

[gizas]

Suggested change

	When it comes to container logs collection, the kubernetes autodiscover provider is used. It watches for pod resources
	When it comes to container logs collection, the <<kubernetes-provider>> is used. It watches for pod resources

[gizas]

Suggested change

	When it comes to container logs collection, the kubernetes autodiscover provider is used. It watches for pod resources
	When it comes to container logs collection, the kubernetes autodiscover provider is used. It watches for pod events

[gizas]

Suggested change

	in the cluster and associates each container log file under the log path provided with a pod's container object.
	in the cluster and associates each container log file with a corresponding pod container object.

[gizas]

Suggested change

	container this log file belongs to. The link is achieved through the container's ID which is part of the log file name.
	container this log file belongs to. The linkage is established by the container's ID, which forms an integral part of the filename for the log.

[gizas]

Suggested change

	In order to configure the metadata collection, the kubernetes provider needs to be configured.
	In more details:
	* For **Standalone {agent}**
	Follow information of `add_resource_metadata` parameter of <<kubernetes-provider>>
	[source,yaml]
	.Example how to configure kubernetes metadata enrichment
	------------------------------------------------
	apiVersion: v1
	kind: ConfigMap
	metadata:
	name: agent-node-datastreams
	namespace: kube-system
	labels:
	k8s-app: elastic-agent
	data:
	agent.yml: |-
	kubernetes.provider
	add_resource_metadata:
	namespace:
	#use_regex_include: false
	include_labels: ["namespacelabel1"]
	#use_regex_exclude: false
	#exclude_labels: ["namespacelabel2"]
	node:
	#use_regex_include: false
	include_labels: ["nodelabel2"]
	include_annotations: ["nodeannotation1"]
	#use_regex_exclude: false
	#exclude_labels: ["nodelabel3"]
	#deployment: false
	#cronjob: false
	---------------------------------------------------
	* For **Managed {agent}**:
	See <<advanced-kubernetes-managed-by-fleet>>

[gizas]

NOTE: This document focuses on {agent} configuration solely. In case beats metadata enrichment for kubernetes needed advice relevant filebeat and metricbeat links

[MichaelKatsoulis]

after the beats prs

[gizas]

I would delete this line

[gizas]

Suggested change

	The kubernetes metricsets use the so called kubernetes metadata enrichers mechanism. What this does is that the different metricsets share a set of resource watchers. Those watchers(pod, node, namespace, deployment, daemonset etc.) are responsible for watching for all the different resources creation, update and deletion by subscribing to kubernetes watch API.
	The {agent} metrics collection implements metadata enrichment based on watchers, a mechanisms used in Kubernetes to observe changes to resources in the Kubernetes cluster.
	In more details, the different datasets share a set of resource watchers. Those watchers(pod, node, namespace, deployment, daemonset etc.) are responsible for watching for all different resource events ( creation, update and deletion) by subscribing to kubernetes watch API. This enables real-time synchronization of application state with the state of the Kubernetes cluster

[gizas]

Suggested change

	So they keep in an up to date shared store all the resource informations and metadata. Whenever metrics are collected by the different sources (kubelet, kube-state-metrics), before they get published to ES as events, they get enriched with needed metadata.
	So they keep in an up to date shared cache store for all the resource informations and metadata. Whenever metrics are collected by the different sources (kubelet, kube-state-metrics), before they get published to {es} as events, they get enriched with needed metadata.

[gizas]

Suggested change

	node, namespace labels and annotations, as well as deployment and cronjob information can be configured per metricset.
	node, namespace labels and annotations, as well as deployment and cronjob information can be configured per dataset.

[gizas]

NOTE: add_resource_metadata block needs to be configured to all datasets that are enabled

[gizas]

Suggested change

	It can be disabled by switching off the `Add Metadata` toggle in every metricset. Exrta resource metadata like
	**Only in metrics collection**, metadata enrichement can be disabled by switching off the `Add Metadata` toggle in every dataset. Exrta resource metadata like

[gizas]

Suggested change

	- For **Managed {agent}**:

[gizas]

Suggested change

	- For **Standalone {agent}**:
	[source,yaml]
	.Elastic Agent Standalone manifest sample
	------------------------------------------------
	[output trunctated ...]
	- data_stream:
	dataset: kubernetes.state_pod
	type: metrics
	metricsets:
	- state_pod
	add_metadata: true
	hosts:
	- 'kube-state-metrics:8080'
	period: 10s
	add_resource_metadata:
	namespace:
	enabled: true
	#use_regex_include: false
	include_labels: ["namespacelabel1"]
	#use_regex_exclude: false
	#exclude_labels: ["namespacelabel2"]
	node:
	enabled: true
	#use_regex_include: false
	include_labels: ["nodelabel2"]
	include_annotations: ["nodeannotation1"]
	#use_regex_exclude: false
	#exclude_labels: ["nodelabel3"]
	#deployment: false
	#cronjob: false
	---------------------------------------------------
	The `add_resource_metadata` block configures the watchers enrichment functionality. See <<kubernetes-provider>> for full description of add_resource_metadata. Same configuration parameters apply.

[kilfoyle]

This looks great @MichaelKatsoulis! I hope you don't mind a bunch of suggestions.

[MichaelKatsoulis]

This looks great @MichaelKatsoulis! I hope you don't mind a bunch of suggestions.

@kilfoyle Thanks a lot! I updated the documentation with your suggestions!

[kilfoyle]

Thanks a lot for adding this @MichaelKatsoulis! The docs are really nice. 👍

I added just a few more super minor suggestions.

I'm not sure if you'll be able to merge in this repo (due to our newly heightened security settings), so whenever it's ready to merge just let me know.

Also, if this should appear in the current 8.13 docs let me know and I'll make sure it backports succcessfully.

[MichaelKatsoulis]

@kilfoyle I think it is ready now. Yes the documentation applies also for 8.13. I added the backport label. I believe I can merge it myself, the merge button is green !

[kilfoyle]

Sounds good Michael! And thanks for letting me know about the merge button. Other people have had problems so I'm really glad to know it's not universal. :-)