Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LOG-5931: enable stream (stdout/stderr) information in logs for Vector #2736

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

LOG-5931: enable stream (stdout/stderr) information in logs for Vector #2736

wants to merge 1 commit into from

Conversation

vparfonov
Copy link
Contributor

@vparfonov vparfonov commented Aug 7, 2024 •
edited
Loading

Description

This PR introduces handling of kubernetes_log stream field (e.g., stdout, stderr). https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/#line-fields

  • moved the original stream field to kubernetes.stream to better align with the log event structure
  • added kubernetes.stream to the group_by parameter of the Detect Multiline Exceptions filter, ensures that exception detection is correctly handled
  • included kubernetes.stream in the group_by parameter for the OTLP output, to improves the granularity of log data sent through OTLP by differentiating between different streams.
  • modified the functional tests to verify that log events are handled correctly for both stdout and stderr outputs.

/cc @cahartma @Clee2691
/assign @jcantrill

/cherry-pick

Links

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 7, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 7, 2024 •
edited by openshift-ci bot
Loading

@vparfonov: This pull request references LOG-5931 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.8.0" version, but no target version was set.

In response to this:

Description

/cc
/assign

/cherry-pick

Links

  • Depending on PR(s):
  • Bugzilla:
  • Github issue:
  • JIRA:
  • Enhancement proposal:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 7, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 7, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 7, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vparfonov
Once this PR has been reviewed and has the lgtm label, please assign jcantrill for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@vparfonov vparfonov changed the title LOG-5931: keep original stream field in log event WIP:LOG-5931: keep original stream field in log event Aug 7, 2024
cahartma
cahartma reviewed Aug 8, 2024
View reviewed changes
internal/generator/vector/output/otlp/transform.go
@@ -28,7 +28,8 @@ resource.attributes = append( resource.attributes,
resource.attributes = append( resource.attributes,
[{"key": "k8s.pod.name", "value": {"stringValue": get!(.,["kubernetes","pod_name"])}},
{"key": "k8s.container.name", "value": {"stringValue": get!(.,["kubernetes","container_name"])}},
{"key": "k8s.namespace.name", "value": {"stringValue": get!(.,["kubernetes","namespace_name"])}}]
{"key": "k8s.namespace.name", "value": {"stringValue": get!(.,["kubernetes","namespace_name"])}},
{"key": "k8s.stream", "value": {"stringValue": get!(.,["kubernetes","stream"])}}]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For OTLP output, if you are adding to the resource attributes, then it also needs to be included in the group_by for container logs. https://github.com/openshift/cluster-logging-operator/blob/master/internal/generator/vector/output/otlp/group_by.go#L39

@vparfonov vparfonov force-pushed the log5931 branch 2 times, most recently from 7daadae to b8e813b Compare August 8, 2024 14:53
@vparfonov
Copy link
Contributor Author

/test functional-target

@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 8, 2024 •
edited by openshift-ci bot
Loading

@vparfonov: This pull request references LOG-5931 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.8.0" version, but no target version was set.

In response to this:

Description

This PR introduces handling of kubernetes_log stream field (e.g., stdout, stderr). https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/#line-fields

  • moved the original stream field to kubernetes.stream to better align with the log event structure
  • added kubernetes.stream to the group_by parameter of the Detect Multiline Exceptions filter, ensures that exception detection is correctly handled
  • included kubernetes.stream in the group_by parameter for the OTLP output, to improves the granularity of log data sent through OTLP by differentiating between different streams.
  • modified the functional tests to verify that log events are handled correctly for both stdout and stderr outputs.

/cc @cahartma @Clee2691
/assign @jcantrill

/cherry-pick

Links

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@vparfonov vparfonov changed the title WIP:LOG-5931: keep original stream field in log event LOG-5931: enable stream (stdout/stderr) information in logs for Vector Aug 8, 2024
@vparfonov vparfonov marked this pull request as ready for review August 8, 2024 16:41
@vparfonov
Copy link
Contributor Author

/hold

@openshift-ci openshift-ci bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Aug 8, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 8, 2024

@vparfonov: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@vparfonov vparfonov requested a review from cahartma August 9, 2024 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Clee2691 Clee2691 Awaiting requested review from Clee2691

@jcantrill jcantrill Awaiting requested review from jcantrill

@cahartma cahartma Awaiting requested review from cahartma

Assignees
No one assigned
Labels
do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. release/6.1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@vparfonov @openshift-ci-robot @cahartma @jcantrill