CrowdStrike Logs Multiline Json Not Working Properly

[navein-kumar]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hi Team,
I am attempting to integrate CrowdStrike SIEM logs into Elasticsearch using Vector. I am encountering issues with transforming multilined JSON format logs. I have attached an error screenshot and configuration details for reference. Please assist.

Configuration

sources:
  apilogs:
    type: "file"
    include:
      - "./test.json"
    multiline:
       start_pattern: ".*"
       condition_pattern: "}$"
       mode: "halt_with"
       timeout_ms: 1000
transforms:
  remap_syslog:
    inputs: ["apilogs"]
    type: "remap"
    source: |
      .message  = parse_json!(.message)
      .timestamp = now()
sinks:
  print:
    type: "console"
    inputs:
      - "remap_syslog"
    encoding:
      codec: "json"
  my_sink_id:
    type: elasticsearch
    inputs:
      - "remap_syslog"
    endpoints: ["https://11.11.1.1:9200"]
    auth:
      user: "111111"
      password: "1111111"
      strategy: "basic"
    tls:
      verify_certificate: false

Version

vector 0.39.0 (x86_64-unknown-linux-gnu 73da9bb 2024-06-17 16:00:23.791735272)

Debug Output

2024-07-26T02:19:12.411635Z ERROR transform{component_kind="transform" component_id=remap_syslog component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (12:33): unable to parse json: EOF while parsing a list at line 36 column 6" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true
2024-07-26T02:19:12.411741Z ERROR transform{component_kind="transform" component_id=remap_syslog component_type=remap}: vector::internal_events::remap: Internal log [Mapping failed with event.] is being suppressed to avoid flooding.
{"file":"test.json","host":"vmi1767413.contaboserver.net","message":"{\n  \"metadata\":{\n   \"customerIDString\":\"956f887xxxxxx\",\n   \"offset\":6068,\n   \"eventType\":\"AuthActivityAuditEvent\",\n   \"eventCreationTime\":1656862026109,\n   \"version\":\"1.0\"\n  },\n  \"event\":{\n   \"UserId\":\"api-client-id:24c6e2395axxxxxxx\",\n   \"UserIp\":\"101.0.60.100\",\n   \"OperationName\":\"streamStarted\",\n   \"ServiceName\":\"Crowdstrike Streaming API\",\n   \"Success\":true,\n   \"UTCTimestamp\":1656862026,\n   \"AuditKeyValues\":[\n     {\n      \"Key\":\"APIClientID\",\n      \"ValueString\":\"24c6e2395a0c4a2eb2xxxxxxx\"\n     },\n     {\n      \"Key\":\"partition\",\n      \"ValueString\":\"0\"\n     },\n     {\n      \"Key\":\"offset\",\n      \"ValueString\":\"6067\"\n     },\n     {\n      \"Key\":\"appId\",\n      \"ValueString\":\"siem-connect\"\n     },\n     {\n      \"Key\":\"eventType\",\n      \"ValueString\":\"[MobileDetectionSummaryEvent LoginAuditEvent DetectionSummaryEvent FirewallMatchEvent ReconNotificationSummaryEvent IdentityProtectionEvent ScheduledReportNotificationEvent IncidentSummaryEvent CSPMIOAStreamingEvent CustomerIOCEvent XdrDetectionSummaryEvent RemoteResponseSessionEndEvent AuthActivityAuditEvent RemoteResponseSessionStartEvent HashSpreadingEvent CSPMSearchStreamingEvent UserActivityAuditEvent]\"\n     }","source_type":"file","timestamp":"2024-07-26T02:19:12.411318553Z"}
{"file":"test.json","host":"vmi1767413.contaboserver.net","message":"   ]\n  }","source_type":"file","timestamp":"2024-07-26T02:19:12.411356353Z"}
{"file":"test.json","host":"vmi1767413.contaboserver.net","message":"}","source_type":"file","timestamp":"2024-07-26T02:19:13.412922522Z"}
2024-07-26T02:19:13.602466Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=2}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_200" response=Response { status: 200, version: HTTP/1.1, headers: {"content-type": "application/json; charset=UTF-8", "content-length": "506"}, body: b"{\"took\":1,\"errors\":true,\"items\":[{\"index\":{\"_index\":\"vector-2024.07.26\",\"_id\":\"J4nV7JABgEIWWdEvzpvy\",\"status\":400,\"error\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"object mapping for [message] tried to parse field [message] as object, but found a concrete value\"}}},{\"index\":{\"_index\":\"vector-2024.07.26\",\"_id\":\"KInV7JABgEIWWdEvzpvy\",\"status\":400,\"error\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"object mapping for [message] tried to parse field [message] as object, but found a concrete value\"}}}]}" }
2024-07-26T02:19:13.602711Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=2}: vector::sinks::util::retries: Not retriable; dropping the request. reason="error type: mapper_parsing_exception, reason: object mapping for [message] tried to parse field [message] as object, but found a concrete value" internal_log_rate_limit=true
2024-07-26T02:19:13.602738Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=2}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=2 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2024-07-26T02:19:13.602766Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=2}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=2 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2024-07-26T02:19:14.602308Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=3}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_200" response=Response { status: 200, version: HTTP/1.1, headers: {"content-type": "application/json; charset=UTF-8", "content-length": "270"}, body: b"{\"took\":1,\"errors\":true,\"items\":[{\"index\":{\"_index\":\"vector-2024.07.26\",\"_id\":\"KYnV7JABgEIWWdEv0pva\",\"status\":400,\"error\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"object mapping for [message] tried to parse field [message] as object, but found a concrete value\"}}}]}" }
2024-07-26T02:19:14.602442Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=3}: vector::sinks::util::retries: Internal log [Not retriable; dropping the request.] is being suppressed to avoid flooding.
2024-07-26T02:19:14.602472Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=3}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.
2024-07-26T02:19:14.602495Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=3}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

Example Data

{
"metadata":{
"customerIDString":"956f887xxxxxx",
"offset":6068,
"eventType":"AuthActivityAuditEvent",
"eventCreationTime":1656862026109,
"version":"1.0"
},
"event":{
"UserId":"api-client-id:24c6e2395axxxxxxx",
"UserIp":"101.0.60.100",
"OperationName":"streamStarted",
"ServiceName":"Crowdstrike Streaming API",
"Success":true,
"UTCTimestamp":1656862026,
"AuditKeyValues":[
{
"Key":"APIClientID",
"ValueString":"24c6e2395a0c4a2eb2xxxxxxx"
},
{
"Key":"partition",
"ValueString":"0"
},
{
"Key":"offset",
"ValueString":"6067"
},
{
"Key":"appId",
"ValueString":"siem-connect"
},
{
"Key":"eventType",
"ValueString":"[MobileDetectionSummaryEvent LoginAuditEvent DetectionSummaryEvent FirewallMatchEvent ReconNotificationSummaryEvent IdentityProtectionEvent ScheduledReportNotificationEvent IncidentSummaryEvent CSPMIOAStreamingEvent CustomerIOCEvent XdrDetectionSummaryEvent RemoteResponseSessionEndEvent AuthActivityAuditEvent RemoteResponseSessionStartEvent HashSpreadingEvent CSPMSearchStreamingEvent UserActivityAuditEvent]"
}
]
}
}

Additional Context

No response

References

No response

[jszwedko]

Hi @navein-kumar ,

The issue here is that your pattern to end the multi-line grouping }$ will match many lines in the JSON object you have there. Is it possible to configure CloudStrike to write the JSON objects as a single line?

Otherwise, you could try a pattern like ^} which should match }s that are at the beginning of the line. If the objects are pretty-printed like you have here that should match the } that ends the object.

[jszwedko]

I converted this to a discussion since I don't think there is a bug here, just a misconfiguration.

[navein-kumar]

Thanks for quick response.

1. I did check with crowdstrike they didn't have options to change format in console.
2. I have changed config as per your recommended, its taking as single new line per value, Pls refer the screenshot.

And is there any remap options to find and merge multi-line JSON into sing-line JSON format.

[jszwedko]

Try:

start_pattern: "^{"
mode: "continue_through"
condition_pattern: "^}"

[navein-kumar]

I have tried the above config and facing regex error " error: unclosed counted repetition"
Pls refer the attached error screenshot and config.

[jszwedko]

Ah, you need to escape it: ^\{ and ^\}.

[navein-kumar]

I tested this config and it takes as new line, pls refer screenshot

[jszwedko]

Hmm, that's odd since there is leading whitespace that should have caused the pattern not to be matched. Could you share the config you are trying now?

[navein-kumar]

Pls find below config details.

[jszwedko]

Thanks! It seems like there might be a bug here where it is ignoring whitespace when matching.

I was able to get it working using a reduce transform instead of the multiline configuration on the file source. See:

data_dir: /tmp/vector
sources:
  source0:
    type: file
    include:
    - /tmp/bar/*
transforms:
  transform0:
    type: reduce
    inputs:
      - source0
    group_by:
      - file
    merge_strategies:
      message: concat_newline
    ends_when: match(string!(.message), r'^\}')
sinks:
  sink0:
    inputs:
    - transform0
    target: stdout
    type: console
    encoding:
      codec: json

[navein-kumar]

the config works fine for now. thanks for quick support.