VRL: coerce string into array

[ADovgalyuk]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Hello everyone!
I have a csv file where some fields can be either empty or array. For example:

2022-12-23 13:01:16;111;['Main','Rule','Velocity'];['routerprotocol','rule','rule'];[2,1,0];[0,0,0];'','VelocityCheck',''];0;;3;-1.00;174;-1;tr-4;13;2;

I'm using function parse_csv that results in array with strings inside, but array fields from csv are also doublequoted as strings.
Code:

arr = parse_csv!(.message, delimiter: ";")
        .=compact(
            {
                "host" : "Test",
                "app" : "Test",
                "dateTime" : {{ arr[0]}},
                "Id" : {{ arr[1]}},
                "OneFieldWithArr" : {{ arr[2]}},
                "AnotherFieldWithArr" : {{ arr[3]}},
....

Result

{"app":"BlaBlaApp","dateTime":"2022-12-23 13:01:16","host":"HostExample","OneFieldWithArr":"['Main','Rule','Velocity']", "AnotherFieldWithArr":"['routerprotocol','rule','rule']" ......}

I want to coerce value back to Array from String, because they have incorrect type after VRL transforms.
Thanks for the help in advance!

Attempted Solutions

I do understand that I can do it via Lua script or with additional string manipulations, but still it's better and more "clean" to do it as additional function.

Proposal

No response

References

Based on discussion

Version

0.40.0

[jorgehermo9]

I think the best approach here should be that the arrays are json-compatible ['Main','Rule','Velocity'] -> ["Main","Rule","Velocity"] and then use the parse_json and array vrl functions on those strings (array!(parse_json!(string))).

If you can't change the origin format of those arrays, maybe you can preprocess the string value with replace(key, "'", "\"")

Working example: https://playground.vrl.dev/?state=eyJwcm9ncmFtIjoiYXJyID0gcGFyc2VfY3N2ISgubWVzc2FnZSwgZGVsaW1pdGVyOiBcIjtcIilcbmZpcnN0X2FyciA9IGFyclsyXVxucHJlcHJvY2Vzc2VkID0gcmVwbGFjZSEoZmlyc3RfYXJyLFwiJ1wiLFwiXFxcIlwiKVxuLnBhcnNlZF9hcnJheSA9IGFycmF5IShwYXJzZV9qc29uIShwcmVwcm9jZXNzZWQpKVxuIiwiZXZlbnQiOnsibWVzc2FnZSI6IjIwMjItMTItMjMgMTM6MDE6MTY7MTExO1snTWFpbicsJ1J1bGUnLCdWZWxvY2l0eSddO1sncm91dGVycHJvdG9jb2wnLCdydWxlJywncnVsZSddO1syLDEsMF07WzAsMCwwXTsnJywnVmVsb2NpdHlDaGVjaycsJyddOzA7OzM7LTEuMDA7MTc0Oy0xO3RyLTQ7MTM7MjsifSwiaXNfanNvbmwiOmZhbHNlLCJlcnJvciI6bnVsbH0%3D

[jszwedko]

Thanks @jorgehermo9 . I agree that that seems like a good approach. I'll convert this to a discussion since it isn't really an bug report or feature request (at least not yet).