s3-to-dc glob error

[kieranricardo]

s3-to-dc can't parse glob urls like s3://dea-public-data/fractional-cover/fc-percentile/annual/v2.2.0/combined/**/2020/*.yaml. It seems like specify a subfolder after the * or ** patterns causes parse_query to throw:

ValueError: Bad query: ...

[Kirill888]

not supported currently, /**/ stands for any number of sub-folders, and only allow this to be followed by "file pattern", so ../**/<file-pattern>.

I feel we should probably just support arbitrary grep patterns instead, but then you still need to list all the paths (slow) and then discard those that do not match.