feat(processors.batch): Add batch processor

[LarsStegman]

Summary

This new processor can distribute metrics across batches by adding a tag indicating what batch number it is in. This makes it possible to distribute the load of a high number of metrics across multiple instances of the same output plugin.

Checklist

• No AI generated code was used in this PR

Related issues

resolves #15621
resolves #11707

[LarsStegman]

@srebhan the implementation is a little different than what you suggested. I did not see the added benefit of also specifying the batch size, since any overflow would probably just overflow into the next batch, which then also overflows. Please let me know if it should still be added.

[srebhan]

Thanks @LarsStegman for the contribution! Just two small comments from my side. Furthermore, should we also add a force_rebatch option that will only overwrite the batch tag if it does not already exists? I'm asking because in the current default, Telegraf will run each processor twice, once before and once after aggregators if any.

[LarsStegman]

should we also add a force_rebatch option that will only overwrite the batch tag if it does not already exists? I'm asking because in the current default, Telegraf will run each processor twice, once before and once after aggregators if any.

Hmmm interesting. The results after the second pass will indeed be different, because the processor will already have run a pass and the count will have increased. I think it is better to add that feature indeed. It will be more predictable for users.

[LarsStegman]

I made the rebatching enabled by default, because it is less computational load. By default it will now not check the existing tags.

[srebhan]

Two more comments. Regarding the flag, I'm fine either way but slightly tend to your approach...

[srebhan]

@LarsStegman awesome! Maybe just avoid abbreviations in config options? How about naming this just batches?

[LarsStegman]

@srebhan looks like the test runner timed out or something

[srebhan]

Thanks @LarsStegman!

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

🥳 This pull request decreases the Telegraf binary size by -8.01 % for linux amd64 (new size: 239.8 MB, nightly size 260.6 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz