batcher: major error handling overhaul

[abraithwaite]

We previously were trying to handle errors either in Run() and previous to that: Send(). Unfortunately neither are great.

First when we tried send, the issue is that because batcher is async, the errors returned would have no semantic relation to the messages passed into the Send function call that just occurred. It was a way to pass errors back to the processor, but it leads to error handling which is error prone.

Next, we tried returning it from Run, but similarly the caller of Run has no context as to what it should do with the error.

Now, we're making error handling more explicit by requiring a handler get passed into the batcher. The signature allows for fairly flexible error handling and includes the batch that failed to flush.

Returning an error from this new function will be an indicator to the batcher to stop processing and return the error to Run(), mimicking the current functionality, but with a lot more clarity and explicitness.

No error returned from the error handler means that the events were processed in a manner appropriate to the error, whether that be dropped, archived for later, or retried.

Be wary about retrying at this stage. If you can rely on the source redelivering the message, better that than retrying here and putting backpressure on the queue. Also better to archive or send to a deadletter queue for retries.