Moving a message to DLQ during handling / postProcessing?

[Mask]

Hello,

I have a queue, set to retries=6. This works well, and I see messages which fail (due to an unhandled exception) being retried as expected, then placed in the DLQ.

What I would like to do: for some exceptions, immediately place the message in the DLQ with no further retries.

I tried to simply set job.rqueueMessage.retryCount to 0 in a middleware class, and this does have the effect of putting the message in the DLQ, but it is additionally puts it into scheduled, and retrys again and again, even exceeding the queue's retry limit of 6.

So I must be missing something. Any tips on what else I can try?

Update: I figured out why this approach is failing: moveMessageToDlq doesn't delete the message from the processing queue, since I changed it and the bytes[] don't match.

trying to find a different approach. And ideas?

[sonus21]

I think this is not straight forward, we have to make some change how do we propagate and handle errors.

[sonus21]

@Mask which version of Rqueue you're using? I'm working on a fix, that could solve your use case. see #231

[Mask]

Wow, that would be great. Using 3.1.1-RELEASE right now.

[Mask]

added some comments to the PR

[sonus21]

Please check updated PR, I've merged it for doc update. If you find this is working fine, I;ll make a release

[Mask]

Something is not quite right with the ExecutionStatus - depending on whether rqueue.retry.per.poll has been increased, or left at the default of 1, the ExecutionStatus is either FAILED or FAILED_IGNORED. Not sure why yet - 'll look into this tomorrow and get back with a proper analysis. May not be a problem though - in PostProcessing it lands in handleFailedIgnoredMessage, although through different code-paths (directly through the switch in handle, or through the initial if in handleFailure)

Question about handleFailedIgnoredMessage: why discard the message and not place it in the DLQ? My use-case (and I think it is not that unusual) is a typical "poison pill" message - some message which, for whatever reason, simply can not be handled (maybe it can't be parsed, maybe some ENUM value is unknown, etc). Retrying makes no sense, the poison pill will still be poison no matter how often we retry, but it would be nice to have them collected in the DLQ for further examination.

[sonus21]

I see, I was in impression that we don't want to store in DLQ too, but we can store them in DLQ and process it later.

Check master branch once and test it in local, if it works well. I;ll make a release

[Mask]

I can confirm that the master branch behaves as I described above:

1. ExecutionStatus is either FAILED or FAILED_IGNORED, depending on rqueue.retry.per.poll > 1
2. Message is discarded, not placed in DLQ

(see correction below)

[sonus21]

@Mask I had made the change in the morning based on your feedback, I've removed the FAILED_IGNORED and messages would be sent to DLQ.

Please do a local testing and let me know if you need snapshot version

[Mask]

Didn't install your last commit correctly - now I have it.

Yes! This works perfectly now.

I see you also added the throwable to TaskExecutionBackOff - nice touch - this gives us even more flexibility to dynamically decide when and how often to retry - I like it!

[sonus21]

A new version has been released, 3.2.0.