Workflow withItems fails if 1 of the actions fail

[glafkosch]

I am making dynamically generated worker requests with make_worker_requests task and passing those to another task run_linked_workers which runs for each item the action tf.run-worker-and-store-result.

The run_linked_workers task runs various worker checks for my project such as sslcheck, webcheck, dbcheck, etc. Everything works well if all return success but if even one check fails which is normal to fail in my case then the whole task fail.

As per documentation: "All action executions must complete successfully for the task to reach a succeeded state. If one or more action executions fail, then the task will result in a failed state."
ref: https://docs.stackstorm.com/orquesta/languages/orquesta.html#with-items-model

Is there any solution to solve this?

make_worker_requests:
action: tf.make_worker_requests
input:
target: <% ctx().target %>
target_type: <% ctx().target_type %>
services: "{{ ctx().services }}"
step: <% ctx().step %>
scan_id: <% ctx().scan_id %>
next:
- when: <% succeeded() %>
publish:
- portscan_linked_worker_requests: <% result().result %>
do: run_linked_workers
- when: <% failed() %>
publish:
- errors: <% ctx().errors + [dict(summary=>"Unable to make worker requests.", detail=>result())] %>
do:
- fail

run_linked_workers:
with:
items: <% ctx().portscan_linked_worker_requests %>
action: tf.run-worker-and-store-result
input:
worker_request: <% item() %>
next:
- when: <% succeeded() %>
publish:
- lwo: <% result() %>
- when: <% failed() %>
publish:
- errors: <% ctx().errors + [dict(summary=>"Unable to run linked workers.", detail=>result())] %>
do:
- fail

[mrpaws]

From what I've read (see below), the task transition model (i.e. fail,continue,noop,retry) and Orquesta enginer do not support when using with-items without failing the workflow.

This to me seems a major inconsistency that compromises the Orquesta engine a bit, but there are a couple dirty workarounds. All of them are variations of dirty because effectively, you make the action not fail but track its state. In my use case, a secondary workflow is called with-items, which has several actions, each of which can fail individually.

The workaround that works for me is to use a var like "workflow_state='success'" and create an action at the join within the secondary workflow that assesses that variable. The actions in the second workflow can fail individually since they are concurrent, but at the join of those actions, the workflow_state is assessed, like this:

  finish:
    join: all
    action: core.echo message="Assessing success of workflow for item <% ctx(item_id) %>"
    next:
      - when: <% succeeded() and (regex_match(ctx(workflow_state), 'success') = true) %>
         do: (handle success)
      - when: <% succeeded() and (regex_match(ctx(workflow_state), 'fail') = true) %>
         do: (handle fail)

So an action does this when it fails in the parallel workflow:

      - when: <% failed() %>
        publish:
           - workflow_state: 'fail'

It's only so useful because if the actions are built according to how the engine is supposed to work, they will fail or succeed. This effectively circumvents the failure like a 'noop' would.

it may not even work in your case without changing the success criteria of the action. In your case here, you would need "tf.run-worker-and-store-result" to essentially never produce a <% failed() %>, and instead update the workflow_state variable like I mentioned, and assess it later.

Like I said, dirty workaround. And to be sure, you lose all the benefits of error handling with Orquesta flow control and St2 as a whole. And it won't show up in your portal as a failure. In my case, the indivudal job still fails and an alert/incident is created in the log and the field of a tracking spreadsheet is updated.

Further context:

I found a response to a similar question on the old board from 2019 on another board, and want to share my solution and also suggest this be addressed by a St2 developer. (https://forum.stackstorm.com/t/orquesta-workflow-should-continue-if-with-item-step-fails/1586.html)

The response on why this happens from there:
By virtue of the workflow engine works, single failure at any given step will bring the workflow to stop. The only way to handle or continue is to add handler for failure by using when <% failed() %>. Though, this is not feasible in the with-items at the time of writing. The entire loop is finished and then only the flow proceeds to the next step. If possible, you might want to consider making this call from python-script with loop or shell-script making curl calls where you have handle on each request and continue the iteration while logging cases that fail. Hope that helps.

===
I hope this helps and hope someone else comes and tells me a better way! :)
mp