Failing logic calls continually retried, potentially preventing future logic calls from being relayed

[cbrit]

While working on steward/cork integration testing, I ran into a situation where the logic call to the cellar contract was failing due to a gas estimation failure which itself was likely due to a malformed method signature or signatures in the call. In the Orchestrators this resulted in in the following logs:

// -- snip
2022-02-14T22:40:38.554763Z  WARN relayer::logic_call_relaying: LogicCall a68edf806c550caa08f6d8d6c9d96b9b4bdf040e1137d8e13e10ffa04e9ced7c/1 can not be submitted yet, waiting for more signatures
2022-02-14T22:40:40.383940Z  INFO orchestrator::main_loop: No unsigned batches! Everything good!
2022-02-14T22:40:40.386423Z  INFO orchestrator::main_loop: Sending Logic call confirm for a68edf806c550caa08f6d8d6c9d96b9b4bdf040e1137d8e13e10ffa04e9ced7c:1
2022-02-14T22:40:51.403563Z  INFO orchestrator::main_loop: No unsigned batches! Everything good!
2022-02-14T22:40:55.542586Z  INFO relayer::valset_relaying: New valset hash b4687de48b64aab4a8942801e36d540339ff25cd8ca7c884388d81b418308e54
2022-02-14T22:40:55.547103Z  INFO relayer::valset_relaying: Consideration: looks good
2022-02-14T22:40:55.547148Z  INFO relayer::valset_relaying: Considering nonces: latest_cosmos_valset.nonce 1 current_eth_valset.nonce 1 should_relay false
2022-02-14T22:40:55.673287Z ERROR relayer::logic_call_relaying: LogicCall cost estimate failed with Err(EthersContractError(MiddlewareError(MiddlewareError(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32603, message: "Error: VM Exception while processing transaction: reverted with reason string 'Address: low-level call failed'", data: None }))))))
2022-02-14T22:41:02.398809Z  INFO orchestrator::main_loop: No unsigned batches! Everything good!
2022-02-14T22:41:12.678382Z  INFO relayer::valset_relaying: New valset hash b4687de48b64aab4a8942801e36d540339ff25cd8ca7c884388d81b418308e54
2022-02-14T22:41:12.683768Z  INFO relayer::valset_relaying: Consideration: looks good
2022-02-14T22:41:12.683836Z  INFO relayer::valset_relaying: Considering nonces: latest_cosmos_valset.nonce 1 current_eth_valset.nonce 1 should_relay false
2022-02-14T22:41:12.801884Z ERROR relayer::logic_call_relaying: LogicCall cost estimate failed with Err(EthersContractError(MiddlewareError(MiddlewareError(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32603, message: "Error: VM Exception while processing transaction: reverted with reason string 'Address: low-level call failed'", data: None }))))))
2022-02-14T22:41:13.399651Z  INFO orchestrator::main_loop: No unsigned batches! Everything good!
2022-02-14T22:41:24.400763Z  INFO orchestrator::main_loop: No unsigned batches! Everything good!
2022-02-14T22:41:29.653273Z  INFO relayer::valset_relaying: New valset hash b4687de48b64aab4a8942801e36d540339ff25cd8ca7c884388d81b418308e54
2022-02-14T22:41:29.658331Z  INFO relayer::valset_relaying: Consideration: looks good
2022-02-14T22:41:29.658417Z  INFO relayer::valset_relaying: Considering nonces: latest_cosmos_valset.nonce 1 current_eth_valset.nonce 1 should_relay false
2022-02-14T22:41:29.907001Z ERROR relayer::logic_call_relaying: LogicCall cost estimate failed with Err(EthersContractError(MiddlewareError(MiddlewareError(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32603, message: "Error: VM Exception while processing transaction: reverted with reason string 'Address: low-level call failed'", data: None }))))))
2022-02-14T22:41:35.398668Z  INFO orchestrator::main_loop: No unsigned batches! Everything good!
2022-02-14T22:41:46.396859Z  INFO orchestrator::main_loop: No unsigned batches! Everything good!
2022-02-14T22:41:46.643836Z  INFO relayer::valset_relaying: New valset hash b4687de48b64aab4a8942801e36d540339ff25cd8ca7c884388d81b418308e54
2022-02-14T22:41:46.649157Z  INFO relayer::valset_relaying: Consideration: looks good
2022-02-14T22:41:46.649236Z  INFO relayer::valset_relaying: Considering nonces: latest_cosmos_valset.nonce 1 current_eth_valset.nonce 1 should_relay false
2022-02-14T22:41:46.925170Z ERROR relayer::logic_call_relaying: LogicCall cost estimate failed with Err(EthersContractError(MiddlewareError(MiddlewareError(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32603, message: "Error: VM Exception while processing transaction: reverted with reason string 'Address: low-level call failed'", data: None }))))))
// -- snip

It seems like the logic_call_relaying loop continues to try the doomed-to-fail logic call. There doesn't seem to be a retry limit or timeout.

Since this loop only tries to relay the oldest signed call, if a logic call fails with a non-transient error this could prevent future logic calls from being relayed and effectively halt logic call relaying.