[QT-709] Validate scenario samples #134
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A week or two ago we changed some scenario variants in an older Vault branch but failed to update samples to remove some variants that had been deleted.
Because the matrices on sample subsets are inclusive instead of exclusive, any subset that contained the non-existent variants ended up with an empty subset sample field. When we tried to take an observation from an empty field it would result in a singular sample of just the scenario name. In practice that meant we tried to run every variant combination instead of a single sample element in the pipeline.
This changes our behavior to disallow sample fields that are empty by default. We now specifically consider whether or not the scenario has a matrix when determining whether a sample field is a scenario with no variants or subsets have excluded all of them.
We add support for validating sample fields into both
enos scenario validate
andenos scenario sample observe
commands.By default
enos scenario validate
will now ensure that all sample and sample subset fields are valid. You can now disable either scenario or sample validation by using the--no-samples
or--no-scenarios
flag.The validation command still expects a scenario filter as its default arg for filtering scenarios. If you wish to target sample validation to specific sample or sample subset you can use the new
--sample-name
,--include
and--exclude
flags.Checklist