Commits on May 2, 2024

1. [QT-709] Validate scenario samples …

   A week or two ago we changes some scenario variants in an older Vault
   branch but failed to update samples that expected deleted variants.
   
   Because the matrices on sample subsets are _inclusive_ instead of
   _exclusive_, any subset that contained the non-existent variants ended
   up with an empty subset sample field. When we tried to take an
   observation from an empty field it would result in a singular sample of
   just the scenario name. In practice that meant we tried to run _every_
   variant combination instead of a single sample in the pipeline.
   
   This changes our behavior to disallow sample fields that are empty by
   default. We now specifically consider whether or not the scenario has a
   matrix when determining whether a sample field is a scenario with no
   variants or subsets have excluded all of them.
   
   We add support for validating sample fields into both `enos scenario
   validate` and `enos scenario sample observe` commands.
   
   By default `enos scenario validate` will now ensure that all sample and
   sample subset fields are valid. You can now disable either scenario or
   sample validation by using the `--no-samples` or `--no-scenarios` flag.
   
   The validation command still expects a scenario filter as its default
   arg for filtering scenarios. If you wish to target sample validation to
   specific sample or sample subset you can use the new `--sample-name`,
   `--include` and `--exclude` flags.
   
   * validate: add support for validating samples into sample command
   * acceptance: add invalid scenario sample combination to validate suite
   * acceptance: add invalid scenario sample combination to sample observe
     suite
   * make: Don't run Go test in verbose mode by default
   * acceptance: split stdout and stderr in the enos acceptance runner
   
   Signed-off-by: Ryan Cragun <[email protected]>

   

   ryancragun committed May 2, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 17a8382
   • Browse repository at this point

   Copy the full SHA

   17a8382 View commit details

   Browse the repository at this point in the history