Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QT-577] Execute scenario sub-commands faster #101

Merged
merged 1 commit into from
Jul 26, 2023
Merged

[QT-577] Execute scenario sub-commands faster #101

merged 1 commit into from
Jul 26, 2023

Conversation

ryancragun
Copy link
Collaborator

Our initial implementation of flightplan decoding would always decode and validate all scenarios and their corresponding variant combinations. While this was nice to ensure that all possible combinations are valid, with large matrices this becomes quite a burden. Vault and Vault Enterprise are two examples where you can see the "generate" step (which is actually decoding, validating, and transpiling the scenario into a Terraform root module) take upwards of a minute for both enos scenario run and enos scenario destroy. It also makes local development difficult because it is run every time you try and test a module.

Instead of decoding all flightplan plans and selecting those that match the filter, we instead decode the matrix, create a sub-matrix with all combinations that match a given filter, then decode only those scenarios that that would match the filter. This greatly improves our time because we don't act on scenarios that are not targeted. For cases where we do match lots of scenarios, we've also implemented concurrent decoding, which allows us to parallelize the decoding process across all available CPU cores. In vault we see validation of all scenarios go from ~43 seconds on my machine to ~5. We also see the "generate" step finish almost immediately. This should shave anywhere from 1.5 -> 2 minutes from all enos scenario runs in CI.

As part of this change I also updated all of the Go modules and modernized our golangci-lint configuration. This makes our CI standards for enos-provider and enos the same. Updating the linters required a number of fixes. Most of small, the most significant were how we could parallelize the acceptance tests. That required further work on the gRPC listener automatically accepting free ports when one is not provided.

  • Update Go modules.
  • Add support for filtering a matrix with a scenario filter.
  • Allow for decoding scenarios in serially or concurrently.
  • Update golangci-lint configuration.
  • Please the simple linters.
  • Fix parallelization of all tests. Execution of of make test-acc was cut down by 66%.
  • Thread the context further through flightplan decoding to better support concurrent decoding.
  • Simplify creating a connection to the enos server in the client.
  • Add support for dynamic port selection in the gRPC server listner.

Checklist

  • The commit message includes an explanation of the changes
  • Manual validation of the changes have been performed (if possible)
  • New or modified code has requisite test coverage (if possible)
  • I have performed a self-review of the changes
  • I have made necessary changes and/or pull requests for documentation
  • I have written useful comments in the code

Our initial implementation of flightplan decoding would always decode
and validate all scenarios and their corresponding variant combinations.
While this was nice to ensure that all possible combinations are valid,
with large matrices this becomes quite a burden. Vault and Vault
Enterprise are two examples where you can see the "generate" step (which
is actually decoding, validating, and transpiling the scenario into a
Terraform root module) take upwards of a minute for both `enos scenario
run` and `enos scenario destroy`. It also makes local development
difficult because it is run every time you try and test a module.

Instead of decoding all flightplan plans and selecting those that match
the filter, we instead decode the matrix, create a sub-matrix with all
combinations that match a given filter, then decode only those scenarios
that that would match the filter. This greatly improves our time because
we don't act on scenarios that are not targeted. For cases where we do
match lots of scenarios, we've also implemented concurrent decoding,
which allows us to parallelize the decoding process across all available
CPU cores. In vault we see validation of all scenarios go from ~43
seconds on my machine to ~5. We also see the "generate" step finish
almost immediately. This should shave anywhere from 1.5 -> 2 minutes
from all enos scenario runs in CI.

As part of this change I also updated all of the Go modules and
modernized our golangci-lint configuration. This makes our CI standards
for enos-provider and enos the same. Updating the linters required
a number of fixes. Most of small, the most significant were how we could
parallelize the acceptance tests. That required further work on the gRPC
listener automatically accepting free ports when one is not provided.

* Update Go modules.
* Add support for filtering a matrix with a scenario filter.
* Allow for decoding scenarios in serially or concurrently.
* Update golangci-lint configuration.
* Please the simple linters.
* Fix parallelization of all tests. Execution of of `make test-acc` was
  cut down by 66%.
* Thread the context further through flightplan decoding to better
  support concurrent decoding.
* Simplify creating a connection to the enos server in the client.
* Add support for dynamic port selection in the gRPC server listner.

Signed-off-by: Ryan Cragun <[email protected]>
@ryancragun ryancragun added the changelog/feat New feature or enhancement. Will be included in "New Features" category in release notes. label Jul 26, 2023
@ryancragun ryancragun requested a review from a team as a code owner July 26, 2023 16:23
@ryancragun ryancragun merged commit 63fbb38 into main Jul 26, 2023
6 checks passed
@ryancragun ryancragun deleted the qt-577 branch July 26, 2023 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog/feat New feature or enhancement. Will be included in "New Features" category in release notes.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant