-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[QT-602] Further optimize scenario listing and sampling
As we move to expand the variant list to tens and even hundreds of thousands of scenarios and variants in vault-enterprise, we ran into a few issues: * We hit our maximim gRPC message size when listing all the scenarios * Listing scenarios with extremely large matricies was too slow We make the following changes to help address those problems * gRPC max send and receive are now configurable in the CLI and have 5x larger defaults. * We identified a few hot spots in the program with profiling and further optimized them to reduce allocations and speed up comparisons. * We introduce a three-step build for artifacts. First we build the program with default compiler optimizations and profile the program. Then we rebuild it again using profile guided optimization. We then profile the program again and built it for a third time using profiles from our default and guided stages. Signed-off-by: Ryan Cragun <[email protected]>
- Loading branch information
1 parent
03cd231
commit daf6cdf
Showing
20 changed files
with
583 additions
and
151 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| name: build | ||
| description: build enos | ||
|
|
||
| inputs: | ||
| bundle: | ||
| description: bundle the enos binary into a zip archive | ||
| default: true | ||
| goarch: | ||
| description: the GOARCH value | ||
| required: true | ||
| goos: | ||
| description: the GOOS value | ||
| required: true | ||
| version: | ||
| description: the version of the package | ||
| required: true | ||
|
|
||
| outputs: | ||
| artifact-name: | ||
| description: where the built artifact is located | ||
| value: ${{ steps.build.outputs.artifact-name }} | ||
| artifact-path: | ||
| description: where the built artifact is located | ||
| value: ${{ steps.build.outputs.artifact-path }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Build | ||
| id: build | ||
| env: | ||
| ARTIFACT_NAME: ${{ env.PKG_NAME}}_${{ inputs.version }}_${{ inputs.goos }}_${{ inputs.goarch }}.zip | ||
| GOARCH: ${{ inputs.goarch }} | ||
| GOOS: ${{ inputs.goos }} | ||
| shell: bash | ||
| run: | | ||
| mkdir -p dist out | ||
| make | ||
| echo "artifact-name=${{ env.ARTIFACT_NAME }}" >> "$GITHUB_OUTPUT" | ||
| echo "artifact-path=out/${{ env.ARTIFACT_NAME }}" >> "$GITHUB_OUTPUT" | ||
| - if: ${{ inputs.bundle == 'true' }} | ||
| name: Bundle | ||
| shell: bash | ||
| run: zip -r -j ${{ steps.build.outputs.artifact-path }} dist/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| --- | ||
| name: profile-build | ||
| description: profile enos build | ||
|
|
||
| inputs: | ||
| build-path: | ||
| description: where the enos artifact to profile is | ||
| default: dist/enos | ||
| profile-out: | ||
| description: where to write the union cpu profile | ||
| default: cpu.pprof | ||
| merge-to-default: | ||
| description: merge all cpu profiles to default.pgo after profiling | ||
| default: true | ||
| upload-profile: | ||
| description: upload default.pgo after profiling | ||
| default: true | ||
|
|
||
| outputs: | ||
| profile-path: | ||
| value: default.pgo | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: profile | ||
| shell: bash | ||
| run: | | ||
| ${{ inputs.build-path }} scenario list --format json --chdir ./acceptance/scenarios/build_pgo/ --profile | jq '.scenarios | length' | ||
| mv cpu.pprof cpu.pprof.scenario.list | ||
| ${{ inputs.build-path }} scenario sample observe complex --max 10 --chdir ./acceptance/scenarios/build_pgo/ --profile | ||
| mv cpu.pprof cpu.pprof.sample.observe | ||
| go tool pprof -proto cpu.pprof.scenario.list cpu.pprof.sample.observe > ${{ inputs.profile-out }} | ||
| - if: ${{ inputs.merge-to-default == 'true' }} | ||
| name: merge-to-default | ||
| shell: bash | ||
| run: | | ||
| if test -f default.pgo; then | ||
| cp default.pgo default.pprof | ||
| go tool pprof -proto ${{ inputs.profile-out }} -proto default.pprof > default.pgo | ||
| exit 0 | ||
| fi | ||
| cp ${{ inputs.profile-out }} default.pgo | ||
| - if: ${{ inputs.upload-profile == 'true' }} | ||
| uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3 | ||
| with: | ||
| name: default.pgo | ||
| path: default.pgo | ||
| retention-days: 1 | ||
| if-no-files-found: error |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.