Speed up service start up time by reducing runner registration / extension loading time #5198
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ad of creating new DriverManager instance per extension. Creating new instance is not needed and results in disk / pkg resources scan per runner (extension) which is slow. This also substantially reduces debug log noise on service start up.
pull-request-size
bot
added
the
size/S
amanda11
approved these changes
Mar 21, 2021
|
|
||
| for name in extension_names: | ||
| # NOTE: We use ExtensionManager directly instead of DriverManager per extension since that is | ||
| # much faster and allows us to reduce stevedore loading overead for each runner |
There was a problem hiding this comment.
typo - overhead not overead
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was looking at st2api profiling data some more and noticed we spend more time in stevedore code then we should.
It turns on that we were not loading all the extensions just once, but X + 1 times where X is number of runners (stevedore extensions).
This is obviously not the fastest thing since loading extensions requires pkg resources scan for available extensions.
This pull request fixes that. It should now also result in much less DEBUG level noise on service start up time.