Simplify analytics logging code#334
Merged
Merged
Conversation
- Don't log requests to the website backends and web app in the analytics database. Only log API requests. This can help eliminate a lot of unnecessary log data and 404 bot traffic from crawlers. - Get rid of several fields that we were collecting but not using. Also remove various complexities with the logging due to the various timers we were trying to gather. Those were really for debugging purposes, but it doesn't seem worth all the extra hoops to jump through. - Shift the timing and debugging details into the nginx access logs, so we still have detailed timing information on the servers for debugging, it's just not part of the analytics database. - Go back to non-gzipped nginx access logs, so it's easier to tail these files. - Make the fields we're logging to Elasticsearch a bit more explicit. This all started with some work to possibly change the Elasticsearch schema. For now, we're keeping the schema mostly the same for backwards compatibility, but the code is now setup to make the schema more explicit so this will be easier to change in the future. - Make the Lua logging code a bit more modular and reusable, so its easier to utilize in other scripts (for example, the recent script we created to reprocess nginx access logs would have been easier if more of the internal Lua functions were more easily available).
Since rsyslog has it's own way to strip invalid utf-8 data to ensure things go into elasticsearch we don't need to perform the same action in Lua code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These changes started with some experimental work while debugging some analytics issues over the past couple weeks. While we're still working on some further changes in a separate branch, the changes here represent the basic cleanup parts of that effort, so we can go ahead and get these merged in for the next release.