feat: Support describing a log type and the variables in the log type.

[rishikeshdevsot]

Request

Apache HTTP server logs are structured as the following:

172.21.0.1 - - [12/Jan/2021:15:52:42 +0000] "GET /js/ajax.js?v=5.0.4 HTTP/1.1" 200 8783 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"

I would like to be able to define a log type with variables as follows:
<ip> -- <timestamp> "<request> <link> <protocol>" <response_code> <int> "-" "<accessor>"
and then define what each variable would encode in regex.

The benefit of specifying an entire log type is that it allows differentiating variables based on their placement in the Log Type.

For example, differentiating <response_code> ("200") and <int> ("8783") and not treating both as <int>.

Possible implementation

[davidlion]

Just to be clear you're requesting to define a full logtype in the schema (based on the original title)?

What is the advantage compared to using capture groups?
E.g. (assuming we fix #175)
if we have a schema (pseudo syntax):

prefix:(?<ip>...) -- (?<timestamp>...)
content:"(?<request>...) (?<link>...) (?<protocol>...)" (?<response_code>...) (?<int>...) "-" "(?<accessor>...)"

At this point it seems more of a question of how we define a logtype. For example, maybe if a variable has captures we treat any non-capture portion of a variable as static test?

[rishikeshdevsot]

I put up this issue when I didn't really have a full understanding of how capture groups work. You are right, this feature can be expressed with just capture groups as you mentioned in you comment .

prefix:(?<ip>...) -- (?<timestamp>...)
content:"(?<request>...) (?<link>...) (?<protocol>...)" (?<response_code>...) (?<int>...) "-" "(?<accessor>...)"

I guess the only limitation is that you can't describe multiple log types that use the same capture group name (based on Capture group name cannot be the same as any other variable name nor repeat across variables issue from Jack's document).

I am okay with closing this issue if there's nothing required to be tracked.