-
Notifications
You must be signed in to change notification settings - Fork 524
[Check Point] Process the packets field in SecureXL format #16235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
7233193
0f68a67
a2ad80f
6c48005
d827624
619eec9
4564cdf
4bcc4e6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1018,6 +1018,29 @@ processors: | |
| tag: remove_checkpoint_subs_exp_19a22c63 | ||
| field: checkpoint.subs_exp | ||
| ignore_missing: true | ||
| - script: | ||
| tag: script_parse_checkpoint_packets_dropped | ||
| description: Parse packets field containing connection tuples into structured packets_dropped array. | ||
| if: ctx.checkpoint?.packets instanceof String && ctx.checkpoint.packets.contains('<') | ||
|
||
| lang: painless | ||
| source: | | ||
| def packetsStr = ctx.checkpoint.packets.trim(); | ||
| def parsed = []; | ||
| def matcher = /<([^,]+),(\d+),([^,]+),(\d+),(\d+)(?:;([^>]+))?>/.matcher(packetsStr); | ||
| while (matcher.find()) { | ||
| def packet = new HashMap(); | ||
| packet.put('source', ['ip': matcher.group(1), 'port': Long.parseLong(matcher.group(2))]); | ||
| packet.put('destination', ['ip': matcher.group(3), 'port': Long.parseLong(matcher.group(4))]); | ||
| packet.put('network', ['iana_number': matcher.group(5)]); | ||
| if (matcher.group(6) != null) { | ||
| packet.put('interface', ['name': matcher.group(6)]); | ||
| } | ||
| parsed.add(packet); | ||
| } | ||
| if (parsed.size() > 0) { | ||
| ctx.checkpoint.packets_dropped = parsed; | ||
| ctx.checkpoint.remove('packets'); | ||
| } | ||
| - convert: | ||
| tag: convert_checkpoint_packets_3af974e8 | ||
| field: checkpoint.packets | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1187,6 +1187,47 @@ | |
| type: integer | ||
| description: | | ||
| Amount of packets dropped. | ||
| - name: packets_dropped | ||
| type: nested | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just leaving it as a note rather than something that needs to be resolved, nested types are something we do try to avoid as much as possible, but at this point I am unsure about any other better alternative
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah it's a tradeoff. With nested types we accurately show the underlying structure, so that they can do "find packets where source is X and destination is Y", but it does come at a cost. The alternative could be to use a group but then we'll only have a list of all sources and a list of all destinations in the event and the information about which source goes to which destination would be lost.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't know the details, but my assumption was that there is no performance penalty as long as nobody is searching on that field.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah nested fields don't induce a performance penalty but you cannot use aggregations on them, so any dashboard visualization etc wont work, the only things you can do is querying for example on the interface name of any of the objects |
||
| description: | | ||
| Connection tuples for dropped packets containing source/destination IP, port, protocol, and interface. | ||
| fields: | ||
| - name: source | ||
| type: group | ||
| fields: | ||
| - name: ip | ||
| type: ip | ||
| description: | | ||
| Source IP address of the dropped packet. | ||
| - name: port | ||
| type: long | ||
| description: | | ||
| Source port of the dropped packet. | ||
| - name: destination | ||
| type: group | ||
| fields: | ||
| - name: ip | ||
| type: ip | ||
| description: | | ||
| Destination IP address of the dropped packet. | ||
| - name: port | ||
| type: long | ||
| description: | | ||
| Destination port of the dropped packet. | ||
| - name: network | ||
| type: group | ||
| fields: | ||
| - name: iana_number | ||
| type: keyword | ||
| description: | | ||
| IANA protocol number of the dropped packet. | ||
| - name: interface | ||
| type: group | ||
| fields: | ||
| - name: name | ||
| type: keyword | ||
| description: | | ||
| Interface name where the packet was dropped. | ||
| - name: packet_capture_unique_id | ||
| type: keyword | ||
| description: | | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be preferred if we could use a common processor to parse this out, as script regex is quite heavy, was this the only way? looking at the data I guess it does seem like that, but its quite troublesome for performance reasons
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I welcome any ideas 😄. I tried
split+grokbut couldn't make it work.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah once I noticed it was meant to produce an array then any other processor mostly goes out the window with the exception of maybe foreach, but that just introduces many other issues, so script was the way to go, I was just hoping we could have managed to skip the regex part, as in theory the data is structured in some ways.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Yeah a review of https://www.elastic.co/docs/reference/scripting-languages/painless/painless-ingest-processor-context has shown to me how this is possible, so I've implemented a non-regexp version.