Skip to content

[Check Point] Process the packets field in SecureXL format #16235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ilyannn wants to merge 2 commits into
base: main
Choose a base branch
from
+76 −1
Draft

[Check Point] Process the packets field in SecureXL format #16235

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7233193
[checkpoint] Add the processor for SecureXL fields
ilyannn Dec 3, 2025
0f68a67
[checkpoint] Update the metadata
ilyannn Dec 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/checkpoint/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.44.0"
changes:
- description: Process the packets field in SecureXL format.
type: enhancement
link: https://github.com/elastic/integrations/pull/16235
- version: "1.43.0"
changes:
- description: Update documentation.
Expand Down
23 changes: 23 additions & 0 deletions packages/checkpoint/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1018,6 +1018,29 @@ processors:
tag: remove_checkpoint_subs_exp_19a22c63
field: checkpoint.subs_exp
ignore_missing: true
- script:
tag: script_parse_checkpoint_packets_dropped
Copy link
Member

@P1llus P1llus Dec 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be preferred if we could use a common processor to parse this out, as script regex is quite heavy, was this the only way? looking at the data I guess it does seem like that, but its quite troublesome for performance reasons

Copy link
Contributor Author

@ilyannn ilyannn Dec 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I welcome any ideas 😄. I tried split + grok but couldn't make it work.

Copy link
Member

@P1llus P1llus Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah once I noticed it was meant to produce an array then any other processor mostly goes out the window with the exception of maybe foreach, but that just introduces many other issues, so script was the way to go, I was just hoping we could have managed to skip the regex part, as in theory the data is structured in some ways.

description: Parse packets field containing connection tuples into structured packets_dropped array.
if: ctx.checkpoint?.packets instanceof String && ctx.checkpoint.packets.contains('<')
Copy link
Member

@P1llus P1llus Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit unsure but it could be startsWith is better performance wise, though if the only times this is a string is when it is the SecureXL format it might be okay.
If we had sample data we could see if there was an already parsed field that showcases that it's SecureXL type data so we wouldn't need to use either

Copy link
Contributor Author

@ilyannn ilyannn Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I don't know if their tuple format is well-documented, for example if it's guaranteed that it does not start with a whitespace. The cleanest way is probably to try a conversion to number first and only run the script if it fails – I'll do that.

lang: painless
source: |
def packetsStr = ctx.checkpoint.packets.trim();
def parsed = [];
def matcher = /<([^,]+),(\d+),([^,]+),(\d+),(\d+)(?:;([^>]+))?>/.matcher(packetsStr);
while (matcher.find()) {
def packet = new HashMap();
packet.put('source', ['ip': matcher.group(1), 'port': Long.parseLong(matcher.group(2))]);
packet.put('destination', ['ip': matcher.group(3), 'port': Long.parseLong(matcher.group(4))]);
packet.put('network', ['iana_number': matcher.group(5)]);
if (matcher.group(6) != null) {
packet.put('interface', ['name': matcher.group(6)]);
}
parsed.add(packet);
}
if (parsed.size() > 0) {
ctx.checkpoint.packets_dropped = parsed;
ctx.checkpoint.remove('packets');
}
- convert:
tag: convert_checkpoint_packets_3af974e8
field: checkpoint.packets
Expand Down
41 changes: 41 additions & 0 deletions packages/checkpoint/data_stream/firewall/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1187,6 +1187,47 @@
type: integer
description: |
Amount of packets dropped.
- name: packets_dropped
type: nested
Copy link
Member

@P1llus P1llus Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just leaving it as a note rather than something that needs to be resolved, nested types are something we do try to avoid as much as possible, but at this point I am unsure about any other better alternative

Copy link
Contributor Author

@ilyannn ilyannn Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it's a tradeoff. With nested types we accurately show the underlying structure, so that they can do "find packets where source is X and destination is Y", but it does come at a cost. The alternative could be to use a group but then we'll only have a list of all sources and a list of all destinations in the event and the information about which source goes to which destination would be lost.

Copy link
Contributor Author

@ilyannn ilyannn Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know the details, but my assumption was that there is no performance penalty as long as nobody is searching on that field.

Copy link
Member

@P1llus P1llus Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah nested fields don't induce a performance penalty but you cannot use aggregations on them, so any dashboard visualization etc wont work, the only things you can do is querying for example on the interface name of any of the objects

description: |
Connection tuples for dropped packets containing source/destination IP, port, protocol, and interface.
fields:
- name: source
type: group
fields:
- name: ip
type: ip
description: |
Source IP address of the dropped packet.
- name: port
type: long
description: |
Source port of the dropped packet.
- name: destination
type: group
fields:
- name: ip
type: ip
description: |
Destination IP address of the dropped packet.
- name: port
type: long
description: |
Destination port of the dropped packet.
- name: network
type: group
fields:
- name: iana_number
type: keyword
description: |
IANA protocol number of the dropped packet.
- name: interface
type: group
fields:
- name: name
type: keyword
description: |
Interface name where the packet was dropped.
- name: packet_capture_unique_id
type: keyword
description: |
Expand Down
6 changes: 6 additions & 0 deletions packages/checkpoint/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -488,6 +488,12 @@ The `firewall` data stream provides events from Check Point devices, including f
| checkpoint.packet_capture_name | | keyword |
| checkpoint.packet_capture_time | | keyword |
| checkpoint.packet_capture_unique_id | Identifier of the packet capture files. | keyword |
| checkpoint.packets_dropped.destination.ip | Destination IP address of the dropped packet. | ip |
| checkpoint.packets_dropped.destination.port | Destination port of the dropped packet. | long |
| checkpoint.packets_dropped.interface.name | Interface name where the packet was dropped. | keyword |
| checkpoint.packets_dropped.network.iana_number | IANA protocol number of the dropped packet. | keyword |
| checkpoint.packets_dropped.source.ip | Source IP address of the dropped packet. | ip |
| checkpoint.packets_dropped.source.port | Source port of the dropped packet. | long |
| checkpoint.parent_file_hash | Archive's hash in case of extracted files. | keyword |
| checkpoint.parent_file_name | Archive's name in case of extracted files. | keyword |
| checkpoint.parent_file_uid | Archive's UID in case of extracted files. | keyword |
Expand Down
2 changes: 1 addition & 1 deletion packages/checkpoint/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: checkpoint
title: Check Point
version: "1.43.0"
version: "1.44.0"
description: Collect logs from Check Point with Elastic Agent.
type: integration
format_version: "3.0.3"
Expand Down