Skip to content

Conversation

@janvi-elastic
Copy link
Contributor

@janvi-elastic janvi-elastic commented Dec 5, 2025

Proposed commit message

The initial release includes asset data stream(Alert, Vulnerability and Finding Class), associated dashboards 
and visualizations.

JupiterOne fields are mapped to their corresponding ECS fields where possible.

Test samples were derived from documentation and live data samples, 
which were subsequently sanitized.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/jupiter_one directory.
  • Run the following command to run tests.

elastic-package test

--- Test results for package: jupiter_one - START ---
╭─────────────┬──────────────────┬───────────┬──────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM      │ TEST TYPE │ TEST NAME                                                            │ RESULT │ TIME ELAPSED │
├─────────────┼──────────────────┼───────────┼──────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ jupiter_one │                  │ asset     │ dashboard jupiter_one-b38d47b8-40d8-4aac-bde8-bf14f1ed64c2 is loaded │ PASS   │      2.613µs │
│ jupiter_one │                  │ asset     │ dashboard jupiter_one-eb28c687-3b1c-43d4-acb3-6e1eb7b7b46e is loaded │ PASS   │        440ns │
│ jupiter_one │ asset            │ asset     │ index_template logs-jupiter_one.asset is loaded                      │ PASS   │        371ns │
│ jupiter_one │ asset            │ asset     │ ingest_pipeline logs-jupiter_one.asset-0.1.0 is loaded               │ PASS   │        279ns │
│ jupiter_one │ risks_and_alerts │ asset     │ index_template logs-jupiter_one.risks_and_alerts is loaded           │ PASS   │        224ns │
╰─────────────┴──────────────────┴───────────┴──────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: jupiter_one - END   ---
Done
--- Test results for package: jupiter_one - START ---
╭─────────────┬─────────────┬───────────┬───────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM │ TEST TYPE │ TEST NAME                                 │ RESULT │ TIME ELAPSED │
├─────────────┼─────────────┼───────────┼───────────────────────────────────────────┼────────┼──────────────┤
│ jupiter_one │ asset       │ pipeline  │ (ingest pipeline warnings test-asset.log) │ PASS   │ 1.478745905s │
│ jupiter_one │ asset       │ pipeline  │ test-asset.log                            │ PASS   │ 597.466219ms │
╰─────────────┴─────────────┴───────────┴───────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: jupiter_one - END   ---
Done
--- Test results for package: jupiter_one - START ---
No test results
--- Test results for package: jupiter_one - END   ---
Done
--- Test results for package: jupiter_one - START ---
No test results
--- Test results for package: jupiter_one - END   ---
Done
--- Test results for package: jupiter_one - START ---
No test results
--- Test results for package: jupiter_one - END   ---
Done

Related issues

Screenshot

image image

Go Code for Ingest Pipeline Generation

The incident data stream pipeline is generated using Go code built on top of the Dispear library.
Below is the code used for generating the pipeline logic:

Asset Pipeline:

package main

import (
    "fmt"
    "strings"

    . "github.com/efd6/dispear"
)

const (
    ECSVersion = "9.2.0"
    PkgRoot    = "json"
)
const errorFormat = "Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}"

func removeErrorHandler(f string) []Renderer {
    return []Renderer{
        REMOVE(f),
        APPEND("error.message", errorFormat),
    }
}

func safeNavigateAndCheck(field string) string {
    parts := strings.Split(field, ".")
    condition := "ctx"
    for i, part := range parts {
        if i > 0 { // Skip the first part which is already included in the condition
            condition += fmt.Sprintf("?.%s", part)
        } else {
            condition += fmt.Sprintf(".%s", part)
        }
    }
    return condition
}

func main() {

    // Initial processors of pipeline
    DESCRIPTION("Pipeline for processing assets.")

    SET("ecs.version").VALUE(ECSVersion)

    TERMINATE("data collection error").
        IF("ctx.error?.message != null && ctx.message == null && ctx.event?.original == null").
        DESCRIPTION("error message set and no data to process.")

    BLANK()

    BLANK().COMMENT("remove agentless metadata")

    REMOVE(
        "organization",
        "division",
        "team",
    ).
        IF("ctx.organization instanceof String && ctx.division instanceof String && ctx.team instanceof String").
        IGNORE_MISSING(true).
        TAG("remove_agentless_tags").
        DESCRIPTION("Removes the fields added by Agentless as metadata, as they can collide with ECS fields.")

    BLANK()

    BLANK().COMMENT("parse the event JSON")

    RENAME("message", "event.original").
        IF("ctx.event?.original == null").
        DESCRIPTION("Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document.").
        IGNORE_MISSING(true)

    REMOVE("message").
        TAG("remove_message").
        IF("ctx.event?.original != null").
        DESCRIPTION("The `message` field is no longer required if the document has an `event.original` field.").
        IGNORE_MISSING(true)

    JSON(PkgRoot, "event.original")

    // Script to rename into snake case

    BLANK()

    BLANK().COMMENT("rename to snake case")

    SCRIPT().
        TAG("script_convert_camelcase_to_snake_case").
        DESCRIPTION("Convert camelCase to snake_case.").
        LANG("painless").
        SOURCE(`
        // Helper function to convert camelCase to snake_case
        String camelToSnake(String str) {
            def result = "";
            for (int i = 0; i < str.length(); i++) {
                char c = str.charAt(i);
                if (Character.isUpperCase(c)) {
                    if (i > 0 && Character.isLowerCase(str.charAt(i - 1))) {
                        result += "_";
                    }
                    result += Character.toLowerCase(c);
                } else {
                    result += c;
                }
            }
            return result;
        }
        // Recursive function to handle nested fields
        def convertToSnakeCase(def obj) {
          if (obj instanceof Map) {
            // Convert each key in the map
            def newObj = [:];
            for (entry in obj.entrySet()) {
              String newKey = camelToSnake(entry.getKey());
              newObj[newKey] = convertToSnakeCase(entry.getValue());
            }
            return newObj;
          } else if (obj instanceof List) {
            // If it's a list, process each item recursively
            def newList = [];
            for (item in obj) {
              newList.add(convertToSnakeCase(item));
            }
            return newList;
          } else {
            return obj;
          }
        }
        // Apply the conversion
        ctx.jupiter_one = ctx.jupiter_one ?: [:];
        if (ctx.json != null) {
          ctx.jupiter_one.asset = convertToSnakeCase(ctx.json);
        }
        // Remove json field
        ctx.remove('json');
        `)
    
    

    // Use Date processors

    BLANK()

    BLANK().COMMENT("Date processors")

    for _, field := range []string{
        "jupiter_one.asset.entity._created_on",
        "jupiter_one.asset.entity._end_on",
        "jupiter_one.asset.entity._begin_on",
        "jupiter_one.asset.properties.created_on",
		"jupiter_one.asset.properties.updated_on",
    } {
        DATE(field, field, "ISO8601").
            IF(safeNavigateAndCheck(field) + " != null" + " && " + "ctx." + field + " != ''").
            ON_FAILURE(removeErrorHandler(field)...)
    }

    // Convert to boolean
    BLANK()
    BLANK().COMMENT("Convert to Boolean Processors")
    for _, field := range []string{
        "jupiter_one.asset.entity._deleted",
        "jupiter_one.asset.properties.active",
        "jupiter_one.asset.properties.public",
        "jupiter_one.asset.properties.validated",
    } {
        CONVERT("", field, "boolean").
            IGNORE_MISSING(true).
            ON_FAILURE(removeErrorHandler(field)...)
    }

    // Convert to string
    BLANK()
    BLANK().COMMENT("Convert to String Processors")
    for _, field := range []string{
        "jupiter_one.asset.entity._version",
    } {
        CONVERT("", field, "string").
            IGNORE_MISSING(true)
    }

    // Set ECS Mapping

    BLANK()

    BLANK().COMMENT("Map custom fields to corresponding ECS and related fields.")

    // Map ECS mapping for top-level fields
    for _, mapping := range []struct {
        ecsField, customField string
    }{
        {ecsField: "event.id", customField: "jupiter_one.asset.id"},
        {ecsField: "event.created", customField: "jupiter_one.asset.entity._created_on"},
        {ecsField: "event.end", customField: "jupiter_one.asset.entity._end_on"},
        {ecsField: "event.start", customField: "jupiter_one.asset.entity._begin_on"},
        {ecsField: "message", customField: "jupiter_one.asset.properties.description"},
        {ecsField: "url.original", customField: "jupiter_one.asset.properties.web_link"},
    } {
        SET(mapping.ecsField).COPY_FROM(mapping.customField).IGNORE_EMPTY(true).TAG(fmt.Sprintf("set %s from %s", mapping.ecsField, mapping.customField))
    }
    SET("vulnerability.description").COPY_FROM("jupiter_one.asset.properties.description").IF(`ctx.jupiter_one?.asset?.entity?._class?.contains('Vulnerability') == true`).IGNORE_EMPTY(true).TAG("set_vulnerability_description_from_jupiter_one_asset_properties_description")

    // Calculate event duration
    SCRIPT().
        TAG("script_to_calculate_event_duration").
        DESCRIPTION("Calculate event.duration.").
        LANG("painless").
        SOURCE(`
        Instant eventstart = ZonedDateTime.parse(ctx.event?.start).toInstant();
        Instant eventend = ZonedDateTime.parse(ctx.event?.end).toInstant();
        ctx.event['duration'] = ChronoUnit.NANOS.between(eventstart, eventend);
        `).IF("ctx.event?.start != null && ctx.event.end != null")

    // URI Part Processor

    URI_PARTS("","url.original").IGNORE_MISSING(true).ON_FAILURE(removeErrorHandler("url.original")...).TAG("uri_parts_url_original")

    // Direct to another pipeline

    PIPELINE("pipeline_risks_and_alerts").IGNORE_MISSING(true).IF(`ctx.jupiter_one.asset.entity._class.contains('Vulnerability') || ctx.jupiter_one.asset.entity._class.contains('Alert') || ctx.jupiter_one.asset.entity._class.contains('Finding')`)

    // Remove Duplicate Fields.

    BLANK()

    BLANK().COMMENT("Remove Duplicate Custom Field if preserve_duplicate_custom_fields are not enabled")

    REMOVE(
		"jupiter_one.asset.id",
		"jupiter_one.asset.entity._created_on",
        "jupiter_one.asset.entity._end_on",
        "jupiter_one.asset.entity._begin_on",
        "jupiter_one.asset.properties.web_link",
        "jupiter_one.asset.properties.level",
		"jupiter_one.asset.properties.device_id",
		"jupiter_one.asset.properties.user_id",
		"jupiter_one.asset.properties.device_mac_address",
		"jupiter_one.asset.properties.device_os_version",
		"jupiter_one.asset.properties.device_hostname",
		"jupiter_one.asset.properties.device_platform_name",
		"jupiter_one.asset.properties.user_name",
    ).
        IF("ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')").
        TAG("remove_custom_duplicate_fields").
        IGNORE_MISSING(true)

    // Clean up script

    BLANK()

    BLANK().COMMENT("Cleanup")

    SCRIPT().
        TAG("script_to_drop_null_values").
        DESCRIPTION("This script processor iterates over the whole document to remove fields with null values.").
        LANG("painless").
        SOURCE(`
        void handleMap(Map map) {
        map.values().removeIf(v -> {
            if (v instanceof Map) {
            handleMap(v);
            } else if (v instanceof List) {
            handleList(v);
            }
            return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0)
        });
        }
        void handleList(List list) {
        list.removeIf(v -> {
            if (v instanceof Map) {
            handleMap(v);
            } else if (v instanceof List) {
            handleList(v);
            }
            return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0)
        });
        }
        handleMap(ctx);
        `)

	// Set and Append processor on last

	SET("event.kind").
		VALUE("pipeline_error").
		IF("ctx.error?.message != null").
		TAG("set event.kind to pipeline_error")

	APPEND("tags", "preserve_original_event").
		IF("ctx.error?.message != null").
		ALLOW_DUPLICATES(false)

	// Global on failure processor

	ON_FAILURE(
		APPEND("error.message", errorFormat),
		SET("event.kind").VALUE("pipeline_error").TAG("set event.kind to pipeline_error"),
		APPEND("tags", "preserve_original_event").
			ALLOW_DUPLICATES(false),
	)

	// Generate the pipeline

	Generate()
}

Risks and Alerts Pipeline:

package main

import (
	"fmt"
	"strings"

	. "github.com/efd6/dispear"
)

const errorFormat = "Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}"

func removeErrorHandler(f string) []Renderer {
	return []Renderer{
		REMOVE(f),
		APPEND("error.message", errorFormat),
	}
}

func safeNavigateAndCheck(field string) string {
	parts := strings.Split(field, ".")
	condition := "ctx"
	for i, part := range parts {
		if i > 0 { // Skip the first part which is already included in the condition
			condition += fmt.Sprintf("?.%s", part)
		} else {
			condition += fmt.Sprintf(".%s", part)
		}
	}
	return condition
}

func main() {

	// Initial processors of pipeline
	DESCRIPTION("Pipeline for processing risks and alerts type.")

	// Setting event.* fields.
	BLANK()
	BLANK().COMMENT("Set event.* fields.")
	SET("event.kind").VALUE("alert").TAG("set_event_kind_alert")
	APPEND("event.category","vulnerability").IF(`ctx.jupiter_one.asset.entity._class.contains('Vulnerability')`).TAG("append_event_category_vulnerability")
	APPEND("event.type","info").IF(`ctx.jupiter_one.asset.entity._class.contains('Vulnerability')`).TAG("append_event_type_info")

    // Dot expander.
    BLANK().COMMENT("Dot expander")
    DOT_EXPANDER("*").PATH("jupiter_one.asset.properties").TAG("dot_expander_jupiter_one_asset_properties")
    

	// Use Date processors

	BLANK().COMMENT("Date processors")

	for _, field := range []string{
		"jupiter_one.asset.properties.approved_on",
		"jupiter_one.asset.properties.reported_on",
		"jupiter_one.asset.properties.detected_on",
		"jupiter_one.asset.properties.published_on",
	} {
		DATE(field, field, "ISO8601").
			IF(safeNavigateAndCheck(field) + " != null" + " && " + "ctx." + field + " != ''").
			ON_FAILURE(removeErrorHandler(field)...)
	}

	// Convert to Long
	BLANK()
	BLANK().COMMENT("Convert to Long Processors")
	for _, field := range []string{
		"jupiter_one.asset.properties.total_number_of_affected_entities",
		"jupiter_one.asset.properties.numeric_severity",
		"jupiter_one.asset.properties.remediation_sla",
		"jupiter_one.asset.properties.exploit_status",
	} {
		CONVERT("", field, "long").
			IGNORE_MISSING(true).
			ON_FAILURE(removeErrorHandler(field)...)
	}

	// Convert to Double
	BLANK()
	BLANK().COMMENT("Convert to Double Processors")
	for _, field := range []string{
		"jupiter_one.asset.properties.exploitability",
		"jupiter_one.asset.properties.impact",
		"jupiter_one.asset.properties.score",
	} {
		CONVERT("", field, "double").
			IGNORE_MISSING(true).
			ON_FAILURE(removeErrorHandler(field)...)
	}

	// Convert to boolean
	BLANK()
	BLANK().COMMENT("Convert to Boolean Processors")
	for _, field := range []string{
		"jupiter_one.asset.properties.open",
		"jupiter_one.asset.properties.approved",
		"jupiter_one.asset.properties.exception",
		"jupiter_one.asset.properties.production",
		"jupiter_one.asset.properties.blocks_production",
		"jupiter_one.asset.properties.tag.production",
		"jupiter_one.asset.properties.blocking",
	} {
		CONVERT("", field, "boolean").
			IGNORE_MISSING(true).
			ON_FAILURE(removeErrorHandler(field)...)
	}

	// Convert to IP
	BLANK()
	BLANK().COMMENT("Convert to IP")
	for _, field := range []string{
		"jupiter_one.asset.properties.device_local_ip",
		"jupiter_one.asset.properties.device_external_ip",
	} {
		CONVERT("", field, "ip").
			IGNORE_MISSING(true).
			IF(safeNavigateAndCheck(field) + " != ''").
			ON_FAILURE(removeErrorHandler(field)...)
	}

	// Map related mapppings
	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "related.user", customField: "jupiter_one.asset.properties.reporter"},
		{ecsField: "related.ip", customField: "jupiter_one.asset.properties.device_local_ip"},
		{ecsField: "related.user", customField: "jupiter_one.asset.properties.user_id"},
		{ecsField: "related.ip", customField: "jupiter_one.asset.properties.device_external_ip"},
		{ecsField: "related.hosts", customField: "jupiter_one.asset.properties.device_hostname"},
		{ecsField: "related.user", customField: "jupiter_one.asset.properties.user_name"},
	} {
		APPEND(mapping.ecsField, "{{{"+mapping.customField+"}}}").
			IF(safeNavigateAndCheck(mapping.customField) + " != null").
			ALLOW_DUPLICATES(false)
	}

	// Map related mapppings for array fields
	FOREACH("jupiter_one.asset.properties.approvers",
		APPEND("related.user", "{{{_ingest._value}}}").
			ALLOW_DUPLICATES(false),
	).IF("ctx.jupiter_one?.asset?.properties?.approvers instanceof List")


	// Set ECS Mapping

	BLANK()

	BLANK().COMMENT("Map custom fields to corresponding ECS and related fields.")

	// Map ECS mapping for top-level fields
	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "log.level", customField: "jupiter_one.asset.properties.level"},
		{ecsField: "host.id", customField: "jupiter_one.asset.properties.device_id"},
		{ecsField: "user.id", customField: "jupiter_one.asset.properties.user_id"},
		{ecsField: "host.os.version", customField: "jupiter_one.asset.properties.device_os_version"},
		{ecsField: "host.hostname", customField: "jupiter_one.asset.properties.device_hostname"},
		{ecsField: "host.os.platform", customField: "jupiter_one.asset.properties.device_platform_name"},
		{ecsField: "user.name", customField: "jupiter_one.asset.properties.user_name"},
	} {
		SET(mapping.ecsField).COPY_FROM(mapping.customField).IGNORE_EMPTY(true).TAG(fmt.Sprintf("set %s from %s", mapping.ecsField, mapping.customField))
	}

	LOWERCASE("","log.level").IGNORE_MISSING(true).TAG("lowecase_log_level")

	for _, mapping := range []struct {
		ecsField, customField string
	}{
        {ecsField: "vulnerability.id", customField: "jupiter_one.asset.properties.cve_id"},
        {ecsField: "vulnerability.score.base", customField: "jupiter_one.asset.properties.score"},
        {ecsField: "vulnerability.severity", customField: "jupiter_one.asset.properties.severity"},
	} {
		SET(mapping.ecsField).COPY_FROM(mapping.customField).IGNORE_EMPTY(true).IF(`ctx.jupiter_one?.asset?.entity?._class?.contains('Vulnerability') == true`).TAG(fmt.Sprintf("set %s from %s", mapping.ecsField, mapping.customField))
	}

	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "threat.indicator.file.name", customField: "jupiter_one.asset.properties.filename"},
		{ecsField: "threat.indicator.file.path", customField: "jupiter_one.asset.properties.filepath"},
		{ecsField: "threat.indicator.ip", customField: "jupiter_one.asset.properties.device_external_ip"},
	} {
		SET(mapping.ecsField).COPY_FROM(mapping.customField).IGNORE_EMPTY(true).IF(`ctx.jupiter_one?.asset?.entity?._class?.contains('Finding') == true`).TAG(fmt.Sprintf("set %s from %s", mapping.ecsField, mapping.customField))
	}

	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "host.mac", customField: "jupiter_one.asset.properties.device_mac_address"},
	} {
		APPEND(mapping.ecsField, "{{{"+mapping.customField+"}}}").
			IF(safeNavigateAndCheck(mapping.customField) + " != null").
			ALLOW_DUPLICATES(false).TAG(fmt.Sprintf("append %s from %s", mapping.ecsField, mapping.customField))
	}

	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "vulnerability.category", customField: "jupiter_one.asset.properties.category"},
	} {
		APPEND(mapping.ecsField, "{{{"+mapping.customField+"}}}").
			IF(safeNavigateAndCheck(mapping.customField) + " != null" + " && " + "ctx.jupiter_one?.asset?.entity?._class?.contains('Vulnerability') == true").
			ALLOW_DUPLICATES(false).TAG(fmt.Sprintf("append %s from %s", mapping.ecsField, mapping.customField))
	}

	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "threat.technique.id", customField: "jupiter_one.asset.properties.technique_id"},
		{ecsField: "threat.tactic.id", customField: "jupiter_one.asset.properties.tactic_id"},
		{ecsField: "threat.tactic.name", customField: "jupiter_one.asset.properties.tactic"},
		{ecsField: "threat.technique.name", customField: "jupiter_one.asset.properties.technique"},
	} {
		APPEND(mapping.ecsField, "{{{"+mapping.customField+"}}}").
			IF(safeNavigateAndCheck(mapping.customField) + " != null" + " && " + "ctx.jupiter_one?.asset?.entity?._class?.contains('Finding') == true").
			ALLOW_DUPLICATES(false).TAG(fmt.Sprintf("append %s from %s", mapping.ecsField, mapping.customField))
	}

	SET("vulnerability.enumeration").
		VALUE("CVE").
		IF("ctx.jupiter_one?.asset?.properties?.cve_id != null").TAG("set_vulnerability_enumeration_from_cve_id")


	// Global on failure processor
	ON_FAILURE(
		APPEND("error.message", errorFormat),
		SET("event.kind").VALUE("pipeline_error"),
		APPEND("tags", "preserve_original_event").
			ALLOW_DUPLICATES(false),
	)

	// Generate the pipeline
	Generate()
}

@janvi-elastic janvi-elastic requested a review from a team as a code owner December 5, 2025 10:35
@andrewkroh andrewkroh added Crest Contributions from Crest developement team. New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Dec 5, 2025
@elastic-vault-github-plugin-prod

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link

💚 Build Succeeded

History

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Crest Contributions from Crest developement team. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. New Integration Issue or pull request for creating a new integration package.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants