ti_misp: add daily refetch option to threat attributes

packages/ti_misp/_dev/build/docs/README.md

@@ -23,8 +23,34 @@ This data stream uses the `/attributes/restSearch` API endpoint which returns mo
 #### Expiration of Indicators of Compromise (IOCs)
 The ingested IOCs expire after certain duration which is indicated by the `decayed` field. An [Elastic Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) is created to faciliate only active IOCs be available to the end users. This transform creates destination indices named `logs-ti_misp_latest.dest_threat_attributes-*` which only contains active and unexpired IOCs. The latest destination index also has an alias named `logs-ti_misp_latest.threat_attributes`. When querying for active indicators or setting up indicator match rules, only use the latest destination indices or the alias to avoid false positives from expired IOCs. Dashboards for `Threat Attributes` datastream are also pointing to the latest destination indices containing active IoCs. Please read [ILM Policy](#ilm-policy) below which is added to avoid unbounded growth on source datastream `.ds-logs-ti_misp.threat_attributes-*` indices.
 
+#### Daily Refresh Mode
+By default, the integration uses incremental updates, only fetching attributes that have been modified since the last poll (tracked via an internal cursor). However, MISP's decay scores are dynamic and decrease over time, which means an attribute's decay status may change without the attribute itself being modified. In such cases, incremental updates would not capture the updated decay state.
+
+To address this, users can enable the `Enable Daily Refresh` toggle. When enabled, the integration will:
+1. **Perform a daily full refresh**: Every 24 hours, the cursor is reset and all attributes from the configured `Initial Interval` are re-fetched from MISP.
+2. **Set 24-hour expiration**: Attributes ingested during a daily refresh will have their `decayed_at` set to 24 hours after ingestion, ensuring they expire before the next refresh cycle.
+3. **Update decay states**: The next daily refresh will re-ingest attributes with their current decay scores from MISP, removing any that have since been marked as decayed.
+
+This approach ensures that:
+- The destination indices stay aligned with MISP's current view of valid indicators
+- Attributes that become decayed in MISP are automatically removed in the next refresh cycle
+- No stale indicators remain in the destination indices beyond 24 hours
+
+**Note**: Daily refreshes will re-ingest all attributes within the `Initial Interval` window, which may result in higher data volume during the refresh period. The transform handles deduplication via unique keys. Attributes already marked as decayed by MISP's decay models during ingestion are not affected by the 24-hour expiration and will be removed immediately.
+
+#### IOC Expiration Duration
+The `IOC Expiration Duration` parameter controls when ingested IOCs are marked as expired when **Daily Refresh is disabled**. This setting applies to all ingested attributes that are not decayed, not just orphaned IOCs. The expiration date for each attribute is calculated as `max(last_seen, timestamp) + IOC Expiration Duration`, which defaults to 90 days.
+
+**Note**: When `Enable Daily Refresh` is enabled, this setting is ignored and all non-decayed attributes will expire 24 hours after ingestion instead. This ensures attributes are refreshed with current decay scores from MISP in the next daily cycle.
+
+When Daily Refresh is disabled, this setting serves as a fail-safe expiration mechanism that works independently of MISP's decay models. Even if MISP does not mark an attribute as decayed, Elastic will expire the attribute after the configured duration.
+
 #### Handling Orphaned IOCs
-Some IOCs may never get decayed/expired and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. To avoid any false positives from such orphaned IOCs, users are allowed to configure `IOC Expiration Duration` parameter while setting up the integration. This parameter deletes all data inside the destination indices `logs-ti_misp_latest.dest_threat_attributes-*` after this specified duration is reached, defaults to `90d` after attribute's `max(last_seen, timestamp)`. Note that `IOC Expiration Duration` parameter only exists to add a fail-safe default expiration in case IOCs never expire.
+Some IOCs may never get decayed/expired by MISP's decay models and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`.
+
+When `Enable Daily Refresh` is **disabled**, the `IOC Expiration Duration` parameter ensures these orphaned IOCs are eventually removed from destination indices after the specified duration from the attribute's `max(last_seen, timestamp)`.
+
+When `Enable Daily Refresh` is **enabled**, orphaned IOCs are handled automatically by the 24-hour expiration cycle. Each daily refresh re-ingests all attributes with their current decay state from MISP, ensuring the destination indices remain aligned with MISP's view of valid indicators.
 
 #### ILM Policy
 To facilitate IOC expiration, source datastream-backed indices `.ds-logs-ti_misp.threat_attributes-*` are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after `5 days` from ingested date. 

packages/ti_misp/changelog.yml

@@ -1,4 +1,12 @@
 # newer versions go on top
+- version: "1.40.0"
+  changes:
+    - description: Clarify documentation about the data retention policy.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/16491
+    - description: Add option to refresh ingested indicators daily.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/16491
 - version: "1.39.0"
   changes:
     - description: Prevent updating fleet health status to degraded when pagination completes.

packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refresh-config.yml

@@ -0,0 +1,15 @@
+input: httpjson
+service: misp
+vars: ~
+data_stream:
+  vars:
+    preserve_original_event: true
+    url: http://{{Hostname}}:{{Port}}
+    api_token: test
+    interval: 1s
+    initial_interval: 10s
+    enable_request_tracer: true
+    daily_refresh: true
+    ioc_expiration_duration: 5d
+assert:
+  hit_count: 10

packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs

@@ -46,6 +46,39 @@ request.transforms:
 - set:
     target: body.returnFormat
     value: json
+{{#if daily_refresh}}
+# Daily refresh mode: every 24 hours, reset cursor to re-fetch all attributes from initial_interval.
+# When a daily refresh occurs, the timestamp is reset to initial_interval and last_daily_refresh is updated.
+- set:
+    target: body.timestamp
+    value: |-
+      [[- $dailyRefreshDuration := (parseDuration "24h") -]]
+      [[- $initialInterval := (parseDuration "-{{initial_interval}}") -]]
+      [[- $initialTimestamp := (now $initialInterval).Unix -]]
+      [[- $isDailyRefresh := true -]]
+      [[- if index .cursor "last_daily_refresh" -]]
+        [[- $lastRefresh := (parseDate .cursor.last_daily_refresh "RFC3339") -]]
+        [[- $nextRefresh := 0 -]]
+        [[- with $lastRefresh -]]
+          [[- $nextRefresh = .Add $dailyRefreshDuration -]]
+        [[- end -]]
+        [[- with $nextRefresh -]]
+          [[- if .Before now -]]
+            [[- $isDailyRefresh = true -]]
+          [[- else -]]
+            [[- $isDailyRefresh = false -]]
+          [[- end -]]
+        [[- end -]]
+      [[- end -]]
+      [[- if $isDailyRefresh -]]
+        [[- $initialTimestamp -]]
+      [[- else if index .cursor "timestamp" -]]
+        [[- .cursor.timestamp -]]
+      [[- else -]]
+        [[- $initialTimestamp -]]
+      [[- end -]]
+    default: '[[ (now (parseDuration "-{{initial_interval}}")).Unix ]]'
+{{else}}
 - set:
     target: body.timestamp
     value: >-
@@ -55,6 +88,7 @@ request.transforms:
         [[- .last_response.url.params.Get "timestamp" -]]
       [[- end -]]
     default: '[[ (now (parseDuration "-{{initial_interval}}")).Unix ]]'
+{{/if}}
 - set:
     # Ignored by MISP, set as a workaround to make it available in response.pagination.
     target: url.params.timestamp
@@ -87,10 +121,35 @@ response.pagination:
 cursor:
   timestamp:
     value: '[[.last_event.timestamp]]'
+{{#if daily_refresh}}
+  last_daily_refresh:
+    # Track when the last daily refresh started. Updated when 24h has passed since previous refresh.
+    value: |-
+      [[- $dailyRefreshDuration := (parseDuration "24h") -]]
+      [[- if index .cursor "last_daily_refresh" -]]
+        [[- $lastRefresh := (parseDate .cursor.last_daily_refresh "RFC3339") -]]
+        [[- $nextRefresh := 0 -]]
+        [[- with $lastRefresh -]]
+          [[- $nextRefresh = .Add $dailyRefreshDuration -]]
+        [[- end -]]
+        [[- with $nextRefresh -]]
+          [[- if .Before now -]]
+            [[- formatDate now "RFC3339" -]]
+          [[- else -]]
+            [[- $.cursor.last_daily_refresh -]]
+          [[- end -]]
+        [[- end -]]
+      [[- else -]]
+        [[- formatDate now "RFC3339" -]]
+      [[- end -]]
+{{/if}}
 tags:
 {{#if preserve_original_event}}
   - preserve_original_event
 {{/if}}
+{{#if daily_refresh}}
+  - daily_refresh
+{{/if}}
 {{#each tags as |tag i|}}
   - {{tag}}
 {{/each}}

packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml

@@ -148,10 +148,11 @@ processors:
 # Add default decayed_at to expire all IOCs after duration `ioc_expiration_duration` from `_tmp_max_time`. 
 # If user-provided value of `ioc_expiration_duration` is not in d, h, or m, default to 90d.
 # Set `decayed` as true if decayed_at is before ingest time i.e., already expired during ingestion.
+# This script is skipped when daily_refresh tag is present.
   - script:
       lang: painless
       tag: script-default-decayed_at
-      if: (ctx.misp?.attribute?.decayed_at == null && ctx._conf?.ioc_expiration_duration != null && ctx._conf.ioc_expiration_duration != '')
+      if: (ctx.misp?.attribute?.decayed_at == null && ctx._conf?.ioc_expiration_duration != null && ctx._conf.ioc_expiration_duration != '' && !(ctx.tags instanceof List && ctx.tags.contains('daily_refresh')))
       description: >
         Add default decayed_at
       source: >
@@ -195,6 +196,24 @@ processors:
         - append:
             field: error.message
             value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
+# Script for daily refresh mode: set decayed_at to 24 hours after ingestion.
+# When daily_refresh tag is present, all non-decayed attributes will expire 24 hours after ingestion,
+# allowing the next daily refresh cycle to re-ingest them with updated decay scores from MISP.
+# Attributes already marked as decayed by MISP (script-misp_decayed) are not affected.
+  - script:
+      lang: painless
+      tag: script-daily_refresh-decayed_at
+      if: (ctx.misp?.attribute?.decayed != true && ctx.misp?.attribute?.decayed_at == null && ctx.tags instanceof List && ctx.tags.contains('daily_refresh'))
+      description: >
+        Set decayed_at to 24 hours after ingestion for daily refresh mode
+      source: |
+        ZonedDateTime _tmp_event_ingested = ZonedDateTime.parse(ctx._tmp.event_ingested);
+        ctx.misp.attribute.decayed_at = _tmp_event_ingested.plusHours(24L);
+        ctx.tags.remove(ctx.tags.indexOf('daily_refresh'));
+      on_failure:
+        - append:
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
   - date:
       if: ctx.misp?.event?.timestamp != null
       field: misp.event.timestamp

packages/ti_misp/data_stream/threat_attributes/manifest.yml

@@ -35,6 +35,15 @@ streams:
         show_user: true
         default: 120h
         description: How far back to look for indicators the first time the agent is started. Supported units for this parameter are h/m/s.
+      - name: daily_refresh
+        type: bool
+        title: Enable Daily Refresh
+        multi: false
+        required: false
+        show_user: true
+        default: false
+        description: >-
+          When enabled, the integration performs a daily full refresh of all attributes from the MISP API (every 24 hours), ignoring the cursor and re-fetching from Initial Interval. This ensures decay scores are updated and attributes ingested during a daily refresh will expire 24 hours after ingestion, allowing the next refresh cycle to update their decay state from MISP.
       - name: ioc_expiration_duration
         type: text
         title: IOC Expiration Duration
@@ -43,7 +52,7 @@ streams:
         show_user: true
         default: "90d"
         description: >-
-          Enforces all IOCs to expire after this duration. This setting is required to avoid "orphaned" IOCs that never expire. Use [Elasticsearch time units](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#time-units) in days, hours, or minutes (e.g 10d)
+          Enforces all IOCs to expire after this duration. This setting applies to ALL ingested attributes (not just orphaned IOCs) and serves as a fail-safe expiration for "orphaned" IOCs that never expire. Use [Elasticsearch time units](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#time-units) in days, hours, or minutes (e.g 10d). When `Enable Daily Refresh` is enabled, this setting is ignored and all non-decayed attributes will expire 24 hours after ingestion instead.
       - name: http_client_timeout
         type: text
         title: HTTP Client Timeout

packages/ti_misp/data_stream/threat_attributes/sample_event.json

@@ -1,34 +1,35 @@
 {
     "@timestamp": "2014-10-03T07:14:05.000Z",
     "agent": {
-        "ephemeral_id": "98efca5d-4e4c-4bab-b557-dccd2aa01ed0",
-        "id": "b20dde43-9229-4544-be2f-fc8d8a4f5450",
-        "name": "elastic-agent-78638",
+        "ephemeral_id": "f755233a-3146-44ed-8822-7de70dab3b39",
+        "id": "7b07b3d9-db6c-4091-a7c4-bbbd22fc8bfe",
+        "name": "elastic-agent-44381",
         "type": "filebeat",
-        "version": "8.19.4"
+        "version": "9.2.1"
     },
     "data_stream": {
         "dataset": "ti_misp.threat_attributes",
-        "namespace": "20988",
+        "namespace": "23477",
         "type": "logs"
     },
     "ecs": {
         "version": "8.11.0"
     },
     "elastic_agent": {
-        "id": "b20dde43-9229-4544-be2f-fc8d8a4f5450",
+        "id": "7b07b3d9-db6c-4091-a7c4-bbbd22fc8bfe",
         "snapshot": false,
-        "version": "8.19.4"
+        "version": "9.2.1"
     },
     "event": {
         "agent_id_status": "verified",
         "category": [
             "threat"
         ],
-        "created": "2025-12-03T07:52:43.898Z",
+        "created": "2025-12-03T17:32:54.494Z",
         "dataset": "ti_misp.threat_attributes",
-        "ingested": "2025-12-03T07:52:46Z",
+        "ingested": "2025-12-03T17:32:57Z",
         "kind": "enrichment",
+        "module": "ti_misp",
         "original": "{\"Event\":{\"distribution\":\"3\",\"id\":\"1\",\"info\":\"OSINT ShellShock scanning IPs from OpenDNS\",\"org_id\":\"1\",\"orgc_id\":\"2\",\"uuid\":\"542e4c9c-cadc-4f8f-bb11-6d13950d210b\"},\"category\":\"External analysis\",\"comment\":\"\",\"deleted\":false,\"disable_correlation\":false,\"distribution\":\"5\",\"event_id\":\"1\",\"first_seen\":null,\"id\":\"1\",\"last_seen\":null,\"object_id\":\"0\",\"object_relation\":null,\"sharing_group_id\":\"0\",\"timestamp\":\"1412320445\",\"to_ids\":false,\"type\":\"link\",\"uuid\":\"542e4cbd-ee78-4a57-bfb8-1fda950d210b\",\"value\":\"http://labs.opendns.com/2014/10/02/opendns-and-bash/\"}",
         "type": [
             "indicator"
@@ -37,6 +38,9 @@
     "input": {
         "type": "httpjson"
     },
+    "labels": {
+        "is_ioc_transform_source": "true"
+    },
     "misp": {
         "attribute": {
             "category": "External analysis",
@@ -72,6 +76,7 @@
     ],
     "threat": {
         "feed": {
+            "dashboard_id": "ti_misp-56ed8040-6c7d-11ec-9bce-f7a4dc94c294",
             "name": "MISP"
         },
         "indicator": {