Skip to content

Commit d917e3a

Browse files
sspainksspaink
authored
plugin/decision: check if event is too large after compression (#7521)
`upload_size_limit_bytes` now only represents the compressed upload limit. Dropping the ND cache only happens if the event is too large after compression. Signed-off-by: sspaink <[email protected]>
1 parent 817b663 commit d917e3a

File tree

8 files changed

+874
-419
lines changed

8 files changed

+874
-419
lines changed

‎docs/docs/configuration.md‎

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -828,24 +828,24 @@ included in the actual bundle gzipped tarball.
828828

829829
## Decision Logs
830830

831-
| Field | Type | Required | Description |
832-
| -------------------------------------------------- | --------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
833-
| `decision_logs.service` | `string` | No | Name of the service to use to contact remote server. If no `plugin` is specified, and `console` logging is disabled, this will default to the first `service` name defined in the Services configuration. |
834-
| `decision_logs.partition_name` | `string` | No | Deprecated: Use `resource` instead. Path segment to include in status updates. |
835-
| `decision_logs.resource` | `string` | No (default: `/logs`) | Full path to use for sending decision logs to a remote server. |
836-
| `decision_logs.reporting.buffer_type` | `string` | No (default: `size`) | Toggles the type of buffer to use. The two available options are "size" or "event". Refer to the [Decision Log Plugin README](https://github.com/open-policy-agent/opa/tree/main/v1/plugins/logs/README.md) for for a detailed comparison. |
837-
| `decision_logs.reporting.buffer_size_limit_events` | `int64` | No (default: `10000`) | Decision log buffer size limit by events. OPA will drop old events from the log if this limit is exceeded. By default, 100 events are held. This number has to be greater than zero. Only works with "event" buffer type. |
838-
| `decision_logs.reporting.buffer_size_limit_bytes` | `int64` | No (default: `unlimited`) | Decision log buffer size limit in bytes. OPA will drop old events from the log if this limit is exceeded. By default, no limit is set. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. Only works with "size" buffer type. |
839-
| `decision_logs.reporting.max_decisions_per_second` | `float64` | No | Maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. |
840-
| `decision_logs.reporting.upload_size_limit_bytes` | `int64` | No (default: `32768`) | Decision log upload size limit in bytes. OPA will chunk uploads to cap message body to this limit. |
841-
| `decision_logs.reporting.min_delay_seconds` | `int64` | No (default: `300`) | Minimum amount of time to wait between uploads. |
842-
| `decision_logs.reporting.max_delay_seconds` | `int64` | No (default: `600`) | Maximum amount of time to wait between uploads. |
843-
| `decision_logs.reporting.trigger` | `string` | No (default: `periodic`) | Controls how decision logs are reported to the remote server. Allowed values are `periodic` and `manual` (`manual` triggers are only possible when using OPA as a Go package). |
844-
| `decision_logs.mask_decision` | `string` | No (default: `/system/log/mask`) | Set path of masking decision. |
845-
| `decision_logs.drop_decision` | `string` | No (default: `/system/log/drop`) | Set path of drop decision. |
846-
| `decision_logs.plugin` | `string` | No | Use the named plugin for decision logging. If this field exists, the other configuration fields are not required. |
847-
| `decision_logs.console` | `boolean` | No (default: `false`) | Log the decisions locally to the console. When enabled alongside a remote decision logging API the `service` must be configured, the default `service` selection will be disabled. |
848-
| `decision_logs.request_context.http.headers` | `array` | No | List of HTTP headers to include in the decision log. OPA will include the values for these headers in the decision log if they exist in the incoming HTTP request. |
831+
| Field | Type | Required | Description |
832+
|----------------------------------------------------|-----------|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
833+
| `decision_logs.service` | `string` | No | Name of the service to use to contact remote server. If no `plugin` is specified, and `console` logging is disabled, this will default to the first `service` name defined in the Services configuration. |
834+
| `decision_logs.partition_name` | `string` | No | Deprecated: Use `resource` instead. Path segment to include in status updates. |
835+
| `decision_logs.resource` | `string` | No (default: `/logs`) | Full path to use for sending decision logs to a remote server. |
836+
| `decision_logs.reporting.buffer_type` | `string` | No (default: `size`) | Toggles the type of buffer to use. The two available options are "size" or "event". Refer to the [Decision Log Plugin README](https://github.com/open-policy-agent/opa/tree/main/v1/plugins/logs/README.md) for for a detailed comparison. |
837+
| `decision_logs.reporting.buffer_size_limit_events` | `int64` | No (default: `10000`) | Decision log buffer size limit by events. OPA will drop old events from the log if this limit is exceeded. By default, 100 events are held. This number has to be greater than zero. Only works with "event" buffer type. |
838+
| `decision_logs.reporting.buffer_size_limit_bytes` | `int64` | No (default: `unlimited`) | Decision log buffer size limit in bytes. OPA will drop old events from the log if this limit is exceeded. By default, no limit is set. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. Only works with "size" buffer type. |
839+
| `decision_logs.reporting.max_decisions_per_second` | `float64` | No | Maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. |
840+
| `decision_logs.reporting.upload_size_limit_bytes` | `int64` | No (default: `32768`) | Decision log upload size limit in bytes. This limit enforces the maximum size of a gzip compressed payload of events within the message body. |
841+
| `decision_logs.reporting.min_delay_seconds` | `int64` | No (default: `300`) | Minimum amount of time to wait between uploads. |
842+
| `decision_logs.reporting.max_delay_seconds` | `int64` | No (default: `600`) | Maximum amount of time to wait between uploads. |
843+
| `decision_logs.reporting.trigger` | `string` | No (default: `periodic`) | Controls how decision logs are reported to the remote server. Allowed values are `periodic` and `manual` (`manual` triggers are only possible when using OPA as a Go package). |
844+
| `decision_logs.mask_decision` | `string` | No (default: `/system/log/mask`) | Set path of masking decision. |
845+
| `decision_logs.drop_decision` | `string` | No (default: `/system/log/drop`) | Set path of drop decision. |
846+
| `decision_logs.plugin` | `string` | No | Use the named plugin for decision logging. If this field exists, the other configuration fields are not required. |
847+
| `decision_logs.console` | `boolean` | No (default: `false`) | Log the decisions locally to the console. When enabled alongside a remote decision logging API the `service` must be configured, the default `service` selection will be disabled. |
848+
| `decision_logs.request_context.http.headers` | `array` | No | List of HTTP headers to include in the decision log. OPA will include the values for these headers in the decision log if they exist in the incoming HTTP request. |
849849

850850
## Discovery
851851

‎docs/docs/management-decision-logs.md‎

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -87,18 +87,22 @@ during the next upload event. OPA also performs an exponential backoff to calcul
8787
when the remote service responds with a non-2xx status.
8888

8989
OPA periodically uploads decision logs to the remote service. In order to conserve network and memory resources, OPA
90-
attempts to fill up each upload chunk with as many events as possible while respecting the user-specified
91-
`upload_size_limit_bytes` config option. OPA defines an adaptive (`soft`) limit that acts as a measure for encoding
92-
as many decisions into each chunk as possible. It uses the below algorithm to optimize the number of log events to
93-
include in a chunk. The algorithm features three phases namely:
94-
95-
`Scale Up`: If the current chunk size is within 90% of the user-configured (`hard`) limit, exponentially increase the
96-
soft limit. The exponential function is 2^x where x has a minimum value of 1
97-
98-
`Scale Down`: If the current chunk size exceeds the hard limit, decrease the soft limit and re-encode the decisions in
99-
the last chunk.
100-
101-
`Equilibrium`: If the chunk size is between 90% and 100% of the user-configured limit, maintain soft limit value.
90+
attempts to fill up each message body with as many events as possible while respecting the user-specified
91+
`upload_size_limit_bytes` config option. Each message body is a gzip compressed JSON array and the `upload_size_limit_bytes`
92+
config option represents the gzip compressed size, it can be referred to as the compressed limit. To avoid compressing
93+
each incoming event to get its compressed size to see if the compressed limit is reached, OPA tries to make an educated
94+
guess what the uncompressed limit could be. It does so by using an adaptive limit, referred to as the uncompressed limit,
95+
that gets adjusted by measuring incoming events. This does mean that initially the chunk sizes will most likely be smaller
96+
than the compressed limit, but as OPA consumes more decision events it will adjust the adaptive uncompressed limit to
97+
optimize the messages. The algorithm to adjust the uncompressed limit uses the following criteria:
98+
99+
`Scale Up`: If the current chunk size is below 90% of the user-configured compressed limit, exponentially increase the
100+
uncompressed limit. The exponential function is 2^x where x has a minimum value of 1
101+
102+
`Scale Down`: If the current chunk size exceeds the compressed limit, decrease the uncompressed limit and re-encode the
103+
decisions in the last chunk.
104+
105+
`Equilibrium`: If the chunk size is between 90% and 100% of the user-configured limit, maintain uncompressed limit value.
102106

103107
When an event containing `nd_builtin_cache` cannot fit into a chunk smaller than `upload_size_limit_bytes`, OPA will
104108
drop the `nd_builtin_cache` key from the event, and will retry encoding the chunk without the non-deterministic

0 commit comments

Comments
 (0)