v1.17.8

• Added experimental support for partial blocks (ex: Base's Flash Blocks) -- only supported on https://base-mainnet-flash.streamingfast.io endpoint
• See more details in the documentation

CLI

• commands run and sink webhook now support these flags:
  • --include-partial-blocks: sends every block as partial(s) but also as real block
  • --partial-blocks-only: only sends partials (for every block, every bit of data should be there)

Sink library

• To use the new partial blocks in a sink that uses github.com/streamingfast/substreams/sink, simply:
  • Define your sink flags with sink.FlagIncludePartialBlocks and/or sink.FlagPartialBlocksOnly under FlagIncludeOptional()
  • Implement the function HandlePartialBlockData(...) and pass it to NewSinkerFullHandlersWithPartial(...) when creating the sinker.

Server

• Server accepts new include_partial_blocks and partial_blocks_only boolean params in the request body.
• Response, when requested with above params, now include new message PartialBlockData, containing the usual "map module output", the clock and the index of that partial.
• Server accepts environment variable SUBSTREAMS_BIGGEST_PARTIAL_BLOCK_INDEX (default 10) -- it will emit a partial with this index when it gets the final part of a block.

v1.17.8-partialblocks

[This is an alpha release]

• Added experimental support for partial blocks (ex: flash blocks) -- only supported on https://base-mainnet-flash.streamingfast.io endpoint

[CLI]

• commands run and sink webhook now support these flags:
  • --include-partial-blocks: sends every block as partial(s) but also as real block
  • --partial-blocks-only: only sends partials (for every block, every bit of data should be there)

v1.17.5

CLI

• substreams GUI: fix setting (only) the start block with a relative value: will now change the default stop block from +1000 to 0
  instead of returning the error relative end block is supported only with an absolute start block

v1.17.4

CLI

• Fix handling of relative stop block (regression) in GUI, RUN and other sinks.
• Fix GUI handling of substreams that are not built yet.

v1.17.3

CLI

• Fix running from manifests that were created with tools unpack and contain imported modules, preventing "failed to get entrypoint" error.

Server

• Added opt-in memory limits related to loading FullKV stores, gated by environment variables:

  • "SUBSTREAMS_STORE_SIZE_LIMIT_PER_REQUEST" (default allows 5GiB: 5368709120): limit size of all loaded stores for a single request, in bytes. Set to a numeric value in bytes.
  • "SUBSTREAMS_ENFORCE_STORE_SIZE_LIMIT_PER_REQUEST" (default false): if set to true, enforce the limit above instead of just logging a warning
  • "SUBSTREAMS_TOTAL_STORE_SIZE_LIMIT_PERCENT" (default: 75): limit the size in-memory of all loaded stores concurrently on the instance, in percentage of usable memory (cgroup or system total -- regardless of free or available)
  • "SUBSTREAMS_ENFORCE_TOTAL_STORE_SIZE_LIMIT" (default: false): if set to true, enforce the limit above instead of just logging a warning

• Fixed an edge case where substreams with modules depending on stores that start on the future would fail and incorrectly report an error about "tier2 version being incompatible"

v1.17.2

Server

• Reduced memory usage associated with reading and writing large stores by streaming the marshalling process. (~45% reduction peak usage)
• New environment variable SUBSTREAMS_TIER1_DEBUG_API_ADDR now enables the debug API on the tier1 service.
• Renamed environment variable SUBSTREAMS_DEBUG_API_ADDR to SUBSTREAMS_TIER2_DEBUG_API_ADDR, since it only affected tier2.

CLI

• Improved substreams init to expand ~ and environment variable when resolving local file path.
• Improved substreams init reporting of error when local file cannot be uploaded/read correctly.
• Improved substreams init rendering of list items and some other elements.
• Fixed substreams init to correctly showing selected label when selecting from a list of items.
• Fixed substreams protogen when params are defined on networks for imported modules that are not dependencies to any local module

v1.17.1

CLI

• Fixed substreams publish alias command not fully aligned with substreams registry publish "main" version.

v1.17.0

New sf.substreams.rpc.v3.Stream/Blocks endpoint

• This new endpoint removes the need for complex "mangling" of the package on the client side.
• Instead of expecting sf.substreams.v1.Modules (with the client having to apply parameters, network, etc.), the sf.substreams.rpc.v3.Request now expects:
  • a sf.substreams.v1.Package.
  • a map<string, string> of params
  • the network string
    which will all be applied to the package server-side.
• It returns the same object as the v2 endpoint, i.e. a stream of sf.substreams.rpc.v2.Response

Server

• Watch for releases for firehose-core, firehose-ethereum, etc. to include this new endpoint.
• It is added on top of the existing 'v2' endpoint, both being active at the same time.
• To enable it, operators will simply need to ensure that their routing allows the /sf.substreams.rpc.v3.Stream/* path.
• Cached spkg on the server will now contain protobuf definitions, simplifying debugging of user requests.
• Emitted metrics for requests can now be sf.substreams.rpc.v3/Blocks instead of always sf.substreams.rpc.v2/Blocks, make sure that your metering endpoint can support it.

Clients

• The clients provided in this release (substreams run, gui and sinks that are linked to the 'client' library) will now support both endpoints.
• Without any flag, they will use the v3 endpoint by default and automatically fallback to v2 if they hit a "404 Not Found" or "Not Implemented" error.
• The --force-protocol-version has been added to all clients. Set it to 2 to force only v2, set it to 3 to force only v3, or leave it unset (0) to use the default "v3 with fallback to v2"

Build

• Added filesystem-backed caching for Buf BSR API requests to improve build performance and prevent rate limit errors. Cache uses SHA256 keys based on module/version/symbols, stores to ~/.config/substreams/buf-cache/, and only caches deterministic semver versions. Falls back to in-memory cache if filesystem unavailable. Warns when descriptor sets lack version specifications, as these cannot be cached and may cause rate limit issues.
• Added support for @version notation in protobuf.descriptorSets section of manifest. You can now specify versions in multiple ways:
  • Separate fields: module: buf.build/streamingfast/substreams-sink-sql
  • Separate fields with explicit latest: module: buf.build/streamingfast/substreams-sink-sql with version: latest
  • Inline notation: module: buf.build/streamingfast/[email protected]
  • Note: @latest inline notation is not allowed; use version: latest or omit the version instead

Bug fixes

• Fixed a bug with BlockFilter: a skipped module would send BlockScopedData (in dev or near HEAD, to follow progress) with an empty module name, breaking some sinks. Module name was present if requesting a module dependent on that skipped module. Now the module name is always included.
• Fix "max-retries" so that it only 'resets' the counter if it receives actual data.

v1.16.6

Server

• Updated Wasmtime runtime from v30.0.0 to v36.0.0, bringing performance improvements, inlining support, Component Model async implementation, and enhanced security features.
• Added WASM bindgen shims support for Wasmtime runtime to handle WASM modules with WASM bindgen imports (when Substreams Module binary is defined as type wasm/rust-v1+wasm-bindgen-shims).

Server and Client

• Added support for foundational-store (in wasmtime and wazero).
• Added foundational-store grpc client to substreams engine.
• Fixed module caching to properly handle modules with different runtime extensions.

CLI

• Added support for http:// and https:// prefixes in the --endpoint flag. Setting the protocol (http/https) in the URL will ignore the --plaintext flag setting. The default (no prefix) is still SSL. The enforcing of --plaintext and --insecure has been relaxed: plaintext+insecure is simply plaintext.
• Fixed the progress logs and prometheus metrics from substreams sink noop when running with an output_module of type "index" in production mode (other sinks will now refuse to run in this mode)
• Removed 'progress_last_contiguous_block' from sink logs, as it was often misleading. Getting a correct value in all cases would require doing a slow lookup on all cached files, which is not desirable.

v1.16.5

Server

Session (stream + workers management)

• BREAKING Concurrent streams and workers limits are now handled under the new session plugin (see CHANGELOG in github.com/streamingfast/firehose-core for details and usage)
  • removed 'WorkerPoolFactory' from Tier1Modules
  • removed 'GlobalRequestPool' from Tier1Modules
  • added 'SessionPool' (dsession.SessionPool) to Tier1Modules

Stability

• BREAKING Add a maximum execution time for a full tier2 segment. By default, this is 60 minutes. It will fail with rpc error: code = DeadlineExceeded desc = request active for too long.
  It can be configured from the SegmentExecutionTimeout configuration option on Tier2Config or disabled by setting it to 0.
• Improve log message for 'request active for a long time', adding stats.
• Fix subscription channel at max capacity error: when the LIVE channel is full (ex: slow module execution or slow client reader), the request will be continued from merged files instead of failing, and gracefully recover if performance is restored.
• Fixed a small context memory leak when using wasmtime (especially with grpc-based metering plugin)

CLI

• BREAKING: Replaced --infinite-retry boolean flag with --max-retries integer flag in sink package for more flexible retry control:
  • --max-retries 0: No retries (fail immediately on first error)
  • --max-retries 3: Default behavior (retry up to 3 times)
  • --max-retries -1: Infinite retries (equivalent to old --infinite-retry flag)