wevt_templates: switch offline cache to single .wevtcache file

.cursor/commands/fix-github-issue.md

@@ -0,0 +1,14 @@
+Use github cli to fetch the issue details and description.
+
+# Important
+
+do NOT respond to the issue with a comment/close it without asking for confirmation first.
+
+Look at the issue - our first goal is to understand if it is a bug or a feature request.
+
+   - If it is a bug, we need to reproduce the issue and then fix it.
+       - It must have sufficient information to reproduce the issue, or some link to a sample file we can parse.
+       - If it does not have sufficient information, we need to ask the user for more information.
+    - If it is a feature request - we need to consider if the feature is reasonable.
+        - Users ask a lot of the times for features which could otherwise have been implemented by chaining existing tools, like piping evtx to jq for filtering.
+        - Suggest a plan of how to implement the feature, and if it is accepted, implement it.

.cursorignore

@@ -0,0 +1 @@
+!/external

.gitattributes

@@ -0,0 +1,2 @@
+samples/dlls/*.dll filter=lfs diff=lfs merge=lfs -text
+samples/dlls/*.exe filter=lfs diff=lfs merge=lfs -text

.github/workflows/test.yml

@@ -13,6 +13,7 @@ jobs:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 1
+          lfs: true
       - uses: dtolnay/rust-toolchain@stable
       - uses: Swatinem/rust-cache@v2
       - name: Run tests

CHANGELOG.md

@@ -6,6 +6,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.11.0 - 2026-01-03]
+
+### Breaking changes (WEVT_TEMPLATE cache)
+- The offline WEVT template cache is now a **single `.wevtcache` file** (directory + `index.jsonl` is no longer supported).
+  - `evtx_dump extract-wevt-templates --input <provider.{dll,exe,sys}> --output /tmp/wevt_cache.wevtcache --overwrite`
+  - `evtx_dump --wevt-cache /tmp/wevt_cache.wevtcache <log.evtx>`
+- `WevtCache` is now **pure in-memory** (no internal filesystem I/O). Load cache blobs at your boundary and pass an `Arc<WevtCache>` into `ParserSettings`.
+- CLI flag renames:
+  - `--wevt-cache-index` → `--wevt-cache`
+  - `apply-wevt-cache --cache-index` → `apply-wevt-cache --cache`
+
+### Fixed
+- Fix MAPS parsing for providers whose MAPS offsets are not monotonic (e.g. `wevtsvc.dll`).
+  - Parse maps deterministically (implied first map + offsets array order).
+  - Use each VMAP’s declared `size` field (no “next offset” boundary guessing / sorting).
+
+### Added
+- Git-LFS tracked DLL/EXE fixtures under `samples/dlls/` + insta snapshot tests for canonical WEVT_TEMPLATE extraction stats (validated against libfwevt via pyfwevt).
+
+**Full Changelog**: [`v0.10.0...v0.11.0`](https://github.com/omerbenamram/evtx/compare/v0.10.0...v0.11.0)
+
 ## [0.10.0 - 2025-12-31]
 
 ### Highlights

Cargo.toml

@@ -6,7 +6,7 @@ repository = "https://github.com/omerbenamram/EVTX"
 license = "MIT/Apache-2.0"
 readme = "README.md"
 
-version = "0.10.0"
+version = "0.11.0"
 authors = ["Omer Ben-Amram <omerbenamram@gmail.com>"]
 edition = "2024"
 

Makefile

@@ -12,3 +12,10 @@ flamegraph-prod:
 	BIN="$(BIN)" FLAME_FILE="$(FLAME_FILE)" FORMAT="$(FORMAT)" DURATION="$(DURATION)" \
 		bash scripts/flamegraph_prod.sh
 
+# Generate Python stub files (.pyi) for the evtx Python module
+# The stub_gen binary must be built WITHOUT extension-module feature
+# to allow linking against Python
+.PHONY: pyi-stubs
+pyi-stubs:
+	cd external/pyevtx-rs && cargo run --bin stub_gen --no-default-features --features wevt_templates
+

README.md

@@ -95,16 +95,16 @@ EVTX records can reference template definitions stored in provider binaries (EXE
 
 **Note:** this functionality requires building `evtx_dump` with the Cargo feature `wevt_templates` (release binaries may already include it).
 
-- Build a cache (writes extracted blobs under `/tmp/wevt_cache/` and emits an index JSONL on stdout):
-  - `evtx_dump extract-wevt-templates --input <provider.dll> --output-dir /tmp/wevt_cache --overwrite > /tmp/wevt_cache/index.jsonl`
+- Build a cache (single portable `.wevtcache` file):
+  - `evtx_dump extract-wevt-templates --input <provider.dll> --output /tmp/wevt_cache.wevtcache --overwrite`
 - Dump an EVTX file while using the cache (deterministic rule: only applies when a record fails due to an explicit missing/corrupt template GUID):
-  - `evtx_dump --wevt-cache-index /tmp/wevt_cache/index.jsonl <log.evtx>`
+  - `evtx_dump --wevt-cache /tmp/wevt_cache.wevtcache <log.evtx>`
 
 Debugging helpers:
 - Dump a record’s `TemplateInstance` substitution values (JSONL):
   - `evtx_dump dump-template-instances --input <log.evtx> --record-id <ID> | head -n1`
 - Render a specific template GUID with substitutions (XML to stdout):
-  - `evtx_dump apply-wevt-cache --cache-index /tmp/wevt_cache/index.jsonl --template-guid <GUID> --evtx <log.evtx> --record-id <ID>`
+  - `evtx_dump apply-wevt-cache --cache /tmp/wevt_cache.wevtcache --template-guid <GUID> --evtx <log.evtx> --record-id <ID>`
 
 See [`docs/wevt_templates.md`](docs/wevt_templates.md) for details and background (issue #103).
 

docs/wevt_templates.md

@@ -123,66 +123,46 @@ cargo run --release --features wevt_templates --bin evtx_dump -- \
   extract-wevt-templates --help
 ```
 
-Example using the public `services.exe` sample (stored as a `.gif` in this repo):
+Example using the committed `services.exe` fixture (tracked via git-lfs):
 
 ```bash
 cargo run --release --features wevt_templates --bin evtx_dump -- \
   extract-wevt-templates \
-  --input samples_local/services.exe.gif \
-  --output-dir /tmp/wevt_cache \
-  --overwrite \
-  --split-ttbl \
-  --dump-temp-xml \
-  --dump-events \
-  --dump-items \
-  > /tmp/wevt_cache/index.jsonl
+  --input samples/dlls/services.exe \
+  --output /tmp/wevt_cache.wevtcache \
+  --overwrite
 ```
 
 What you get:
 
-- `/tmp/wevt_cache/*.bin`: raw `WEVT_TEMPLATE` resource blobs (CRIM payloads)
-- `/tmp/wevt_cache/temp/*.bin`: raw `TEMP` slices
-- `/tmp/wevt_cache/temp_xml/*.xml`: rendered template XML skeletons
-- `/tmp/wevt_cache/index.jsonl`: JSONL describing resources, events, template GUIDs, and template items
+- `/tmp/wevt_cache.wevtcache`: a single portable cache file containing raw `WEVT_TEMPLATE` resource blobs (CRIM payloads)
 
-### 2) Look up a template GUID for an event (offline join)
+Notes:
 
-Assuming you know:
+- You **do not** need the original provider DLL/EXE once the cache is built.
+- To move the cache across machines/OSes, copy the `.wevtcache` file.
 
-- `provider_guid` (from the record’s `<System><Provider Guid="...">`)
-- `event_id` and `version` (from the record’s `<System><EventID>` and version field)
+### 2) Use the cache
 
-You can find the template GUID from the JSONL:
+Use the cache as a fallback when dumping EVTX (only used when embedded templates are missing/corrupt).
+
+This is especially useful for **DFIR / carving** workflows, where you may have:
+- Records reconstructed from raw disk/memory without their original embedded templates
+- Logs copied to an analysis workstation without the original provider binaries
 
 ```bash
-jq -r '
-  select(has("provider_guid")) |
-  select(.provider_guid=="{PROVIDER_GUID}" and .event_id=={EVENT_ID} and .version=={VERSION}) |
-  .template_guid
-' /tmp/wevt_cache/index.jsonl | head -n1
+evtx_dump --wevt-cache /tmp/wevt_cache.wevtcache /path/to/log.evtx
 ```
 
-Then locate the corresponding rendered template XML skeleton:
+You can also render a single record/template offline using the cache + substitution values:
 
 ```bash
-jq -r '
-  select(has("output_path") and (.output_path|endswith(".xml"))) |
-  select(.guid=="{TEMPLATE_GUID}") |
-  .output_path
-' /tmp/wevt_cache/index.jsonl | head -n1
+evtx_dump apply-wevt-cache \
+  --cache /tmp/wevt_cache.wevtcache \
+  --evtx /path/to/log.evtx \
+  --record-id 12345
 ```
 
-### 3) Apply it to a carved record (what remains)
-
-The cache solves the hard part: **offline extraction and parsing of provider templates**, plus **stable joins**.
-
-To fully render a carved record end-to-end you still need the record’s **substitution values array** (the `{sub:N}` values). Once you have those values (from the record’s BinXML TemplateInstance data), you can:
-
-- pick the template (by GUID or by event→template join)
-- substitute `{sub:N(:Name)?}` slots with actual values (with proper escaping)
-
-This last “apply substitutions” step is not yet wired as a single CLI command, but the format pieces are now in place to build it cleanly without heuristics.
-
 ## Implementation map (where to read the code)
 
 - Template extraction + CLI wiring:\n
@@ -203,11 +183,11 @@ This last “apply substitutions” step is not yet wired as a single CLI comman
 
 ## Testing strategy
 
-We avoid shipping proprietary Windows binaries:
-
 - Committed minimal synthetic PE fixture for `WEVT_TEMPLATE` extraction.
 - Synthetic CRIM/WEVT/TTBL/TEMP blobs for structural correctness + join tests.
-- Ignored integration test against the `services.exe` sample if present locally (or downloaded by the test when enabled).
+- Committed provider fixtures under `samples/dlls/` (git-lfs) + insta snapshot tests (canonical stats validated against libfwevt):
+  - `adtschema.dll`, `lsasrv.dll`, `scesrv.dll`, `services.exe`, `wevtsvc.dll`
+- Optional ignored integration test against the public `services.exe.gif` sample (downloadable by the test when enabled).
 
 ## Future work
 

src/bin/evtx_dump.rs

@@ -24,6 +24,31 @@ mod dump_template_instances;
 #[path = "evtx_dump/extract_wevt_templates.rs"]
 mod extract_wevt_templates;
 
+#[cfg(feature = "wevt_templates")]
+fn load_wevt_cache_from_wevtcache_file(
+    path: &str,
+) -> Result<std::sync::Arc<evtx::wevt_templates::WevtCache>> {
+    use std::sync::Arc;
+
+    let cache = Arc::new(evtx::wevt_templates::WevtCache::new());
+    {
+        use evtx::wevt_templates::wevtcache::{EntryKind, WevtCacheReader};
+
+        let mut reader = WevtCacheReader::open(Path::new(path))?;
+        while let Some((kind, bytes)) = reader.next_entry()? {
+            match kind {
+                EntryKind::Crim => {
+                    cache
+                        .add_wevt_blob(Arc::new(bytes))
+                        .map_err(|e| format_err!("{e}"))?;
+                }
+            }
+        }
+    }
+
+    Ok(cache)
+}
+
 #[cfg(all(not(target_env = "msvc"), feature = "fast-alloc"))]
 use tikv_jemallocator::Jemalloc;
 
@@ -160,8 +185,8 @@ impl EvtxDump {
 
         #[cfg(feature = "wevt_templates")]
         let wevt_cache = matches
-            .get_one::<String>("wevt-cache-index")
-            .map(|p| evtx::wevt_templates::WevtCache::load(p).map(std::sync::Arc::new))
+            .get_one::<String>("wevt-cache")
+            .map(|p| load_wevt_cache_from_wevtcache_file(p))
             .transpose()?;
 
         let mut parser_settings = ParserSettings::new()
@@ -510,10 +535,10 @@ fn main() -> Result<()> {
     // whose embedded EVTX templates are missing/corrupt (common in carved/dirty logs).
     #[cfg(feature = "wevt_templates")]
     let cmd = cmd.arg(
-        Arg::new("wevt-cache-index")
-            .long("wevt-cache-index")
-            .value_name("INDEX_JSONL")
-            .help("Path to a WEVT template cache index JSONL (from `extract-wevt-templates`). When set, evtx_dump will try to render records using this cache if the embedded EVTX template expansion fails."),
+        Arg::new("wevt-cache")
+            .long("wevt-cache")
+            .value_name("WEVTCACHE")
+            .help("Path to a WEVT template cache file (`.wevtcache`). When set, evtx_dump will try to render records using this cache if the embedded EVTX template expansion fails."),
     );
 
     let matches = cmd