Conversation
Recovered from filesystem after data loss. This squashes ~58 commits originally made between 2026-03-23 and 2026-03-28. The full original reflog is preserved in docs/recovered-git-history.md. New OoT-specific factories: - OoTSceneFactory (OOT:SCENE, OOT:ROOM) — scene command parsing and binary export - OoTSkeletonFactory — skeleton, limb, and skin vertex support - OoTAnimationFactory — normal, curve, legacy, and player animations - OoTCollisionFactory — collision mesh with camera data and waterboxes - OoTArrayFactory — Shipwright-compatible VTX and Vec3s arrays Modified upstream: - DisplayListFactory — OoT cross-segment DList handling, VTX consolidation, virtual segment 0x80, G_BRANCH_Z discovery, ZAPD compatibility fixes - Companion — OoT factory registration, BUILD_OOT cmake option - ResourceType — OoT type codes (OSKL, OSLB, OANM, OROM, OCOL, OPTH, OTXT) Tooling (soh/): - zapd_to_torch.py — converts ZAPDTR/OTRExporter XML to Torch YAML - test_assets.sh, check.sh, verify.sh, manifest.sh, lib.sh — test harness - list_assets.py — asset manifest query tool Status at time of loss: 20,432 assets passing, 0 failures. 14,355 scene assets in progress (scene/room factory implemented, iterating on binary format correctness). OoTTextFactory was not recovered and needs recreation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- identify_roms.sh: identifies OoT ROMs by SHA1, renames to standardized format, handles duplicates - extract_dma.py: extracts DMA tables from all 17 ROM versions using Shipwright filelists, outputs JSON keyed by filename - Pre-computed DMA tables for all 17 versions (14 unique) - Manifests directory with gitignore for generated hash files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
config.yml moved from soh/ to soh/assets/yml/ where Torch expects it. Generated per-version YAML dirs are gitignored via local .gitignore rather than the top-level one. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add lib/libyaz0/ with decode support following libmio0/libyay0 pattern - Wire YAZ0 into Decompressor::Decode and AutoDecode - Add missing PendingVtx struct in DeferredVtx namespace - Add missing IS_VIRTUAL_SEGMENT macro in BaseFactory.h - Add libyaz0 to CMake C_FILES glob Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- TranslateAddr now recognizes high segments (>= 0x80) when they exist in the segment map, not just standard segments (0x01-0x1F) - ASSET_PTR extracts segment offset for virtual segments too, preventing raw 0x80XXXXXX addresses from being used as buffer offsets Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OTRExporter writes 0-byte files for LimbTable entries. BlobFactory crashed when trying to Write() a null buffer. Guard the write with an empty check. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Empty blobs (e.g. LimbTable) now write 0 bytes to match OTRExporter reference output instead of writing a header with size 0 - test_assets.sh auto-logs to soh/logs/ with timestamp - New compare_asset.sh tool for hex-diffing individual assets between reference and generated O2R Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
*.o2r are generated archive files. torch.hash.yml is a Torch build cache tracking which YAMLs have been processed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Hash all extracted files in a single sha256sum call instead of one process per file - Redirect torch output to a log file instead of piping through grep - Collapse duplicate jq reduce into one pass with inline fail count Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrites the asset test script in Python to avoid per-file process spawning. YAML collection, O2R extraction, and hashing are all done in-process. Hashes assets directly from the zip without extracting to disk. 107s → 1.6s for 17,516 object assets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add BUILD_OOT option (default ON) following pattern of other games, defines OOT_SUPPORT so OoT factories are registered - Stub OoTTextFactory so it compiles (real impl is task HarbourMasters#5) - Expose DeferredVtx::BeginDefer in DisplayListFactory.h so OoTSceneFactory can call it Enables 16,952 additional assets: 12,377 → 29,329 passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Enable GFX auto-discovery for auto-discovered limbs (previously disabled, causing 573 limbs to have empty DList paths) - Fix LOD limb DList suffix: use "FarDL" instead of "DL2" to match OTRExporter/ZAPDTR naming convention - Fix Curve limb DList suffixes: "CurveDL"/"Curve2DL" to match ZAPDTR - Resolve LOD far DList before near, so shared-address limbs use the Far name for both fields (matches OTRExporter behavior) - Rewrite compare_asset.sh as compare_asset.py (takes two O2Rs, no torch run needed) - test_assets.py now saves generated.o2r to soh/o2r/ by default Objects: 17,322 passed, 1 failed (MTX), 193 not generated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OTRExporter writes a 0-byte file for each skeleton's limb array (e.g. gKeeseSkeletonLimbs). Add this to the skeleton factory's parse to match. Objects: 17,515 passed, 1 failed (MTX), 0 not generated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OTRExporter/ZAPDTR reads the N64 Mtx as 16 sequential int32 BE values and writes them back as-is. Our exporter was writing individual uint16 int-part values, which produced byte-swapped output within each 32-bit word. Now reads and stores the raw int32 values in the parser and writes them in the binary exporter, matching the reference format. Objects: 17,516 passed, 0 failed. Code: 11 passed, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When multiple segments map to the same physical ROM address (common for overlays which alias segments 8-13 to their code data), the virtual address patcher was returning a segment 0x0D address instead of segment 0x80. This caused texture lookups to fail because textures are registered under segment 0x80 offsets in the YAML. Now explicitly prefers segment 0x80 when it maps to the same physical address, matching how YAML offsets are declared. Overlays: 325 passed, 0 failed (was 101 failures). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scene/room DLists are auto-discovered by the scene factory with room-prefixed names matching OTRExporter output. Pre-declared DList entries from ZAPDTR XMLs used different naming (gXxxDL_ vs xxx_room_0DL_) causing mismatches. Scenes: 10,729 passed, 0 failed (was 27 failures). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Room mesh DLists are auto-discovered by the scene factory with correct room-prefixed names. Pre-declared DLists from ZAPDTR XMLs (both room-named and scene-named) conflict with auto-discovery. 18 scene-level DLists declared in room files (e.g. gKinsutaDL_0030B0) are now missing — these need to be handled by the scene factory or a separate mechanism. Tracked as part of scene work. 31,156 passed, 1 failed (version), 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scene/room alternate headers (SetAlternateHeaders command) are now recursively processed as sub-assets. Processing is deferred until after the primary header's commands (especially SetMesh) complete, so primary DLists are registered first and alternate headers reuse their names for shared ROM addresses. DeferredVtx state is saved/restored around each alternate header to prevent VTX consolidation corruption. Exposes SaveAndClearPending/RestorePending and PendingVtx struct in DisplayListFactory.h for use by scene factory. 31,436 passed (+280), 128 scene failures (Sets/Cutscenes), 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Alternate headers pass parent's baseName for sub-asset naming (DLists, backgrounds, cutscenes, pathways) so names match OTRExporter which doesn't prefix with Set_ - Fix cutscene suffix: "CutsceneData" instead of "Cs" to match OTRExporter's GetSegmentedPtrName convention 31,501 passed (+345 from session start), 108 failed, 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Cutscenes use entryName (with Set_ prefix) matching OTRExporter - Pathways use baseName (parent name) matching OTRExporter - Fix cutscene suffix: CutsceneData instead of Cs 31,583 passed, 109 failed (84 Set command data, 24 cutscenes, 1 version). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use getNeighborSize to limit pathway entry scanning instead of a hard 256 maximum. This helps some alternate headers with tight boundaries, though pathway count inference remains imperfect without XML metadata. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OTRExporter creates empty placeholder files for actor list data (e.g. Bmori1_room_0ActorEntry_000054). Add these as companion files in the scene factory. 32,151 passed (+568), 109 failed, 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OoT alternate headers reference the same DLists as primary headers under Set_-prefixed names. OTRExporter creates both files with identical content. - Add RegisterAssetAlias to Companion for creating duplicate O2R entries with the same binary data under different names - Scene factory uses entryName for DList symbols and ResolveGfxWithAlias to register aliases when an existing DList is found at the same offset - Alias files are written during the export phase using the already-serialized binary data (zero re-parsing overhead) 34,539 passed (+2,388), 109 failed, 738 not generated. Session total: 12,377 → 34,539 (34.9% → 97.6%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace naive 0xFFFFFFFF scan with a command-aware parser that correctly determines cutscene boundaries by parsing the command structure (ID + entry count + entry size per type). Handles camera splines (terminated by continueFlag), scene transitions (0x2D), destinations (0x3E8), and standard commands. Cutscene sizes are now correct, but content still differs from reference because OTRExporter re-serializes with different byte ordering (ROM is BE, O2R is LE with CMD_HH packing). Full re-serialization is the next step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document the BE→LE field re-packing needed for each command type. Raw copy doesn't work because OTRExporter uses CMD_HH/CMD_BBH/CMD_HBB macros to pack fields into uint32 words differently than ROM layout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace raw cutscene copy with proper BE→LE re-serialization using CMD_HH/CMD_BBH/CMD_HBB field packing to match OTRExporter output. Handles camera splines, actor cues, misc/lighting/BGM, textbox, rumble, settime, transition, and destination commands. 33 additional cutscenes now match. 76 failures remain (likely a subtle issue with uint16/uint32 field reading in some entries). 34,572 passed (97.7%), 76 failed, 738 not generated. Session total: 12,377 → 34,572 (34.9% → 97.7%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Actor cue entries have rotY/rotZ as the 3rd word packed with CMD_HH, not a raw uint32. Differentiate actor cues from misc/lighting/BGM commands to apply correct packing. 34,602 passed (97.8%), 46 failed (44 cutscene, 1 pathway, 1 version). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
34,602/35,386 (97.8%) passing. Remaining: 44 cutscene format issues, 598 audio (no factory), 135 scene sub-assets, 4 text. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace inline tuple with DrumEntry struct - Move SFXEntry and InstEntry structs to header - Add FontResidueState struct for cross-font stack residue tracking - Extract all three parsers as private methods Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace FontResidueState struct with FontResidue class owning Reset, SeedFromDrums, ApplyToInstrument, UpdateFromInstrument - Seed residue in ParseDrums instead of ParseInstruments - Remove drums param from ParseInstruments - Move DrumEntry, SFXEntry, InstEntry structs to header Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…riter Extract's font loop is now: parse drums → parse sfx → parse instruments → write counts → WriteDrums → WriteInstruments → WriteSFXEntries → register companion. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add FontWriteContext struct bundling audioBank, sampleBankTable, sampleMap to reduce parameter counts across all font methods - Reorder loop body: parse all data first, then write via WriteFontCompanion - Rename fi/fe to fontIndex/fontEntry for clarity Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Extract CalculateCutsceneSize into separate method
- Add helper functions IsCameraCmd, IsSingleEntryCmd, IsSmallEntryCmd
- Replace csParseOk flag with direct return {} on corrupt data
- Replace else-if chain with early continues
- Rename variables for clarity (csMaxBytes→endOffset, csSizeCalc→reader,
csCmdWord2→entryCount, cf→marker, totalCsBytes→csSize)
- Add comments and spacing throughout
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace free SerializeCutscene function with CutsceneSerializer class - Serialize orchestrates: CalculateSize → Write - Rename variables for clarity (calculatedSize/csSize → size) - Add comment explaining two-phase approach Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move command helpers (IsCameraCmd, IsSmallEntryCmd, etc.) from file-static to CutsceneSerializer private static members - Extract IsHandledCmd with comments for each category - Extract WriteCameraCmd, WriteSingleEntryCmd, WriteEntryCountCmd - Remove dead code (0x07/0x08 branch inside unhandled block) - Replace else-if chains with early continues throughout - Pull shared header fields (base, startFrame, endFrame) above per-command branches in WriteEntryCountCmd - Clean up variable names and add comments/spacing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Extract GetLimbDataSize helper for limb type → data size mapping - Remove unused canAutoDiscoverGfx variable - Remove unused autoDiscover parameter from ResolveGfxPointer Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Extract ParseLimbHeader, ParseCurveLimb, ParseLegacyLimb, ParseStandardLimb, ParseLODLimb, ParseSkinLimb as private methods - Replace else-if chain with early returns per type - Use AutoDecode offset overload in ParseSkinLimb (remove throwaway YAML nodes) - Remove unused autoDiscover param from ResolveGfxPointer - Add T& vs shared_ptr cross-cutting concern to checklist Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Extract ParseAnimatedSkinData for SkinType_Animated handling - Extract ParseSkinVertices and ParseSkinTransformations helpers - Flatten nesting with early returns in ParseSkinLimb - Move skinSegmentAddr null check into ParseAnimatedSkinData Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename variables (np→numPoints, ptsAddr→pointsAddr), add comments, improve spacing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add #ifdef OOT_SUPPORT guard - Split parseMessages into ParseMessagesNTSC and ParseMessagesPAL - Extract readMessageText, readMessageMetadata, readMessageOffsetNTSC, readMessageOffsetPAL helpers - Extract IsEndOfMessageCode, GetTrailingBytes with named control codes - Simplify message text loop: no flags, inline trailing byte consumption - Move all static functions to private methods on OoTTextFactory - Move MessageEntry struct to header - Pass DataChunk instead of unpacked data/size - Improve variable names, comments, and spacing throughout Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add guards to DeferredVtx.h/.cpp and OoTTextFactory.h/.cpp. All OoT factory files now consistently guarded. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reviewed all changes in DisplayListFactory.cpp diff from main. Documented cleanup needed in docs/oot-dlist-cleanup.md: - Extract OoT-specific handlers from shared Export/parse paths - Replace GBIMinorVersion gates with config-driven flags - Deduplicate alias segment detection, address resolution helpers - Consolidate gSunDL/sShadowMaterialDL name-based hacks - Investigate G_SETOTHERMODE_H LUT encoding, NOOP zeroing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First step of DisplayListFactory OoT extraction — move OoT-specific code to a separate file to reduce diff noise in shared code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restore main's SearchVtx in DisplayListFactory, OoT version (with OOT:ARRAY and cross-segment support) dispatched from helpers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract the gSunDL ranged-match VTX hack from DListBinaryExporter::Export into OoT::DListHelpers::HandleGSunDLVtx. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract alias segment detection, cross-file VTX check, OOT:ARRAY lookup, virtual segment handling, and cross-segment fallback into HandleExportVtx. Restore main's G_VTX path as the fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract segment 8-13 skip, RemapSegmentedAddr fallback, and cross-segment DL fallback. Restore main's G_DL path as fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract sShadowMaterialDL hack, unresolved texture fallback, and gSunDL SETTILE/LOADBLOCK format corrections. Restore main's G_SETTIMG path as fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract OOT:MTX type chain, RemapSegmentedAddr fallback, and cross-segment matrix fallback. Restore main's G_MTX path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract G_RDPHALF_1/G_BRANCH_Z handler, G_SETOTHERMODE_H LUT re-encoding, G_NOOP zeroing, and unhandled opcode zeroing into HandleExportOpcodeFixups. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract all OoT-specific parse logic: - HandleParseDL: child DList symbol derivation + AddAsset skip - HandleParseOpcodes: G_RDPHALF_1 DeferredVtx scanning, G_MTX logging - HandleParseVtx: cross-segment comparison, alias skip, DeferredVtx - FlushParseVtx: deferred VTX merge at end of parse - ShouldSkipAutoDiscovery: gate for light AddAsset skip - Remove dead isAutogen variable Restore main's original paths as fallbacks for all opcodes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Keep the original main code at its original indent level inside if (!OoT::DListHelpers::Handle...) blocks to minimize the diff against main. Add comments explaining the intentional non-indentation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace per-opcode wrapper blocks in DisplayListFactory with three method-level replacements (SearchVtx, Export, Parse) that early-return when OoT is active. DisplayListFactory.cpp diff vs main is now just 10 insertions (1 include + 3 early-return blocks), with main's code completely untouched. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strip all OoT work-in-progress documentation and SoH-specific tooling, manifests, DMA tables, VTX data, and test infrastructure to produce a clean Torch-only branch for PR review. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
inspectredc
left a comment
There was a problem hiding this comment.
First of all this is really cool to see! I haven't gone through and tested the factories yet but left a few questions about the PR overall in the major sections (the main major blocker being the issue in the DisplayListFactory)
|
|
||
| // Deferred VTX consolidation state (ZAPD-style MergeConnectingVertexLists). | ||
| // ZAPD merges VTX per-DList (each DList has its own vertices map and merge pass). | ||
| // We collect VTX during each DList parse call and flush at the end of that parse. |
There was a problem hiding this comment.
Would this be beneficial for us to have in the main Torch factories folder?
| @@ -0,0 +1,122 @@ | |||
| #ifdef OOT_SUPPORT | |||
There was a problem hiding this comment.
Out of curiousity, is there any benefit to having this #ifdef since my understanding is that the cmakelists already handles the exclusion of this file
| // Not all 0x80XXXXXX addresses are VRAM pointers — segment 0x80 addresses also have | ||
| // bit 31 set. We distinguish them by checking if the address falls within the current | ||
| // file's VRAM range. |
There was a problem hiding this comment.
I'm not too sure this is really necessary since the segment range doesn't go this high but it can't hurt to have
| const auto symbol = GetSafeNode<std::string>(asset, "symbol", ""); | ||
| const auto decl = this->GetNodeByAddr(offset); | ||
|
|
||
| // For OoT, all assets should be pre-declared in enriched YAML. |
There was a problem hiding this comment.
Perhaps we should just make this a general setting we add to the config as I think this would make sense based on project's own preferences
| reader.SetEndianness(Torch::Endianness::Big); | ||
| reader.Seek(0x10, LUS::SeekOffsetType::Start); | ||
| this->gRomCRC = BSWAP32(reader.ReadUInt32()); | ||
| this->gRomCRC = reader.ReadUInt32(); |
There was a problem hiding this comment.
May have to require some decomps/ports to check this doesn't break any checks they make
| #endif | ||
|
|
||
| std::optional<std::tuple<std::string, YAML::Node>> SearchVtx(uint32_t ptr) { | ||
| auto result = OoT::DListHelpers::SearchVtx(ptr); |
There was a problem hiding this comment.
Does this break without OOT_SUPPORT? And would using Torch's own VTX factory remove the need for this?
| mAliases[primaryPath].push_back(aliasPath); | ||
| } | ||
|
|
||
| void AliasManager::WriteAliases(const std::string& primaryPath, BinaryWrapper* wrapper, |
There was a problem hiding this comment.
Just a quick question about this system, is this a way for one file's data to be written to multiple locations in the output binary, and if so what is the use case for it?
this doesn't include all of the test scaffolding i've been using. that still exists in https://github.com/briaguya0/Torch/tree/oot-simplify-addasset
i'm very happy with almost every factory in here. the one big chunk left to figure out is
DisplayListFactory. i originally had a bunch of oot specific changes littered throughout that file, now i just haveOoTDListHelpersthat basically just reimplements it with oot specific changes.the other thing i'd like to do is verify this against other roms, but once the dlist stuff is sorted i see no reason to wait for every rom to work before moving this out of draft