0121 maxsim support#211

Open

FarmersWrap wants to merge 31 commits intotexttron:mainfrom

FarmersWrap:0121-chunk-encode

Commits on Dec 10, 2025

rewrote passage chunking
Ryan Yu
committed
added logic for left padding
Ryan Yu
committed

Commits on Dec 11, 2025

added search
Ryan Yu
committed

Commits on Dec 14, 2025

changed the tokenizer logic
Ryan Yu
committed
added train collator debug
Ryan Yu
committed
traincollator is done
Ryan Yu
committed
fixed some comments
Ryan Yu
committed
modified chunkedencoder
Ryan Yu
committed
Modified forward and model
Ryan Yu
committed
Modified inference on chunked passage, in progress
Ryan Yu
committed
fixed a chunk size not passed to model
Ryan Yu
committed

Commits on Dec 15, 2025

changed eos to sep
Ryan Yu
committed

Commits on Dec 16, 2025

added logs
Ryan Yu
committed

Commits on Dec 19, 2025

added some scripts
Ryan Yu
committed

Commits on Dec 22, 2025

added tests
Ryan Yu
committed
added tests
Ryan Yu
committed
added log
Ryan Yu
committed

Commits on Dec 23, 2025

review
Ryan Yu
committed
added collator helper functions
Ryan Yu
committed
added padding helper
Ryan Yu
committed
Tested the collator
Ryan Yu
committed
Reviewed forward and maxsim
Ryan Yu
committed

Commits on Dec 24, 2025

dataset uses random negative for cases have less negatives
Ryan Yu
committed
added some prints
Ryan Yu
committed
removed one breakpoint
Ryan Yu
committed

Commits on Dec 29, 2025

Added random chunks
Ryan Yu
committed
Added full randomization
Ryan Yu
committed
removed useless variables
Ryan Yu
committed
Refactored the randomization
Ryan Yu
committed

Commits on Jan 6, 2026

added search time passage encoding with chunks
Ryan Yu
committed

Commits on Jan 10, 2026

Added random chunk size for eval, and tests for prechunked passages and random chunking
Ryan Yu
committed