Learnable Contrastive Decoding in State-Space LLMs

Environment Build

We recommmend using uv to build the environment.

uv sync

check uv installation

Code Usage

main.ipynb is the training scripts. We train the LCD module from RWKV.\ test.ipynb is for testing the result over 3 models: RWKV, Transformer (here we use pythia-70m) and our approach LCD

Datasets

The used training/testing dataset are provided in link.