expressivity-of-transformers

This repository is in accompany with the submission: Characterizing the Expressivity of Transformer Language Models.

Dependencies

• python 3.11.6
• jaxlib 0.4.30

Setup

Install required packages:

pip install -r requirements.txt

To run language recognition tasks:

cd neural_networks_chomsky_hierarchy/
python experiments/example.py \
        --task last \
        --architecture transformer_encoder \
        --causal_masking \
        --strict_masking \

To run language modeling tasks:

cd neural_networks_chomsky_hierarchy/
python experiments/example.py \
        --task right_deterministic_plm \
        --num_states 2 \
        --architecture transformer_encoder \
        --causal_masking \
        --strict_masking \
        --probabilistic_language_model \