Bidirectionality

[saydemr]

Hello,

Have you tried any specific structure for bidirectional processing of the data? Currently there seems to be no option for using a bidirectional model architecture and paper does not touch on this aside from couple of references to other bidirectional models.

Thanks

[kpoeppel]

Bi-directionality is possible, but currently not natively supported in the code.
For a naive implementation, you can just use two Cells in a layer, flip the inputs for one of the cells and flip its outputs again. Then you can sum them, or apply any other aggregation.
Still, there would be more efficient ways to implement this, but should be definitely good for first experiments and as a reference for tests of a more efficient implementation.

[stefan-it]

I think good inspiration could be this PR from Mamba ( state-spaces/mamba#52 ) and in general BiGS (https://github.com/jxiw/BiGS) that is a bi-directional LM with SSM :)

[saydemr]

Bi-directionality is possible, but currently not natively supported in the code. For a naive implementation, you can just use two Cells in a layer, flip the inputs for one of the cells and flip its outputs again. Then you can sum them, or apply any other aggregation. Still, there would be more efficient ways to implement this, but should be definitely good for first experiments and as a reference for tests of a more efficient implementation.

Thanks very much. I was trying to understand if you had already experimented with certain architectures. Thank you for the suggestions as well.

thanks @stefan-it too.