Have RAG return generator cross-attentions when output_attentions=True

[dblakely]

🚀 Have RAG return generator cross-attentions when output_attentions=True

This feature request is for the RAG code to be modified so that if output_attentions=True, it returns the generator's cross-attentions in addition to the attentions it already returns.

Motivation

I'm interested in extracting the generator's attentions from a RAG generator model. Currently, transformers allows you to extract the generator's encoder attentions and decoder attentions, but not it's cross attentions. For example, inside modeling_rag.py, the return objects such as RetrievAugLMMarginOutput, have fields for these other attentions, but not the cross-attentions.

Because both T5 and BART can output cross-attentions, I think they could simply propagate up through the RAG code. Is there a reason this isn't already the case? Or could I do a PR to include the cross attentions along with the other attentions in the model output?

Your contribution

On my own fork of transformers, I've already added this feature and would happily submit a PR!

[dblakely]

@patrickvonplaten, @lhoestq Any feedback on this?

[lhoestq]

Feel free to open a PR indeed :)
What do you think @patrickvonplaten ? I guess it can be part of the RetrievAugLMMarginOutput attributes.

[patrickvonplaten]

Hey @dblakely,

It would be great if you could open a PR. Both Bart and T5 already return the cross_attentions, so it should be a pretty easy change by just adding

generator_cross_attentions=gen_outputs.cross_attentions,

here:

transformers/src/transformers/models/rag/modeling_rag.py

Line 657 in fac7cfb

generator_dec_attentions=gen_outputs.decoder_attentions,

and then adding generator_cross_attentions to all output classes as suggested by @lhoestq