[GSoC 2025] Explore Transformer-Based Architectures

[singjc]

Background

Our current diffusion model for DIA-MS data deconvolution uses a U-Net-based architecture. While effective, recent advances in Vision Transformers (ViTs) and other transformer-based architectures have shown promising results in many domains. This task involves exploring whether transformer-based architectures could provide performance improvements for our specific use case of MS signal deconvolution. We have previously tried a custom transformer backbone.

Task Objectives

• Implement one or more transformer-based architectures (e.g., ViT, Swin Transformer) as alternative backbones for our diffusion model
• Adapt these architectures to work with our 1D/2D MS data representation
• Train and evaluate the transformer-based models against our U-Net baseline
• Analyze trade-offs in performance, training time, and resource requirements

Technical Details

• MS data has unique characteristics that may require architectural adaptations of standard transformer models
• Consider attention mechanisms specifically suited for spectral data
• Focus initially on smaller transformer variants to enable rapid experimentation

Deliverables

• Implementation of at least one transformer-based architecture compatible with our existing pipeline
• Training and evaluation scripts for the new architecture(s)
• Performance comparison with our current U-Net baseline, including:
  • Deconvolution quality metrics
  • Convergence speed
  • Inference time
  • Memory requirements
• Analysis of the strengths and weaknesses of different architectures for our specific problem

Resources

• Current model implementation in our repository
• Vision Transformer documentation

Difficulty

Advanced - This task requires deep understanding of model architectures and their adaptation to specialized data formats.

[CoderOMaster]

can i explore this issue? i will start off with studing in depth about swin transformer,vit,and similar transformer based architectures.
then i will move back to custom tranformer again to see how layers for working wrt to avilable transformers and how i can swap them. is it possible for u to share a small snippet for MS data too? @singjc

[CoderOMaster]

i was also thinking how if its possible to replace attention unet/deeplab maybe with the standard 1d unet? what are your thoughts on that?

[singjc]

can i explore this issue? i will start off with studing in depth about swin transformer,vit,and similar transformer based architectures. then i will move back to custom tranformer again to see how layers for working wrt to avilable transformers and how i can swap them. is it possible for u to share a small snippet for MS data too? @singjc

Of course you can, that would be great! We initially started with the custom transformer, but it didn't seem to perform well, which is why we decided to test using a Unet as the back bone. At the time, we didn't really have much time to experiment/optimize the transformer backbone. So it would be great to look into.

Yes, I can share a small sample of the data. The training data is currently broken up into many small parquet files across all the DIA isolation windows (~1-5Gb per file). We still need to figure out how we're going to share the data. So I will have to get back to you on that.

i was also thinking how if its possible to replace attention unet/deeplab maybe with the standard 1d unet? what are your thoughts on that?

Yes, that would be possible too, we can explore different backbones. Not sure how a standard 1d unet would perform though, we might loose/not learn the context relationship between peak shape points and related fragment ion co-elution context. Doesn't hurt to try as a learning process though 🤷 . What do you think @LLYX

[CoderOMaster]

I do have a gut feeling that attention unet will be able to preserve peaks well or atleast better than current unet. Could experiment more with the attention gates if needed. But we do have unet with different encoder options too. Had used unet in depth in my last research project. densenet 121 also works well in finer details along with inception+resnet combo. Whole lotta options with unet though. For deeplabv3 I am excited to try out if it's possible to input timestamp + data appropriately anyways doesn't hurt to try. I would like to request some dataset.

If possible can we connect over email thread too for sharing dataset?
Also can you please review the PR.
@singjc @LLYX
Thanks
Email: keshavarorasci@gmail.com

[singjc]

I do have a gut feeling that attention unet will be able to preserve peaks well or atleast better than current unet. Could experiment more with the attention gates if needed. But we do have unet with different encoder options too. Had used unet in depth in my last research project. densenet 121 also works well in finer details along with inception+resnet combo. Whole lotta options with unet though. For deeplabv3 I am excited to try out if it's possible to input timestamp + data appropriately anyways doesn't hurt to try. I would like to request some dataset.

If possible can we connect over email thread too for sharing dataset? Also can you please review the PR. @singjc @LLYX Thanks Email: keshavarorasci@gmail.com

Yeah, it could be useful to explore more with the attention gates. I think the current Unet does have a transformer1d layer with attention. This is pretty exploratory, so could try whatever makes sense/would be interesting. We would need to keep in mind the size and scale of the model though, because we don't have access to amazing compute resources. The current DDIM(UNet1d) model is ~1.2B params, which we can just about fit into an A100 with 40Gb of vram.

I uploaded some example processed data for training here.

[CoderOMaster]

I totally forgot about the compute. But ig it should run on 16gb vram(Kaggle) too since it's 1.2 B only with full precision? ( I am taking into account of the formula we use for LLMs to get vram req)
@singjc

[singjc]

I totally forgot about the compute. But ig it should run on 16gb vram(Kaggle) too since it's 1.2 B only with full precision? ( I am taking into account of the formula we use for LLMs to get vram req) @singjc

Yeah could try to see if it will run. I will ask to see if we're able to get access to more resources or not.

[nithin200417]

I've been following the discussion on exploring transformer-based architectures for the diffusion model, and I'd love to contribute as well. It seems like there's a lot of potential in testing ViT, Swin Transformer, and attention-based U-Net variants for MS signal deconvolution.

Would it be possible to include me in this effort? I'd be happy to help with implementation, experimentation, and analyzing the trade-offs. Also, if there’s any dataset or setup details I should get access to, let me know!

[singjc]

I've been following the discussion on exploring transformer-based architectures for the diffusion model, and I'd love to contribute as well. It seems like there's a lot of potential in testing ViT, Swin Transformer, and attention-based U-Net variants for MS signal deconvolution.

Would it be possible to include me in this effort? I'd be happy to help with implementation, experimentation, and analyzing the trade-offs. Also, if there’s any dataset or setup details I should get access to, let me know!

That would be great. You and @CoderOMaster can create a sub-issue from this one to work on different backbone architectures, to avoid working on the same one.

See the message above for where you can find sample processed training data. See the README for setting up the environment and model training.

[bhautik-pithadiya]

Hey, I'd love to join as well! Having more people on the team would help us explore a wider range of architectures more efficiently

[singjc]

Hey, I'd love to join as well! Having more people on the team would help us explore a wider range of architectures more efficiently

Of course, feel free! You can also discuss with @CoderOMaster and @nithin200417 to see which architectures they are trying. I just posted a temporary link to the laterst pretrained model here