HSI-Adapter

arXiv | Project Page

This repository is the official implementation of the paper:

Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models

Juana Valeria Hurtado*, Rohit Mohan*, and Abhinav Valada.

IEEE Robotics and Automation Letters (RA-L), 2026

If you find this work useful, please cite:

@article{hurtado2026hyperspectral,
  title={Hyperspectral adapter for semantic segmentation with vision foundation models},
  author={Hurtado, Juana Valeria and Mohan, Rohit and Valada, Abhinav},
  journal={IEEE Robotics and Automation Letters},
  year={2026},
  publisher={IEEE}
}

📔 Abstract

Hyperspectral imaging (HSI) captures spatial information along with dense spectral measurements across numerous narrow wavelength bands. This rich spectral content has the potential to facilitate robust robotic perception, particularly in environments with complex material compositions, varying illumination, or other visually challenging conditions. However, current HSI semantic segmentation methods underperform due to their reliance on architectures and learning frameworks optimized for RGB inputs. In this work, we propose a novel hyperspectral adapter that leverages pretrained vision foundation models to effectively learn from hyperspectral data. Our architecture incorporates a spectral transformer and a spectrum-aware spatial prior module to extract rich spatial-spectral features. Additionally, we introduce a modality-aware interaction block that facilitates effective integration of hyperspectral representations and frozen vision Transformer features through dedicated extraction and injection mechanisms. Extensive evaluations on three benchmark autonomous driving datasets demonstrate that our architecture achieves state-of-the-art semantic segmentation performance while directly using HSI inputs, outperforming both vision-based and hyperspectral segmentation methods.

🏗 Setup

💻 System Requirements

• Linux
• Python 3.10.19
• CUDA 11.8
• PyTorch 2.0.1
• torchvision 0.15.2
• mmengine 0.9.0
• mmcv 2.1.0

IMPORTANT NOTE: These requirements are not necessarily mandatory. However, we have only tested the code under the above settings and cannot provide support for other setups.

⚙️ Installation

1. Create the environment:

   source ~/miniconda3/etc/profile.d/conda.sh
   conda create -n hsi-adapter-release python=3.10 -y
   conda activate hsi-adapter-release

2. Install PyTorch for CUDA 11.8:

   pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 \
     --index-url https://download.pytorch.org/whl/cu118

3. Set CUDA paths:

   export CUDA_HOME=/usr/local/cuda-11.8
   export PATH="$CUDA_HOME/bin:$PATH"
   export LD_LIBRARY_PATH="$CUDA_HOME/lib64:$LD_LIBRARY_PATH"

4. Install OpenMMLab dependencies:

   pip install -U openmim
   mim install mmengine==0.9.0
   mim install mmcv==2.1.0
   pip install -e .

   mmdet is not required for the released HSI-Adapter training and evaluation path. Some upstream MMSeg components still support optional MMDetection-based heads, but the provided HSI-Adapter configs do not depend on them.

5. Install additional Python packages:

   pip install ftfy regex timm scikit-learn h5py imageio prettytable scipy matplotlib packaging \
     numpy==1.26.4 torchmetrics==0.11.4

6. Build the custom multi-scale deformable attention extension:

   cd external/ms_deformable_attention
   python setup.py build install
   cd ../..

7. Verify the installation:

   python - <<'PY'
   import torch, mmcv, mmengine, mmseg
   print(torch.__version__)
   print(mmcv.__version__, mmengine.__version__, mmseg.__version__)
   PY

💾 Data Preparation

This repository contains configs for three datasets:

• HSI-DriveV2
• Hyperspectral City V2 (HCV2)
• HyKo2

All default configs in this release expect dataset roots directly under data/. If your datasets live elsewhere, override the paths through EXTRA_ARGS in tools/hsi_train.sh or tools/hsi_eval.sh.

Expected dataset layouts and download sources are:

HSI-DriveV2

1. Download the official HSI-Drive release from the official download page.

2. Place the prepared HDF5 files under:

   data/HSI-DriveV2/
   ├── hsidrive_train.h5
   ├── hsidrive_val.h5
   └── hsidrive_test.h5
   

   This release expects the HSI-DriveV2 data in the HDF5 layout shown above. If your downloaded package is in a different raw format, prepare or reorganize it into these files before training or evaluation.

HCV2

1. Download the official HCV2 release from the official data host.

2. Extract it under:

   data/HyperspectralCityV2/
   ├── train/
   │   ├── npy/
   │   └── labels/
   └── test/
       ├── npy/
       └── labels/
   

HyKo2

1. Download HyKo2 from the official dataset page.

2. Place the prepared files under:

   data/HyKo2-VIS/
   ├── hyko2_train.h5
   └── hyko2_test.h5
   

   This release expects the benchmark data in the HDF5 layout shown above.

🏃 Running the Code

🚂 Training

To train HSI-Adapter:

1. Open tools/hsi_train.sh and fill in the placeholder values:

   • <CONDA_ENV>: Conda environment name for this project
   • <GPU_IDS>: Comma-separated GPU IDs to use, such as 0,1,2,3
   • <NUM_GPUS>: Number of GPUs in GPU_IDS
   • <MASTER_PORT>: Port for distributed training, such as 29500
   • <CONFIG_FILE>: Config file for the dataset/experiment you want to train
   • <WORK_DIR>: Output directory for logs and checkpoints
   • EXTRA_ARGS: Optional --cfg-options overrides for dataset-specific paths or run settings

2. Run the training script:

   bash tools/hsi_train.sh

📊 Evaluation

To evaluate a trained model:

1. Open tools/hsi_eval.sh and fill in the placeholder values:

   • <CONDA_ENV>: Conda environment name for this project
   • <GPU_IDS>: Comma-separated GPU IDs to use for evaluation
   • <NUM_GPUS>: Number of GPUs in GPU_IDS
   • <MASTER_PORT>: Port for distributed evaluation, such as 29500
   • <CONFIG_FILE>: Config file for the dataset/experiment you want to evaluate
   • <CHECKPOINT_PATH>: Path to the checkpoint file
   • <WORK_DIR>: Output directory for evaluation logs
   • EXTRA_ARGS: Optional --cfg-options overrides, for example to switch between validation and test files

2. Run the evaluation script:

   bash tools/hsi_eval.sh

👩‍⚖️ License

For academic usage, the code is released under the GPLv3 license. For any commercial purpose, please contact the authors.

🙏 Acknowledgment

This codebase builds adapted from the open-source VIT-Adapter repository: https://github.com/czczup/ViT-Adapter

We sincerely thank the authors for making their implementation publicly available.

This work was financed by the Baden-Württemberg Stiftung gGmbH within the programm Automone Robotik.

📬 Contacts

• Juana Valeria Hurtado
• Rohit Mohan