AttributeError: 'NoneType' object has no attribute 'GetAtoms' with the drugs dataset

[pikmin5656]

Hi! Thank you for this great work!
While training your model on the drugs dataset, I got the following error after about 2% of the first epoch. I would be grateful if you could tell me how to solve the problem.

AttributeError: 'NoneType' object has no attribute 'GetAtoms'

Also, just before the error occurred, the following message was displayed:

Explicit valence for atom # 37 S, 5, is greater than permitted

By the way, the error did not occur with the QM9 dataset.
Thanks.

[shenoynikhil]

I haven't faced this issue. Seems like there was something wrong with the pre-processing of the dataset. Could you re-try the preprocessing step.

[sspeedtail]

Hi! Thank you for this great work! While training your model on the drugs dataset, I got the following error after about 2% of the first epoch. I would be grateful if you could tell me how to solve the problem.

AttributeError: 'NoneType' object has no attribute 'GetAtoms'

Also, just before the error occurred, the following message was displayed:

Explicit valence for atom # 37 S, 5, is greater than permitted

By the way, the error did not occur with the QM9 dataset. Thanks.

Hi! Have you solved this problem? Thanks.

[Rich-XGK]

I also encountered the same problem.

[shenoynikhil]

I can double check this again.

[lutupix]

I also encountered the same problem when training on the drugs dataset.

[shenoynikhil]

I am working on a revamp of the dataset processing to fix the errors. Should be done in a couple of days.

[shenoynikhil]

Please let me know if the new preprocessing pipeline works with the model.

[sspeedtail]

Please let me know if the new preprocessing pipeline works with the model.

Thanks for your reply. But after modification, a new error appeared:

[rank0]: File "/home/legion-w3/LabData/zxp/ETFlow-main/scripts/train.py", line 103, in
[rank0]: run(config)
[rank0]: File "/home/legion-w3/LabData/zxp/ETFlow-main/scripts/train.py", line 85, in run
[rank0]: trainer.fit(model, datamodule=datamodule, ckpt_path=resume_ckpt_path)
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/trainer/trainer.py", line 539, in fit
[rank0]: call._call_and_handle_interrupt(
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/trainer/call.py", line 46, in _call_and_handle_interrupt
[rank0]: return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
[rank0]: return function(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/trainer/trainer.py", line 575, in _fit_impl
[rank0]: self._run(model, ckpt_path=ckpt_path)
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/trainer/trainer.py", line 982, in _run
[rank0]: results = self._run_stage()
[rank0]: ^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/trainer/trainer.py", line 1026, in _run_stage
[rank0]: self.fit_loop.run()
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/fit_loop.py", line 216, in run
[rank0]: self.advance()
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/fit_loop.py", line 455, in advance
[rank0]: self.epoch_loop.run(self._data_fetcher)
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 150, in run
[rank0]: self.advance(data_fetcher)
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 320, in advance
[rank0]: batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 192, in run
[rank0]: self._optimizer_step(batch_idx, closure)
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 270, in _optimizer_step
[rank0]: call._call_lightning_module_hook(
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/trainer/call.py", line 171, in _call_lightning_module_hook
[rank0]: output = fn(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/core/module.py", line 1302, in optimizer_step
[rank0]: optimizer.step(closure=optimizer_closure)
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/core/optimizer.py", line 154, in step
[rank0]: step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/strategies/ddp.py", line 270, in optimizer_step
[rank0]: optimizer_output = super().optimizer_step(optimizer, closure, model, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/strategies/strategy.py", line 239, in optimizer_step
[rank0]: return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/plugins/precision/precision.py", line 123, in optimizer_step
[rank0]: return optimizer.step(closure=closure, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/optim/lr_scheduler.py", line 130, in wrapper
[rank0]: return func.get(opt, opt.class)(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/optim/optimizer.py", line 484, in wrapper
[rank0]: out = func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/optim/optimizer.py", line 89, in _use_grad
[rank0]: ret = func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/optim/adamw.py", line 204, in step
[rank0]: loss = closure()
[rank0]: ^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/plugins/precision/precision.py", line 109, in _wrap_closure
[rank0]: closure_result = closure()
[rank0]: ^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 146, in call
[rank0]: self._result = self.closure(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 131, in closure
[rank0]: step_output = self._step_fn()
[rank0]: ^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 319, in _training_step
[rank0]: training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/trainer/call.py", line 323, in _call_strategy_hook
[rank0]: output = fn(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/strategies/strategy.py", line 390, in training_step
[rank0]: return self._forward_redirection(self.model, self.lightning_module, "training_step", *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/strategies/strategy.py", line 641, in call
[rank0]: wrapper_output = wrapper_module(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/nn/parallel/distributed.py", line 1636, in forward
[rank0]: else self._run_ddp_forward(*inputs, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/nn/parallel/distributed.py", line 1454, in _run_ddp_forward
[rank0]: return self.module(*inputs, **kwargs) # type: ignore[index]
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]: return forward_call(args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/anaconda3/envs/etflow/lib/python3.12/site-packages/lightning/pytorch/strategies/strategy.py", line 634, in wrapped_forward
[rank0]: out = method(_args, **_kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/LabData/zxp/ETFlow-main/etflow/models/base.py", line 75, in training_step
[rank0]: return self.generic_step(batch, batch_idx, "train")
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/legion-w3/LabData/zxp/ETFlow-main/etflow/models/model.py", line 470, in generic_step
[rank0]: raise ValueError("x0 is NaN. Fix bug in harmonic alignment!")
[rank0]: ValueError: x0 is NaN. Fix bug in harmonic alignment!

[shenoynikhil]

Can you share which config you used? I am unable to reproduce the issue with drugs-base.yaml

[FranklinHu1]

I am also encountering this problem when trying to train on the drugs dataset. I am using the drugs-so3.yaml configuration file and trying to fine-tune starting from the drugs-so3.ckpt file released on Zenodo. My configuration is as follows:

# task name for logging
task_name: flow/drugs-so3

# pretrained checkpoint to continue from
pretrained_ckpt: pretrained_ckpt/drugs-so3.ckpt

# unique seed for experiment reproducibility
seed: 42

# data config
datamodule: BaseDataModule
datamodule_args:
    partition: drugs
    data_dir: processed
    dataloader_args:
        batch_size: 1
        num_workers: 1
        pin_memory: false
        persistent_workers: true

# model config
model: BaseFlow
model_args:
    # network args
    network_type: TorchMDDynamics
    hidden_channels: 160
    num_layers: 20
    num_rbf: 64
    rbf_type: expnorm
    trainable_rbf: true
    activation: silu
    neighbor_embedding: true
    cutoff_lower: 0.0
    cutoff_upper: 10.0
    max_z: 100
    node_attr_dim: 10
    edge_attr_dim: 1
    attn_activation: silu
    num_heads: 8
    distance_influence: both
    reduce_op: sum
    qk_norm: true
    so3_equivariant: true
    clip_during_norm: true

    # flow matching specific
    sigma: 0.1
    prior_type: harmonic
    interpolation_type: linear

    # optimizer args
    optimizer_type: AdamW
    lr: 8.e-4
    weight_decay: 1.e-8

    # lr scheduler args
    lr_scheduler_type: CosineAnnealingWarmupRestarts
    first_cycle_steps: 375_000
    cycle_mult: 1.0
    max_lr: 5.e-4
    min_lr: 1.e-8
    warmup_steps: 0
    gamma: 0.05
    last_epoch: -1
    lr_scheduler_monitor: val/loss
    lr_scheduler_interval: step
    lr_scheduler_frequency: 1

# callbacks
callbacks:
    -   callback: ModelCheckpoint
        callback_args:
            dirpath: './checkpoint'
            monitor: val/loss
            mode: min
            save_last: true
            every_n_epochs: 1
            save_top_k: 3

    -   callback: LearningRateMonitor
        callback_args:
            log_momentum: false
            logging_interval: null


# logger
logger: WandbLogger
logger_args:
    project: ETFlow
    entity: franklinhu

# trainer
trainer: Trainer
trainer_args:
    max_epochs: 150
    devices: 1
    limit_train_batches: 5000
    strategy: ddp
    accelerator: auto

# eval config
eval_args:
    batch_size: 32
    sampler_args:
        sampler_type: ode
        n_timesteps: 50
        s_churn: 1.0
        t_min: 0.0001
        t_max: 0.9999
        std: 1.0

Any help would be greatly appreciated. I followed the provided instructions for processing the raw GEOM data and downloaded the QM9 + DRUGS splits from the google drive linked in the README. I tried just training with a batch size of 1 on 1 device as a preliminary test but encountered the NoneType error as described above. My current workaround is just to filter out the .pt files corresponding to the molecules that fail the featurization in the EuclideanDataset getitem method, but this is sub-optimal.

Thanks!

[shenoynikhil]

The error was due to disconnected graphs in the ground truth GEOM data. One way to address this is to remove any mols that have disconnected components. I am doing a few checks to remove these mols in #25. Post which this error should be resolved.

[shenoynikhil]

Update: Have checked on my end and the errors don't seem to occur anymore. You will have to re-run the pre-processing.

cc. @FranklinHu1 @sspeedtail