Training PartA2 network

[hcyoo93]

Hello @sshaoshuai,

Thank you for your great work!

I have trouble training PartA2 network.

Evaluating PartA2 and training PV-RCNN, PointRCNN, and SECOND work fine.

x = self.conv_input(input_sp_tensor) of UNetV2 gives nan values for some batches, and the nan value yields 0 max_overlap, 0 foreground and 0 background.

How can I resolve this issue?

Does anyone have a similar experience?

Here is the log.

Please note that my data set is Nusecenes exported to KITTI.

2020-09-07 00:43:50,426   INFO  CUDA_VISIBLE_DEVICES=ALL
2020-09-07 00:43:50,426   INFO  cfg_file         /home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/cfgs/kitti_models/PartA2.yaml
2020-09-07 00:43:50,426   INFO  batch_size       1
2020-09-07 00:43:50,426   INFO  epochs           85
2020-09-07 00:43:50,427   INFO  workers          8
2020-09-07 00:43:50,427   INFO  extra_tag        default
2020-09-07 00:43:50,427   INFO  ckpt             /home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/checkpoints/kitti/PartA2_7940.pth
2020-09-07 00:43:50,427   INFO  pretrained_model None
2020-09-07 00:43:50,427   INFO  launcher         none
2020-09-07 00:43:50,427   INFO  tcp_port         18888
2020-09-07 00:43:50,427   INFO  sync_bn          False
2020-09-07 00:43:50,427   INFO  fix_random_seed  False
2020-09-07 00:43:50,427   INFO  ckpt_save_interval 1
2020-09-07 00:43:50,427   INFO  local_rank       0
2020-09-07 00:43:50,427   INFO  max_ckpt_save_num 30
2020-09-07 00:43:50,427   INFO  merge_all_iters_to_one_epoch False
2020-09-07 00:43:50,427   INFO  set_cfgs         None
2020-09-07 00:43:50,427   INFO  max_waiting_mins 0
2020-09-07 00:43:50,427   INFO  start_epoch      0
2020-09-07 00:43:50,427   INFO  save_to_file     False
2020-09-07 00:43:50,427   INFO  cfg.ROOT_DIR: /home/hcyoo/anaconda3/envs/pcdet/OpenPCDet
2020-09-07 00:43:50,427   INFO  cfg.LOCAL_RANK: 0
2020-09-07 00:43:50,427   INFO  cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
2020-09-07 00:43:50,427   INFO  
cfg.DATA_CONFIG = edict()
2020-09-07 00:43:50,427   INFO  cfg.DATA_CONFIG.DATASET: KittiDataset
2020-09-07 00:43:50,427   INFO  cfg.DATA_CONFIG.DATA_PATH: ../data/kitti
2020-09-07 00:43:50,427   INFO  cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [0, -40, -3, 70.4, 40, 1]
2020-09-07 00:43:50,427   INFO  
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2020-09-07 00:43:50,427   INFO  cfg.DATA_CONFIG.DATA_SPLIT.train: train
2020-09-07 00:43:50,427   INFO  cfg.DATA_CONFIG.DATA_SPLIT.test: val
2020-09-07 00:43:50,427   INFO  
cfg.DATA_CONFIG.INFO_PATH = edict()
2020-09-07 00:43:50,427   INFO  cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_train.pkl']
2020-09-07 00:43:50,427   INFO  cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_val.pkl']
2020-09-07 00:43:50,428   INFO  cfg.DATA_CONFIG.FOV_POINTS_ONLY: True
2020-09-07 00:43:50,428   INFO  
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
2020-09-07 00:43:50,428   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2020-09-07 00:43:50,428   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['kitti_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Car:5', 'Pedestrian:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Car:20', 'Pedestrian:15'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
2020-09-07 00:43:50,428   INFO  
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2020-09-07 00:43:50,428   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2020-09-07 00:43:50,428   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity']
2020-09-07 00:43:50,428   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
2020-09-07 00:43:50,428   INFO  cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.05, 0.05, 0.1], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 16000, 'test': 40000}}]
2020-09-07 00:43:50,428   INFO  cfg.DATA_CONFIG._BASE_CONFIG_: cfgs/dataset_configs/kitti_dataset.yaml
2020-09-07 00:43:50,428   INFO  
cfg.MODEL = edict()
2020-09-07 00:43:50,428   INFO  cfg.MODEL.NAME: PartA2Net
2020-09-07 00:43:50,428   INFO  
cfg.MODEL.VFE = edict()
2020-09-07 00:43:50,428   INFO  cfg.MODEL.VFE.NAME: MeanVFE
2020-09-07 00:43:50,428   INFO  
cfg.MODEL.BACKBONE_3D = edict()
2020-09-07 00:43:50,428   INFO  cfg.MODEL.BACKBONE_3D.NAME: UNetV2
2020-09-07 00:43:50,428   INFO  
cfg.MODEL.MAP_TO_BEV = edict()
2020-09-07 00:43:50,428   INFO  cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression
2020-09-07 00:43:50,428   INFO  cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256
2020-09-07 00:43:50,428   INFO  
cfg.MODEL.BACKBONE_2D = edict()
2020-09-07 00:43:50,428   INFO  cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2020-09-07 00:43:50,428   INFO  cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5]
2020-09-07 00:43:50,428   INFO  cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2]
2020-09-07 00:43:50,428   INFO  cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256]
2020-09-07 00:43:50,428   INFO  cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2]
2020-09-07 00:43:50,428   INFO  cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256]
2020-09-07 00:43:50,428   INFO  
cfg.MODEL.DENSE_HEAD = edict()
2020-09-07 00:43:50,428   INFO  cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2020-09-07 00:43:50,428   INFO  cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2020-09-07 00:43:50,428   INFO  cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2020-09-07 00:43:50,429   INFO  
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2020-09-07 00:43:50,429   INFO  
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2020-09-07 00:43:50,429   INFO  
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2020-09-07 00:43:50,429   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2020-09-07 00:43:50,429   INFO  
cfg.MODEL.POINT_HEAD = edict()
2020-09-07 00:43:50,429   INFO  cfg.MODEL.POINT_HEAD.NAME: PointIntraPartOffsetHead
2020-09-07 00:43:50,429   INFO  cfg.MODEL.POINT_HEAD.CLS_FC: []
2020-09-07 00:43:50,429   INFO  cfg.MODEL.POINT_HEAD.PART_FC: []
2020-09-07 00:43:50,429   INFO  cfg.MODEL.POINT_HEAD.CLASS_AGNOSTIC: True
2020-09-07 00:43:50,429   INFO  
cfg.MODEL.POINT_HEAD.TARGET_CONFIG = edict()
2020-09-07 00:43:50,429   INFO  cfg.MODEL.POINT_HEAD.TARGET_CONFIG.GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
2020-09-07 00:43:50,429   INFO  
cfg.MODEL.POINT_HEAD.LOSS_CONFIG = edict()
2020-09-07 00:43:50,429   INFO  cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_REG: smooth-l1
2020-09-07 00:43:50,429   INFO  
cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2020-09-07 00:43:50,429   INFO  cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.point_cls_weight: 1.0
2020-09-07 00:43:50,429   INFO  cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.point_part_weight: 1.0
2020-09-07 00:43:50,429   INFO  
cfg.MODEL.ROI_HEAD = edict()
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NAME: PartA2FCHead
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.CLASS_AGNOSTIC: True
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.SHARED_FC: [256, 256, 256]
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.CLS_FC: [256, 256]
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.REG_FC: [256, 256]
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.DP_RATIO: 0.3
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.SEG_MASK_SCORE_THRESH: 0.3
2020-09-07 00:43:50,430   INFO  
cfg.MODEL.ROI_HEAD.NMS_CONFIG = edict()
2020-09-07 00:43:50,430   INFO  
cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN = edict()
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_TYPE: nms_gpu
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.MULTI_CLASSES_NMS: False
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_PRE_MAXSIZE: 9000
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_POST_MAXSIZE: 512
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_THRESH: 0.8
2020-09-07 00:43:50,430   INFO  
cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST = edict()
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_TYPE: nms_gpu
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.MULTI_CLASSES_NMS: False
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_PRE_MAXSIZE: 1024
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_POST_MAXSIZE: 100
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_THRESH: 0.7
2020-09-07 00:43:50,430   INFO  
cfg.MODEL.ROI_HEAD.ROI_AWARE_POOL = edict()
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.ROI_AWARE_POOL.POOL_SIZE: 12
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.ROI_AWARE_POOL.NUM_FEATURES: 128
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.ROI_AWARE_POOL.MAX_POINTS_PER_VOXEL: 128
2020-09-07 00:43:50,430   INFO  
cfg.MODEL.ROI_HEAD.TARGET_CONFIG = edict()
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.BOX_CODER: ResidualCoder
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.ROI_PER_IMAGE: 128
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.FG_RATIO: 0.5
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.SAMPLE_ROI_BY_EACH_CLASS: True
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_SCORE_TYPE: roi_iou
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_FG_THRESH: 0.75
2020-09-07 00:43:50,430   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH: 0.25
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH_LO: 0.1
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.HARD_BG_RATIO: 0.8
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.REG_FG_THRESH: 0.65
2020-09-07 00:43:50,431   INFO  
cfg.MODEL.ROI_HEAD.LOSS_CONFIG = edict()
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CLS_LOSS: BinaryCrossEntropy
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.REG_LOSS: smooth-l1
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CORNER_LOSS_REGULARIZATION: True
2020-09-07 00:43:50,431   INFO  
cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_cls_weight: 1.0
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_reg_weight: 1.0
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_corner_weight: 1.0
2020-09-07 00:43:50,431   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2020-09-07 00:43:50,431   INFO  
cfg.MODEL.POST_PROCESSING = edict()
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2020-09-07 00:43:50,431   INFO  
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.1
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2020-09-07 00:43:50,431   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2020-09-07 00:43:50,431   INFO  
cfg.OPTIMIZATION = edict()
2020-09-07 00:43:50,431   INFO  cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2020-09-07 00:43:50,431   INFO  cfg.OPTIMIZATION.NUM_EPOCHS: 80
2020-09-07 00:43:50,431   INFO  cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2020-09-07 00:43:50,431   INFO  cfg.OPTIMIZATION.LR: 0.01
2020-09-07 00:43:50,431   INFO  cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2020-09-07 00:43:50,431   INFO  cfg.OPTIMIZATION.MOMENTUM: 0.9
2020-09-07 00:43:50,431   INFO  cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2020-09-07 00:43:50,432   INFO  cfg.OPTIMIZATION.PCT_START: 0.4
2020-09-07 00:43:50,432   INFO  cfg.OPTIMIZATION.DIV_FACTOR: 10
2020-09-07 00:43:50,432   INFO  cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2020-09-07 00:43:50,432   INFO  cfg.OPTIMIZATION.LR_DECAY: 0.1
2020-09-07 00:43:50,432   INFO  cfg.OPTIMIZATION.LR_CLIP: 1e-07
2020-09-07 00:43:50,432   INFO  cfg.OPTIMIZATION.LR_WARMUP: False
2020-09-07 00:43:50,432   INFO  cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2020-09-07 00:43:50,432   INFO  cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2020-09-07 00:43:50,432   INFO  cfg.TAG: PartA2
2020-09-07 00:43:50,432   INFO  cfg.EXP_GROUP_PATH: home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/cfgs/kitti_models
2020-09-07 00:43:51,385   INFO  Database filter by min points Car: 116656 => 47595
2020-09-07 00:43:51,391   INFO  Database filter by min points Pedestrian: 52521 => 13192
2020-09-07 00:43:51,444   INFO  Database filter by difficulty Car: 47595 => 33349
2020-09-07 00:43:51,458   INFO  Database filter by difficulty Pedestrian: 13192 => 10978
2020-09-07 00:43:51,458   INFO  Database filter by difficulty Cyclist: 0 => 0
2020-09-07 00:43:51,592   INFO  Loading KITTI dataset
2020-09-07 00:43:52,198   INFO  Total samples for KITTI dataset: 26291
2020-09-07 00:43:54,469   INFO  ==> Loading parameters from checkpoint /home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/checkpoints/kitti/PartA2_7940.pth to CPU
2020-09-07 00:43:54,606   INFO  ==> Done
2020-09-07 00:43:54,614   INFO  PartA2Net(
  (vfe): MeanVFE()
  (backbone_3d): UNetV2(
    (conv_input): SparseSequential(
      (0): SubMConv3d()
      (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (conv1): SparseSequential(
      (0): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
    (conv2): SparseSequential(
      (0): SparseSequential(
        (0): SparseConv3d()
        (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (2): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
    (conv3): SparseSequential(
      (0): SparseSequential(
        (0): SparseConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (2): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
    (conv4): SparseSequential(
      (0): SparseSequential(
        (0): SparseConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (2): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
    (conv_out): SparseSequential(
      (0): SparseConv3d()
      (1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (conv_up_t4): SparseBasicBlock(
      (conv1): SubMConv3d()
      (bn1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (relu): ReLU()
      (conv2): SubMConv3d()
      (bn2): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
    )
    (conv_up_m4): SparseSequential(
      (0): SubMConv3d()
      (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (inv_conv4): SparseSequential(
      (0): SparseInverseConv3d()
      (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (conv_up_t3): SparseBasicBlock(
      (conv1): SubMConv3d()
      (bn1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (relu): ReLU()
      (conv2): SubMConv3d()
      (bn2): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
    )
    (conv_up_m3): SparseSequential(
      (0): SubMConv3d()
      (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (inv_conv3): SparseSequential(
      (0): SparseInverseConv3d()
      (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (conv_up_t2): SparseBasicBlock(
      (conv1): SubMConv3d()
      (bn1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (relu): ReLU()
      (conv2): SubMConv3d()
      (bn2): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
    )
    (conv_up_m2): SparseSequential(
      (0): SubMConv3d()
      (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (inv_conv2): SparseSequential(
      (0): SparseInverseConv3d()
      (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (conv_up_t1): SparseBasicBlock(
      (conv1): SubMConv3d()
      (bn1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (relu): ReLU()
      (conv2): SubMConv3d()
      (bn2): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
    )
    (conv_up_m1): SparseSequential(
      (0): SubMConv3d()
      (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (conv5): SparseSequential(
      (0): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
  )
  (map_to_bev_module): HeightCompression()
  (pfe): None
  (backbone_2d): BaseBEVBackbone(
    (blocks): ModuleList(
      (0): Sequential(
        (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
        (1): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), bias=False)
        (2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (3): ReLU()
        (4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (6): ReLU()
        (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (9): ReLU()
        (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (12): ReLU()
        (13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (15): ReLU()
        (16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (18): ReLU()
      )
      (1): Sequential(
        (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
        (1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), bias=False)
        (2): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (3): ReLU()
        (4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (5): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (6): ReLU()
        (7): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (8): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (9): ReLU()
        (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (11): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (12): ReLU()
        (13): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (14): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (15): ReLU()
        (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (17): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (18): ReLU()
      )
    )
    (deblocks): ModuleList(
      (0): Sequential(
        (0): ConvTranspose2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): Sequential(
        (0): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
  )
  (dense_head): AnchorHeadSingle(
    (cls_loss_func): SigmoidFocalClassificationLoss()
    (reg_loss_func): WeightedSmoothL1Loss()
    (dir_loss_func): WeightedCrossEntropyLoss()
    (conv_cls): Conv2d(512, 18, kernel_size=(1, 1), stride=(1, 1))
    (conv_box): Conv2d(512, 42, kernel_size=(1, 1), stride=(1, 1))
    (conv_dir_cls): Conv2d(512, 12, kernel_size=(1, 1), stride=(1, 1))
  )
  (point_head): PointIntraPartOffsetHead(
    (cls_loss_func): SigmoidFocalClassificationLoss()
    (cls_layers): Sequential(
      (0): Linear(in_features=16, out_features=1, bias=True)
    )
    (part_reg_layers): Sequential(
      (0): Linear(in_features=16, out_features=3, bias=True)
    )
  )
  (roi_head): PartA2FCHead(
    (proposal_target_layer): ProposalTargetLayer()
    (reg_loss_func): WeightedSmoothL1Loss()
    (SA_modules): ModuleList()
    (conv_part): SparseSequential(
      (0): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
    (conv_rpn): SparseSequential(
      (0): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): SparseSequential(
        (0): SubMConv3d()
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
    (shared_fc_layer): Sequential(
      (0): Conv1d(221184, 256, kernel_size=(1,), stride=(1,), bias=False)
      (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Dropout(p=0.3, inplace=False)
      (4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
      (5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (6): ReLU()
      (7): Dropout(p=0.3, inplace=False)
      (8): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
      (9): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (10): ReLU()
    )
    (cls_layers): Sequential(
      (0): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
      (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Dropout(p=0.3, inplace=False)
      (4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
      (5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (6): ReLU()
      (7): Conv1d(256, 1, kernel_size=(1,), stride=(1,))
    )
    (reg_layers): Sequential(
      (0): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
      (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Dropout(p=0.3, inplace=False)
      (4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
      (5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (6): ReLU()
      (7): Conv1d(256, 7, kernel_size=(1,), stride=(1,))
    )
    (roiaware_pool3d_layer): RoIAwarePool3d()
  )
)
2020-09-07 00:43:54,615   INFO  **********************Start training home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/cfgs/kitti_models/PartA2(default)**********************
epochs:   0%|          | 0/5 [00:00<?, ?it/s]
train:   0%|          | 0/26291 [00:00<?, ?it/s]
train:   0%|          | 1/26291 [00:00<6:22:25,  1.15it/s]
epochs:   0%|          | 0/5 [00:01<?, ?it/s, loss=4.68, lr=0.001]
train:   0%|          | 2/26291 [00:01<4:59:45,  1.46it/s, total_it=12401]
epochs:   0%|          | 0/5 [00:01<?, ?it/s, loss=3.62, lr=0.001]
train:   0%|          | 3/26291 [00:01<4:02:46,  1.80it/s, total_it=12402]
epochs:   0%|          | 0/5 [00:01<?, ?it/s, loss=4.07, lr=0.001]
train:   0%|          | 4/26291 [00:01<3:22:12,  2.17it/s, total_it=12403]
epochs:   0%|          | 0/5 [00:01<?, ?it/s, loss=3.53, lr=0.001]
train:   0%|          | 5/26291 [00:01<2:54:01,  2.52it/s, total_it=12404]
epochs:   0%|          | 0/5 [00:02<?, ?it/s, loss=4.19, lr=0.001]
train:   0%|          | 6/26291 [00:02<2:31:47,  2.89it/s, total_it=12405]
epochs:   0%|          | 0/5 [00:02<?, ?it/s, loss=2.52, lr=0.001]
train:   0%|          | 7/26291 [00:02<2:16:11,  3.22it/s, total_it=12406]
epochs:   0%|          | 0/5 [00:02<?, ?it/s, loss=2.7, lr=0.001]
train:   0%|          | 8/26291 [00:02<2:05:12,  3.50it/s, total_it=12407]
epochs:   0%|          | 0/5 [00:02<?, ?it/s, loss=3.27, lr=0.001]
train:   0%|          | 9/26291 [00:02<1:56:13,  3.77it/s, total_it=12408]
epochs:   0%|          | 0/5 [00:03<?, ?it/s, loss=4.17, lr=0.001]
train:   0%|          | 10/26291 [00:02<1:50:29,  3.96it/s, total_it=12409]
epochs:   0%|          | 0/5 [00:03<?, ?it/s, loss=3.17, lr=0.001]
train:   0%|          | 11/26291 [00:03<1:47:31,  4.07it/s, total_it=12410]
epochs:   0%|          | 0/5 [00:03<?, ?it/s, loss=2.57, lr=0.001]
train:   0%|          | 12/26291 [00:03<1:44:55,  4.17it/s, total_it=12411]
epochs:   0%|          | 0/5 [00:03<?, ?it/s, loss=4.33, lr=0.001]
train:   0%|          | 13/26291 [00:03<1:43:06,  4.25it/s, total_it=12412]
epochs:   0%|          | 0/5 [00:04<?, ?it/s, loss=2.88, lr=0.001]
train:   0%|          | 14/26291 [00:03<1:41:09,  4.33it/s, total_it=12413]
epochs:   0%|          | 0/5 [00:04<?, ?it/s, loss=4.2, lr=0.001]
train:   0%|          | 15/26291 [00:04<1:40:32,  4.36it/s, total_it=12414]
epochs:   0%|          | 0/5 [00:04<?, ?it/s, loss=nan, lr=0.001]WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
Warning: Sparse_Idx_Shape(torch.Size([0, 4])) 
train:   0%|          | 16/26291 [00:04<1:35:34,  4.58it/s, total_it=12415]
epochs:   0%|          | 0/5 [00:04<?, ?it/s, loss=nan, lr=0.001]WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
maxoverlaps:(min=nan, max=nan)
ERROR: FG=0, BG=0
epochs:   0%|          | 0/5 [00:04<?, ?it/s, loss=nan, lr=0.001]
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/hcyoo/Programs/pycharm/pycharm-professional-2020.2/pycharm-2020.2/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/hcyoo/Programs/pycharm/pycharm-professional-2020.2/pycharm-2020.2/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/train.py", line 206, in <module>
    main()
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/train.py", line 178, in main
    merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/train_utils/train_utils.py", line 93, in train_model
    dataloader_iter=dataloader_iter
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/train_utils/train_utils.py", line 38, in train_one_epoch
    loss, tb_dict, disp_dict = model_func(model, batch)
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/__init__.py", line 30, in model_func
    ret_dict, tb_dict, disp_dict = model(batch_dict)
  File "/home/hcyoo/anaconda3/envs/pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/detectors/PartA2_net.py", line 11, in forward
    batch_dict = cur_module(batch_dict)
  File "/home/hcyoo/anaconda3/envs/pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/partA2_head.py", line 175, in forward
    targets_dict = self.assign_targets(batch_dict)
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/roi_head_template.py", line 104, in assign_targets targets_dict = self.proposal_target_layer.forward(batch_dict)
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 33, in forward
    batch_dict=batch_dict
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 107, in sample_rois_for_rcnn
    sampled_inds = self.subsample_rois(max_overlaps=max_overlaps)
  File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 159, in subsample_rois
    raise NotImplementedError
NotImplementedError

[hcyoo93]

Changing version from "pytorch 1.5, spconv 1.2" to "pytorch1.1, spconv1.0" resolves the issue.
I think, anyone who has trouble with spconv can consider downgrading it

[yustaub]

Hi, sir. Do you convert the nuscenes dataset to kitti format and trained with PartA2? May I ask how I can transform nuscenes to kitti, I want to train pv-rcnn on nuscenes but has no idea about it. Very appreciate for your reply. @hcyoo93

[hcyoo93]

Hi, @yustaub

Yes, I convert Nuscenes to KITTI since I want to do cross-evaluation of models on the dataset.

I think you can refer https://github.com/nutonomy/nuscenes-devkit/blob/02e9200218977193a1058dd7234f935834378319/python-sdk/nuscenes/scripts/export_kitti.py

[yustaub]

Sorry to bother you again. Do you also change the categories of the original PartA2 config, or just use the original config? @hcyoo93

[hcyoo93]

Hi, @yustaub

I think it depends on you.

You need to change anchor size, point cloud range, etc.
Note that the anchor size is the average instance size of the dataset.
Also, Nuscenes predict 360 degrees while KITTI only cares about front cam FOV.

you can refer to this paper for cross-dataset training and evaluation
https://arxiv.org/pdf/2005.08139.pdf

[yustaub]

thanks for your reply! Appreciate!

[JianyyuWang]

Changing version from "pytorch 1.5, spconv 1.2" to "pytorch1.1, spconv1.0" resolves the issue. I think, anyone who has trouble with spconv can consider downgrading it

hi, i used your way to solve the problem,but failed. do you have some other methods?

[lcc815]

Changing version from "pytorch 1.5, spconv 1.2" to "pytorch1.1, spconv1.0" resolves the issue. I think, anyone who has trouble with spconv can consider downgrading it

hi, i used your way to solve the problem,but failed. do you have some other methods?

same question! Ask for help.

[Tottowich]

Did you find any other solution to

OpenPCDet/tools/../pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 159, in subsample_rois
    raise NotImplementedError

@JianyyuWang @lcc815 ?