BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities

Boris Meden · Asma Brazi · Fabrice Mayran de Chamisso
Steve Bourgeois · Vincent Lepetit

WACV 2026 (Oral)

Arxiv · Publication · Supplementary

BOP-Distrib Toolkit

Official implementation of the paper "BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities", Meden B., Brazi A., Mayran de Chamisso F., Bourgeois S. and Lepetit V., WACV 2026 (https://cea-list.github.io/BOP-Distrib).

• bop_distrib_toolkit_lib - The core Python library for calculation of pose errors.
• external - Original BOP Toolkit.
• scripts - Scripts for evaluation, rendering of training images, visualization of 6D object poses etc.

Extends the standard BOP Toolkit to evaluate methods that output a distribution of pose hypotheses rather than a single pose per object. Supports distribution-aware recall and precision metrics with MSD (Maximum Surface Distance) and MPD (Maximum Projection Distance) error functions.

---

Table of Contents

• Installation
  • Environment
  • Dataset
  • Dataset Directory Structure Novelty
• Usage
  • 1. Generate and format results
  • 2. Run evaluation
  • Command-line arguments
  • Error types
  • Output
• Reference Method Evaluation
  • Reference data
  • Reference results
• Project Structure
• References
• Dependencies and License

---

Installation

Environment

Download the code with the git submodules and navigate to the folder:

git clone --recurse-submodules https://github.com/CEA-LIST/BOP-Distrib_code
cd BOP-Distrib_code

Follow installation of the BOP Toolkit https://github.com/thodan/bop_toolkit.

Next, use the following environment variables to set your directories:

#!/bin/sh

export REPO_PATH=/path/to/bop-distrib/repository  # Replace with the path to the BOP-Distrib repository.
export BOP_PATH=/path/to/bop/datasets  # Replace with the path to BOP datasets (https://bop.felk.cvut.cz/datasets).
export PYTHONPATH=$REPO_PATH:$REPO_PATH/external/bop_toolkit

If you use conda, you can create (or update) the conda environment activation script to set the necessary environment variables. This script will run automatically when you activate the environment.

The activation script is typically located at $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh. You can find $CONDA_PREFIX by running:

conda info --envs

If the env_vars.sh file does not exist, create it.

Dataset

Download the BOP datasets from here and ensure they are organized in the expected folder structure.

Update the output_path in the BOP config file located at external/bop_toolkit/bop_toolkit_lib/config.py to point to the root directory of your BOP datasets.

Directory Structure Novelty

Your BOP datasets should be organized in the following directory structure:

bop_datasets/                                     # This is your $BOP_PATH
├── tless/                                        # Dataset directory for T-LESS
│   ├── camera.json
│   ├── dataset_info.md
│   ├── models_cad/                               # 3D models of the objects
│   ├── models_eval/                              # Simplified models for evaluation
│   ├── test_primesense/                          # Test images and annotations
│       ├── 000001/
│              ├── depth/
│              ├── mask/
│              ├── mask_visib/
│              ├── rgb/
│              ├── scene_camera.json
│              ├── scene_gt.json
│              ├── scene_gt_info.json
│              ├── scene_gt_distribution.json   # Ground truth distribution per instance in each image. Available [here](https://huggingface.co/datasets/CEAai/bop_distrib_tless)
│       └── ...
├── tudl/ 
├── ...
├── detections/
│   └── cnos-fastsam/
│       ├── cnos-fastsam_lmo_test.json
│       ├── cnos-fastsam_tudl_test.json
│       └── ...
└── ...

Right now, only T-LESS is supported. Annotations are available for download at https://huggingface.co/datasets/CEAai/bop_distrib_tless

---

Usage

1. Generate and format results

Your method must produce:

Per-scene pose distribution JSON files (scene{N}.json):

{
  "{im_id}": [
    {
      "obj_id": 4,
      "score": 0.95,
      "cam_R_m2c": [[...], [...], [...]],    # List of the rotations matrices of the distribution
      "cam_t_m2c": [..., ..., ...],          # List of the translation vectors of the distribution
      "p": [0.4, 0.35, 0.25]                 # List of per-mode probabilities (must sum to 1 across modes for the same `(im_id, obj_id, instance_id)`)
    },
    ...
  ]
}

2. Run evaluation

python scripts/eval_bop_distributionPose.py \
  --result_filenames=/path/to/distributions_directory \
  --results_path=/path/to/results \
  --eval_path=/path/to/eval_output \
  --num_workers=10

Command-line arguments

Argument	Default	Description
--result_filenames	(required)	Comma-separated CSV filenames
--results_path	config.results_path	Folder containing the CSV files
--eval_path	config.eval_path	Output folder for errors and scores
--targets_filename	test_targets_bop19.json	BOP test targets file
--num_workers	config.num_workers	Parallel workers for error computation
--renderer_type	vispy	Renderer: vispy, cpp, python
--error_types	distRecallMPD,distPrecisionMPD,distRecallMSD,distPrecisionMSD	Comma-separated error types
--msd_thresholds	0.05,0.10,0.15,0.20	MSD thresholds (normalized by diameter)
--mpd_thresholds	5,10,15,20	MPD thresholds (pixels, normalized to 640px width)

Error types

Error type	Description	Units
distRecallMSD	Distribution recall w.r.t. Maximum Surface Distance	mm (normalized by object diameter)
distPrecisionMSD	Distribution precision w.r.t. MSD	mm (normalized by object diameter)
distRecallMPD	Distribution recall w.r.t. Maximum Projection Distance	px (normalized to 640px width)
distPrecisionMPD	Distribution precision w.r.t. MPD	px (normalized to 640px width)

Recall: Fraction of GT poses covered by at least one estimate within threshold. Precision: Weighted (by mode probability) fraction of estimates that match a GT within threshold.

Output

The evaluation produces:

{eval_path}/
└── {result_name}/
    ├── {error_sign}/
    │   ├── errors_{scene_id:06d}.json    # Per-scene pose errors
    │   ├── matches_{score_sign}.json     # Established GT↔estimate matches
    │   └── scores_{score_sign}.json      # Per-scene/per-object scores
    └── scores_bop19.json                 # Final aggregated scores

scores_bopDistrib.json contains:

{
  "bop19_average_distRecallMPD": -,
  "bop19_average_distPrecisionMPD": -,
  "bop19_average_distRecallMSD": -,
  "bop19_average_distPrecisionMSD": --
}

---

Reference Method Evaluation

Reference data

As an example, we provide estimation results here: https://huggingface.co/datasets/CEAai/bop_distrib_tless/blob/main/referenceEstimatedDistributions_tless-test.zip

Reference results

Error type	Description
bop19_average_distRecallMPD	0.4541203379426082
bop19_average_distPrecisionMPD	0.4390807218107545
bop19_average_distRecallMSD	0.30488544409832424
bop19_average_distPrecisionMSD	0.30288745616571394

---

Project Structure

bop_distrib_toolkit/
├── bop_distrib_toolkit_lib/
│   ├── pose_distrib_error.py                 # Error functions (MSD, MPD, recall, precision)
│   └── vis_SO3.py                            # Distribution visualization helper
├── scripts/
│   ├── eval_bop_distributionPose.py          # Entry point — orchestrates evaluation
│   ├── eval_calc_errors_distributionPose.py  # Worker — computes per-pose errors
│   └── eval_calc_scores_distributionPose.py  # Worker — matches poses, computes scores
└── README.md

---

References

• BOP Benchmark: http://bop.felk.cvut.cz
• BOP Toolkit: https://github.com/thodan/bop_toolkit
• Meden et al., "BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities", WACV 2026

---

Citation

If you find this work useful, please cite our paper:

@article{meden2026bopd,
title={BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities},
author={Meden, Boris and Brazi, Asma and Mayran de Chamisso, Fabrice and Bourgeois, Steve and Lepetit, Vincent},
journal={Winter Conference on Applications of Computer Vision (WACV)},
year={2026}
}

---

Dependencies and Licenses

This project includes modified code from Apache Commons Lang, licensed under the Apache License 2.0. This project also builds on the BOP Toolkit, licensed under the [MIT license]