Skip to content

Conversation

@chinmay0301ucsd
Copy link

This PR is regarding the issue #79. One of the reasons for performance regression on doors and windows comes from the evaluation script eval.py. I found that in lines 181 :

attach_wall = wall_id_lookup.get(entity.id, None)

and 200

attach_wall = wall_id_lookup.get(entity.id, None)

of the file eval.py, entity.wall_id should be used instead of entity.id, as that would actually fetch the correct wall id of the attached entity (door / window) from the wall id lookup dictionary. This change affects only the door and window evaluation numbers, not the wall numbers. On making this change, I re-ran 3 evaluations

  1. Finetuned the SpatiaLM Model SpatialLM1.1-Qwen-0.5B on structured3D train set with the labels and gt provided by the authors, and evaluated it on the structured3D test set -
    | Layouts | F1 @.25 IoU | F1 @.50 IoU |
    |---------|-------------|-------------|
    | wall | 0.9300944007413725 | 0.9158097842802695 |
    | door | 0.9301076299430355 | 0.9245620838810897 |
    | window | 0.892983727373873 | 0.8768681686048276 |

  2. SpatialLM model finetuned on Structured3D provided by the authors https://huggingface.co/ysmao/SpatialLM1.1-Qwen-0.5B-Structured3D-SFT evaluated on structured3D test set.
    | Layouts | F1 @.25 IoU | F1 @.50 IoU |
    |---------|-------------|-------------|
    | wall | 0.9418111124360891 | 0.9318676751934882 |
    | door | 0.9503952201855035 | 0.9459975133713943 |
    | window | 0.9055098651958046 | 0.892037246299918 |

  3. SpatialLM Model SpatialLM1.1-Qwen-0.5B without any finetuning evaluated on the structured3D test set.
    | Layouts | F1 @.25 IoU | F1 @.50 IoU |
    |---------|-------------|--------------|
    | wall | 0.7516214808780001 | 0.6963917824685191 |
    | door | 0.4925566276008396 | 0.4578708554555917 |
    | window | 0.6665149238502482 | 0.27879356972595465 |

For the all the 3 results above, the performance for doors and windows improves significantly to the previous results obtained in the discussion in #79.

@chinmay0301ucsd chinmay0301ucsd marked this pull request as ready for review August 29, 2025 21:46
@SamMaoYS
Copy link
Collaborator

Thank you for carefully reviewing and catching the issues in the evaluation script for the door and window instances!

@SamMaoYS SamMaoYS merged commit 576b543 into manycore-research:dev Aug 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants