using entity.wall_id instead of entity.id for wall lookup #88
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is regarding the issue #79. One of the reasons for performance regression on doors and windows comes from the evaluation script
eval.py. I found that in lines 181 :and 200
of the file
eval.py,entity.wall_idshould be used instead ofentity.id, as that would actually fetch the correct wall id of the attached entity (door / window) from the wall id lookup dictionary. This change affects only the door and window evaluation numbers, not the wall numbers. On making this change, I re-ran 3 evaluationsFinetuned the SpatiaLM Model SpatialLM1.1-Qwen-0.5B on structured3D train set with the labels and gt provided by the authors, and evaluated it on the structured3D test set -
| Layouts | F1 @.25 IoU | F1 @.50 IoU |
|---------|-------------|-------------|
| wall | 0.9300944007413725 | 0.9158097842802695 |
| door | 0.9301076299430355 | 0.9245620838810897 |
| window | 0.892983727373873 | 0.8768681686048276 |
SpatialLM model finetuned on Structured3D provided by the authors https://huggingface.co/ysmao/SpatialLM1.1-Qwen-0.5B-Structured3D-SFT evaluated on structured3D test set.
| Layouts | F1 @.25 IoU | F1 @.50 IoU |
|---------|-------------|-------------|
| wall | 0.9418111124360891 | 0.9318676751934882 |
| door | 0.9503952201855035 | 0.9459975133713943 |
| window | 0.9055098651958046 | 0.892037246299918 |
SpatialLM Model SpatialLM1.1-Qwen-0.5B without any finetuning evaluated on the structured3D test set.
| Layouts | F1 @.25 IoU | F1 @.50 IoU |
|---------|-------------|--------------|
| wall | 0.7516214808780001 | 0.6963917824685191 |
| door | 0.4925566276008396 | 0.4578708554555917 |
| window | 0.6665149238502482 | 0.27879356972595465 |
For the all the 3 results above, the performance for doors and windows improves significantly to the previous results obtained in the discussion in #79.