Unified Query Learning for 3D Plane Recovery from a Single View

Jingjia Shi
Shuaifeng Zhi
Kai Xu

National University of Defense Technology

ICCV 2023

[video Y / B]

Our core idea is to use unified query learning to jointly model all subtasks of single-view plane recovery and output plane-level predictions, which achieves mutual benefits among tasks and enables higher performance.

3D plane recovery from a single image can usually be divided into several subtasks of plane detection, segmentation, parameter estimation and possibly depth estimation. Previous works tend to solve this task by either extending the RCNN-based segmentation network or the dense pixel embedding-based clustering framework. However, none of them tried to integrate above related subtasks into a unified framework but treat them separately and sequentially, which we suspect is potentially a main source of performance limitation for existing approaches. Motivated by this finding and the success of query-based learning in enriching reasoning among semantic entities, in this paper, we propose PlaneRecTR, a Transformer-based architecture, which for the first time unifies all subtasks related to single-view plane recovery with a single compact model. Extensive quantitative and qualitative experiments demonstrate that our proposed unified learning achieves mutual benefits across subtasks, obtaining a new state-of-the-art performance on public ScanNet and NYUv2-Plane datasets.


[bilibili video]

Interactive Results

Ours Seg
GT Seg
Ours Models
GT Models


  author={Shi, Jingjia and Zhi, Shuaifeng and Xu, Kai},
  title={PlaneRecTR: Unified Query learning for 3D Plane Recovery from a Single View}, 


We thank the anonymous reviewers and AC of ICCV23 for their valuable comments, and authors of PlaneTR for the helpful comments on evaluations.

This work was supported in part by the National Key Research and Development Program of China (2018AAA0102200), NSFC (62325211, 62132021, 62201603) and Research Program of National University of Defense Technology (Grant No. ZK22-04).

This webpage was adapted from Sparse Planes. The interactive examples are powered by model-viewer.