SfM ToAEval: Trade-off Aware Evaluation of Feature Extraction Algorithms in Structure from Motion

Abstract

This paper presents SfM ToAEval, a framework for evaluating different feature extraction algorithms in Structure from Motion (SfM) pipelines. SfM ToAEval allows automatic evaluation of the effect of using different feature detectors and descriptors combinations on the quality of the 3D reconstruction of stationary objects or scenes from a given collection of image sequences. In addition, SfM ToAEval evaluates the 3D reconstruction without the need for ground truth. Moreover, SfM ToAEval is aware of the reconstruction density-accuracy trade-off, and it supports visualizing it to enable deciding the “best” reconstruction transparently. Furthermore, SfM ToAEval allows quantifying the quality of each 3D reconstruction compared to others. SfM ToAEval was used to evaluate 98 feature detectors and descriptors combinations on six image sequences, and it was able to identify four promising combinations. Experimental results comparing the proposed combinations with related work are presented in this research. The complete source code of the proposed framework as well as a minimal Jupyter Notebook demonstrating how different functionalities can be used are released under the MIT license.