The KITTI Vision Benchmark Suite

We propose a deep neural network architecture to
infer dense depth from an image and a sparse
point
cloud. It is trained using a video stream and
corresponding synchronized sparse point cloud, as
obtained from a LIDAR or other range sensor,
along
with the intrinsic calibration parameters of the
camera. At inference time, the calibration of the
camera, which can be different than the one used
for
training, is fed as an input to the network along
with the sparse point cloud and a single image.
A Calibrated Backprojection Layer backprojects
each
pixel in the image to three-dimensional space
using
the calibration matrix and a depth feature
descriptor. The resulting 3D positional encoding
is
concatenated with the image descriptor and the
previous layer output to yield the input to the
next
layer of the encoder. A decoder, exploiting skip-
connections, produces a dense depth map. The
resulting Calibrated Backprojection Network, or
KBNet, is trained without supervision.

w_{ph} = 1, w_{co}=0.15, w_{st}=0.95, w_{sz} = 0.6,
and w_{sm} = 0.04

@inproceedings{wong2021unsupervised,
title={Unsupervised Depth Completion with
Calibrated Backprojection Layers},
author={Wong, Alex and Soatto, Stefano},
booktitle={Proceedings of the IEEE International
Conference on Computer Vision (ICCV)},
year={2021}
}