基于深度學(xué)習(xí)的單目深度估計(jì)在近幾年是比較熱門(mén)的研究方向之一，MIT的Diana Wofk等人在ICRA 2019上提出了一種用于嵌入式系統(tǒng)的深度估計(jì)算法FastDepth，在保證準(zhǔn)確率的情況下，大大提高了模型的計(jì)算效率。
論文：FastDepth: Fast Monocular Depth Estimation on Embedded Systems
Offical Pytorch：https://github.com/dwofk/fast-depth
方法
模型
模型的整體結(jié)構(gòu)比較簡(jiǎn)單，采用了Encoder-Decoder的架構(gòu)。Encoder部分采用了MobileNet模型提取到7x7x1024的特征；Decoder部分采用了5次上采樣，中間三次上采樣結(jié)果通過(guò)Skip Connections的方法分別與Encoder部分的特征進(jìn)行了特征融合，為了減小上采樣部分的通道特征，還使用了5x5的卷積來(lái)降維；最后使用1*1的卷積得到深度圖。
Model
使用Keras實(shí)現(xiàn)基本的FastDepth模型：
from keras.layers import Conv2D, UpSampling2D, SeparableConv2D, BatchNormalization, Activation, add
from keras.models import Model
from keras.applications.mobilenet import MobileNet
class FastDepth:
def __init__(self):
self.build_net()
def _SDWConv(self, filtres, kernel):
def f(x):
x = SeparableConv2D(filtres, kernel, padding='same')(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)
return x
return f
def _encoder(self):
self.MN = MobileNet(input_shape=(224, 224, 3),
weights=None,
include_top='False')
# 7*7*1024
latent = self.MN.get_layer('conv_pw_13_relu').output
return latent
def _decoder(self, x):
# 14*14*512
x1 = self._SDWConv(512, (5, 5))(x)
x1 = UpSampling2D()(x1)
# 28*28*256
x2 = self._SDWConv(256, (5, 5))(x1)
x2 = UpSampling2D()(x2)
s2 = self.MN.get_layer('conv_pw_5_relu').output
x2 = add([x2, s2])
# 56*56*128
x3 = self._SDWConv(128, (5, 5))(x2)
x3 = UpSampling2D()(x3)
s3 = self.MN.get_layer('conv_pw_3_relu').output
x3 = add([x3, s3])
# 112*112*64
x4 = self._SDWConv(64, (5, 5))(x3)
x4 = UpSampling2D()(x4)
s4 = self.MN.get_layer('conv_pw_1_relu').output
x4 = add([x4, s4])
# 224*224*32
x5 = self._SDWConv(32, (5, 5))(x4)
x5 = UpSampling2D()(x5)
return x5
def build_net(self):
latent = self._encoder()
out = self._decoder(latent)
out_dense = Conv2D(1, (1, 1))(out)
self.model = Model(inputs=self.MN.input, outputs=out_dense)
if __name__ == '__main__':
net = FastDepth()
net.model.summary()
Decoder部分的結(jié)構(gòu)如下所示：
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
省略MobileNet...
__________________________________________________________________________________________________
conv_pw_13_relu (ReLU) (None, 7, 7, 1024) 0 conv_pw_13_bn[0][0]
__________________________________________________________________________________________________
separable_conv2d_3 (SeparableCo (None, 7, 7, 512) 550400 conv_pw_13_relu[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 7, 7, 512) 2048 separable_conv2d_3[0][0]
__________________________________________________________________________________________________
activation_1 (Activation) (None, 7, 7, 512) 0 batch_normalization_1[0][0]
__________________________________________________________________________________________________
up_sampling2d_3 (UpSampling2D) (None, 14, 14, 512) 0 activation_1[0][0]
__________________________________________________________________________________________________
separable_conv2d_4 (SeparableCo (None, 14, 14, 256) 144128 up_sampling2d_3[0][0]
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 14, 14, 256) 1024 separable_conv2d_4[0][0]
__________________________________________________________________________________________________
activation_2 (Activation) (None, 14, 14, 256) 0 batch_normalization_2[0][0]
__________________________________________________________________________________________________
up_sampling2d_4 (UpSampling2D) (None, 28, 28, 256) 0 activation_2[0][0]
__________________________________________________________________________________________________
add_2 (Add) (None, 28, 28, 256) 0 up_sampling2d_4[0][0]
conv_pw_5_relu[0][0]
__________________________________________________________________________________________________
separable_conv2d_5 (SeparableCo (None, 28, 28, 128) 39296 add_2[0][0]
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 28, 28, 128) 512 separable_conv2d_5[0][0]
__________________________________________________________________________________________________
activation_3 (Activation) (None, 28, 28, 128) 0 batch_normalization_3[0][0]
__________________________________________________________________________________________________
up_sampling2d_5 (UpSampling2D) (None, 56, 56, 128) 0 activation_3[0][0]
__________________________________________________________________________________________________
add_3 (Add) (None, 56, 56, 128) 0 up_sampling2d_5[0][0]
conv_pw_3_relu[0][0]
__________________________________________________________________________________________________
separable_conv2d_6 (SeparableCo (None, 56, 56, 64) 11456 add_3[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 56, 56, 64) 256 separable_conv2d_6[0][0]
__________________________________________________________________________________________________
activation_4 (Activation) (None, 56, 56, 64) 0 batch_normalization_4[0][0]
__________________________________________________________________________________________________
up_sampling2d_6 (UpSampling2D) (None, 112, 112, 64) 0 activation_4[0][0]
__________________________________________________________________________________________________
add_4 (Add) (None, 112, 112, 64) 0 up_sampling2d_6[0][0]
conv_pw_1_relu[0][0]
__________________________________________________________________________________________________
separable_conv2d_7 (SeparableCo (None, 112, 112, 32) 3680 add_4[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 112, 112, 32) 128 separable_conv2d_7[0][0]
__________________________________________________________________________________________________
activation_5 (Activation) (None, 112, 112, 32) 0 batch_normalization_5[0][0]
__________________________________________________________________________________________________
up_sampling2d_7 (UpSampling2D) (None, 224, 224, 32) 0 activation_5[0][0]
__________________________________________________________________________________________________
conv2d_1 (Conv2D) (None, 224, 224, 1) 33 up_sampling2d_7[0][0]
==================================================================================================
Total params: 3,981,825
Trainable params: 3,957,953
Non-trainable params: 23,872
__________________________________________________________________________________________________
網(wǎng)絡(luò)裁剪
為了減小模型體積，提高運(yùn)算效率，使得模型更適用于嵌入式設(shè)備，使用NetAdapt算法對(duì)FastDepth進(jìn)行了裁剪。
NerAdapt
實(shí)驗(yàn)結(jié)果
模型在NYU Depth V2 dataset上進(jìn)行了訓(xùn)練，基本實(shí)驗(yàn)結(jié)果如下圖所示?？梢钥闯稣撐奶岢龅腇astDepth算法相較當(dāng)前準(zhǔn)確率最高的算法低了4%，但是運(yùn)算速度有著大幅提升，因此特別適用于嵌入式設(shè)備。
GPU
error.png
下圖是深度估計(jì)的可視化效果：
vis
下圖是不同方法下Encoder和Decoder部分的運(yùn)算效率和準(zhǔn)確率，可以看出論文提出的方法運(yùn)算速度非?？欤褼epthwise、Skip Connections和網(wǎng)絡(luò)裁剪這三個(gè)技巧可以大幅提高運(yùn)算效率而且對(duì)準(zhǔn)確率的影響比較小。
speed
