Approach

We propose a simple two-step approach for speeding up convolution layers within large convolutional neural networks based on tensor decomposition and discriminative fine-tuning. Given a layer, we use non-linear least squares to compute a low-rank CP-decomposition of the 4D convolution kernel tensor into a sum of a small number of rank-one tensors. At the second step, this decomposition is used to replace the original convolutional layer with a sequence of four convolutional layers with small kernels. After such replacement, the entire network is fine tuned on the training data using standard backpropagation process.

The rank-R CP-decomposition (2) of the 4D kernel tensor has the form:

where Kx, Ky, Ks, Kt are the four components of the composition representing 2D tensors (matrices) of sizes d×R, d×R, S×R, and T×R respectively.

Experiment

References:
SPEEDING-UP CONVOLUTIONAL NEURAL NETWORKS USING FINE-TUNED CP-DECOMPOSITION, Vadim Lebedev, 2015, ICLR