聲明：作者翻譯論文僅為學(xué)習(xí)，如有侵權(quán)請聯(lián)系作者刪除博文，謝謝！
翻譯論文匯總：https://github.com/SnailTyan/deep-learning-papers-translation
Deep Residual Learning for Image Recognition
Abstract
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers——8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers.
摘要
更深的神經(jīng)網(wǎng)絡(luò)更難訓(xùn)練。我們提出了一種殘差學(xué)習(xí)框架來減輕網(wǎng)絡(luò)訓(xùn)練，這些網(wǎng)絡(luò)比以前使用的網(wǎng)絡(luò)更深。我們明確地將層變?yōu)閷W(xué)習(xí)關(guān)于層輸入的殘差函數(shù)，而不是學(xué)習(xí)未參考的函數(shù)。我們提供了全面的經(jīng)驗證據(jù)說明這些殘差網(wǎng)絡(luò)很容易優(yōu)化，并可以顯著增加深度來提高準(zhǔn)確性。在ImageNet數(shù)據(jù)集上我們評估了深度高達152層的殘差網(wǎng)絡(luò)——比VGG[40]深8倍但仍具有較低的復(fù)雜度。這些殘差網(wǎng)絡(luò)的集合在ImageNet測試集上取得了3.57%的錯誤率。這個結(jié)果在ILSVRC 2015分類任務(wù)上贏得了第一名。我們也在CIFAR-10上分析了100層和1000層的殘差網(wǎng)絡(luò)。
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
對于許多視覺識別任務(wù)而言，表示的深度是至關(guān)重要的。僅由于我們非常深度的表示，我們便在COCO目標(biāo)檢測數(shù)據(jù)集上得到了28%的相對提高。深度殘差網(wǎng)絡(luò)是我們向ILSVRC和COCO 2015競賽提交的基礎(chǔ)，我們也贏得了ImageNet檢測任務(wù)，ImageNet定位任務(wù)，COCO檢測和COCO分割任務(wù)的第一名。
1. Introduction
Deep convolutional neural networks [22, 21] have led to a series of breakthroughs for image classification [21, 49, 39]. Deep networks naturally integrate low/mid/high-level features [49] and classifiers in an end-to-end multi-layer fashion, and the “l(fā)evels” of features can be enriched by the number of stacked layers (depth). Recent evidence [40, 43] reveals that network depth is of crucial importance, and the leading results [40, 43, 12, 16] on the challenging ImageNet dataset [35] all exploit “very deep” [40] models, with a depth of sixteen [40] to thirty [16]. Many other non-trivial visual recognition tasks [7, 11, 6, 32, 27] have also greatly benefited from very deep models.
1. 引言
深度卷積神經(jīng)網(wǎng)絡(luò)[22, 21]導(dǎo)致了圖像分類[21, 49, 39]的一系列突破。深度網(wǎng)絡(luò)自然地將低/中/高級特征[49]和分類器以端到端多層方式進行集成，特征的“級別”可以通過堆疊層的數(shù)量（深度）來豐富。最近的證據(jù)[40, 43]顯示網(wǎng)絡(luò)深度至關(guān)重要，在具有挑戰(zhàn)性的ImageNet數(shù)據(jù)集上領(lǐng)先的結(jié)果都采用了“非常深”[40]的模型，深度從16 [40]到30 [16]之間。許多其它重要的視覺識別任務(wù)[7, 11, 6, 32, 27]也從非常深的模型中得到了極大受益。
Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers? An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [14, 1, 8], which hamper convergence from the beginning. This problem, however, has been largely addressed by normalized initialization [23, 8, 36, 12] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].
在深度重要性的推動下，出現(xiàn)了一個問題：學(xué)些更好的網(wǎng)絡(luò)是否像堆疊更多的層一樣容易？回答這個問題的一個障礙是梯度消失/爆炸[14, 1, 8]這個眾所周知的問題，它從一開始就阻礙了收斂。然而，這個問題通過標(biāo)準(zhǔn)初始化[23, 8, 36, 12]和中間標(biāo)準(zhǔn)化層[16]在很大程度上已經(jīng)解決，這使得數(shù)十層的網(wǎng)絡(luò)能通過具有反向傳播的隨機梯度下降（SGD）開始收斂。
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [10, 41] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
Figure 1. Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper network has higher training error, and thus test error. Similar phenomena on ImageNet is presented in Fig. 4.
當(dāng)更深的網(wǎng)絡(luò)能夠開始收斂時，暴露了一個退化問題：隨著網(wǎng)絡(luò)深度的增加，準(zhǔn)確率達到飽和（這可能并不奇怪）然后迅速下降。意外的是，這種下降不是由過擬合引起的，并且在適當(dāng)?shù)纳疃饶Ｐ蜕咸砑痈嗟膶訒?dǎo)致更高的訓(xùn)練誤差，正如[10, 41]中報告的那樣，并且由我們的實驗完全證實。圖1顯示了一個典型的例子。
圖1 20層和56層的“簡單”網(wǎng)絡(luò)在CIFAR-10上的訓(xùn)練誤差（左）和測試誤差（右）。更深的網(wǎng)絡(luò)有更高的訓(xùn)練誤差和測試誤差。ImageNet上的類似現(xiàn)象如圖4所示。
The degradation (of training accuracy) indicates that not all systems are similarly easy to optimize. Let us consider a shallower architecture and its deeper counterpart that adds more layers onto it. There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model. The existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart. But experiments show that our current solvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time).
退化（訓(xùn)練準(zhǔn)確率）表明不是所有的系統(tǒng)都很容易優(yōu)化。讓我們考慮一個較淺的架構(gòu)及其更深層次的對象，為其添加更多的層。存在通過構(gòu)建得到更深層模型的解決方案：添加的層是恒等映射，其他層是從學(xué)習(xí)到的較淺模型的拷貝。 這種構(gòu)造解決方案的存在表明，較深的模型不應(yīng)該產(chǎn)生比其對應(yīng)的較淺模型更高的訓(xùn)練誤差。但是實驗表明，我們目前現(xiàn)有的解決方案無法找到與構(gòu)建的解決方案相比相對不錯或更好的解決方案（或在合理的時間內(nèi)無法實現(xiàn)）。
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as $H(x)$, we let the stacked nonlinear layers fit another mapping of $F(x) := H(x) ? x$. The original mapping is recast into $F(x) + x$. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
在本文中，我們通過引入深度殘差學(xué)習(xí)框架解決了退化問題。我們明確地讓這些層擬合殘差映射，而不是希望每幾個堆疊的層直接擬合期望的基礎(chǔ)映射。形式上，將期望的基礎(chǔ)映射表示為$H(x)$，我們將堆疊的非線性層擬合另一個映射$F(x) := H(x) ? x$。原始的映射重寫為$F(x) + x$。我們假設(shè)殘差映射比原始的、未參考的映射更容易優(yōu)化。在極端情況下，如果一個恒等映射是最優(yōu)的，那么將殘差置為零比通過一堆非線性層來擬合恒等映射更容易。
The formulation of $F (x) + x$ can be realized by feedforward neural networks with “shortcut connections” (Fig. 2). Shortcut connections [2, 33, 48] are those skipping one or more layers. In our case, the shortcut connections simply perform identity mapping, and their outputs are added to the outputs of the stacked layers (Fig. 2). Identity shortcut connections add neither extra parameter nor computational complexity. The entire network can still be trained end-to-end by SGD with backpropagation, and can be easily implemented using common libraries (e.g., Caffe [19]) without modifying the solvers.
Figure 2. Residual learning: a building block.
公式$F (x) + x$可以通過帶有“快捷連接”的前向神經(jīng)網(wǎng)絡(luò)（圖2）來實現(xiàn)?？旖葸B接[2, 33, 48]是那些跳過一層或更多層的連接。在我們的案例中，快捷連接簡單地執(zhí)行恒等映射，并將其輸出添加到堆疊層的輸出（圖2）。恒等快捷連接既不增加額外的參數(shù)也不增加計算復(fù)雜度。整個網(wǎng)絡(luò)仍然可以由帶有反向傳播的SGD進行端到端的訓(xùn)練，并且可以使用公共庫（例如，Caffe [19]）輕松實現(xiàn)，而無需修改求解器。
圖2. 殘差學(xué)習(xí)：構(gòu)建塊
We present comprehensive experiments on ImageNet [35] to show the degradation problem and evaluate our method. We show that: 1) Our extremely deep residual nets are easy to optimize, but the counterpart “plain” nets (that simply stack layers) exhibit higher training error when the depth increases; 2) Our deep residual nets can easily enjoy accuracy gains from greatly increased depth, producing results substantially better than previous networks.
我們在ImageNet[35]上進行了綜合實驗來顯示退化問題并評估我們的方法。我們發(fā)現(xiàn)：1）我們極深的殘差網(wǎng)絡(luò)易于優(yōu)化，但當(dāng)深度增加時，對應(yīng)的“簡單”網(wǎng)絡(luò)（簡單堆疊層）表現(xiàn)出更高的訓(xùn)練誤差；2）我們的深度殘差網(wǎng)絡(luò)可以從大大增加的深度中輕松獲得準(zhǔn)確性收益，生成的結(jié)果實質(zhì)上比以前的網(wǎng)絡(luò)更好。
Similar phenomena are also shown on the CIFAR-10 set [20], suggesting that the optimization difficulties and the effects of our method are not just akin to a particular dataset. We present successfully trained models on this dataset with over 100 layers, and explore models with over 1000 layers.
CIFAR-10數(shù)據(jù)集上[20]也顯示出類似的現(xiàn)象，這表明了優(yōu)化的困難以及我們的方法的影響不僅僅是針對一個特定的數(shù)據(jù)集。我們在這個數(shù)據(jù)集上展示了成功訓(xùn)練的超過100層的模型，并探索了超過1000層的模型。
On the ImageNet classification dataset [35], we obtain excellent results by extremely deep residual nets. Our 152-layer residual net is the deepest network ever presented on ImageNet, while still having lower complexity than VGG nets [40]. Our ensemble has 3.57% top-5 error on the ImageNet test set, and won the 1st place in the ILSVRC 2015 classification competition. The extremely deep representations also have excellent generalization performance on other recognition tasks, and lead us to further win the 1st places on: ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation in ILSVRC & COCO 2015 competitions. This strong evidence shows that the residual learning principle is generic, and we expect that it is applicable in other vision and non-vision problems.
在ImageNet分類數(shù)據(jù)集[35]中，我們通過非常深的殘差網(wǎng)絡(luò)獲得了很好的結(jié)果。我們的152層殘差網(wǎng)絡(luò)是ImageNet上最深的網(wǎng)絡(luò)，同時還具有比VGG網(wǎng)絡(luò)[40]更低的復(fù)雜性。我們的模型集合在ImageNet測試集上有3.57% top-5的錯誤率，并在ILSVRC 2015分類比賽中獲得了第一名。極深的表示在其它識別任務(wù)中也有極好的泛化性能，并帶領(lǐng)我們在進一步贏得了第一名：包括ILSVRC & COCO 2015競賽中的ImageNet檢測，ImageNet定位，COCO檢測和COCO分割。堅實的證據(jù)表明殘差學(xué)習(xí)準(zhǔn)則是通用的，并且我們期望它適用于其它的視覺和非視覺問題。
2. Related Work
Residual Representations. In image recognition, VLAD [18] is a representation that encodes by the residual vectors with respect to a dictionary, and Fisher Vector [30] can be formulated as a probabilistic version [18] of VLAD. Both of them are powerful shallow representations for image retrieval and classification [4, 47]. For vector quantization, encoding residual vectors [17] is shown to be more effective than encoding original vectors.
2. 相關(guān)工作
殘差表示。在圖像識別中，VLAD[18]是一種通過關(guān)于字典的殘差向量進行編碼的表示形式，F(xiàn)isher矢量[30]可以表示為VLAD的概率版本[18]。它們都是圖像檢索和圖像分類[4,47]中強大的淺層表示。對于矢量量化，編碼殘差矢量[17]被證明比編碼原始矢量更有效。
In low-level vision and computer graphics, for solving Partial Differential Equations (PDEs), the widely used Multigrid method [3] reformulates the system as subproblems at multiple scales, where each subproblem is responsible for the residual solution between a coarser and a finer scale. An alternative to Multigrid is hierarchical basis preconditioning [44, 45], which relies on variables that represent residual vectors between two scales. It has been shown [3, 44, 45] that these solvers converge much faster than standard solvers that are unaware of the residual nature of the solutions. These methods suggest that a good reformulation or preconditioning can simplify the optimization.
在低級視覺和計算機圖形學(xué)中，為了求解偏微分方程（PDE），廣泛使用的Multigrid方法[3]將系統(tǒng)重構(gòu)為在多個尺度上的子問題，其中每個子問題負責(zé)較粗尺度和較細尺度的殘差解。Multigrid的替代方法是層次化基礎(chǔ)預(yù)處理[44,45]，它依賴于表示兩個尺度之間殘差向量的變量。已經(jīng)被證明[3,44,45]這些求解器比不知道解的殘差性質(zhì)的標(biāo)準(zhǔn)求解器收斂得更快。這些方法表明，良好的重構(gòu)或預(yù)處理可以簡化優(yōu)化。
Shortcut Connections. Practices and theories that lead to shortcut connections [2, 33, 48] have been studied for a long time. An early practice of training multi-layer perceptrons (MLPs) is to add a linear layer connected from the network input to the output [33, 48]. In [43, 24], a few intermediate layers are directly connected to auxiliary classifiers for addressing vanishing/exploding gradients. The papers of [38, 37, 31, 46] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections. In [43], an “inception” layer is composed of a shortcut branch and a few deeper branches.
快捷連接。導(dǎo)致快捷連接[2,33,48]的實踐和理論已經(jīng)被研究了很長時間。訓(xùn)練多層感知機（MLP）的早期實踐是添加一個線性層來連接網(wǎng)絡(luò)的輸入和輸出[33,48]。在[43,24]中，一些中間層直接連接到輔助分類器，用于解決梯度消失/爆炸。論文[38,37,31,46]提出了通過快捷連接實現(xiàn)層間響應(yīng)，梯度和傳播誤差的方法。在[43]中，一個“inception”層由一個快捷分支和一些更深的分支組成。
Concurrent with our work, “highway networks” [41, 42] present shortcut connections with gating functions [15]. These gates are data-dependent and have parameters, in contrast to our identity shortcuts that are parameter-free. When a gated shortcut is “closed” (approaching zero), the layers in highway networks represent non-residual functions. On the contrary, our formulation always learns residual functions; our identity shortcuts are never closed, and all information is always passed through, with additional residual functions to be learned. In addition, highway networks have not demonstrated accuracy gains with extremely increased depth (e.g., over 100 layers).
和我們同時進行的工作，“highway networks” [41, 42]提出了門功能[15]的快捷連接。這些門是數(shù)據(jù)相關(guān)且有參數(shù)的，與我們不具有參數(shù)的恒等快捷連接相反。當(dāng)門控快捷連接“關(guān)閉”（接近零）時，高速網(wǎng)絡(luò)中的層表示非殘差函數(shù)。相反，我們的公式總是學(xué)習(xí)殘差函數(shù)；我們的恒等快捷連接永遠不會關(guān)閉，所有的信息總是通過，還有額外的殘差函數(shù)要學(xué)習(xí)。此外，高速網(wǎng)絡(luò)還沒有證實極度增加的深度（例如，超過100個層）帶來的準(zhǔn)確性收益。
3. Deep Residual Learning
3.1. Residual Learning
Let us consider $H(x)$ as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with $x$ denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions, then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., $H(x) ? x$ (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate $H(x)$, we explicitly let these layers approximate a residual function $F(x) := H(x) ? x$. The original function thus becomes $F(x) + x$. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.
3. 深度殘差學(xué)習(xí)
3.1. 殘差學(xué)習(xí)
我們考慮$H(x)$作為幾個堆疊層（不必是整個網(wǎng)絡(luò)）要擬合的基礎(chǔ)映射，$x$表示這些層中第一層的輸入。假設(shè)多個非線性層可以漸近地近似復(fù)雜函數(shù)，它等價于假設(shè)它們可以漸近地近似殘差函數(shù)，即$H(x) ? x$(假設(shè)輸入輸出是相同維度)。因此，我們明確讓這些層近似參數(shù)函數(shù) $F(x) := H(x) ? x$，而不是期望堆疊層近似$H(x)$。因此原始函數(shù)變?yōu)?F(x) + x$。盡管兩種形式應(yīng)該都能漸近地近似要求的函數(shù)（如假設(shè)），但學(xué)習(xí)的難易程度可能是不同的。
This reformulation is motivated by the counterintuitive phenomena about the degradation problem (Fig. 1, left). As we discussed in the introduction, if the added layers can be constructed as identity mappings, a deeper model should have training error no greater than its shallower counterpart. The degradation problem suggests that the solvers might have difficulties in approximating identity mappings by multiple nonlinear layers. With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings.
關(guān)于退化問題的反直覺現(xiàn)象激發(fā)了這種重構(gòu)（圖1左）。正如我們在引言中討論的那樣，如果添加的層可以被構(gòu)建為恒等映射，更深模型的訓(xùn)練誤差應(yīng)該不大于它對應(yīng)的更淺版本。退化問題表明求解器通過多個非線性層來近似恒等映射可能有困難。通過殘差學(xué)習(xí)的重構(gòu)，如果恒等映射是最優(yōu)的，求解器可能簡單地將多個非線性連接的權(quán)重推向零來接近恒等映射。
In real cases, it is unlikely that identity mappings are optimal, but our reformulation may help to precondition the problem. If the optimal function is closer to an identity mapping than to a zero mapping, it should be easier for the solver to find the perturbations with reference to an identity mapping, than to learn the function as a new one. We show by experiments (Fig. 7) that the learned residual functions in general have small responses, suggesting that identity mappings provide reasonable preconditioning.
Figure 7. Standard deviations (std) of layer responses on CIFAR-10. The responses are the outputs of each 3×3 layer, after BN and before nonlinearity. Top: the layers are shown in their original order. Bottom: the responses are ranked in descending order.
在實際情況下，恒等映射不太可能是最優(yōu)的，但是我們的重構(gòu)可能有助于對問題進行預(yù)處理。如果最優(yōu)函數(shù)比零映射更接近于恒等映射，則求解器應(yīng)該更容易找到關(guān)于恒等映射的抖動，而不是將該函數(shù)作為新函數(shù)來學(xué)習(xí)。我們通過實驗（圖7）顯示學(xué)習(xí)的殘差函數(shù)通常有更小的響應(yīng)，表明恒等映射提供了合理的預(yù)處理。
圖7。層響應(yīng)在CIFAR-10上的標(biāo)準(zhǔn)差（std）。這些響應(yīng)是每個3×3層的輸出，在BN之后非線性之前。上面：以原始順序顯示層。下面：響應(yīng)按降序排列。
3.2. Identity Mapping by Shortcuts
We adopt residual learning to every few stacked layers. A building block is shown in Fig. 2. Formally, in this paper we consider a building block defined as:
$$y = F(x, {W_i}) + x$$ (1)
Here $x$ and $y$ are the input and output vectors of the layers considered. The function $F(x, {W_i})$ represents the residual mapping to be learned. For the example in Fig. 2 that has two layers, $F = W_2 \sigma(W_1x)$ in which $\sigma$ denotes ReLU [29] and the biases are omitted for simplifying notations. The operation $F + x$ is performed by a shortcut connection and element-wise addition. We adopt the second nonlinearity after the addition (i.e., $\sigma(y)$, see Fig. 2).
3.2. 快捷恒等映射
我們每隔幾個堆疊層采用殘差學(xué)習(xí)。構(gòu)建塊如圖2所示。在本文中我們考慮構(gòu)建塊正式定義為：
$$y = F(x, {W_i}) + x$$ (1)
$x$和$y$是考慮的層的輸入和輸出向量。函數(shù)$F(x, {W_i})$表示要學(xué)習(xí)的殘差映射。圖2中的例子有兩層，$F = W_2 \sigma(W_1x)$中$\sigma$表示ReLU[29]，為了簡化寫法忽略偏置項。$F + x$操作通過快捷連接和各個元素相加來執(zhí)行。在相加之后我們采納了第二種非線性（即$\sigma(y)$，看圖2）。
The shortcut connections in Eqn.(1) introduce neither extra parameter nor computation complexity. This is not only attractive in practice but also important in our comparisons between plain and residual networks. We can fairly compare plain/residual networks that simultaneously have the same number of parameters, depth, width, and computational cost (except for the negligible element-wise addition).
方程(1)中的快捷連接既沒有引入外部參數(shù)又沒有增加計算復(fù)雜度。這不僅在實踐中有吸引力，而且在簡單網(wǎng)絡(luò)和殘差網(wǎng)絡(luò)的比較中也很重要。我們可以公平地比較同時具有相同數(shù)量的參數(shù)，相同深度，寬度和計算成本的簡單/殘差網(wǎng)絡(luò)（除了不可忽略的元素加法之外）。
The dimensions of $x$ and $F$ must be equal in Eqn.(1). If this is not the case (e.g., when changing the input/output channels), we can perform a linear projection $W_s$ by the shortcut connections to match the dimensions:
$$y = F(x, {W_i }) + W_sx.$$ (2)
We can also use a square matrix $W_s$ in Eqn.(1). But we will show by experiments that the identity mapping is sufficient for addressing the degradation problem and is economical, and thus $W_s$ is only used when matching dimensions.
方程(1)中$x$和$F$的維度必須是相等的。如果不是這種情況（例如，當(dāng)更改輸入/輸出通道時），我們可以通過快捷連接執(zhí)行線性投影$W_s$來匹配維度：
$$y = F(x, {W_i }) + W_sx.$$ (2)
我們也可以使用方程(1)中的方陣$W_s$。但是我們將通過實驗表明，恒等映射足以解決退化問題，并且是合算的，因此$W_s$僅在匹配維度時使用。
The form of the residual function $F$ is flexible. Experiments in this paper involve a function $F$ that has two or three layers (Fig. 5), while more layers are possible. But if $F$ has only a single layer, Eqn.(1) is similar to a linear layer: $y = W_1x + x$, for which we have not observed advantages.
Figure 5. A deeper residual function $F$ for ImageNet. Left: a building block (on 56×56 feature maps) as in Fig. 3 for ResNet-34. Right: a “bottleneck” building block for ResNet-50/101/152.
殘差函數(shù)$F$的形式是可變的。本文中的實驗包括有兩層或三層（圖5）的函數(shù)$F$，同時可能有更多的層。但如果$F$只有一層，方程(1)類似于線性層：$y = W_1x + x$，我們沒有看到優(yōu)勢。
圖5。ImageNet的深度殘差函數(shù)$F$。左：ResNet-34的構(gòu)建塊（在56×56的特征圖上），如圖3。右：ResNet-50/101/152的“bottleneck”構(gòu)建塊。
We also note that although the above notations are about fully-connected layers for simplicity, they are applicable to convolutional layers. The function $F(x,{W_i})$ can represent multiple convolutional layers. The element-wise addition is performed on two feature maps, channel by channel.
我們還注意到，為了簡單起見，盡管上述符號是關(guān)于全連接層的，但它們同樣適用于卷積層。函數(shù)$F(x，{W_i})$可以表示多個卷積層。元素加法在兩個特征圖上逐通道進行。
3.3. Network Architectures
We have tested various plain/residual nets, and have observed consistent phenomena. To provide instances for discussion, we describe two models for ImageNet as follows.
3.3. 網(wǎng)絡(luò)架構(gòu)
我們測試了各種簡單/殘差網(wǎng)絡(luò)，并觀察到了一致的現(xiàn)象。為了提供討論的實例，我們描述了ImageNet的兩個模型如下。
Plain Network. Our plain baselines (Fig. 3, middle) are mainly inspired by the philosophy of VGG nets [40](Fig. 3, left). The convolutional layers mostly have 3×3 filters and follow two simple design rules: (i) for the same output feature map size, the layers have the same number of filters; and (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer. We perform downsampling directly by convolutional layers that have a stride of 2. The network ends with a global average pooling layer and a 1000-way fully-connected layer with softmax. The total number of weighted layers is 34 in Fig. 3 (middle).
Figure 3. Example network architectures for ImageNet. Left: the VGG-19 model [40] (19.6 billion FLOPs) as a reference. Middle: a plain network with 34 parameter layers (3.6 billion FLOPs). Right: a residual network with 34 parameter layers (3.6 billion FLOPs). The dotted shortcuts increase dimensions. Table 1 shows more details and other variants.
Table 1. Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down-
sampling is performed by conv3_1, conv4_1, and conv5_1 with a stride of 2.
簡單網(wǎng)絡(luò)。 我們簡單網(wǎng)絡(luò)的基準(zhǔn)（圖3，中間）主要受到VGG網(wǎng)絡(luò)[40]（圖3，左圖）的哲學(xué)啟發(fā)。卷積層主要有3×3的濾波器，并遵循兩個簡單的設(shè)計規(guī)則：（i）對于相同的輸出特征圖尺寸，層具有相同數(shù)量的濾波器；（ii）如果特征圖尺寸減半，則濾波器數(shù)量加倍，以便保持每層的時間復(fù)雜度。我們通過步長為2的卷積層直接執(zhí)行下采樣。網(wǎng)絡(luò)以全局平均池化層和具有softmax的1000維全連接層結(jié)束。圖3（中間）的加權(quán)層總數(shù)為34。
圖3。ImageNet的網(wǎng)絡(luò)架構(gòu)例子。左：作為參考的VGG-19模型40。中：具有34個參數(shù)層的簡單網(wǎng)絡(luò)（36億FLOPs）。右：具有34個參數(shù)層的殘差網(wǎng)絡(luò)（36億FLOPs）。帶點的快捷連接增加了維度。表1顯示了更多細節(jié)和其它變種。
表1。ImageNet架構(gòu)。構(gòu)建塊顯示在括號中（也可看圖5），以及構(gòu)建塊的堆疊數(shù)量。下采樣通過步長為2的conv3_1, conv4_1和conv5_1執(zhí)行。
It is worth noticing that our model has fewer filters and lower complexity than VGG nets [40](Fig. 3, left). Our 34-layer baseline has 3.6 billion FLOPs (multiply-adds), which is only 18% of VGG-19 (19.6 billion FLOPs).
值得注意的是我們的模型與VGG網(wǎng)絡(luò)（圖3左）相比，有更少的濾波器和更低的復(fù)雜度。我們的34層基準(zhǔn)有36億FLOP(乘加)，僅是VGG-19（196億FLOP）的18%。
Residual Network. Based on the above plain network, we insert shortcut connections (Fig. 3, right) which turn the network into its counterpart residual version. The identity shortcuts (Eqn.(1)) can be directly used when the input and output are of the same dimensions (solid line shortcuts in Fig. 3). When the dimensions increase (dotted line shortcuts in Fig. 3), we consider two options: (A) The shortcut still performs identity mapping, with extra zero entries padded for increasing dimensions. This option introduces no extra parameter; (B) The projection shortcut in Eqn.(2) is used to match dimensions (done by 1×1 convolutions). For both options, when the shortcuts go across feature maps of two sizes, they are performed with a stride of 2.
殘差網(wǎng)絡(luò)。 基于上述的簡單網(wǎng)絡(luò)，我們插入快捷連接（圖3，右），將網(wǎng)絡(luò)轉(zhuǎn)換為其對應(yīng)的殘差版本。當(dāng)輸入和輸出具有相同的維度時（圖3中的實線快捷連接）時，可以直接使用恒等快捷連接（方程（1））。當(dāng)維度增加（圖3中的虛線快捷連接）時，我們考慮兩個選項：（A）快捷連接仍然執(zhí)行恒等映射，額外填充零輸入以增加維度。此選項不會引入額外的參數(shù)；（B）方程（2）中的投影快捷連接用于匹配維度（由1×1卷積完成）。對于這兩個選項，當(dāng)快捷連接跨越兩種尺寸的特征圖時，它們執(zhí)行時步長為2。
3.4. Implementation
Our implementation for ImageNet follows the practice in [21, 40]. The image is resized with its shorter side randomly sampled in [256, 480] for scale augmentation [40]. A 224×224 crop is randomly sampled from an image or its horizontal flip, with the per-pixel mean subtracted [21]. The standard color augmentation in [21] is used. We adopt batch normalization (BN) [16] right after each convolution and before activation, following [16]. We initialize the weights as in [12] and train all plain/residual nets from scratch. We use SGD with a mini-batch size of 256. The learning rate starts from 0.1 and is divided by 10 when the error plateaus, and the models are trained for up to $60 × 10^4$ iterations. We use a weight decay of 0.0001 and a momentum of 0.9. We do not use dropout [13], following the practice in [16].
3.4. 實現(xiàn)
ImageNet中我們的實現(xiàn)遵循[21，40]的實踐。調(diào)整圖像大小，其較短的邊在[256,480]之間進行隨機采樣，用于尺度增強[40]。224×224裁剪是從圖像或其水平翻轉(zhuǎn)中隨機采樣，并逐像素減去均值[21]。使用了[21]中的標(biāo)準(zhǔn)顏色增強。在每個卷積之后和激活之前，我們采用批量歸一化（BN）[16]。我們按照[12]的方法初始化權(quán)重，從零開始訓(xùn)練所有的簡單/殘差網(wǎng)絡(luò)。我們使用批大小為256的SGD方法。學(xué)習(xí)速度從0.1開始，當(dāng)誤差穩(wěn)定時學(xué)習(xí)率除以10，并且模型訓(xùn)練高達$60 × 10^4$次迭代。我們使用的權(quán)重衰減為0.0001，動量為0.9。根據(jù)[16]的實踐，我們不使用丟棄[13]。
In testing, for comparison studies we adopt the standard 10-crop testing [21]. For best results, we adopt the fully-convolutional form as in [40, 12], and average the scores at multiple scales (images are resized such that the shorter side is in {224, 256, 384, 480, 640}).
在測試階段，為了比較學(xué)習(xí)我們采用標(biāo)準(zhǔn)的10-crop測試[21]。對于最好的結(jié)果，我們采用如[40, 12]中的全卷積形式，并在多尺度上對分數(shù)進行平均（圖像歸一化，短邊位于{224, 256, 384, 480, 640}中）。
4. Experiments
4.1. ImageNet Classification
We evaluate our method on the ImageNet 2012 classification dataset [35] that consists of 1000 classes. The models are trained on the 1.28 million training images, and evaluated on the 50k validation images. We also obtain a final result on the 100k test images, reported by the test server. We evaluate both top-1 and top-5 error rates.
4. 實驗
4.1. ImageNet分類
我們在ImageNet 2012分類數(shù)據(jù)集[35]對我們的方法進行了評估，該數(shù)據(jù)集由1000個類別組成。這些模型在128萬張訓(xùn)練圖像上進行訓(xùn)練，并在5萬張驗證圖像上進行評估。我們也獲得了測試服務(wù)器報告的在10萬張測試圖像上的最終結(jié)果。我們評估了top-1和top-5錯誤率。
Plain Networks. We first evaluate 18-layer and 34-layer plain nets. The 34-layer plain net is in Fig. 3 (middle). The 18-layer plain net is of a similar form. See Table 1 for detailed architectures.
簡單網(wǎng)絡(luò)。我們首先評估18層和34層的簡單網(wǎng)絡(luò)。34層簡單網(wǎng)絡(luò)在圖3（中間）。18層簡單網(wǎng)絡(luò)是一種類似的形式。有關(guān)詳細的體系結(jié)構(gòu)，請參見表1。
The results in Table 2 show that the deeper 34-layer plain net has higher validation error than the shallower 18-layer plain net. To reveal the reasons, in Fig. 4 (left) we compare their training/validation errors during the training procedure. We have observed the degradation problem —— the 34-layer plain net has higher training error throughout the whole training procedure, even though the solution space of the 18-layer plain network is a subspace of that of the 34-layer one.
Table 2. Top-1 error (%, 10-crop testing) on ImageNet validation. Here the ResNets have no extra parameter compared to their plain counterparts. Fig. 4 shows the training procedures.
Figure 4. Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain networks of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to their plain counterparts.
表2中的結(jié)果表明，較深的34層簡單網(wǎng)絡(luò)比較淺的18層簡單網(wǎng)絡(luò)有更高的驗證誤差。為了揭示原因，在圖4（左圖）中，我們比較訓(xùn)練過程中的訓(xùn)練/驗證誤差。我們觀察到退化問題——雖然18層簡單網(wǎng)絡(luò)的解空間是34層簡單網(wǎng)絡(luò)解空間的子空間，但34層簡單網(wǎng)絡(luò)在整個訓(xùn)練過程中具有較高的訓(xùn)練誤差。
表2。ImageNet驗證集上的Top-1錯誤率(%，10個裁剪圖像測試)。相比于對應(yīng)的簡單網(wǎng)絡(luò)，ResNet沒有額外的參數(shù)。圖4顯示了訓(xùn)練過程。
圖4。在ImageNet上訓(xùn)練。細曲線表示訓(xùn)練誤差，粗曲線表示中心裁剪圖像的驗證誤差。左：18層和34層的簡單網(wǎng)絡(luò)。右：18層和34層的ResNet。在本圖中，殘差網(wǎng)絡(luò)與對應(yīng)的簡單網(wǎng)絡(luò)相比沒有額外的參數(shù)。
We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN [16], which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish. In fact, the 34-layer plain net is still able to achieve competitive accuracy (Table 3), suggesting that the solver works to some extent. We conjecture that the deep plain nets may have exponentially low convergence rates, which impact the reducing of the training error. The reason for such optimization difficulties will be studied in the future.
Table 3. Error rates (%, 10-crop testing) on ImageNet validation. VGG-16 is based on our test. ResNet-50/101/152 are of option B that only uses projections for increasing dimensions.
我們認為這種優(yōu)化難度不可能是由于梯度消失引起的。這些簡單網(wǎng)絡(luò)使用BN[16]訓(xùn)練，這保證了前向傳播信號有非零方差。我們還驗證了反向傳播的梯度，結(jié)果顯示其符合BN的正常標(biāo)準(zhǔn)。因此既不是前向信號消失也不是反向信號消失。實際上，34層簡單網(wǎng)絡(luò)仍能取得有競爭力的準(zhǔn)確率（表3），這表明在某種程度上來說求解器仍工作。我們推測深度簡單網(wǎng)絡(luò)可能有指數(shù)級低收斂特性，這影響了訓(xùn)練誤差的降低。這種優(yōu)化困難的原因?qū)頃芯俊?/p>
表3。ImageNet驗證集錯誤率（%，10個裁剪圖像測試）。VGG16是基于我們的測試結(jié)果的。ResNet-50/101/152的選擇B僅使用投影增加維度。
Residual Networks. Next we evaluate 18-layer and 34-layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3×3 filters as in Fig. 3 (right). In the first comparison (Table 2 and Fig. 4 right), we use identity mapping for all shortcuts and zero-padding for increasing dimensions (option A). So they have no extra parameter compared to the plain counterparts.
殘差網(wǎng)絡(luò)。接下來我們評估18層和34層殘差網(wǎng)絡(luò)（ResNets）?；鶞?zhǔn)架構(gòu)與上述的簡單網(wǎng)絡(luò)相同，如圖3（右）所示，預(yù)計每對3×3濾波器都會添加快捷連接。在第一次比較（表2和圖4右側(cè)）中，我們對所有快捷連接都使用恒等映射和零填充以增加維度（選項A）。所以與對應(yīng)的簡單網(wǎng)絡(luò)相比，它們沒有額外的參數(shù)。
We have three major observations from Table 2 and Fig. 4. First, the situation is reversed with residual learning —— the 34-layer ResNet is better than the 18-layer ResNet (by 2.8%). More importantly, the 34-layer ResNet exhibits considerably lower training error and is generalizable to the validation data. This indicates that the degradation problem is well addressed in this setting and we manage to obtain accuracy gains from increased depth.
我們從表2和圖4中可以看到三個主要的觀察結(jié)果。首先，殘留學(xué)習(xí)的情況變了——34層ResNet比18層ResNet更好（2.8％）。更重要的是，34層ResNet顯示出較低的訓(xùn)練誤差，并且可以泛化到驗證數(shù)據(jù)。這表明在這種情況下，退化問題得到了很好的解決，我們從增加的深度中設(shè)法獲得了準(zhǔn)確性收益。
Second, compared to its plain counterpart, the 34-layer ResNet reduces the top-1 error by 3.5% (Table 2), resulting from the successfully reduced training error (Fig. 4 right vs. left). This comparison verifies the effectiveness of residual learning on extremely deep systems.
第二，與對應(yīng)的簡單網(wǎng)絡(luò)相比，由于成功的減少了訓(xùn)練誤差，34層ResNet降低了3.5%的top-1錯誤率。這種比較證實了在極深系統(tǒng)中殘差學(xué)習(xí)的有效性。
Last, we also note that the 18-layer plain/residual nets are comparably accurate (Table 2), but the 18-layer ResNet converges faster (Fig. 4 right vs. left). When the net is “not overly deep” (18 layers here), the current SGD solver is still able to find good solutions to the plain net. In this case, the ResNet eases the optimization by providing faster convergence at the early stage.
最后，我們還注意到18層的簡單/殘差網(wǎng)絡(luò)同樣地準(zhǔn)確（表2），但18層ResNet收斂更快（圖4右和左）。當(dāng)網(wǎng)絡(luò)“不過度深”時（18層），目前的SGD求解器仍能在簡單網(wǎng)絡(luò)中找到好的解。在這種情況下，ResNet通過在早期提供更快的收斂簡便了優(yōu)化。
Identity vs. Projection Shortcuts. We have shown that parameter-free, identity shortcuts help with training. Next we investigate projection shortcuts (Eqn.(2)). In Table 3 we compare three options: (A) zero-padding shortcuts are used for increasing dimensions, and all shortcuts are parameter-free (the same as Table 2 and Fig. 4 right); (B) projection shortcuts are used for increasing dimensions, and other shortcuts are identity; and (C) all shortcuts are projections.
恒等和投影快捷連接我們已經(jīng)表明沒有參數(shù)，恒等快捷連接有助于訓(xùn)練。接下來我們調(diào)查投影快捷連接（方程2）。在表3中我們比較了三個選項：(A) 零填充快捷連接用來增加維度，所有的快捷連接是沒有參數(shù)的（與表2和圖4右相同）；(B)投影快捷連接用來增加維度，其它的快捷連接是恒等的；（C）所有的快捷連接都是投影。
Table 3 shows that all three options are considerably better than the plain counterpart. B is slightly better than A. We argue that this is because the zero-padded dimensions in A indeed have no residual learning. C is marginally better than B, and we attribute this to the extra parameters introduced by many (thirteen) projection shortcuts. But the small differences among A/B/C indicate that projection shortcuts are not essential for addressing the degradation problem. So we do not use option C in the rest of this paper, to reduce memory/time complexity and model sizes. Identity shortcuts are particularly important for not increasing the complexity of the bottleneck architectures that are introduced below.
表3顯示，所有三個選項都比對應(yīng)的簡單網(wǎng)絡(luò)好很多。選項B比A略好。我們認為這是因為A中的零填充確實沒有殘差學(xué)習(xí)。選項C比B稍好，我們把這歸因于許多（十三）投影快捷連接引入了額外參數(shù)。但A/B/C之間的細微差異表明，投影快捷連接對于解決退化問題不是至關(guān)重要的。因為我們在本文的剩余部分不再使用選項C，以減少內(nèi)存/時間復(fù)雜性和模型大小。恒等快捷連接對于不增加下面介紹的瓶頸結(jié)構(gòu)的復(fù)雜性尤為重要。
Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design. For each residual function $F$ , we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×1, 3×3, and 1×1 convolutions, where the 1×1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.
更深的瓶頸結(jié)構(gòu)。接下來我們描述ImageNet中我們使用的更深的網(wǎng)絡(luò)網(wǎng)絡(luò)。由于關(guān)注我們能承受的訓(xùn)練時間，我們將構(gòu)建塊修改為瓶頸設(shè)計。對于每個殘差函數(shù)$F$，我們使用3層堆疊而不是2層（圖5）。三層是1×1，3×3和1×1卷積，其中1×1層負責(zé)減小然后增加（恢復(fù)）維度，使3×3層成為具有較小輸入/輸出維度的瓶頸。圖5展示了一個示例，兩個設(shè)計具有相似的時間復(fù)雜度。
The parameter-free identity shortcuts are particularly important for the bottleneck architectures. If the identity shortcut in Fig. 5 (right) is replaced with projection, one can show that the time complexity and model size are doubled, as the shortcut is connected to the two high-dimensional ends. So identity shortcuts lead to more efficient models for the bottleneck designs.
無參數(shù)恒等快捷連接對于瓶頸架構(gòu)尤為重要。如果圖5（右）中的恒等快捷連接被投影替換，則可以顯示出時間復(fù)雜度和模型大小加倍，因為快捷連接是連接到兩個高維端。因此，恒等快捷連接可以為瓶頸設(shè)計得到更有效的模型。
50-layer ResNet: We replace each 2-layer block in the 34-layer net with this 3-layer bottleneck block, resulting in a 50-layer ResNet (Table 1). We use option B for increasing dimensions. This model has 3.8 billion FLOPs.
50層ResNet：我們用3層瓶頸塊替換34層網(wǎng)絡(luò)中的每一個2層塊，得到了一個50層ResNet（表1）。我們使用選項B來增加維度。該模型有38億FLOP。
101-layer and 152-layer ResNet: We construct 101-layer and 152-layer ResNets by using more 3-layer blocks (Table 1). Remarkably, although the depth is significantly increased, the 152-layer ResNet (11.3 billion FLOPs) still has lower complexity than VGG-16/19 nets (15.3/19.6 billion FLOPs).
101層和152層ResNet：我們通過使用更多的3層瓶頸塊來構(gòu)建101層和152層ResNets（表1）。值得注意的是，盡管深度顯著增加，但152層ResNet（113億FLOP）仍然比VGG-16/19網(wǎng)絡(luò)（153/196億FLOP）具有更低的復(fù)雜度。
The 50/101/152-layer ResNets are more accurate than the 34-layer ones by considerable margins (Table 3 and 4). We do not observe the degradation problem and thus enjoy significant accuracy gains from considerably increased depth. The benefits of depth are witnessed for all evaluation metrics (Table 3 and 4).
50/101/152層ResNet比34層ResNet的準(zhǔn)確性要高得多（表3和4）。我們沒有觀察到退化問題，因此可以從顯著增加的深度中獲得顯著的準(zhǔn)確性收益。所有評估指標(biāo)都能證明深度的收益（表3和表4）。
Comparisons with State-of-the-art Methods. In Table 4 we compare with the previous best single-model results. Our baseline 34-layer ResNets have achieved very competitive accuracy. Our 152-layer ResNet has a single-model top-5 validation error of 4.49%. This single-model result outperforms all previous ensemble results (Table 5). We combine six models of different depth to form an ensemble (only with two 152-layer ones at the time of submitting). This leads to 3.57% top-5 error on the test set (Table 5). This entry won the 1st place in ILSVRC 2015.
Table 4. Error rates (%) of single-model results on the ImageNet validation set (except reported on the test set).
Table 5. Error rates (%) of ensembles. The top-5 error is on the test set of ImageNet and reported by the test server.
與最先進的方法比較。在表4中，我們與以前最好的單一模型結(jié)果進行比較。我們基準(zhǔn)的34層ResNet已經(jīng)取得了非常有競爭力的準(zhǔn)確性。我們的152層ResNet具有單模型4.49％的top-5錯誤率。這種單一模型的結(jié)果勝過以前的所有綜合結(jié)果（表5）。我們結(jié)合了六種不同深度的模型，形成一個集合（在提交時僅有兩個152層）。這在測試集上得到了3.5％的top-5錯誤率（表5）。這次提交在2015年ILSVRC中榮獲了第一名。
表4。單一模型在ImageNet驗證集上的錯誤率（%）(除了?是測試集上報告的錯誤率)。
表5。模型綜合的錯誤率(%)。top-5錯誤率是ImageNet測試集上的并由測試服務(wù)器報告的。
4.2. CIFAR-10 and Analysis
We conducted more studies on the CIFAR-10 dataset [20], which consists of 50k training images and 10k testing images in 10 classes. We present experiments trained on the training set and evaluated on the test set. Our focus is on the behaviors of extremely deep networks, but not on pushing the state-of-the-art results, so we intentionally use simple architectures as follows.
4.2. CIFAR-10和分析
我們對CIFAR-10數(shù)據(jù)集[20]進行了更多的研究，其中包括10個類別中的5萬張訓(xùn)練圖像和1萬張測試圖像。我們介紹了在訓(xùn)練集上進行訓(xùn)練和在測試集上進行評估的實驗。我們的焦點在于極深網(wǎng)絡(luò)的行為，但不是推動最先進的結(jié)果，所以我們有意使用如下的簡單架構(gòu)。
The plain/residual architectures follow the form in Fig. 3 (middle/right). The network inputs are 32×32 images, with the per-pixel mean subtracted. The first layer is 3×3 convolutions. Then we use a stack of 6n layers with 3×3 convolutions on the feature maps of sizes {32, 16, 8} respectively, with 2n layers for each feature map size. The numbers of filters are {16, 32, 64} respectively. The subsampling is performed by convolutions with a stride of 2. The network ends with a global average pooling, a 10-way fully-connected layer, and softmax. There are totally 6n+2 stacked weighted layers. The following table summarizes the architecture:
When shortcut connections are used, they are connected to the pairs of 3×3 layers (totally 3n shortcuts). On this dataset we use identity shortcuts in all cases (i.e., option A), so our residual models have exactly the same depth, width, and number of parameters as the plain counterparts.
簡單/殘差架構(gòu)遵循圖3（中/右）的形式。網(wǎng)絡(luò)輸入是32×32的圖像，每個像素減去均值。第一層是3×3卷積。然后我們在大小為{32,16,8}的特征圖上分別使用了帶有3×3卷積的6n個堆疊層，每個特征圖大小使用2n層。濾波器數(shù)量分別為{16,32,64}。下采樣由步長為2的卷積進行。網(wǎng)絡(luò)以全局平均池化，一個10維全連接層和softmax作為結(jié)束。共有6n+2個堆疊的加權(quán)層。下表總結(jié)了這個架構(gòu)：
當(dāng)使用快捷連接時，它們連接到成對的3×3卷積層上（共3n個快捷連接）。在這個數(shù)據(jù)集上，我們在所有案例中都使用恒等快捷連接（即選項A），因此我們的殘差模型與對應(yīng)的簡單模型具有完全相同的深度，寬度和參數(shù)數(shù)量。
We use a weight decay of 0.0001 and momentum of 0.9, and adopt the weight initialization in [12] and BN [16] but with no dropout. These models are trained with a mini-batch size of 128 on two GPUs. We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and terminate training at 64k iterations, which is determined on a 45k/5k train/val split. We follow the simple data augmentation in [24] for training: 4 pixels are padded on each side, and a 32×32 crop is randomly sampled from the padded image or its horizontal flip. For testing, we only evaluate the single view of the original 32×32 image.
我們使用的權(quán)重衰減為0.0001和動量為0.9，并采用[12]和BN[16]中的權(quán)重初始化，但沒有使用丟棄。這些模型在兩個GPU上進行訓(xùn)練，批處理大小為128。我們開始使用的學(xué)習(xí)率為0.1，在32k次和48k次迭代后學(xué)習(xí)率除以10，并在64k次迭代后終止訓(xùn)練，這是由45k/5k的訓(xùn)練/驗證集分割決定的。我們按照[24]中的簡單數(shù)據(jù)增強進行訓(xùn)練：每邊填充4個像素，并從填充圖像或其水平翻轉(zhuǎn)圖像中隨機采樣32×32的裁剪圖像。對于測試，我們只評估原始32×32圖像的單一視圖。
We compare $n = {3, 5, 7, 9}$, leading to 20, 32, 44, and 56-layer networks. Fig. 6 (left) shows the behaviors of the plain nets. The deep plain nets suffer from increased depth, and exhibit higher training error when going deeper. This phenomenon is similar to that on ImageNet (Fig. 4, left) and on MNIST (see [41]), suggesting that such an optimization difficulty is a fundamental problem.
Figure 6. Training on CIFAR-10. Dashed lines denote training error, and bold lines denote testing error. Left: plain networks. The error of plain-110 is higher than 60% and not displayed. Middle: ResNets. Right: ResNets with 110 and 1202 layers.
我們比較了$n = {3, 5, 7, 9}$，得到了20層，32層，44層和56層的網(wǎng)絡(luò)。圖6（左）顯示了簡單網(wǎng)絡(luò)的行為。深度簡單網(wǎng)絡(luò)經(jīng)歷了深度增加，隨著深度增加表現(xiàn)出了更高的訓(xùn)練誤差。這種現(xiàn)象類似于ImageNet中（圖4，左）和MNIST中（請看[41]）的現(xiàn)象，表明這種優(yōu)化困難是一個基本的問題。
圖6。在CIFAR-10上訓(xùn)練。虛線表示訓(xùn)練誤差，粗線表示測試誤差。左：簡單網(wǎng)絡(luò)。簡單的110層網(wǎng)絡(luò)錯誤率超過60%沒有展示。中間：ResNet。右：110層ResNet和1202層ResNet。
Fig. 6 (middle) shows the behaviors of ResNets. Also similar to the ImageNet cases (Fig. 4, right), our ResNets manage to overcome the optimization difficulty and demonstrate accuracy gains when the depth increases.
圖6（中）顯示了ResNet的行為。與ImageNet的情況類似（圖4，右），我們的ResNet設(shè)法克服優(yōu)化困難并隨著深度的增加展示了準(zhǔn)確性收益。
We further explore $n = 18$ that leads to a 110-layer ResNet. In this case, we find that the initial learning rate of 0.1 is slightly too large to start converging. So we use 0.01 to warm up the training until the training error is below 80% (about 400 iterations), and then go back to 0.1 and continue training. The rest of the learning schedule is as done previously. This 110-layer network converges well (Fig. 6, middle). It has fewer parameters than other deep and thin networks such as FitNet [34] and Highway [41] (Table 6), yet is among the state-of-the-art results (6.43%, Table 6).
Table 6. Classification error on the CIFAR-10 test set. All methods are with data augmentation. For ResNet-110, we run it 5 times and show “best (mean±std)” as in [42].
我們進一步探索了$n = 18$得到了110層的ResNet。在這種情況下，我們發(fā)現(xiàn)0.1的初始學(xué)習(xí)率對于收斂來說太大了。因此我們使用0.01的學(xué)習(xí)率開始訓(xùn)練，直到訓(xùn)練誤差低于80%（大約400次迭代），然后學(xué)習(xí)率變回到0.1并繼續(xù)訓(xùn)練。學(xué)習(xí)過程的剩余部分與前面做的一樣。這個110層網(wǎng)絡(luò)收斂的很好（圖6，中）。它與其它的深且窄的網(wǎng)絡(luò)例如FitNet[34]和Highway41相比有更少的參數(shù)，但結(jié)果仍在目前最好的結(jié)果之間（6.43%，表6）。
表6。在CIFAR-10測試集上的分類誤差。所有的方法都使用了數(shù)據(jù)增強。對于ResNet-110，像論文[42]中那樣，我們運行了5次并展示了“最好的(mean±std)”。
Analysis of Layer Responses. Fig. 7 shows the standard deviations (std) of the layer responses. The responses are the outputs of each 3×3 layer, after BN and before other nonlinearity (ReLU/addition). For ResNets, this analysis reveals the response strength of the residual functions. Fig. 7 shows that ResNets have generally smaller responses than their plain counterparts. These results support our basic motivation (Sec.3.1) that the residual functions might be generally closer to zero than the non-residual functions. We also notice that the deeper ResNet has smaller magnitudes of responses, as evidenced by the comparisons among ResNet-20, 56, and 110 in Fig. 7. When there are more layers, an individual layer of ResNets tends to modify the signal less.
層響應(yīng)分析。圖7顯示了層響應(yīng)的標(biāo)準(zhǔn)偏差（std）。這些響應(yīng)每個3×3層的輸出，在BN之后和其他非線性（ReLU/加法）之前。對于ResNets，該分析揭示了殘差函數(shù)的響應(yīng)強度。圖7顯示ResNet的響應(yīng)比其對應(yīng)的簡單網(wǎng)絡(luò)的響應(yīng)更小。這些結(jié)果支持了我們的基本動機（第3.1節(jié)），殘差函數(shù)通常具有比非殘差函數(shù)更接近零。我們還注意到，更深的ResNet具有較小的響應(yīng)幅度，如圖7中ResNet-20，56和110之間的比較所證明的。當(dāng)層數(shù)更多時，單層ResNet趨向于更少地修改信號。
Exploring Over 1000 layers. We explore an aggressively deep model of over 1000 layers. We set $n = 200$ that leads to a 1202-layer network, which is trained as described above. Our method shows no optimization difficulty, and this $10^3$-layer network is able to achieve training error <0.1% (Fig. 6, right). Its test error is still fairly good (7.93%, Table 6).
探索超過1000層。我們探索超過1000層的過深的模型。我們設(shè)置$n = 200$，得到了1202層的網(wǎng)絡(luò)，其訓(xùn)練如上所述。我們的方法顯示沒有優(yōu)化困難，這個$10^3$層網(wǎng)絡(luò)能夠?qū)崿F(xiàn)訓(xùn)練誤差<0.1％（圖6，右圖）。其測試誤差仍然很好（7.93％，表6）。
But there are still open problems on such aggressively deep models. The testing result of this 1202-layer network is worse than that of our 110-layer network, although both have similar training error. We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset. Strong regularization such as maxout [9] or dropout [13] is applied to obtain the best results ([9, 25, 24, 34]) on this dataset. In this paper, we use no maxout/dropout and just simply impose regularization via deep and thin architectures by design, without distracting from the focus on the difficulties of optimization. But combining with stronger regularization may improve results, which we will study in the future.
但是，這種極深的模型仍然存在著開放的問題。這個1202層網(wǎng)絡(luò)的測試結(jié)果比我們的110層網(wǎng)絡(luò)的測試結(jié)果更差，雖然兩者都具有類似的訓(xùn)練誤差。我們認為這是因為過擬合。對于這種小型數(shù)據(jù)集，1202層網(wǎng)絡(luò)可能是不必要的大（19.4M）。在這個數(shù)據(jù)集應(yīng)用強大的正則化，如maxout[9]或者dropout[13]來獲得最佳結(jié)果（[9,25,24,34]）。在本文中，我們不使用maxout/dropout，只是簡單地通過設(shè)計深且窄的架構(gòu)簡單地進行正則化，而不會分散集中在優(yōu)化難點上的注意力。但結(jié)合更強的正規(guī)化可能會改善結(jié)果，我們將來會研究。
4.3. Object Detection on PASCAL and MS COCO
Our method has good generalization performance on other recognition tasks. Table 7 and 8 show the object detection baseline results on PASCAL VOC 2007 and 2012 [5] and COCO [26]. We adopt Faster R-CNN [32] as the detection method. Here we are interested in the improvements of replacing VGG-16 [40] with ResNet-101. The detection implementation (see appendix) of using both models is the same, so the gains can only be attributed to better networks. Most remarkably, on the challenging COCO dataset we obtain a 6.0% increase in COCO’s standard metric (mAP@[.5, .95]), which is a 28% relative improvement. This gain is solely due to the learned representations.
Table 7. Object detection mAP (%) on the PASCAL VOC 2007/2012 test sets using baseline Faster R-CNN. See also appendix for better results.
Table 8. Object detection mAP (%) on the COCO validation set using baseline Faster R-CNN. See also appendix for better results.
4.3. 在PASCAL和MS COCO上的目標(biāo)檢測
我們的方法對其他識別任務(wù)有很好的泛化性能。表7和表8顯示了PASCAL VOC 2007和2012[5]以及COCO[26]的目標(biāo)檢測基準(zhǔn)結(jié)果。我們采用更快的R-CNN[32]作為檢測方法。在這里，我們感興趣的是用ResNet-101替換VGG-16[40]。使用這兩種模式的檢測實現(xiàn)（見附錄）是一樣的，所以收益只能歸因于更好的網(wǎng)絡(luò)。最顯著的是，在有挑戰(zhàn)性的COCO數(shù)據(jù)集中，COCO的標(biāo)準(zhǔn)度量指標(biāo)（mAP@[.5，.95]）增長了6.0％，相對改善了28％。這種收益完全是由于學(xué)習(xí)表示。
表7。在PASCAL VOC 2007/2012測試集上使用基準(zhǔn)Faster R-CNN的目標(biāo)檢測mAP(%)。更好的結(jié)果請看附錄。
表8。在COCO驗證集上使用基準(zhǔn)Faster R-CNN的目標(biāo)檢測mAP(%)。更好的結(jié)果請看附錄。
Based on deep residual nets, we won the 1st places in several tracks in ILSVRC & COCO 2015 competitions: ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. The details are in the appendix.
基于深度殘差網(wǎng)絡(luò)，我們在ILSVRC & COCO 2015競賽的幾個任務(wù)中獲得了第一名，分別是：ImageNet檢測，ImageNet定位，COCO檢測，COCO分割。跟多細節(jié)請看附錄。
References
[1] Y.Bengio,P.Simard,andP.Frasconi.Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994.
[2] C. M. Bishop. Neural networks for pattern recognition. Oxford university press, 1995.
[3] W. L. Briggs, S. F. McCormick, et al. A Multigrid Tutorial. Siam, 2000.
[4] K. Chatfield, V. Lempitsky, A. Vedaldi, and A. Zisserman. The devil is in the details: an evaluation of recent feature encoding methods. In BMVC, 2011.
[5] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV, pages 303–338, 2010.
[6] R. Girshick. Fast R-CNN. In ICCV, 2015.
[7] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In
CVPR, 2014.
[8] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
[9] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. arXiv:1302.4389, 2013.
[10] K.Heand J.Sun. Convolutional neural networks at constrained time cost. In CVPR, 2015.
[11] K.He, X.Zhang, S.Ren, and J.Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV, 2014.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In
ICCV, 2015.
[13] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580, 2012.
[14] S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma thesis, TU Munich, 1991.
[15] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
[16] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[17] H.Jegou, M.Douze, and C.Schmid. Product quantization for nearest neighbor search. TPAMI, 33, 2011.
[18] H. Jegou, F. Perronnin, M. Douze, J. Sanchez, P. Perez, and C. Schmid. Aggregating local image descriptors into compact codes.
TPAMI, 2012.
[19] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for
fast feature embedding. arXiv:1408.5093, 2014.
[20] A. Krizhevsky. Learning multiple layers of features from tiny images. Tech Report, 2009.
[21] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[22] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to hand-written zip code recognition. Neural computation, 1989.
[23] Y.LeCun,L.Bottou,G.B.Orr,and K.-R.Muller. Efficient back prop. In Neural Networks: Tricks of the Trade, pages 9–50. Springer, 1998.
[24] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. arXiv:1409.5185, 2014.
[25] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv:1312.4400,2013.
[26] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick. Microsoft COCO: Common objects in
context. In ECCV. 2014.
[27] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
[28] G. Montufar, R. Pascanu, K. Cho, and Y. Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014.
[29] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
[30] F. Perronnin and C. Dance. Fisher kernels on visual vocabularies for image categorization. In CVPR, 2007.
[31] T. Raiko, H. Valpola, and Y. LeCun. Deep learning made easier by linear transformations in perceptrons. In AISTATS, 2012.
[32] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
[33] B. D. Ripley. Pattern recognition and neural networks. Cambridge university press, 1996.
[34] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[35] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. arXiv:1409.0575, 2014.
[36] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120, 2013.
[37] N.N.Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical report, 1998.
[38] N. N. Schraudolph. Centering neural network gradient factors. In Neural Networks: Tricks of the Trade, pages 207–226. Springer, 1998.
[39] P.Sermanet, D.Eigen, X.Zhang, M.Mathieu, R.Fergus, and Y.LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In ICLR, 2014.
[40] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[41] R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks. arXiv:1505.00387, 2015.
[42] R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. 1507.06228, 2015.
[43] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[44] R. Szeliski. Fast surface interpolation using hierarchical basis functions. TPAMI, 1990.
[45] R. Szeliski. Locally adapted hierarchical basis preconditioning. In SIGGRAPH, 2006.
[46] T. Vatanen, T. Raiko, H. Valpola, and Y. LeCun. Pushing stochastic gradient towards second-order methods–backpropagation learning with transformations in nonlinearities. In Neural Information Processing, 2013.
[47] A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library of computer vision algorithms, 2008.
[48] W. Venables and B. Ripley. Modern applied statistics with s-plus. 1999.
[49] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional neural networks. In ECCV, 2014.
