飛機異常檢測
aircraft fuselage 飛機機身
Aircraft inspection and maintenance 飛機檢查和維護
ABSTRACT
To ensure flight safety of aircraft structures, it is necessary to have regular maintenance using visual and nondestructive inspection (NDI) methods. In this paper, we propose an automatic image-based aircraft defect detection using Deep Neural Networks (DNNs). To the best of our knowledge, this is the first work for aircraft defect detection using DNNs. We perform a comprehensive evaluation of state of-the-art feature descriptors and show that the best performance is achieved by vgg-f DNN as feature extractor with a linear SVM classifier. To reduce the processing time, we propose to apply SURF key point detector to identify defect patch candidates. Our experiment results suggest that we can achieve over 96% accuracy at around 15s processing time for a high-resolution (20-megapixel) image on a laptop
為確保飛機結構的飛行安全，必須使用視覺和非破壞性檢查（NDI）方法進行定期維護。 在本文中，我們提出了一種使用深度神經網絡（DNN）進行基于圖像的自動飛機缺陷檢測。 據我們所知，這是使用DNN進行飛機缺陷檢測的第一項工作。 我們對最先進的特征描述符進行了全面評估，并表明vgg-f DNN作為具有線性SVM分類器的特征提取器可以實現最佳性能。 為了減少處理時間，我們建議應用SURF關鍵點檢測器來識別缺陷補丁候選者。 我們的實驗結果表明，在筆記本電腦上拍攝高分辨率（20萬像素）圖像時，我們可以在大約15秒的處理時間內獲得超過96％的準確度
1. INTRODUCTION
Aircraft inspection and maintenance is an essential to safe air transportation [1-3]. A fully automated system to monitor the structural health of an aircraft has the potential to reduce operating costs, increase flight safety and improve aircraft availability [4]. This paper makes a contribution to the field of automatic defect detection of an aircraft fuselage with computer vision techniques.?
飛機檢查和維護對安全航空運輸至關重要[1-3]。 用于監(jiān)控飛機結構健康狀況的全自動系統(tǒng)有可能降低運營成本，提高飛行安全性并提高飛機可用性[4]。 本文利用計算機視覺技術為飛機機身的自動缺陷檢測領域做出了貢獻。
In [5], they research computer-simulated visual inspection (VI) and non-destructive inspection (NDI) tasks. However, these visual inspection tasks were performed by human inspectors who searched for defect manually. Our proposed algorithm is a completely automatic inspection.
在[5]中，他們研究了計算機模擬視覺檢測（VI）和非破壞性檢查（NDI）任務。 然而，這些視覺檢查任務由人工檢查員執(zhí)行，他們手動搜索缺陷。 我們提出的算法是一種全自動檢測。
[5] K. Latorella, A. Gramopadhye, P. Prabhu, C. Drury, M. Smith, and D. Shanahan, "Computer-simulated aircraft inspection tasks for off-line experimentation," in Proceedings of the Human Factors and Ergonomics Society Annual Meeting , 1992, pp. 92-96
In recent years, deep neural networks (DNN) have shown promising results in different classification tasks [6-10]. Although DNNs can be used to perform classification directly using the output of the last network layer, they can also be used as a feature extractor combined with a classifier [11].
近年來，深度神經網絡（DNN）在不同的分類任務中顯示出有希望的結果[6-10]。 雖然DNN可以用于直接使用最后一個網絡層的輸出進行分類，但它們也可以用作與分類器結合的特征提取器[11]。
In this paper, we investigate a classification system that employs a DNN, pretrained using natural images, to extract features suitable to another domain, i.e., aircraft fuselage defect detection, where there are few samples available. The contributions of this study are:
在本文中，我們研究了一種采用DNN的分類系統(tǒng)，該系統(tǒng)使用自然圖像預訓練，以提取適合于另一個域的特征，即飛機機身缺陷檢測，其中可用的樣本很少。 這項研究的貢獻是：
1) To the best of our knowledge, this is the first work for automatic defect detection of aircraft fuselage using visual images and deep learning.
2) We propose a fast and accurate detection algorithm with selection of region of interest using SURF interest point extractor.
3) We propose techniques to handle washed and unwashed fuselage based on pre- and post-processing.
1）據我們所知，這是使用視覺圖像和深度學習進行飛機機身自動缺陷檢測的第一項工作。
2）我們提出了一種快速準確的檢測算法，使用SURF興趣點提取器選擇感興趣的區(qū)域。
surf
3）我們提出了基于預處理和后處理來處理洗滌和未洗滌機身的技術。
To the best of our knowledge, there is no previous work on automatic image-based aircraft defect detection. Image based defect detection has been investigated for other problems: In [12] X-ray images of metallic components are used as a non-destructive testing method, to detect the defects within casting components. In [13] they propose a deep convolutional neural network solution to the analysis of image data for the detection of rail surface defects. The images were obtained from many hours of automated video recordings. However, the image and defect characteristics of these problems are rather different from ours.
據我們所知，以前沒有關于基于自動圖像的飛機缺陷檢測的工作。 已經針對其他問題研究了基于圖像的缺陷檢測：在[12]中，金屬部件的X射線圖像被用作非破壞性測試方法，以檢測鑄造部件內的缺陷。 在[13]中，他們提出了一種深度卷積神經網絡解決方案，用于分析圖像數據，以檢測軌道表面缺陷。 這些圖像是從許多小時的自動視頻錄制中獲得的。 然而，這些問題的形象和缺陷特征與我們的不同。
[12] D. Mery and C. Arteta, "Automatic Defect Recognition in X-Ray Testing Using Computer Vision," in Applications of Computer Vision (WACV), 2017 IEEE Winter Conference on , 2017, pp. 1026-1035.
[13] S. Faghih-Roohi, S. Hajizadeh, A. Nú?ez, R. Babuska, and B. De Schutter, "Deep convolutional neural networks for detection of rail surface defects," in Neural Networks (IJCNN), 2016 International Joint Conference on , 2016, pp. 2584-2589.
The rest of the paper is organized as follows. In Section 2 we give a detailed explanation of our datasets. Section 3 explains the proposed algorithm and the DNN-derived features generated automatically from a dataset of fuselage images. The performance evaluation of the proposed algorithm is provided in Section 4, while Section 5 presents the conclusion and the feature work
本文的其余部分安排如下。 在第2節(jié)中，我們詳細解釋了我們的數據集。 第3節(jié)解釋了所提出的算法以及從機身圖像數據集自動生成的DNN衍生特征。 第4節(jié)提供了算法的性能評估，第5節(jié)給出了結論和特征工作
2. DATASETS
Our dataset images are taken in a straight view of the airplane fuselage. During the inspection, a drone can be used to capture these images automatically. Images are stored in JPEG format. All images have three color channels and 3888×5184 resolution. Some examples of aircraft fuselage images with defects are illustrated in Figure 1. For each image, a binary mask is created by an experienced inspector to represent defects. Considering?
我們的數據集圖像是在飛機機身的直視圖中拍攝的。 在檢查過程中，可以使用無人機自動捕獲這些圖像。 圖像以JPEG格式存儲。 所有圖像都有三個顏色通道和3888×5184分辨率。 圖1中示出了具有缺陷的飛機機身圖像的一些示例。對于每個圖像，由經驗豐富的檢查員創(chuàng)建二元掩模以表示缺陷。
3. METHODOLOGY
In this work, we propose a patch-based scheme for detection of defects. Specifically, we partition the image into 65x65 patches and classify each patch into defect / non-defect class. The classification is a two-step process: First, we compute a set of features for each patch; second, we build a classification model based on the extracted features.
在這項工作中，我們提出了一種基于補丁的方法來檢測缺陷。 具體來說，我們將圖像分區(qū)為65x65補丁，并將每個補丁分類為缺陷/非缺陷類。 分類分為兩步：首先，我們?yōu)槊總€補丁計算一組特征; 第二，我們基于提取的特征構建分類模型。
For computing the discriminative features, we evaluate and compare a set of techniques including deep neural network, local descriptors, and texture features. Our experiments show that using the pretrained convolutional neural network (CNN) results in the best performance. For the classification step, we use SVM with a linear kernel.
為了計算判別特征，我們評估和比較一組技術，包括深度神經網絡，局部描述符和紋理特征。 我們的實驗表明，使用預訓練卷積神經網絡（CNN）可以獲得最佳性能。 對于分類步驟，我們將SVM與線性內核一起使用。
In the experiment, we split the data into disjoint training and testing sets, in a manner that the data which is present in the training set is not allowed to be in the testing set. But in order to make these two sets completely disjoint, we employ 10-fold cross validation on the images rather than the patches, i.e. the patches of a particular image have the same crossvalidation index as their parent image. This approach prevents having highly correlated data in both training and testing sets, that results in high accuracy which is not the case
在實驗中，我們將數據分成不相交的訓練和測試集，其方式是不允許訓練集中存在的數據在測試集中。 但是為了使這兩組完全不相交，我們對圖像而不是貼片采用10倍交叉驗證，即特定圖像的貼片具有與其父圖像相同的交叉驗證指數。 這種方法可以防止在訓練集和測試集中都有高度相關的數據，從而導致高精度，而事實并非如此
Considering the high resolution of images in our dataset and the maximum allowed size of a patch, it needs high computational complexity to evaluate all the patches in a single image. Since this work addresses an industrial application, processing time is a critical factor. We propose to boost our algorithm by analyzing the region of interest which is extracted by SURF [15] detector.
考慮到我們的數據集中圖像的高分辨率和補丁的最大允許大小，它需要高計算復雜度來評估單個圖像中的所有補丁。 由于這項工作涉及工業(yè)應用，處理時間是一個關鍵因素。 我們建議通過分析由SURF [15]檢測器提取的感興趣區(qū)域來增強我們的算法。
Another critical problem in defect detection of airplane fuselage is about washed or unwashed fuselage. Unwashed fuselage with dirt causes some problems in detection of defects. We propose to extend our algorithm for both washed and unwashed conditions.
飛機機身缺陷檢測中的另一個關鍵問題是洗滌或未洗滌的機身。 沒有洗滌的機身帶有污垢會導致檢測缺陷的一些問題。 我們建議擴展我們的洗滌和未洗滌條件的算法。
3.1. Feature Extraction
As we will show in the results, using CNN as feature extractor achieves the best performance. Therefore, we have used a convolutional neural network (CNN) pre-trained on ImageNet as a feature extractor for our dataset. Transferring the knowledge of an existing CNN to a new domain has been studied and proved successful in several applications [11, 16, 17]. This approach is more appropriate for our application rather than fine-tuning the CNN [18] due to several reasons such as size and types of our dataset. Considering the limited size of our dataset, we propose to build a classifier model on top of the output (activations) of the hidden layers [19]. Furthermore, since the dataset (ImageNet) that was used to train CNN is quite different from our dataset, it is better to use the activations of the earlier layers of the network to construct the classifier. The block diagram of the proposed method for defect detection is shown in figure 3. As discussed in section 2, an equally balanced set of patches is used for training.
正如我們將在結果中顯示的那樣，使用CNN作為特征提取器可以實現最佳性能。因此，我們使用在ImageNet上預訓練的卷積神經網絡（CNN）作為我們數據集的特征提取器。將現有CNN的知識轉移到新域已經在若干應用中被研究并證明是成功的[11,16,17]。由于我的數據集的大小和類型等多種原因，這種方法更適合我們的應用而不是微調CNN [18]。考慮到我們的數據集的大小有限，我們建議在隱藏層的輸出（激活）之上構建分類器模型[19]。此外，由于用于訓練CNN的數據集（ImageNet）與我們的數據集完全不同，因此最好使用網絡早期層的激活來構建分類器。所提出的缺陷檢測方法的框圖如圖3所示。如第2節(jié)所述，使用一組同等平衡的補丁進行訓練。
In this work, we evaluate two CNN models related to ImageNet: AlexNet [6] and VGG-F networks [7]. As illustrated in figure 3, the nets comprise of eight layers; the first five are convolutional layers and the remaining three are fully connected layers. The size of the descriptors is 4096 for ‘fc6’ and ‘fc7’ layers and 1000 for ‘fc8’ layer. The input image of these models are images of 244 ×244 ×3 pixels. For this reason, our 65 × 65 pixel-patches were resized to the required size (all three channels are equal). Considering K neurons in fully connected layers, we consider the extracted layer as a feature vector?
在這項工作中，我們評估了兩個與ImageNet相關的CNN模型：AlexNet [6]和VGG-F網絡[7]。 如圖3所示，網由八層組成; 前五個是卷積層，其余三個是完全連接的層。 'fc6'和'fc7'層的描述符大小為4096，'fc8'層為1000。 這些模型的輸入圖像是244×244×3像素的圖像。 出于這個原因，我們將65×65像素的貼片尺寸調整為所需的尺寸（所有三個通道都相同）。 考慮完全連接層中的K神經元，我們將提取的層視為特征向量
3.2. Boosting Defect Detection
As discussed in 3.1, the SVM classifier is fed with a set of discriminative features, extracted for each patch. Considering our high-resolution images (20-megapixel), there are lots of patches to be evaluated by the feature extractor and the classifier. To speed up the processing, one approach is to decrease the number of patches by increasing the patch size, but this affects the accuracy of detection. In general, increasing the patch size reduces the computational complexity but degrades the accuracy and vice versa. Therefore, there is a trade-off between the accuracy and the computational complexity to choose patch size. We have tested a variation of patch sizes from 20x20 to 100x100 pixels and the best results are achieved by patch size 65x65 pixels.
如3.1中所討論的，SVM分類器被提供有一組判別特征，為每個補丁提取。 考慮到我們的高分辨率圖像（20萬像素），功能提取器和分類器需要評估許多補丁。 為了加快處理速度，一種方法是通過增加補丁大小來減少補丁數量，但這會影響檢測的準確性。 通常，增加貼片尺寸會降低計算復雜度，但會降低精度，反之亦然。 因此，在準確度和計算復雜度之間存在選擇補丁大小的權衡。 我們已經測試了從20x20到100x100像素的貼片尺寸的變化，并且通過貼片尺寸65x65像素實現了最佳結果。
Considering this patch size and the resolution of our images, it is a time-consuming task to evaluate all the patches within an image. We propose to boost our algorithm via enforcing the evaluation to some regions of interest. The regions of interest must include all the probable defect areas.
考慮到此修補程序大小和圖像的分辨率，評估圖像中的所有修補程序是一項耗時的任務。 我們建議通過對某些感興趣的區(qū)域執(zhí)行評估來提升我們的算法。 感興趣的區(qū)域必須包括所有可能的缺陷區(qū)域。
Through our experiments, we found that, in most images, speeded up robust feature (SURF) is able to detect all the defect regions together with some normal regions which are similar to the defects. Therefore, we propose to apply SURF interest point detector to select some patches to be included in the evaluation procedure. A patch is included in the defect evaluation procedure if it contains at least one SURF key point. In this way, lots of homogenous regions of the fuselage are excluded from the evaluation step. Our results show that evaluating only the selected patches of the regions of interest can boost defect detection algorithm by a 6x speed-up. Figure 4 shows the block diagram of the boosted defect detection algorithm.
通過我們的實驗，我們發(fā)現，在大多數圖像中，加速魯棒特征（SURF）能夠檢測所有缺陷區(qū)域以及一些與缺陷類似的正常區(qū)域。 因此，我們建議應用SURF興趣點檢測器來選擇要包括在評估程序中的一些補丁。 如果補丁包含至少一個SURF關鍵點，則補丁包含在缺陷評估過程中。 以這種方式，機身的許多均勻區(qū)域被排除在評估步驟之外。 我們的結果表明，僅評估感興趣區(qū)域的選定塊可以通過6倍加速來增強缺陷檢測算法。 圖4顯示了增強缺陷檢測算法的框圖。
3.3. Post-processing
As this work is an industrial application, we have to make the algorithm applicable for different conditions. Washing status of the aircraft is an important factor, which affects the defect detection procedure. As aircraft exterior cleaning procedure is time and effort consuming, it is usually done occasionally. As a result, an aircraft could be unwashed with dirty spots on it which mislead the defect detection process. In order to overcome this issue, we employ a user interface to choose between two different conditions of washed or unwashed aircraft. In the condition of washed aircraft, the detection pipeline is the same as discussed in section 3.2. But for an unwashed aircraft with dirty spots on the fuselage, we propose to apply a low-pass Gaussian filter to reduce the noise-like spots on the fuselage images, in a manner that has the minimum smoothing effect on the real defects. But, there is a trade-off between reducing the high-pass components of the image and retaining the defects thoroughly. Our approach to overcome this problem is to have a post-processing after classification of the patches, which is done by a similarity comparison of the adjacent patches. The intensity range of the patch is used as the similarity metric. Specifically, if a patch is detected as a defect patch, it is most likely to be in a defect region, so we also test its adjacent patches with a postprocessing scheme to ensure all the patches in a defect region are detected. If an adjacent patch satisfies the similarity threshold, it is classified into the defect class.
由于這項工作是工業(yè)應用，我們必須使算法適用于不同的條件。飛機的清洗狀態(tài)是影響缺陷檢測程序的重要因素。由于飛機外部清潔程序耗費時間和精力，通常偶爾進行。結果，飛機可能沒有清洗，上面有臟點，誤導了缺陷檢測過程。為了克服這個問題，我們采用用戶界面在兩種不同的洗滌或未洗滌飛機條件之間進行選擇。在水洗飛機的情況下，檢測管道與3.2節(jié)中討論的相同。但是對于機身上有污點的未經洗滌的飛機，我們建議采用低通高斯濾波器來減少機身圖像上的類似噪聲的斑點，其方式對真實缺陷具有最小的平滑效果。但是，在減少圖像的高通分量和徹底保留缺陷之間需要進行權衡。我們克服該問題的方法是在對片進行分類之后進行后處理，這通過相鄰片的相似性比較來完成。貼片的強度范圍用作相似性度量。具體而言，如果將補丁檢測為缺陷補丁，則最有可能位于缺陷區(qū)域，因此我們還使用后處理方案測試其相鄰補丁，以確保檢測到缺陷區(qū)域中的所有補丁。如果相鄰的補丁滿足相似性閾值，則將其分類為缺陷類。
