本文運行環(huán)境為XCode9 beta2, iOS 11 beta2

1. CoreML

1. CoreML是蘋果在WWDC2017 新發(fā)布的Framework，方便了Machine Learning在蘋果自家平臺的接接入與使用，同時蘋果提供了Python的coremltools，方便將各大開源模型訓(xùn)練工具的現(xiàn)有模型轉(zhuǎn)化為MLModel。

2. 模型訓(xùn)練

   • Caffe
   • Karas
   • XGBoost
   • scikit-learn

2. vision

vision是一個新的，強大的，易于使用的框架，是蘋果于WWDC 2017上針對CoreML使用所提出的新Framework，能快速有效的用于面部檢測、面部特征點、文字、矩形、條形碼和物體。

3. 集成機器學(xué)習(xí)

我們將構(gòu)建一個通過AVCaptureSession捕獲到當(dāng)前圖像，并通過MLModel分析，獲取到與圖像最匹配的物品名字。
界面大概是

那么我們就正式開始

1. 創(chuàng)建Single工程

2. 可以從蘋果的“機器學(xué)習(xí)”頁面下載Inception v3。

3. 在Info.plist中添加Privacy - Camera Usage Description

4. 代碼編寫

1. 首先我們創(chuàng)建一個AVCaptureSession用來獲取攝像頭的圖像

   
       lazy var avSession: AVCaptureSession = AVCaptureSession()
       lazy var preViewLayer: AVCaptureVideoPreviewLayer = {
       return AVCaptureVideoPreviewLayer(session: self.avSession)
       }()
   
       override func viewDidLoad() {
           super.viewDidLoad()
       
           setupAVSession()
       
           preViewLayer.frame = view.bounds
           self.view.layer.insertSublayer(preViewLayer, at: 0)
       
           avSession.startRunning()
       }
   
   
       fileprivate func setupAVSession() {
       
           guard let device = AVCaptureDevice.default(for: .video) else {
               fatalError("this application cannot be run on simulator")
           }
       
           do {
           
               let input = try AVCaptureDeviceInput(device: device)
               avSession.addInput(input)
           
               let output = AVCaptureVideoDataOutput()
               avSession.addOutput(output)
           
               let queue = DispatchQueue(label: "video queue", qos: .userInteractive)
               output.setSampleBufferDelegate(self, queue: queue)
           } catch let error {
           
               print(error)
           }
   }
   

2. 實現(xiàn)AVCaptureVideoDataOutputSampleBufferDelegate代理

   extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
   
       func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
   
       //FIXME: 添加CoreML相關(guān)邏輯
       }
   }
   

3. 為工程引入MLModel
   直接拖拽，點擊Inceptionv3
   

   
   可以看到模型的詳細信息

4. 添加模型處理代碼

   lazy var inceptionv3ClassificationRequest: VNCoreMLRequest = {
       // Load the ML model through its generated class and create a Vision request for it.
       do {
           let model = try VNCoreMLModel(for: Inceptionv3().model)
           return VNCoreMLRequest(model: model, completionHandler: self.inceptionv3ClassificationHandler)
       } catch {
           fatalError("can't load Vision ML model: \(error)")
       }
   }()
       
   
   extension ViewController {
   
       func inceptionv3ClassificationHandler(request: VNRequest, error: Error?) {
           guard let observations = request.results as? [VNClassificationObservation]
               else { fatalError("unexpected result type from VNCoreMLRequest") }
       
           guard let best = observations.first
            else { fatalError("can't get best result") }
       
           DispatchQueue.main.async {
               print("Classification: \"\(best.identifier)\" Confidence: \(best.confidence)")
               self.classifyLabel.text = best.identifier
           }
       }
   }
   

5. 傳入MLModel參數(shù)

   func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
       guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
           return
       }
       
       let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])
       
       do {
           try handler.perform([inceptionv3ClassificationRequest])
       } catch _ {
           
       }
   }
   

4. 效果展示

至此已完成了機器學(xué)習(xí)的集成，代碼已上傳到git

參考資料

1. WWDC Session 506
2. WWDC Session 703
3. WWDC Session 710