AI Meets Beauty

Perfect Half Million Beauty Product Image Recognition Challenge

分析：比賽給的數(shù)據(jù)集是一個(gè)csv文件，包括了名稱和下載地址，因此首先要下載數(shù)據(jù)。
數(shù)據(jù)集總共包含520k張圖像，來自14個(gè)電商網(wǎng)站。

data.csv

Dataset

數(shù)據(jù)下載腳本如下

# -*- coding: utf-8 -*-
# download.py --下載數(shù)據(jù)集

from __future__ import print_function 
import os, csv
from skimage import io
from multiprocessing import Pool

def signal_handler(signum, frame, img_url):
    raise Exception('Time out--', img_url)

def getFile(url):
    url = url.split(',')
    imgname, imgurl = url[0], url[1].strip('"')
    imgformat = '.jpg'
    
    try: 
        image = io.imread(imgurl)
        io.imsave("./dataset/" + imgname + imgformat, image)
    except Exception as e:
        failures = open('failure.txt', 'a')
        print (imgname + ' ' + imgurl)
        failures.write(imgname + ' ' + imgurl + '\n')
    
if __name__ == "__main__":
    os.chdir(os.path.join(os.getcwd()))
    if not os.path.exists('./dataset'):
        os.mkdir('./dataset')
    
    with open ('data.csv', 'r') as f:
        lines=f.readlines()
        csvf = lines[:]
        #csvf.extend(lines[:])
        parallel_num = 10
        line_num = len(csvf)
        p = Pool(parallel_num)
        for i in range(int(line_num/parallel_num)):
            p.map(getFile, csvf[i*parallel_num:i*parallel_num+parallel_num])
    f.close()
    # failures.close()