本篇文章帶來(lái)的是TensorFlow框架下常見的數(shù)據(jù)讀取方式。

1、Preloaded data: 預(yù)加載數(shù)據(jù)
就是我們常見的寫在程序里面的數(shù)據(jù)格式。

#coding=utf8
import tensorflow as tf

a = tf.constant([[1,2],[3,4]])
b = tf.constant([[1,2],[3,4]])

c = tf.matmul(a,b)
with tf.Session() as sess:
    print(sess.run(c))

2、Feeding: Python產(chǎn)生數(shù)據(jù)，再把數(shù)據(jù)喂給后端。
這種方法經(jīng)常用到。

#coding=utf8
import tensorflow as tf

a = tf.placeholder(tf.int32,shape=[2,2])
b = tf.placeholder(tf.int32,shape=[2,2])
##a,b為占位符，只有在程序運(yùn)行的時(shí)候才載入數(shù)據(jù)

c = tf.matmul(a,b)

a1 = [[1,2],[3,4]]
b1 = [[1,2],[3,4]]
##這里的a1，b1是我們要傳入a，b的數(shù)據(jù)，也可以從文件讀取
with tf.Session() as sess:
    print(sess.run(c,feed_dict={a:a1,b:b1}))

3、Reading from file: 從文件中直接讀取
（1）、read from CSV or txt
有時(shí)我們遇到的數(shù)據(jù)文件是CSV或者txt格式。
單reader，單樣本（batch_size=1）

#coding=utf8
import tensorflow as tf

#創(chuàng)建文件隊(duì)列
filenames = ['datas/A.csv','datas/B.csv']
filename_queue = tf.train.string_input_producer(filenames,shuffle=True)
#shuffle=True 文件隊(duì)列隨機(jī)讀取，默認(rèn)

TFReader = tf.TextLineReader()
key,value = TFReader.read(filename_queue)

example, label = tf.decode_csv(value, record_defaults=[[], []])
##record_defaults=[[], []]文件讀取后的數(shù)據(jù)默認(rèn)格式，文件有幾列返回值就有幾個(gè)，
##默認(rèn)是英文逗號(hào)分隔，可以指定
##關(guān)于tf.decode_csv()的具體用法可以查看https://www.tensorflow.org/versions/master/api_docs/python/tf/decode_csv

with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess,coord=coord)
    for i in range(100):
        ##循環(huán)讀取，即使所有文件沒有那么多行
        print(example.eval(),label.eval())
    coord.request_stop()
    coord.join(threads)

單reader，多樣本（batch_size）

#coding=utf8
import tensorflow as tf

#創(chuàng)建文件隊(duì)列
filenames = ['datas/A.csv','datas/B.csv']
filename_queue = tf.train.string_input_producer(filenames,shuffle=False)
#shuffle=True 文件隊(duì)列隨機(jī)讀取，默認(rèn)

TFReader = tf.TextLineReader()
key,value = TFReader.read(filename_queue)

example, label = tf.decode_csv(value, record_defaults=[[], []])
##record_defaults=[[], []]文件讀取后的數(shù)據(jù)默認(rèn)格式，文件有幾列返回值就有幾個(gè)，
##默認(rèn)是英文逗號(hào)分隔，可以指定

example_batch,label_batch = tf.train.batch([example,label],
                                           batch_size=5,
                                           capacity=100,
                                           num_threads=2)
# ###隨機(jī)讀取
# example_batch,label_batch = tf.train.shuffle_batch([example,label],
#                                                     batch_size=5,
#                                                     capacity=100,
#                                                    min_after_dequeue=50,
#                                                     num_threads=2)

with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess,coord=coord)
    for i in range(10):
        ##循環(huán)讀取，即使所有文件沒有那么多行
        print(example_batch.eval())
    coord.request_stop()
    coord.join(threads)

多reader，多樣本

#coding=utf8
import tensorflow as tf

#創(chuàng)建文件隊(duì)列
filenames = ['datas/A.csv','datas/B.csv']
filename_queue = tf.train.string_input_producer(filenames,shuffle=False)
#shuffle=True 文件隊(duì)列隨機(jī)讀取，默認(rèn)

TFReader = tf.TextLineReader()
key,value = TFReader.read(filename_queue)

example_list = [tf.decode_csv(value, record_defaults=[[], []]) for _ in range(2)]
##2表示創(chuàng)建兩個(gè)reader

example_batch,label_batch = tf.train.batch_join(example_list,batch_size=5)
# 使用tf.train.batch_join()，可以使用多個(gè)reader，并行讀取數(shù)據(jù)。每個(gè)Reader使用一個(gè)線程。

with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess,coord=coord)
    for i in range(10):
        ##循環(huán)讀取，即使所有文件沒有那么多行
        print(example_batch.eval(),label_batch.eval())
    coord.request_stop()
    coord.join(threads)

tf.train.batch與tf.train.shuffle_batch函數(shù)是單個(gè)Reader讀取，但是可以多線程。tf.train.batch_join與tf.train.shuffle_batch_join可設(shè)置多Reader讀取，每個(gè)Reader使用一個(gè)線程。至于兩種方法的效率，單Reader時(shí)，2個(gè)線程就達(dá)到了速度的極限。多Reader時(shí)，2個(gè)Reader就達(dá)到了極限。所以并不是線程越多越快，甚至更多的線程反而會(huì)使效率下降。
（2）、read from tfrecords
這種讀取方式常被用來(lái)讀取圖片數(shù)據(jù)，先是將圖片數(shù)據(jù)寫入到tfrecords文件中，當(dāng)要使用時(shí)再?gòu)闹凶x取，速度很快，但是將圖片格式的文件寫入tfrecords文件后所占用的磁盤內(nèi)存更大？有弊有利，在圖像處理時(shí)要先將其處理成相同大小的圖片保存（很重要，我在測(cè)試過程中沒有找到這么儲(chǔ)存不同大小的圖片）。

#coding=utf8
import tensorflow as tf
import os
import numpy as np
from PIL import Image

## 寫入tfrecords文件
def write_tfrecods():
    np.random.seed(100)
    path = 'E:/cats_dogs/'
    savepath = 'datas/test.tfrecords'
    files = [path+item for item in os.listdir(path) if item.endswith('.jpg')]
    np.random.shuffle(files)
    train_files = files[:23000]
    test_files = files[23000:]
    TFWriter = tf.python_io.TFRecordWriter(savepath)
    for i,file in enumerate(test_files):
        if i%1000==0:
            print(i)
        lab = file.split('/')[-1].split('.')[0].strip()
        if lab=='cat':
            label = 1
        else:
            label = 0
        image = Image.open(file)
        image = image.resize((208,208))
        imagerow = image.tobytes()
        sample = tf.train.Example(features=tf.train.Features(feature={
            'label':tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),
            'image':tf.train.Feature(bytes_list=tf.train.BytesList(value=[imagerow]))
        }))
        TFWriter.write(sample.SerializeToString())
    TFWriter.close()


# write_tfrecods()


## 從tfrecords文件讀取
def read_tfrecords():
    filepath = 'datas/test.tfrecords'
    filename_queue = tf.train.string_input_producer([filepath])
    TFReader = tf.TFRecordReader()
    _,serialize_sample = TFReader.read(filename_queue)
    features = tf.parse_single_example(serialize_sample,features={
        'label':tf.FixedLenFeature([],tf.int64),
        'image':tf.FixedLenFeature([],tf.string)
    })
    image = tf.decode_raw(features['image'],tf.uint8)
    image = tf.reshape(image,shape=[208,208,3])
    label = tf.cast(features['label'],tf.int32)

    return image,label

## 讀取批次
def next_batch(batch_size):
    # import matplotlib.pyplot as plt
    image,label = read_tfrecords()
    image_batch,label_batch = tf.train.shuffle_batch([image,label],
                                                     batch_size=batch_size,
                                                     capacity=200,
                                                     min_after_dequeue=100,
                                                     num_threads=32)

    # with tf.Session() as sess:
    #     sess.run(tf.global_variables_initializer())
    #     coord = tf.train.Coordinator()
    #     threads = tf.train.start_queue_runners(coord=coord)
    #     image,label = sess.run([image_batch,label_batch])
    #     for i in range(2):
    #         print(label[i])
    #         plt.imshow(image[i])
    #         plt.show()
    #     coord.request_stop()
    #     coord.join(threads)
    image_batch = tf.cast(image_batch,tf.float32)
    label_batch = tf.cast(label_batch,tf.int32)
    return image_batch,label_batch

# next_batch(2)

上面這種方法非?？?，而且不占用內(nèi)存，但是缺點(diǎn)是數(shù)據(jù)已定，比如在處理圖像數(shù)據(jù)時(shí)往往為了增加數(shù)據(jù)量，會(huì)對(duì)圖像做一些噪聲，如模糊、亮度、刪除，如果先處理后再寫入tfrecords文件，那么是及其浪費(fèi)磁盤空間的，因此我有時(shí)候就喜歡直接讀取文件，然后對(duì)圖片加噪聲處理，然后再送入訓(xùn)練，這樣做的缺點(diǎn)是高速讀寫圖片文件，會(huì)非常占用CPU。

import tensorflow as tf
import numpy as np
import os
import math

# you need to change this to your data directory
# train_dir = '/home/acrobat/DataSets/cats_vs_dogs/train/'

def get_files(file_dir, ratio):
    """
    Args:
        file_dir: file directory
        ratio:ratio of validation datasets
    Returns:
        list of images and labels
    """
    cats = []
    label_cats = []
    dogs = []
    label_dogs = []
    for file in os.listdir(file_dir):
        name = file.split(sep='.')
        if name[0]=='cat':
            cats.append(file_dir + file)
            label_cats.append(0)
        else:
            dogs.append(file_dir + file)
            label_dogs.append(1)
    print('There are %d cats\nThere are %d dogs' %(len(cats), len(dogs)))

    image_list = np.hstack((cats, dogs))
    label_list = np.hstack((label_cats, label_dogs))

    temp = np.array([image_list, label_list])
    temp = temp.transpose()
    np.random.shuffle(temp)

    all_image_list = temp[:, 0]
    all_label_list = temp[:, 1]

    n_sample = len(all_label_list)
    n_val = math.ceil(n_sample*ratio) # number of validation samples
    n_train = n_sample - n_val # number of trainning samples

    tra_images = all_image_list[0:n_train]
    tra_labels = all_label_list[0:n_train]
    tra_labels = [int(float(i)) for i in tra_labels]
    val_images = all_image_list[n_train:-1]
    val_labels = all_label_list[n_train:-1]
    val_labels = [int(float(i)) for i in val_labels]

    return tra_images,tra_labels,val_images,val_labels


def get_batch(image, label, image_W, image_H, batch_size, capacity):
    """
    Args:
        image: list type
        label: list type
        image_W: image width
        image_H: image height
        batch_size: batch size
        capacity: the maximum elements in queue
    Returns:
        image_batch: 4D tensor [batch_size, width, height, 3], dtype=tf.float32
        label_batch: 1D tensor [batch_size], dtype=tf.int32
    """

    image = tf.cast(image, tf.string)
    label = tf.cast(label, tf.int32)

    # make an input queue
    input_queue = tf.train.slice_input_producer([image, label])

    label = input_queue[1]
    image_contents = tf.read_file(input_queue[0])
    image = tf.image.decode_jpeg(image_contents, channels=3)

    image = tf.image.resize_image_with_crop_or_pad(image, image_W, image_H)

    # if you want to test the generated batches of images, you might want to comment the following line.
    image = tf.image.per_image_standardization(image)

    image_batch, label_batch = tf.train.batch([image, label],
                                                batch_size= batch_size,
                                                num_threads= 64,
                                                capacity = capacity)
    #you can also use shuffle_batch
#    image_batch, label_batch = tf.train.shuffle_batch([image,label],
#                                                      batch_size=BATCH_SIZE,
#                                                      num_threads=64,
#                                                      capacity=CAPACITY,
#                                                      min_after_dequeue=CAPACITY-1)

    label_batch = tf.reshape(label_batch, [batch_size])
    image_batch = tf.cast(image_batch, tf.float32)

    return image_batch, label_batch

（3）、read from bin
有的時(shí)候我們的數(shù)據(jù)是二進(jìn)制格式（bin），因此需要將二進(jìn)制文件讀取出來(lái)。
在官網(wǎng)的cifar的例子中就是從bin文件中讀取的。bin文件需要以一定的size格式存儲(chǔ)，比如每個(gè)樣本的值占多少字節(jié)，label占多少字節(jié)，且這對(duì)于每個(gè)樣本都是固定的，然后一個(gè)挨著一個(gè)存儲(chǔ)。這樣就可以使用tf.FixedLengthRecordReader 類來(lái)每次讀取固定長(zhǎng)度的字節(jié)，正好對(duì)應(yīng)一個(gè)樣本存儲(chǔ)的字節(jié)（包括label）。并且用tf.decode_raw進(jìn)行解析。

import tensorflow as tf
import numpy as np

# 預(yù)定義圖像數(shù)據(jù)信息
labelBytes = 1
witdthBytes = 32
heightBytes = 32
depthBytes = 3
imageBytes = witdthBytes*heightBytes*depthBytes
recordBytes = imageBytes+labelBytes

filename_queue = tf.train.string_input_producer(["./data/train.bin"])
reader = tf.FixedLengthRecordReader(record_bytes=recordBytes) # 按固定長(zhǎng)度讀取二進(jìn)制文件
key,value = reader.read(filename_queue)

bytes = tf.decode_raw(value,out_type=tf.uint8) # 解碼為uint8,0-255 8位3通道圖像
label = tf.cast(tf.strided_slice(bytes,[0],[labelBytes]),tf.int32) # 分割label并轉(zhuǎn)化為int32
##tf.strided_slice() 將讀取的一個(gè)bytes切分
originalImg  = tf.reshape(tf.strided_slice(bytes,[labelBytes],[labelBytes+imageBytes]),[depthBytes,heightBytes,witdthBytes])
# 分割圖像，此時(shí)按照數(shù)據(jù)組織形式深度在前
img = tf.transpose(originalImg,[1,2,0]) # 調(diào)整軸的順序，深度在后

with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)

    for i in range(100):
        imgArr = sess.run(img)
        print (imgArr.shape)

    coord.request_stop()
    coord.join(threads)

tf.strided_slice具體用法見：https://www.tensorflow.org/versions/master/api_docs/python/tf/strided_slice

上面就是我現(xiàn)在遇到過的TensorFlow讀寫數(shù)據(jù)的一些常見方式，后面遇到了會(huì)陸續(xù)添加。

補(bǔ)充：這里有一篇比較詳細(xì)的tfrecords文件的讀寫教程，受益頗多！！
http://blog.csdn.net/u010223750/article/details/70482498

參考文章：
http://honggang.io/2016/08/19/tensorflow-data-reading/
http://blog.csdn.net/freedom098/article/details/56008784