1. 數(shù)據(jù)加載與存儲
1.1. np.save,np.load
In [78]: a = np.arange(10)
In [79]: a
Out[79]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In [80]: np.save('some_array',a)
In [83]: np.load('some_array.npy')
Out[83]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
1.2. 常規(guī)用 pd.read_<tab> 和data.to_<format>走遍天下，新版pandas幾乎什么格式都能讀了。
2. CSV 和 txt 格式
- 讀取.csv格式的文件，直接read_csv不需要加分隔號；用read_table需要制定分隔號
- 關(guān)于用CLI讀數(shù)據(jù)，linux人盡皆知用cat，但是windows用的是type，而且斜杠方向與linux相反
- csv很方便，直接read，然后選擇參數(shù)，例如header，index_col
a) 例子1，csv可以用read_csv或read_table讀取
# windows system
# ex1, csv and text values
In [3]: !type ch06\ex1.csv
a,b,c,d,message
1,2,3,4,hello
5,6,7,8,world
9,10,11,12,foo
In [10]: df = pd.read_csv('ch06/ex1.csv')
In [11]: df
Out[11]:
a b c d message
0 1 2 3 4 hello
1 5 6 7 8 world
2 9 10 11 12 foo
In [12]: df1 = pd.read_table('ch06/ex1.csv')
In [13]: df1
Out[13]:
a,b,c,d,message
0 1,2,3,4,hello
1 5,6,7,8,world
2 9,10,11,12,foo
In [14]: df1 = pd.read_table('ch06/ex1.csv',sep=',')
In [15]: df1
Out[15]:
a b c d message
0 1 2 3 4 hello
1 5 6 7 8 world
2 9 10 11 12 foo
b) 例子2，csv設(shè)置參數(shù)header，index_col
# ex2 csv and header,index_col
In [48]: pd.read_csv('ch06/ex2.csv',header=None)
Out[48]:
0 1 2 3 4
0 1 2 3 4 hello
1 5 6 7 8 world
2 9 10 11 12 foo
In [49]: pd.read_csv('ch06/ex2.csv',names=['a','b','c','d','message'])
Out[49]:
a b c d message
0 1 2 3 4 hello
1 5 6 7 8 world
2 9 10 11 12 foo
In [53]: pd.read_csv('ch06/ex2.csv',names= names,index_col = 'message')
Out[53]:
a b c d
message
hello 1 2 3 4
world 5 6 7 8
foo 9 10 11 12
# csv_mindex.csv
In [57]: !type ch06\csv_mindex.csv
key1,key2,value1,value2
one,a,1,2
one,b,3,4
one,c,5,6
one,d,7,8
two,a,9,10
two,b,11,12
two,c,13,14
two,d,15,16
In [60]: parsed = pd.read_csv('ch06/csv_mindex.csv',index_col=['key1','key2'])
In [61]: parsed
Out[61]:
value1 value2
key1 key2
one a 1 2
b 3 4
c 5 6
d 7 8
two a 9 10
b 11 12
c 13 14
d 15 16
c) 例子3，多個空格時使用正則式\s+
In [62]: list(open('ch06/ex3.txt'))
Out[62]:
[' A B C\n',
'aaa -0.264438 -1.026059 -0.619500\n',
'bbb 0.927272 0.302904 -0.032399\n',
'ccc -0.264273 -0.386314 -0.217601\n',
'ddd -0.871858 -0.348382 1.100491\n']
In [63]:
In [63]:
In [63]: result = pd.read_table('ch06/ex3.txt',sep='\s+')
In [64]: result
Out[64]:
A B C
aaa -0.264438 -1.026059 -0.619500
bbb 0.927272 0.302904 -0.032399
ccc -0.264273 -0.386314 -0.217601
ddd -0.871858 -0.348382 1.100491
d) 例子4，忽略格式不對的行，處理缺失值
In [65]: !type ch06\ex4.csv
# hey!
a,b,c,d,message
# just wanted to make things more difficult for you
# who reads CSV files with computers, anyway?
1,2,3,4,hello
5,6,7,8,world
9,10,11,12,foo
In [66]:
In [66]: pd.read_csv('ch06/ex4.csv',skiprows=[0,2,3])
Out[66]:
a b c d message
0 1 2 3 4 hello
1 5 6 7 8 world
2 9 10 11 12 foo
In [67]: !type ch06\ex5.csv
something,a,b,c,d,message
one,1,2,3,4,NA
two,5,6,,8,world
three,9,10,11,12,foo
In [68]: pd.read_csv('ch06/ex5.csv',na_values='Null')
Out[68]:
something a b c d message
0 one 1 2 3.0 4 NaN
1 two 5 6 NaN 8 world
2 three 9 10 11.0 12 foo
In [69]: setNAvaluse = {'message':['foo','NA'],'something':['two']}
In [70]: pd.read_csv('ch06/ex5.csv',na_values=setNAvaluse)
Out[70]:
something a b c d message
0 one 1 2 3.0 4 NaN
1 NaN 5 6 NaN 8 world
2 three 9 10 11.0 12 NaN
JSON 格式
json 包，直接load就好?？梢钥磒y4e免費(fèi)在線text book
XML tree
python3.6 直接有elementree可以用，數(shù)據(jù)讀出來常規(guī)處理就好。同上
二進(jìn)制
參考官網(wǎng)
7.1. struct — Interpret bytes as packed binary data
HDF5文件
這個好像是hadoop里的文件格式，適用于處理大批量文件，大數(shù)據(jù)上手繼續(xù)學(xué)這部分。
In [39]: store = pd.HDFStore('mydata.h5')
In [41]: frame
Out[41]:
a b c d message
0 1 2 3 4 hello
1 5 6 7 8 world
2 9 10 11 12 foo
In [42]: store['obj1'] = frame
In [43]: store
Out[43]:
<class 'pandas.io.pytables.HDFStore'>
File path: mydata.h5
/obj1 frame (shape->[3,5])
In [44]: store['obj1_col'] = frame['a']
In [45]: store
Out[45]:
<class 'pandas.io.pytables.HDFStore'>
File path: mydata.h5
/obj1 frame (shape->[3,5])
/obj1_col series (shape->[3])
EXCEL
不用按照書里的安裝啥庫了，現(xiàn)在pandas可以直接讀pd.read_excel('ch06/test.xls')
使用HTML和Web API
從網(wǎng)頁中獲取數(shù)據(jù)，暫時我只用過urllib和socket...
可以看py4e網(wǎng)站： Networked programs
request庫好像是高級用法，待做
數(shù)據(jù)庫
簡單的SQL語言可以用內(nèi)置的sqlite3
MongoDB
這是NoSQL數(shù)據(jù)庫，還沒裝，遲點(diǎn)跟著hadoop一起做...
2018.7.2x 大數(shù)據(jù)文件格式，上手后再做。被成功安利request庫處理網(wǎng)頁。
