創(chuàng)建數(shù)據(jù)
Series和python的列表類似。DataFrame則類似值為Series的字典。
create.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# create.py
import pandas as pd
print("\n\n創(chuàng)建序列Series")
s = pd.Series(['banana', 42])
print(s)
print("\n\n指定索引index創(chuàng)建序列Series")
s = pd.Series(['Wes McKinney', 'Creator of Pandas'], index=['Person', 'Who'])
print(s)
# 注意：列名未必為執(zhí)行的順序，通常為按字母排序
print("\n\n創(chuàng)建數(shù)據(jù)幀DataFrame")
scientists = pd.DataFrame({
' Name': ['Rosaline Franklin', 'William Gosset'],
' Occupation': ['Chemist', 'Statistician'],
' Born': ['1920-07-25', '1876-06-13'],
' Died': ['1958-04-16', '1937-10-16'],
' Age': [37, 61]})
print(scientists)
print("\n\n指定順序(index和columns)創(chuàng)建數(shù)據(jù)幀DataFrame")
scientists = pd.DataFrame(
data={'Occupation': ['Chemist', 'Statistician'],
'Born': ['1920-07-25', '1876-06-13'],
'Died': ['1958-04-16', '1937-10-16'],
'Age': [37, 61]},
index=['Rosaline Franklin', 'William Gosset'],
columns=['Occupation', 'Born', 'Died', 'Age'])
print(scientists)
執(zhí)行結(jié)果：
$ ./create.py
創(chuàng)建序列Series
0 banana
1 42
dtype: object
指定索引index創(chuàng)建序列Series
Person Wes McKinney
Who Creator of Pandas
dtype: object
創(chuàng)建數(shù)據(jù)幀DataFrame
Name Occupation Born Died Age
0 Rosaline Franklin Chemist 1920-07-25 1958-04-16 37
1 William Gosset Statistician 1876-06-13 1937-10-16 61
指定順序(index和columns)創(chuàng)建數(shù)據(jù)幀DataFrame
Occupation Born Died Age
Rosaline Franklin Chemist 1920-07-25 1958-04-16 37
William Gosset Statistician 1876-06-13 1937-10-16 61
Series
官方文檔：http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html
Series的屬性
| 屬性 | 描述 |
|---|---|
| loc | 使用索引值獲取子集 |
| iloc | 使用索引位置獲取子集 |
| dtype或dtypes | 類型 |
| T | 轉(zhuǎn)置 |
| shape | 數(shù)據(jù)的尺寸 |
| size | 元素的數(shù)量 |
| values | ndarray或類似ndarray的Series |
Series的方法
| 方法 | 描述 |
|---|---|
| append | 連接2個(gè)或更多系列 |
| corr | 計(jì)算與其他Series的關(guān)聯(lián) |
| cov | 與其他Series計(jì)算協(xié)方差 |
| describe | 計(jì)算匯總統(tǒng)計(jì) |
| drop duplicates | 返回一個(gè)沒(méi)有重復(fù)項(xiàng)的Series |
| equals | Series是否具有相同的元素 |
| get values | 獲取Series的值，與values屬性相同 |
| hist | 繪制直方圖 |
| min | 返回最小值 |
| max | 返回最大值 |
| mean | 返回算術(shù)平均值 |
| median | 返回中位數(shù) |
| mode(s) | 返回mode(s) |
| replace | 用指定值替換系列中的值 |
| sample | 返回Series中值的隨機(jī)樣本 |
| sort values | 排序 |
| to frame | 轉(zhuǎn)換為數(shù)據(jù)幀 |
| transpose | 返回轉(zhuǎn)置 |
| unique | 返回numpy.ndarray唯一值 |
series.py
#!/usr/bin/python3
# -*- coding: utf-8 -*-
# CreateDate: 2018-3-14
# series.py
import pandas as pd
import numpy as np
scientists = pd.DataFrame(
data={'Occupation': ['Chemist', 'Statistician'],
'Born': ['1920-07-25', '1876-06-13'],
'Died': ['1958-04-16', '1937-10-16'],
'Age': [37, 61]},
index=['Rosaline Franklin', 'William Gosset'],
columns=['Occupation', 'Born', 'Died', 'Age'])
print(scientists)
# 從數(shù)據(jù)幀(DataFrame)獲取的行或者列為Series
first_row = scientists.loc['William Gosset']
print(type(first_row))
print(first_row)
# index和keys是一樣的
print(first_row.index)
print(first_row.keys())
print(first_row.values)
print(first_row.index[0])
print(first_row.keys()[0])
# Pandas.Series和numpy.ndarray很類似
ages = scientists['Age']
print(ages)
# 統(tǒng)計(jì)，更多參考http://pandas.pydata.org/pandas-docs/stable/basics.html#descriptive-statistics
print(ages.mean())
print(ages.min())
print(ages.max())
print(ages.std())
scientists = pd.read_csv('../data/scientists.csv')
ages = scientists['Age']
print(ages)
print(ages.mean())
print(ages.describe())
print(ages[ages > ages.mean()])
print(ages > ages.mean())
manual_bool_values = [True, True, False, False, True, True, False, False]
print(ages[manual_bool_values])
print(ages + ages)
print(ages * ages)
print(ages + 100)
print(ages * 2)
print(ages + pd.Series([1, 100]))
# print(ages + np.array([1, 100])) 會(huì)報(bào)錯(cuò)，不同類型相加，大小一定要一樣
print(ages + np.array([1, 100, 1, 100, 1, 100, 1, 100]))
# 排序： 默認(rèn)有自動(dòng)排序
print(ages)
rev_ages = ages.sort_index(ascending=False)
print(rev_ages)
print(ages * 2)
print(ages + rev_ages)
執(zhí)行結(jié)果
$ python3 series.py
Occupation Born Died Age
Rosaline Franklin Chemist 1920-07-25 1958-04-16 37
William Gosset Statistician 1876-06-13 1937-10-16 61
<class 'pandas.core.series.Series'>
Occupation Statistician
Born 1876-06-13
Died 1937-10-16
Age 61
Name: William Gosset, dtype: object
Index(['Occupation', 'Born', 'Died', 'Age'], dtype='object')
Index(['Occupation', 'Born', 'Died', 'Age'], dtype='object')
['Statistician' '1876-06-13' '1937-10-16' 61]
Occupation
Occupation
Rosaline Franklin 37
William Gosset 61
Name: Age, dtype: int64
49.0
37
61
16.97056274847714
0 37
1 61
2 90
3 66
4 56
5 45
6 41
7 77
Name: Age, dtype: int64
59.125
count 8.000000
mean 59.125000
std 18.325918
min 37.000000
25% 44.000000
50% 58.500000
75% 68.750000
max 90.000000
Name: Age, dtype: float64
1 61
2 90
3 66
7 77
Name: Age, dtype: int64
0 False
1 True
2 True
3 True
4 False
5 False
6 False
7 True
Name: Age, dtype: bool
0 37
1 61
4 56
5 45
Name: Age, dtype: int64
0 74
1 122
2 180
3 132
4 112
5 90
6 82
7 154
Name: Age, dtype: int64
0 1369
1 3721
2 8100
3 4356
4 3136
5 2025
6 1681
7 5929
Name: Age, dtype: int64
0 137
1 161
2 190
3 166
4 156
5 145
6 141
7 177
Name: Age, dtype: int64
0 74
1 122
2 180
3 132
4 112
5 90
6 82
7 154
Name: Age, dtype: int64
0 38.0
1 161.0
2 NaN
3 NaN
4 NaN
5 NaN
6 NaN
7 NaN
dtype: float64
0 38
1 161
2 91
3 166
4 57
5 145
6 42
7 177
Name: Age, dtype: int64
0 37
1 61
2 90
3 66
4 56
5 45
6 41
7 77
Name: Age, dtype: int64
7 77
6 41
5 45
4 56
3 66
2 90
1 61
0 37
Name: Age, dtype: int64
0 74
1 122
2 180
3 132
4 112
5 90
6 82
7 154
Name: Age, dtype: int64
0 74
1 122
2 180
3 132
4 112
5 90
6 82
7 154
Name: Age, dtype: int64
數(shù)據(jù)幀(DataFrame)
DataFrame是最常見(jiàn)的Pandas對(duì)象,可認(rèn)為是Python存儲(chǔ)類似電子表格的數(shù)據(jù)的方式。Series多常見(jiàn)功能都包含在DataFrame中。
子集的方法
注意ix現(xiàn)在已經(jīng)不推薦使用。
DataFrame常用的索引操作有：
| 方式 | 描述 |
|---|---|
| df[val] | 選擇單個(gè)列 |
| df [[ column1, column2, ... ]] | 選擇多個(gè)列 |
| df.loc[val] | 選擇行 |
| df. loc [[ label1 , label2 ,...]] | 選擇多行 |
| df.loc[:, val] | 基于行index選擇列 |
| df.loc[val1, val2] | 選擇行列 |
| df.iloc[row number] | 基于行數(shù)選擇行 |
| df. iloc [[ row1, row2, ...]] Multiple rows by row number | 基于行數(shù)選擇多行 |
| df.iloc[:, where] | 選擇列 |
| df.iloc[where_i, where_j] | 選擇行列 |
| df.at[label_i, label_j] | 選擇值 |
| df.iat[i, j] | 選擇值 |
| reindex method | 通過(guò)label選擇多行或列 |
| get_value, set_value | 通過(guò)label選擇耽擱行或列 |
| df[bool] | 選擇行 |
| df [[ bool1, bool2, ...]] | 選擇行 |
| df[ start :stop: step ] | 基于行數(shù)選擇行 |
#!/usr/bin/python3
# -*- coding: utf-8 -*-
# CreateDate: 2018-3-31
# df.py
import pandas as pd
import numpy as np
scientists = pd.read_csv('../data/scientists.csv')
print(scientists[scientists['Age'] > scientists['Age'].mean()])
first_half = scientists[: 4]
second_half = scientists[ 4 :]
print(first_half)
print(second_half)
print(first_half + second_half)
print(scientists * 2)
執(zhí)行結(jié)果
#!/usr/bin/python3
# -*- coding: utf-8 -*-
# df.py
import pandas as pd
import numpy as np
scientists = pd.read_csv('../data/scientists.csv')
print(scientists[scientists['Age'] > scientists['Age'].mean()])
first_half = scientists[: 4]
second_half = scientists[ 4 :]
print(first_half)
print(second_half)
print(first_half + second_half)
print(scientists * 2)
執(zhí)行結(jié)果
$ python3 df.py
Name Born Died Age Occupation
1 William Gosset 1876-06-13 1937-10-16 61 Statistician
2 Florence Nightingale 1820-05-12 1910-08-13 90 Nurse
3 Marie Curie 1867-11-07 1934-07-04 66 Chemist
7 Johann Gauss 1777-04-30 1855-02-23 77 Mathematician
Name Born Died Age Occupation
0 Rosaline Franklin 1920-07-25 1958-04-16 37 Chemist
1 William Gosset 1876-06-13 1937-10-16 61 Statistician
2 Florence Nightingale 1820-05-12 1910-08-13 90 Nurse
3 Marie Curie 1867-11-07 1934-07-04 66 Chemist
Name Born Died Age Occupation
4 Rachel Carson 1907-05-27 1964-04-14 56 Biologist
5 John Snow 1813-03-15 1858-06-16 45 Physician
6 Alan Turing 1912-06-23 1954-06-07 41 Computer Scientist
7 Johann Gauss 1777-04-30 1855-02-23 77 Mathematician
Name Born Died Age Occupation
0 NaN NaN NaN NaN NaN
1 NaN NaN NaN NaN NaN
2 NaN NaN NaN NaN NaN
3 NaN NaN NaN NaN NaN
4 NaN NaN NaN NaN NaN
5 NaN NaN NaN NaN NaN
6 NaN NaN NaN NaN NaN
7 NaN NaN NaN NaN NaN
Name Born \
0 Rosaline FranklinRosaline Franklin 1920-07-251920-07-25
1 William GossetWilliam Gosset 1876-06-131876-06-13
2 Florence NightingaleFlorence Nightingale 1820-05-121820-05-12
3 Marie CurieMarie Curie 1867-11-071867-11-07
4 Rachel CarsonRachel Carson 1907-05-271907-05-27
5 John SnowJohn Snow 1813-03-151813-03-15
6 Alan TuringAlan Turing 1912-06-231912-06-23
7 Johann GaussJohann Gauss 1777-04-301777-04-30
Died Age Occupation
0 1958-04-161958-04-16 74 ChemistChemist
1 1937-10-161937-10-16 122 StatisticianStatistician
2 1910-08-131910-08-13 180 NurseNurse
3 1934-07-041934-07-04 132 ChemistChemist
4 1964-04-141964-04-14 112 BiologistBiologist
5 1858-06-161858-06-16 90 PhysicianPhysician
6 1954-06-071954-06-07 82 Computer ScientistComputer Scientist
7 1855-02-231855-02-23 154 MathematicianMathematician
修改列
#!/usr/bin/python3
# -*- coding: utf-8 -*-
# Author: xurongzhong#126.com wechat:pythontesting qq:37391319
# qq群：144081101 591302926 567351477
# CreateDate: 2018-06-07
# change.py
import pandas as pd
import numpy as np
import random
scientists = pd.read_csv('../data/scientists.csv')
print(scientists['Born'].dtype)
print(scientists['Died'].dtype)
print(scientists.head())
# 轉(zhuǎn)為日期 參考：https://docs.python.org/3.5/library/datetime.html
born_datetime = pd.to_datetime(scientists['Born'], format='%Y-%m-%d')
died_datetime = pd.to_datetime(scientists['Died'], format='%Y-%m-%d')
# 增加列
scientists['born_dt'], scientists['died_dt'] = (born_datetime, died_datetime)
print(scientists.shape)
print(scientists.head())
random.seed(42)
random.shuffle(scientists['Age']) # 此修改會(huì)作用于scientists
print(scientists.head())
scientists['age_days_dt'] = (scientists['died_dt'] - scientists['born_dt'])
print(scientists.head())
執(zhí)行結(jié)果：
$ python3 change.py
object
object
Name Born Died Age Occupation
0 Rosaline Franklin 1920-07-25 1958-04-16 37 Chemist
1 William Gosset 1876-06-13 1937-10-16 61 Statistician
2 Florence Nightingale 1820-05-12 1910-08-13 90 Nurse
3 Marie Curie 1867-11-07 1934-07-04 66 Chemist
4 Rachel Carson 1907-05-27 1964-04-14 56 Biologist
(8, 7)
Name Born Died Age Occupation born_dt \
0 Rosaline Franklin 1920-07-25 1958-04-16 37 Chemist 1920-07-25
1 William Gosset 1876-06-13 1937-10-16 61 Statistician 1876-06-13
2 Florence Nightingale 1820-05-12 1910-08-13 90 Nurse 1820-05-12
3 Marie Curie 1867-11-07 1934-07-04 66 Chemist 1867-11-07
4 Rachel Carson 1907-05-27 1964-04-14 56 Biologist 1907-05-27
died_dt
0 1958-04-16
1 1937-10-16
2 1910-08-13
3 1934-07-04
4 1964-04-14
/usr/lib/python3.5/random.py:272: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
x[i], x[j] = x[j], x[i]
Name Born Died Age Occupation born_dt \
0 Rosaline Franklin 1920-07-25 1958-04-16 66 Chemist 1920-07-25
1 William Gosset 1876-06-13 1937-10-16 56 Statistician 1876-06-13
2 Florence Nightingale 1820-05-12 1910-08-13 41 Nurse 1820-05-12
3 Marie Curie 1867-11-07 1934-07-04 77 Chemist 1867-11-07
4 Rachel Carson 1907-05-27 1964-04-14 90 Biologist 1907-05-27
died_dt
0 1958-04-16
1 1937-10-16
2 1910-08-13
3 1934-07-04
4 1964-04-14
Name Born Died Age Occupation born_dt \
0 Rosaline Franklin 1920-07-25 1958-04-16 66 Chemist 1920-07-25
1 William Gosset 1876-06-13 1937-10-16 56 Statistician 1876-06-13
2 Florence Nightingale 1820-05-12 1910-08-13 41 Nurse 1820-05-12
3 Marie Curie 1867-11-07 1934-07-04 77 Chemist 1867-11-07
4 Rachel Carson 1907-05-27 1964-04-14 90 Biologist 1907-05-27
died_dt age_days_dt
0 1958-04-16 13779 days
1 1937-10-16 22404 days
2 1910-08-13 32964 days
3 1934-07-04 24345 days
4 1964-04-14 20777 days
數(shù)據(jù)導(dǎo)入導(dǎo)出
out.py
#!/usr/bin/python3
# -*- coding: utf-8 -*-
# Author: china-testing#126.com wechat:pythontesting qq群:630011153
# CreateDate: 2018-3-31
# out.py
import pandas as pd
import numpy as np
import random
scientists = pd.read_csv('../data/scientists.csv')
names = scientists['Name']
print(names)
names.to_pickle('../output/scientists_names_series.pickle')
scientists.to_pickle('../output/scientists_df.pickle')
# .p, .pkl, .pickle 是常用的pickle文件擴(kuò)展名
scientist_names_from_pickle = pd.read_pickle('../output/scientists_df.pickle')
print(scientist_names_from_pickle)
names.to_csv('../output/scientist_names_series.csv')
scientists.to_csv('../output/scientists_df.tsv', sep='\t')
# 不輸出行號(hào)
scientists.to_csv('../output/scientists_df_no_index.csv', index=None)
# Series可以轉(zhuǎn)為df再輸出成excel文件
names_df = names.to_frame()
names_df.to_excel('../output/scientists_names_series_df.xls')
names_df.to_excel('../output/scientists_names_series_df.xlsx')
scientists.to_excel('../output/scientists_df.xlsx', sheet_name='scientists',
index=False)
執(zhí)行結(jié)果：
$ python3 out.py
0 Rosaline Franklin
1 William Gosset
2 Florence Nightingale
3 Marie Curie
4 Rachel Carson
5 John Snow
6 Alan Turing
7 Johann Gauss
Name: Name, dtype: object
Name Born Died Age Occupation
0 Rosaline Franklin 1920-07-25 1958-04-16 37 Chemist
1 William Gosset 1876-06-13 1937-10-16 61 Statistician
2 Florence Nightingale 1820-05-12 1910-08-13 90 Nurse
3 Marie Curie 1867-11-07 1934-07-04 66 Chemist
4 Rachel Carson 1907-05-27 1964-04-14 56 Biologist
5 John Snow 1813-03-15 1858-06-16 45 Physician
6 Alan Turing 1912-06-23 1954-06-07 41 Computer Scientist
7 Johann Gauss 1777-04-30 1855-02-23 77 Mathematician
注意：序列一般是直接輸出成excel文件
更多的輸入輸出方法：
| 方式 | 描述 |
|---|---|
| to_clipboard | 將數(shù)據(jù)保存到系統(tǒng)剪貼板進(jìn)行粘貼 |
| to_dense | 將數(shù)據(jù)轉(zhuǎn)換為常規(guī)“密集”DataFrame |
| to_dict | 將數(shù)據(jù)轉(zhuǎn)換為Python字典 |
| to_gbq | 將數(shù)據(jù)轉(zhuǎn)換為Google BigQuery表格 |
| toJidf | 將數(shù)據(jù)保存為分層數(shù)據(jù)格式（HDF） |
| to_msgpack | 將數(shù)據(jù)保存到可移植的類似JSON的二進(jìn)制文件中 |
| toJitml | 將數(shù)據(jù)轉(zhuǎn)換為HTML表格 |
| tojson | 將數(shù)據(jù)轉(zhuǎn)換為JSON字符串 |
| toJatex | 將數(shù)據(jù)轉(zhuǎn)換為L(zhǎng)TEXtabular環(huán)境 |
| to_records | 將數(shù)據(jù)轉(zhuǎn)換為記錄數(shù)組 |
| to_string | 將DataFrame顯示為stdout的字符串 |
| to_sparse | 將數(shù)據(jù)轉(zhuǎn)換為SparceDataFrame |
| to_sql | 將數(shù)據(jù)保存到SQL數(shù)據(jù)庫(kù)中 |
| to_stata | 將數(shù)據(jù)轉(zhuǎn)換為Stata dta文件 |
- 讀CSV文件
read_csv.py
#!/usr/bin/python3
# -*- coding: utf-8 -*-
# Author: china-testing#126.com wechat:pythontesting QQ群:630011153
# CreateDate: 2018-3-9
# read_csv.py
import pandas as pd
df = pd.read_csv("1.csv", header=None) # 不讀取列名
print("df:")
print(df)
print("df.head():")
print(df.head()) # head(self, n=5)，默認(rèn)為5行，類似的有tail
print("df.tail():")
print(df.tail())
df = pd.read_csv("1.csv") # 默認(rèn)讀取列名
print("df:")
print(df)
df = pd.read_csv("1.csv", names=['號(hào)碼','群號(hào)']) # 自定義列名
print("df:")
print(df)
# 自定義列名，去掉第一行
df = pd.read_csv("1.csv", skiprows=[0], names=['號(hào)碼','群號(hào)'])
print("df:")
print(df)
執(zhí)行結(jié)果：
df:
0 1
0 qq qqgroup
1 37391319 144081101
2 37391320 144081102
3 37391321 144081103
4 37391322 144081104
5 37391323 144081105
6 37391324 144081106
7 37391325 144081107
8 37391326 144081108
9 37391327 144081109
10 37391328 144081110
11 37391329 144081111
12 37391330 144081112
13 37391331 144081113
14 37391332 144081114
15 37391333 144081115
df.head():
0 1
0 qq qqgroup
1 37391319 144081101
2 37391320 144081102
3 37391321 144081103
4 37391322 144081104
df.tail():
0 1
11 37391329 144081111
12 37391330 144081112
13 37391331 144081113
14 37391332 144081114
15 37391333 144081115
df:
qq qqgroup
0 37391319 144081101
1 37391320 144081102
2 37391321 144081103
3 37391322 144081104
4 37391323 144081105
5 37391324 144081106
6 37391325 144081107
7 37391326 144081108
8 37391327 144081109
9 37391328 144081110
10 37391329 144081111
11 37391330 144081112
12 37391331 144081113
13 37391332 144081114
14 37391333 144081115
df:
號(hào)碼 群號(hào)
0 qq qqgroup
1 37391319 144081101
2 37391320 144081102
3 37391321 144081103
4 37391322 144081104
5 37391323 144081105
6 37391324 144081106
7 37391325 144081107
8 37391326 144081108
9 37391327 144081109
10 37391328 144081110
11 37391329 144081111
12 37391330 144081112
13 37391331 144081113
14 37391332 144081114
15 37391333 144081115
df:
號(hào)碼 群號(hào)
0 37391319 144081101
1 37391320 144081102
2 37391321 144081103
3 37391322 144081104
4 37391323 144081105
5 37391324 144081106
6 37391325 144081107
7 37391326 144081108
8 37391327 144081109
9 37391328 144081110
10 37391329 144081111
11 37391330 144081112
12 37391331 144081113
13 37391332 144081114
14 37391333 144081115
- 寫(xiě)CSV文件
#!/usr/bin/python3
# -*- coding: utf-8 -*-
# write_csv.py
import pandas as pd
data ={'qq': [37391319,37391320], 'group':[1,2]}
df = pd.DataFrame(data=data, columns=['qq','group'])
df.to_csv('2.csv',index=False)
讀寫(xiě)excel和csv類似，不過(guò)要改用read_excel來(lái)讀，excel_summary_demo, 提供了多個(gè)excel求和的功能，可以做為excel讀寫(xiě)的實(shí)例，這里不再贅述。
使用pandas處理excel有更多的pandas處理excel的資料，深入學(xué)習(xí)可以參考。
- 討論qq群144081101 591302926 567351477 釘釘免費(fèi)群21745728
- 本文涉及的python測(cè)試開(kāi)發(fā)庫(kù) 請(qǐng)?jiān)趃ithub上點(diǎn)贊，謝謝！
- 本文相關(guān)書(shū)籍下載
- 源碼下載
- 本文英文版書(shū)籍下載
