關(guān)于io 密集型 和 cpu密集型
https://blog.csdn.net/youanyyou/article/details/78990156
協(xié)程:Coroutine
一句話說(shuō)就是一種用戶態(tài)的輕量級(jí)線程。實(shí)現(xiàn)了單線程下并發(fā)的效果。
優(yōu)點(diǎn):
1、無(wú)需線程上下文的切換,還是單線程,只是函數(shù)之間的切換。
2、無(wú)需原子操作鎖定及同步的開(kāi)銷(xiāo)。因?yàn)閰f(xié)程就是單線程。不需要鎖,數(shù)據(jù)同步等的需要。
3、方便切換控制流,簡(jiǎn)化編程模型
4、高并發(fā)+高擴(kuò)展+低成本:一個(gè)cpu支持上萬(wàn)的協(xié)程都是沒(méi)有問(wèn)題,所以適合高并發(fā)處理。
缺點(diǎn):
1、無(wú)法利用多核資源,需要和進(jìn)程配合才能運(yùn)行在多cpu上。
2、協(xié)程指的是單個(gè)線程,因而一旦協(xié)程出現(xiàn)阻塞,將會(huì)阻塞整個(gè)線程
協(xié)程,又稱微線程,纖程。協(xié)程是一種用戶態(tài)的輕量級(jí)線程。
協(xié)程擁有自己的寄存器和棧。協(xié)程調(diào)度切換時(shí),將寄存器上下文和棧保存在其他地方,在切回來(lái)的時(shí)候,恢復(fù)先前保存的寄存器上下文和棧。
因此:協(xié)程能保留上一次調(diào)用時(shí)的狀態(tài)(即所有局部狀態(tài)的一個(gè)特定組合),每次過(guò)程重入時(shí),就相當(dāng)于進(jìn)入上一次調(diào)用的狀態(tài),換種說(shuō)法:進(jìn)入上一次離開(kāi)時(shí)所處邏輯流的位置
- 實(shí)例一:利用yield實(shí)現(xiàn)單線程下并發(fā)
import time
import queue
def consumer(name):
print("--->starting eating baozi...")
while True:
new_baozi = yield #
print("[%s] is eating baozi %s" % (name, new_baozi))
# time.sleep(1)
def producer():
r = con.__next__() #開(kāi)始執(zhí)行
r = con2.__next__()
n = 0
while n < 5:
n += 1
con.send(n) #喚醒生成器的同時(shí)并且傳值
con2.send(n)
time.sleep(1)
print("\033[32;1m[producer]\033[0m is making baozi %s" % n)
if __name__ == '__main__':
con = consumer("c1") #只是生成器,next執(zhí)行
con2 = consumer("c2")
producer()
上面這個(gè)程序是我們自己寫(xiě)的一個(gè)類(lèi)似于協(xié)程的例子,它沒(méi)有實(shí)現(xiàn)遇到io自動(dòng)切換。那么,怎么樣的程序才算協(xié)程呢?
1、必須在只有一個(gè)單線程里實(shí)現(xiàn)并發(fā)
2、修改共享數(shù)據(jù)不需要加鎖
3、用戶程序里自己保存多個(gè)控制流的上下棧
4、一個(gè)協(xié)程遇到io操作自動(dòng)切換到其他協(xié)程(那么整個(gè)程序就只有cpu在運(yùn)算)
- 實(shí)例二:greenlet實(shí)現(xiàn)遇到io手動(dòng)切換
from greenlet import greenlet
def test1():
print(12)
gr2.switch() #手動(dòng)切換
print(34)
gr2.switch()
def test2():
print(56)
gr1.switch()
print(78)
gr1 = greenlet(test1) #啟動(dòng)一個(gè)協(xié)程
gr2 = greenlet(test2)
gr1.switch() #先執(zhí)行g(shù)r1
F:\anaconda\python.exe F:/web/s14/進(jìn)程、線程/noke.py
12
56
34
78
- 實(shí)例三:gevent對(duì)greenlet封裝實(shí)現(xiàn)了自動(dòng)切換.
注意:整體是按2s運(yùn)行的程序,但是有50處io,49個(gè)1s,1個(gè)2s,就只需要2s。
import gevent
def func1():
print('\033[31;1m李闖在跟海濤搞...\033[0m')
gevent.sleep(2) #模擬io
print('\033[31;1m李闖又回去跟繼續(xù)跟海濤搞...\033[0m')
def func2():
print('\033[32;1m李闖切換到了跟海龍搞...\033[0m')
gevent.sleep(1) #停頓一秒
print('\033[32;1m李闖搞完了海濤,回來(lái)繼續(xù)跟海龍搞...\033[0m')
def func3():
print("running func3")
gevent.sleep(0)
print("again")
gevent.joinall([
gevent.spawn(func1), #生成一個(gè)協(xié)程
gevent.spawn(func2),
gevent.spawn(func3),
])
F:\anaconda\python.exe F:/web/s14/進(jìn)程、線程/noke.py
李闖在跟海濤搞...
李闖切換到了跟海龍搞...
running func3
again
李闖搞完了海濤,回來(lái)繼續(xù)跟海龍搞...
李闖又回去跟繼續(xù)跟海濤搞...
- 實(shí)例四:通過(guò)gevent實(shí)現(xiàn)協(xié)程并發(fā)爬取網(wǎng)頁(yè)
from urllib import request
import gevent,time
from gevent import monkey
monkey.patch_all() #把當(dāng)前程序的所有的io操作給我單獨(dú)的做上標(biāo)記 打補(bǔ)丁,因?yàn)間event不知道urllib進(jìn)行了io操作
#相當(dāng)于sleep一下,阻塞,
def f(url):
print('GET: %s' % url)
resp = request.urlopen(url)
data = resp.read()
print('%d bytes received from %s.' % (len(data), url))
urls = [
'https://edu.hellobi.com/course/explore?c2=37&c3=26',
'https://www.baidu.com',
'https://www.cnblogs.com/alex3714/articles/5248247.html'
]
time_start = time.time()
for url in urls:
f(url)
print("同步cost",time.time() - time_start)
async_time_start = time.time() #async是異步的縮寫(xiě)
gevent.joinall([
gevent.spawn(f,'https://edu.hellobi.com/course/explore?c2=37&c3=26' ),
gevent.spawn(f,'https://www.baidu.com' ),
gevent.spawn(f, 'https://www.cnblogs.com/alex3714/articles/5248247.html'),
])
print("異步cost",time.time() - async_time_start)
F:\anaconda\python.exe F:/web/s14/進(jìn)程、線程/noke.py
GET: https://edu.hellobi.com/course/explore?c2=37&c3=26
76534 bytes received from https://edu.hellobi.com/course/explore?c2=37&c3=26.
GET: https://www.baidu.com
227 bytes received from https://www.baidu.com.
GET: https://www.cnblogs.com/alex3714/articles/5248247.html
93323 bytes received from https://www.cnblogs.com/alex3714/articles/5248247.html.
同步cost 1.8301048278808594
GET: https://edu.hellobi.com/course/explore?c2=37&c3=26
GET: https://www.baidu.com
GET: https://www.cnblogs.com/alex3714/articles/5248247.html
227 bytes received from https://www.baidu.com.
93323 bytes received from https://www.cnblogs.com/alex3714/articles/5248247.html.
76534 bytes received from https://edu.hellobi.com/course/explore?c2=37&c3=26.
異步cost 0.6120350360870361
- 實(shí)例五:通過(guò)gevent實(shí)現(xiàn)單線程下多socket并發(fā)
服務(wù)端
import sys
import socket
import time
import gevent
from gevent import socket, monkey
monkey.patch_all()
def server(port):
s = socket.socket()
s.bind(('0.0.0.0', port))
s.listen(500)
while True:
cli, addr = s.accept()
gevent.spawn(handle_request, cli)
def handle_request(conn):
try:
while True:
data = conn.recv(1024)
print("recv:", data)
conn.send(data)
if not data:
conn.shutdown(socket.SHUT_WR)
except Exception as ex:
print(ex)
finally:
conn.close()
if __name__ == '__main__':
server(8001)
客戶端
import socket
HOST = 'localhost' # The remote host
PORT = 8001 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
while True:
msg = bytes(input(">>:"), encoding="utf8")
s.sendall(msg)
data = s.recv(1024)
# print(data)
print('Received', repr(data)) #repr格式化輸出
s.close()