uwsgi 關(guān)閉/重啟 夯死問(wèn)題

背景

近期使用uwsgi啟動(dòng)django服務(wù)，發(fā)現(xiàn)在stop/reload uwsgi時(shí)會(huì)出現(xiàn)hangs問(wèn)題，具體日志表現(xiàn)為：

...gracefully killing workers...
Gracefully killing worker 2 (pid: 390129)...
Gracefully killing worker 1 (pid: 390128)...
Gracefully killing worker 4 (pid: 390131)...
Gracefully killing worker 3 (pid: 390130)...
Wed Jul 31 20:13:14 2019 - worker 1 (pid: 390128) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 2 (pid: 390129) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 3 (pid: 390130) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 4 (pid: 390131) is taking too much time to die...NO MERCY !!!
...

在測(cè)試中發(fā)現(xiàn)無(wú)論用什么防線先uwsgi發(fā)送stop/reload信號(hào)，uwsgi都會(huì)進(jìn)入一種假死狀態(tài)，即不接收請(qǐng)求，也不結(jié)束進(jìn)程。

期初以為uwsgi進(jìn)程是在處理未完成的web請(qǐng)求。后來(lái)發(fā)現(xiàn)在沒有任何請(qǐng)求時(shí)，uwsgi也會(huì)進(jìn)入這種夯死狀態(tài)。

問(wèn)題原因

首先說(shuō)明問(wèn)題的原因是因何而起。

出現(xiàn)這種夯死的問(wèn)題是由于在uwsgi中使用了線程導(dǎo)致。

Destroying threads in the POSIX world is basically no-way :) The right approach is letting your thread return when the destroy procedure is triggered. You have a bunch of ways to do it, but personally i am way more brutal and i generally set --worker-reload-mercy to a couple of seconds :)

在POSIX中銷毀一個(gè)線程幾乎是不可能實(shí)現(xiàn)的工作。正確的做法是：當(dāng)接收到銷毀信號(hào)后正確的讓線程返回。通常有很多方式來(lái)實(shí)現(xiàn)著一點(diǎn)，但是更推薦的方式是設(shè)置一個(gè)--worker-reload-mercy時(shí)間。

解決方案

  • 盡量避免在uwsgi中使用線程，首先使用線程會(huì)產(chǎn)生GIL鎖問(wèn)題，其次在使用線程時(shí)還要考慮如何讓其正確退出。
  • 設(shè)置reload-mercy和worker-reload-mercy兩個(gè)參數(shù)。

--reload-mercy - used when reloading/stopping whole uWSGI instance
--worker-reload-mercy - used when reloading/stopping single worker

問(wèn)題復(fù)現(xiàn)

這里創(chuàng)建了一個(gè)最簡(jiǎn)單的django服務(wù)，并用uwsgi來(lái)啟動(dòng)。

進(jìn)程模式

首先，配置uwsgi為進(jìn)程模式啟動(dòng)，這里創(chuàng)建了5個(gè)進(jìn)程。

uwsgi配置文件如下：

[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
processes = 5
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log

現(xiàn)在來(lái)reload uwsgi服務(wù)，并查看uwsgi的日志。

...gracefully killing workers...
Gracefully killing worker 1 (pid: 13055)...
Gracefully killing worker 2 (pid: 13056)...
Gracefully killing worker 3 (pid: 13057)...
Gracefully killing worker 4 (pid: 13058)...
Gracefully killing worker 5 (pid: 13059)...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......

結(jié)論：在uwsgi使用進(jìn)程模式時(shí)，reload uwsgi不會(huì)出現(xiàn)夯死的問(wèn)題。

線程模式

前面以進(jìn)程方式啟動(dòng)uwsgi沒有出現(xiàn)夯死問(wèn)題，那么現(xiàn)在就試一下以線程模式啟動(dòng)wusgi。這里啟動(dòng)了5個(gè)進(jìn)程，每個(gè)進(jìn)程中又包含了兩個(gè)線程。

uwsgi配置文件如下：

[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
enable-threads = True
processes = 5
threads = 2
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log

reload uwsgi服務(wù)，并觀察日志輸出。

...gracefully killing workers...
Gracefully killing worker 1 (pid: 14913)...
Gracefully killing worker 2 (pid: 14914)...
Gracefully killing worker 3 (pid: 14915)...
Gracefully killing worker 4 (pid: 14916)...
Gracefully killing worker 5 (pid: 14917)...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......

結(jié)論：在uwsgi中使用線程模式也不會(huì)造成reload夯死的問(wèn)題。

uwsgi app中使用線程

uwsgi的線程不會(huì)造成任何問(wèn)題，那前文所指的線程究竟是什么？

現(xiàn)在uwsgi啟動(dòng)腳本中創(chuàng)建一個(gè)線程，在這種情況下嘗試reload uwsgi并查看日志輸出。

"""
WSGI config for laji_baclend project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/
"""

import os
import time
import threading

from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'laji_baclend.settings')

def loop():
    while True:
        time.sleep(10)

t = threading.Thread(target=loop, name="loop_thread")
t.start()

application = get_wsgi_application()

隨后以線程方式啟動(dòng)uwsgi，并reload uwsgi。

...gracefully killing workers...
Gracefully killing worker 1 (pid: 24390)...
Gracefully killing worker 2 (pid: 24391)...
Gracefully killing worker 3 (pid: 24394)...
Gracefully killing worker 4 (pid: 24396)...
Gracefully killing worker 5 (pid: 24398)...
Thu Aug  1 16:44:48 2019 - worker 1 (pid: 24390) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:44:48 2019 - worker 2 (pid: 24391) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:44:48 2019 - worker 3 (pid: 24394) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:44:48 2019 - worker 4 (pid: 24396) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:44:48 2019 - worker 5 (pid: 24398) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
......

結(jié)論：在uwsgi app中使用線程就導(dǎo)致reload夯死。

采用reload-mercy和worker-reload-mercy避免

若場(chǎng)景中非要在uwsgi app中使用線程，可以通過(guò)配置reload-mercy和worker-reload-mercy兩個(gè)參數(shù)避免夯死的問(wèn)題。

uwsgi配置文件如下：

[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
enable-threads = True
processes = 5
threads = 2
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log
reload-mercy = 1
worker-reload-mercy = 1

再次reload uwsgi服務(wù)，輸出日志如下：

...gracefully killing workers...
Gracefully killing worker 1 (pid: 24766)...
Gracefully killing worker 2 (pid: 24767)...
Gracefully killing worker 3 (pid: 24768)...
Gracefully killing worker 4 (pid: 24769)...
Gracefully killing worker 5 (pid: 24770)...
Thu Aug  1 16:50:13 2019 - worker 1 (pid: 24766) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:50:13 2019 - worker 2 (pid: 24767) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:50:13 2019 - worker 3 (pid: 24768) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:50:13 2019 - worker 4 (pid: 24769) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:50:13 2019 - worker 5 (pid: 24770) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......

這里雖然也會(huì)出現(xiàn)NO MERCY問(wèn)題，但是uwsgi在reload過(guò)程中并沒有出現(xiàn)夯死的情況。

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容