pyspider代碼結(jié)構(gòu)
主要從以下幾個(gè)模塊看pyspider源碼
- libs里面的工具類。比如最常用的basehandler 等等。
- process，scheduler，resultdb。其中最容易看得是resultdb
- webui部分，web如何調(diào)用數(shù)據(jù)庫的腳本等
debug的一些技巧
pyspider在run.py啟動(dòng)了多個(gè)線程， 多個(gè)線程的debug其實(shí)很麻煩，我們需要根據(jù)上一個(gè)欄目的代碼結(jié)構(gòu)來一個(gè)個(gè)的看。
比如最簡(jiǎn)單的resultdb模塊，只有一個(gè)result_worker.py文件，看懂了這個(gè)文件，其實(shí)就看懂了這個(gè)模塊。
class ResultWorker(object):
"""
do with result
override this if needed.
"""
def __init__(self, resultdb, inqueue):
self.resultdb = resultdb
self.inqueue = inqueue
self._quit = False
def on_result(self, task, result):
'''Called every result'''
if not result:
return
if 'taskid' in task and 'project' in task and 'url' in task:
logger.info('result %s:%s %s -> %.30r' % (
task['project'], task['taskid'], task['url'], result))
return self.resultdb.save(
project=task['project'],
taskid=task['taskid'],
url=task['url'],
result=result
)
else:
logger.warning('result UNKNOW -> %.30r' % result)
return
def quit(self):
self._quit = True
def run(self):
'''Run loop'''
logger.info("result_worker starting...")
while not self._quit:
try:
task, result = self.inqueue.get(timeout=1)
self.on_result(task, result)
except Queue.Empty as e:
continue
except KeyboardInterrupt:
break
except AssertionError as e:
logger.error(e)
continue
except Exception as e:
logger.exception(e)
continue
logger.info("result_worker exiting...")
class OneResultWorker(ResultWorker):
'''Result Worker for one mode, write results to stdout'''
def on_result(self, task, result):
'''Called every result'''
if not result:
return
if 'taskid' in task and 'project' in task and 'url' in task:
logger.info('result %s:%s %s -> %.30r' % (
task['project'], task['taskid'], task['url'], result))
print(json.dumps({
'taskid': task['taskid'],
'project': task['project'],
'url': task['url'],
'result': result,
'updatetime': time.time()
}))
else:
logger.warning('result UNKNOW -> %.30r' % result)
return
OneResultWorker不需要看，功能上，這個(gè)模塊和上面實(shí)現(xiàn)一樣的功能，應(yīng)該是用來單獨(dú)啟動(dòng)的時(shí)候調(diào)用的代碼。同樣的道理，很多one開頭的代碼都可以掠過。
