比 requests 更強(qiáng)大 Python 庫,讓你的爬蟲效率提高一倍!
來源:網(wǎng)絡(luò)

什么是協(xié)程?
協(xié)程比多線程好在哪呢?
協(xié)程的適用 & 不適用場(chǎng)景
初探異步 http 框架 httpx
什么是 httpx
安裝
pip install httpx
最佳實(shí)踐
import asyncioimport httpximport threadingimport timedef sync_main(url, sign):response = httpx.get(url).status_codeprint(f'sync_main: {threading.current_thread()}: {sign}2 + 1{response}')sync_start = time.time()[sync_main(url='http://www.baidu.com', sign=i) for i in range(200)]sync_end = time.time()print(sync_end - sync_start)
sync_main: <_MainThread(MainThread, started 4471512512)>: 192: 200sync_main: <_MainThread(MainThread, started 4471512512)>: 193: 200sync_main: <_MainThread(MainThread, started 4471512512)>: 194: 200sync_main: <_MainThread(MainThread, started 4471512512)>: 195: 200sync_main: <_MainThread(MainThread, started 4471512512)>: 196: 200sync_main: <_MainThread(MainThread, started 4471512512)>: 197: 200sync_main: <_MainThread(MainThread, started 4471512512)>: 198: 200sync_main: <_MainThread(MainThread, started 4471512512)>: 199: 20016.56578803062439
import asyncioimport httpximport threadingimport timeclient = httpx.AsyncClient()async def async_main(url, sign):response = await client.get(url)status_code = response.status_codeprint(f'async_main: {threading.current_thread()}: {sign}:{status_code}')loop = asyncio.get_event_loop()tasks = [async_main(url='http://www.baidu.com', sign=i) for i in range(200)]async_start = time.time()loop.run_until_complete(asyncio.wait(tasks))async_end = time.time()loop.close()print(async_end - async_start)
async_main: <_MainThread(MainThread, started 4471512512)>: 56: 200async_main: <_MainThread(MainThread, started 4471512512)>: 99: 200async_main: <_MainThread(MainThread, started 4471512512)>: 67: 200async_main: <_MainThread(MainThread, started 4471512512)>: 93: 200async_main: <_MainThread(MainThread, started 4471512512)>: 125: 200async_main: <_MainThread(MainThread, started 4471512512)>: 193: 200async_main: <_MainThread(MainThread, started 4471512512)>: 100: 2004.518340110778809

加入知識(shí)星球【我們談?wù)摂?shù)據(jù)科學(xué)】
400+小伙伴一起學(xué)習(xí)!
· 推薦閱讀 ·
Python中JSON結(jié)構(gòu)數(shù)據(jù)的高效增刪改操作
在Python中操縱json數(shù)據(jù)的最佳方式
Python Dash 一個(gè)可以玩轉(zhuǎn)AI的可視化利器
評(píng)論
圖片
表情
