一個(gè)Python爬蟲案例，帶你掌握xpath數(shù)據(jù)解析方法！
xpath基本概念
xpath解析原理
環(huán)境安裝
如何實(shí)例化一個(gè)etree對(duì)象：
xpath(‘xpath表達(dá)式’)
xpath爬取58二手房實(shí)例
爬取網(wǎng)址
完整代碼
效果圖
xpath圖片解析下載實(shí)例
爬取網(wǎng)址
完整代碼
效果圖
xpath爬取全國(guó)城市名稱實(shí)例
爬取網(wǎng)址
完整代碼
效果圖
xpath爬取簡(jiǎn)歷模板實(shí)例
爬取網(wǎng)址
完整代碼
效果圖
xpath基本概念
xpath解析：最常用且最便捷高效的一種解析方式。通用性強(qiáng)。
xpath解析原理
1.實(shí)例化一個(gè)etree的對(duì)象，且需要將被解析的頁(yè)面源碼數(shù)據(jù)加載到該對(duì)象中
2.調(diào)用etree對(duì)象中的xpath方法結(jié)合xpath表達(dá)式實(shí)現(xiàn)標(biāo)簽的定位和內(nèi)容的捕獲。
環(huán)境安裝
pip?install?lxml
如何實(shí)例化一個(gè)etree對(duì)象：
from?lxml?import?etree
1.將本地的html文件中的遠(yuǎn)嗎數(shù)據(jù)加載到etree對(duì)象中：
etree.parse(filePath)
2.可以將從互聯(lián)網(wǎng)上獲取的原碼數(shù)據(jù)加載到該對(duì)象中：
etree.HTML(‘page_text’)
xpath(‘xpath表達(dá)式’)
/:表示的是從根節(jié)點(diǎn)開始定位。表示一個(gè)層級(jí) //:表示多個(gè)層級(jí)?？梢员硎緩娜我馕恢瞄_始定位 屬性定位：//div[@class='song'] tag[@attrName='attrValue'] 索引定位：//div[@class='song']/p[3] 索引從1開始的 取文本： /text()獲取的是標(biāo)簽中直系的文本內(nèi)容 //text()標(biāo)簽中非直系的文本內(nèi)容（所有文本內(nèi)容） 取屬性：/@attrName ==>img/src
xpath爬取58二手房實(shí)例
爬取網(wǎng)址
https://xa.58.com/ershoufang/完整代碼
from?lxml?import?etree
import?requests
if?__name__?==?'__main__':
????headers?=?{
????????'User-Agent':?'Mozilla/5.0?(Windows?NT?10.0;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/84.0.4147.105?Safari/537.36'
????}
????url?=?'https://xa.58.com/ershoufang/'
????page_text?=?requests.get(url=url,headers=headers).text
????tree?=?etree.HTML(page_text)
????div_list?=?tree.xpath('//section[@class="list"]/div')
????fp?=?open('./58同城二手房.txt','w',encoding='utf-8')
????for?div?in?div_list:
????????title?=?div.xpath('.//div[@class="property-content-title"]/h3/text()')[0]
????????print(title)
????????fp.write(title+'\n'+'\n')
xpath圖片解析下載實(shí)例
爬取網(wǎng)址
https://pic.netbian.com/4kmeinv/完整代碼
import?requests,os
from?lxml?import?etree
if?__name__?==?'__main__':
????headers?=?{
????????'User-Agent':?'Mozilla/5.0?(Windows?NT?10.0;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/84.0.4147.105?Safari/537.36'
????}
????url?=?'https://pic.netbian.com/4kmeinv/'
????page_text?=?requests.get(url=url,headers=headers).text
????tree?=?etree.HTML(page_text)
????li_list?=?tree.xpath('//div[@class="slist"]/ul/li/a')
????if?not?os.path.exists('./piclibs'):
????????os.mkdir('./piclibs')
????for?li?in?li_list:
????????detail_url?='https://pic.netbian.com'?+?li.xpath('./img/@src')[0]
????????detail_name?=?li.xpath('./img/@alt')[0]+'.jpg'
????????detail_name?=?detail_name.encode('iso-8859-1').decode('GBK')
????????detail_path?=?'./piclibs/'?+?detail_name
????????detail_data?=?requests.get(url=detail_url,?headers=headers).content
????????with?open(detail_path,'wb')?as?fp:
????????????fp.write(detail_data)
????????????print(detail_name,'seccess!!')
xpath爬取全國(guó)城市名稱實(shí)例
爬取網(wǎng)址?
https://www.aqistudy.cn/historydata/完整代碼
import?requests
from?lxml?import?etree
if?__name__?==?'__main__':
????url?=?'https://www.aqistudy.cn/historydata/'
????headers?=?{
????????'User-Agent':?'Mozilla/5.0?(Windows?NT?10.0;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/87.0.4280.141?Safari/537.36',
????}
????page_text?=?requests.get(url=url,headers=headers).content.decode('utf-8')
????tree?=?etree.HTML(page_text)
????#熱門城市???//div[@class="bottom"]/ul/li
????#全部城市???//div[@class="bottom"]/ul/div[2]/li
????a_list?=?tree.xpath('//div[@class="bottom"]/ul/li?|?//div[@class="bottom"]/ul/div[2]/li')
????fp?=?open('./citys.txt','w',encoding='utf-8')
????i?=?0
????for?a?in?a_list:
????????city_name?=?a.xpath('.//a/text()')[0]
????????fp.write(city_name+'\t')
????????i=i+1
????????if?i?==?6:
????????????i?=?0
????????????fp.write('\n')
????print('爬取成功')
xpath爬取簡(jiǎn)歷模板實(shí)例
爬取網(wǎng)址
https://sc.chinaz.com/jianli/free.html完整代碼
import?requests,os
from?lxml?import?etree
if?__name__?==?'__main__':
????url?=?'https://sc.chinaz.com/jianli/free.html'
????headers?=?{
????????'User-Agent':?'Mozilla/5.0?(Windows?NT?10.0;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/87.0.4280.141?Safari/537.36',
????}
????page_text?=?requests.get(url=url,headers=headers).content.decode('utf-8')
????tree?=?etree.HTML(page_text)
????a_list?=?tree.xpath('//div[@class="box?col3?ws_block"]/a')
????if?not?os.path.exists('./簡(jiǎn)歷模板'):
????????os.mkdir('./簡(jiǎn)歷模板')
????for?a?in?a_list:
????????detail_url?=?'https:'+a.xpath('./@href')[0]
????????detail_page_text?=?requests.get(url=detail_url,headers=headers).content.decode('utf-8')
????????detail_tree?=?etree.HTML(detail_page_text)
????????detail_a_list?=?detail_tree.xpath('//div[@class="clearfix?mt20?downlist"]/ul/li[1]/a')
????????for?a?in?detail_a_list:
????????????download_name?=?detail_tree.xpath('//div[@class="ppt_tit?clearfix"]/h1/text()')[0]
????????????download_url?=?a.xpath('./@href')[0]
????????????download_data?=?requests.get(url=download_url,headers=headers).content
????????????download_path?=?'./簡(jiǎn)歷模板/'+download_name+'.rar'
????????????with?open(download_path,'wb')?as?fp:
????????????????fp.write(download_data)
????????????????print(download_name,'success!!')
評(píng)論
圖片
表情