一年一度的雙十一即將來臨，臨時(shí)接到了一個(gè)任務(wù)：統(tǒng)計(jì)某品牌數(shù)據(jù)銀行中自己品牌分別在2017和2018的10月20日至10月31日之間不同時(shí)間段的AIPL（“認(rèn)知”(Aware)、“興趣”(Interest)、“購買”(Purchase)、“忠誠”(Loyalty)）流轉(zhuǎn)率。

使用Fiddler獲取到目標(biāo)地址為：

https://databank.yushanfang.com/api/ecapi?path=/databank/crowdFullLink/flowInfo&fromCrowdId=3312&beginTheDate=20181020&endTheDate=20181031&toCrowdIdList[0]=3312&toCrowdIdList[1]=3313&toCrowdIdList[2]=3314&toCrowdIdList[3]=3315

本文中以爬取其中的AI流轉(zhuǎn)率數(shù)據(jù)為例。

該地址返回的響應(yīng)內(nèi)容為Json類型，其中紅框標(biāo)記的項(xiàng)即為AI流轉(zhuǎn)率值：

實(shí)現(xiàn)代碼如下：

import requestsimport jsonimport csv # 爬蟲地址url = ’https://databank.yushanfang.com/api/ecapi?path=/databank/crowdFullLink/flowInfo&fromCrowdId=3312&beginTheDate=201810{}&endTheDate=201810{}&toCrowdIdList[0]=3312&toCrowdIdList[1]=3313&toCrowdIdList[2]=3314&toCrowdIdList[3]=3315’ # 攜帶cookie進(jìn)行訪問headers = {’Host’:’databank.yushanfang.com’,’Referer’:’https://databank.yushanfang.com/’,’Connection’:’keep-alive’,’User-Agent’:’Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36’,’Cookie’:’_tb_token_=iNkDeJLdM3MgvKjhsfdW; bs_n_lang=zh_CN; cna=aaj1EViI7x0CATo9kTKvjzgS; ck2=072de851f1c02d5c7bac555f64c5c66d; c_token=c74594b486f8de731e2608cb9526a3f2; an=5YWo5qOJ5pe25Luj5a6Y5pa55peX6Iiw5bqXOnpmeA%3D%3D; lg=true; sg='=19'; lvc=sAhojs49PcqHQQ%3D%3D; isg=BPT0Md7dE_ic5Ie3Oa85RxaMxbLK3UqJMMiN6o5VjH8C-ZRDtt7aRXb3fXGEAVAP’,} rows = []for n in range(20, 31): row = [] row.append(n) for m in range (21, 32): if m < n + 1: row.append('') else: # 格式化請求地址，更換請求參數(shù) reqUrl = url.format(n, m) # 打印本次請求地址 print(url) # 發(fā)送請求，獲取響應(yīng)結(jié)果 response = requests.get(url=reqUrl, headers=headers, verify=False) text = response.text # 打印本次請求響應(yīng)內(nèi)容 print(text) # 將響應(yīng)內(nèi)容轉(zhuǎn)換為Json對象 jsonobj = json.loads(text) # 從Json對象獲取想要的內(nèi)容 toCntPercent = jsonobj[’data’][’interCrowdInfo’][1][’toCntPercent’] # 生成行數(shù)據(jù) row.append(str(toCntPercent)+'%')# 保存行數(shù)據(jù) rows.append(row) # 生成Excel表頭header = [’AI流轉(zhuǎn)率’, ’21’, ’22’, ’23’, ’24’, ’25’, ’26’, ’27’, ’28’, ’29’, ’30’, ’31’] # 將表頭數(shù)據(jù)和爬蟲數(shù)據(jù)導(dǎo)出到Excel文件with open(’D:respachongtmall.csv’, ’w’, encoding=’gb18030’) as f : f_csv = csv.writer(f) f_csv.writerow(header) f_csv.writerows(rows)

import csvimport jsonimport sslimport urllib.request # 爬蟲地址url = ’https://databank.yushanfang.com/api/ecapi?path=/databank/crowdFullLink/flowInfo&fromCrowdId=3312&beginTheDate=201810{}&endTheDate=201810{}&toCrowdIdList[0]=3312&toCrowdIdList[1]=3313&toCrowdIdList[2]=3314&toCrowdIdList[3]=3315’ # 不校驗(yàn)證書ssl._create_default_https_context = ssl._create_unverified_context # 攜帶cookie進(jìn)行訪問headers = {’Host’:’databank.yushanfang.com’,’Referer’:’https://databank.yushanfang.com/’,’Connection’:’keep-alive’,’User-Agent’:’Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36’,’Cookie’:’_tb_token_=iNkDeJLdM3MgvKjhsfdW; bs_n_lang=zh_CN; cna=aaj1EViI7x0CATo9kTKvjzgS; ck2=072de851f1c02d5c7bac555f64c5c66d; c_token=c74594b486f8de731e2608cb9526a3f2; an=5YWo5qOJ5pe25Luj5a6Y5pa55peX6Iiw5bqXOnpmeA%3D%3D; lg=true; sg='=19'; lvc=sAhojs49PcqHQQ%3D%3D; isg=BPT0Md7dE_ic5Ie3Oa85RxaMxbLK3UqJMMiN6o5VjH8C-ZRDtt7aRXb3fXGEAVAP’,} rows = []n = 20while n <31: row = [] row.append(n) m =21 while m <32:if m < n + 1: row.append('') else: # 格式化請求地址，更換請求參數(shù) reqUrl = url.format(n, m) # 打印本次請求地址 print(reqUrl) # 發(fā)送請求，獲取響應(yīng)結(jié)果 request = urllib.request.Request(url=reqUrl, headers=headers) response = urllib.request.urlopen(request) text = response.read().decode(’utf8’) # 打印本次請求響應(yīng)內(nèi)容 print(text) # 將響應(yīng)內(nèi)容轉(zhuǎn)換為Json對象 jsonobj = json.loads(text) # 從Json對象獲取想要的內(nèi)容 toCntPercent = jsonobj[’data’][’interCrowdInfo’][1][’toCntPercent’] # 生成行數(shù)據(jù) row.append(str(toCntPercent) + '%') m = m+1 rows.append(row) n = n+1 # 生成Excel表頭header = [’AI流轉(zhuǎn)率’, ’21’, ’22’, ’23’, ’24’, ’25’, ’26’, ’27’, ’28’, ’29’, ’30’, ’31’] # 將表頭數(shù)據(jù)和爬蟲數(shù)據(jù)導(dǎo)出到Excel文件with open(’D:respachongtmall.csv’, ’w’, encoding=’gb18030’) as f : f_csv = csv.writer(f) f_csv.writerow(header) f_csv.writerows(rows)

導(dǎo)出內(nèi)容如下：

到此這篇關(guān)于使用Python爬取Json數(shù)據(jù)的文章就介紹到這了,更多相關(guān)Python爬取Json數(shù)據(jù)內(nèi)容請搜索好吧啦網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持好吧啦網(wǎng)！