相關(guān)：urllib是python內(nèi)置的http請(qǐng)求庫(kù)，本文介紹urllib三個(gè)模塊：請(qǐng)求模塊urllib.request、異常處理模塊urllib.error、url解析模塊urllib.parse。

1、請(qǐng)求模塊：urllib.request

python2

import urllib2response = urllib2.urlopen(’http://httpbin.org/robots.txt’)

python3

import urllib.requestres = urllib.request.urlopen(’http://httpbin.org/robots.txt’)urllib.request.urlopen(url, data=None, [timeout, ]*, cafile=None, capath=None, cadefault=False, context=None)urlopen()方法中的url參數(shù)可以是字符串，也可以是一個(gè)Request對(duì)象

#url可以是字符串import urllib.requestresp = urllib.request.urlopen(’http://www.baidu.com’)print(resp.read().decode(’utf-8’)) # read()獲取響應(yīng)體的內(nèi)容，內(nèi)容是bytes字節(jié)流，需要轉(zhuǎn)換成字符串

##url可以也是Request對(duì)象import urllib.requestrequest = urllib.request.Request(’http://httpbin.org’)response = urllib.request.urlopen(request)print(response.read().decode(’utf-8’))data參數(shù)：post請(qǐng)求

# coding:utf8import urllib.request, urllib.parsedata = bytes(urllib.parse.urlencode({’word’: ’hello’}), encoding=’utf8’)resp = urllib.request.urlopen(’http://httpbin.org/post’, data=data)print(resp.read())urlopen()中的參數(shù)timeout：設(shè)置請(qǐng)求超時(shí)時(shí)間：

# coding:utf8#設(shè)置請(qǐng)求超時(shí)時(shí)間import urllib.requestresp = urllib.request.urlopen(’http://httpbin.org/get’, timeout=0.1)print(resp.read().decode(’utf-8’))響應(yīng)類型：

# coding:utf8#響應(yīng)類型import urllib.requestresp = urllib.request.urlopen(’http://httpbin.org/get’)print(type(resp))

響應(yīng)的狀態(tài)碼、響應(yīng)頭：

# coding:utf8#響應(yīng)的狀態(tài)碼、響應(yīng)頭import urllib.requestresp = urllib.request.urlopen(’http://www.baidu.com’)print(resp.status)print(resp.getheaders()) # 數(shù)組（元組列表）print(resp.getheader(’Server’)) # 'Server'大小寫不區(qū)分

200[(’Bdpagetype’, ’1’), (’Bdqid’, ’0xa6d873bb003836ce’), (’Cache-Control’, ’private’), (’Content-Type’, ’text/html’), (’Cxy_all’, ’baidu+b8704ff7c06fb8466a83df26d7f0ad23’), (’Date’, ’Sun, 21 Apr 2019 15:18:24 GMT’), (’Expires’, ’Sun, 21 Apr 2019 15:18:03 GMT’), (’P3p’, ’CP=' OTI DSP COR IVA OUR IND COM '’), (’Server’, ’BWS/1.1’), (’Set-Cookie’, ’BAIDUID=8C61C3A67C1281B5952199E456EEC61E:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com’), (’Set-Cookie’, ’BIDUPSID=8C61C3A67C1281B5952199E456EEC61E; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com’), (’Set-Cookie’, ’PSTM=1555859904; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com’), (’Set-Cookie’, ’delPer=0; path=/; domain=.baidu.com’), (’Set-Cookie’, ’BDSVRTM=0; path=/’), (’Set-Cookie’, ’BD_HOME=0; path=/’), (’Set-Cookie’, ’H_PS_PSSID=1452_28777_21078_28775_28722_28557_28838_28584_28604; path=/; domain=.baidu.com’), (’Vary’, ’Accept-Encoding’), (’X-Ua-Compatible’, ’IE=Edge,chrome=1’), (’Connection’, ’close’), (’Transfer-Encoding’, ’chunked’)]BWS/1.1

使用代理：urllib.request.ProxyHandler()：

# coding:utf8proxy_handler = urllib.request.ProxyHandler({’http’: ’http://www.example.com:3128/’})proxy_auth_handler = urllib.request.ProxyBasicAuthHandler()proxy_auth_handler.add_password(’realm’, ’host’, ’username’, ’password’)opener = urllib.request.build_opener(proxy_handler, proxy_auth_handler)# This time, rather than install the OpenerDirector, we use it directly:resp = opener.open(’http://www.example.com/login.html’)print(resp.read())2、異常處理模塊：urllib.error異常處理實(shí)例1：

# coding:utf8from urllib import error, requesttry: resp = request.urlopen(’http://www.blueflags.cn’)except error.URLError as e: print(e.reason)

異常處理實(shí)例2：

# coding:utf8from urllib import error, requesttry: resp = request.urlopen(’http://www.baidu.com’)except error.HTTPError as e: print(e.reason, e.code, e.headers, sep=’n’)except error.URLError as e: print(e.reason)else: print(’request successfully’)

異常處理實(shí)例3：

# coding:utf8import socket, urllib.request, urllib.errortry: resp = urllib.request.urlopen(’http://www.baidu.com’, timeout=0.01)except urllib.error.URLError as e: print(type(e.reason)) if isinstance(e.reason,socket.timeout):print(’time out’)

3、url解析模塊：urllib.parseparse.urlencode

# coding:utf8from urllib import request, parseurl = ’http://httpbin.org/post’headers = { ’Host’: ’httpbin.org’, ’User-Agent’: ’Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36’}dict = {’name’: ’Germey’}data = bytes(parse.urlencode(dict), encoding=’utf8’)req = request.Request(url=url, data=data, headers=headers, method=’POST’)resp = request.urlopen(req)print(resp.read().decode(’utf-8’))

{'args': {},'data': '','files': {},'form': {'name': 'Thanlon'},'headers': {'Accept-Encoding': 'identity','Content-Length': '12','Content-Type': 'application/x-www-form-urlencoded','Host': 'httpbin.org','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36'},'json': null,'origin': '117.136.78.194, 117.136.78.194','url': 'https://httpbin.org/post'}add_header方法添加請(qǐng)求頭：

# coding:utf8from urllib import request, parseurl = ’http://httpbin.org/post’dict = {’name’: ’Thanlon’}data = bytes(parse.urlencode(dict), encoding=’utf8’)req = request.Request(url=url, data=data, method=’POST’)req.add_header(’User-Agent’, ’Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36’)resp = request.urlopen(req)print(resp.read().decode(’utf-8’))parse.urlparse：

# coding:utf8from urllib.parse import urlparseresult = urlparse(’http://www.baidu.com/index.html;user?id=1#comment’)print(type(result))print(result)

<class ’urllib.parse.ParseResult’>ParseResult(scheme=’http’, netloc=’www.baidu.com’, path=’/index.html’, params=’user’, query=’id=1’, fragment=’comment’)

from urllib.parse import urlparseresult = urlparse(’www.baidu.com/index.html;user?id=1#comment’, scheme=’https’)print(type(result))print(result)

<class ’urllib.parse.ParseResult’>ParseResult(scheme=’https’, netloc=’’, path=’www.baidu.com/index.html’, params=’user’, query=’id=1’, fragment=’comment’)

# coding:utf8from urllib.parse import urlparseresult = urlparse(’http://www.baidu.com/index.html;user?id=1#comment’, scheme=’https’)print(result)

ParseResult(scheme=’http’, netloc=’www.baidu.com’, path=’/index.html’, params=’user’, query=’id=1’, fragment=’comment’)

# coding:utf8from urllib.parse import urlparseresult = urlparse(’http://www.baidu.com/index.html;user?id=1#comment’,allow_fragments=False)print(result)

ParseResult(scheme=’http’, netloc=’www.baidu.com’, path=’/index.html’, params=’user’, query=’id=1’, fragment=’comment’)

parse.urlunparse：

# coding:utf8from urllib.parse import urlunparsedata = [’http’, ’www.baidu.com’, ’index.html’, ’user’, ’name=Thanlon’, ’comment’]print(urlunparse(data))

parse.urljoin：

# coding:utf8from urllib.parse import urljoinprint(urljoin(’http://www.bai.com’, ’index.html’))print(urljoin(’http://www.baicu.com’, ’https://www.thanlon.cn/index.html’))#以后面為基準(zhǔn)

urlencode將字典對(duì)象轉(zhuǎn)換成get請(qǐng)求的參數(shù):

# coding:utf8from urllib.parse import urlencodeparams = { ’name’: ’Thanlon’, ’age’: 22}baseUrl = ’http://www.thanlon.cn?’url = baseUrl + urlencode(params)print(url)

4、Cookiecookie的獲取(保持登錄會(huì)話信息)：

# coding:utf8#cookie的獲取(保持登錄會(huì)話信息)import urllib.request, http.cookiejarcookie = http.cookiejar.CookieJar()handler = urllib.request.HTTPCookieProcessor(cookie)opener = urllib.request.build_opener(handler)res = opener.open(’http://www.baidu.com’)for item in cookie: print(item.name + ’=’ + item.value)

MozillaCookieJar(filename)形式保存cookie

# coding:utf8#將cookie保存為cookie.txtimport http.cookiejar, urllib.requestfilename = ’cookie.txt’cookie = http.cookiejar.MozillaCookieJar(filename)handler = urllib.request.HTTPCookieProcessor(cookie)opener = urllib.request.build_opener(handler)res = opener.open(’http://www.baidu.com’)cookie.save(ignore_discard=True, ignore_expires=True)LWPCookieJar(filename)形式保存cookie：

# coding:utf8import http.cookiejar, urllib.requestfilename = ’cookie.txt’cookie = http.cookiejar.LWPCookieJar(filename)handler = urllib.request.HTTPCookieProcessor(cookie)opener = urllib.request.build_opener(handler)res = opener.open(’http://www.baidu.com’)cookie.save(ignore_discard=True, ignore_expires=True)讀取cookie請(qǐng)求，獲取登陸后的信息

# coding:utf8import http.cookiejar, urllib.requestcookie = http.cookiejar.LWPCookieJar()cookie.load(’cookie.txt’, ignore_discard=True, ignore_expires=True)handler = urllib.request.HTTPCookieProcessor(cookie)opener = urllib.request.build_opener(handler)resp = opener.open(’http://www.baidu.com’)print(resp.read().decode(’utf-8’))

以上就是python urllib庫(kù)的使用詳解的詳細(xì)內(nèi)容，更多關(guān)于python urllib庫(kù)的資料請(qǐng)關(guān)注好吧啦網(wǎng)其它相關(guān)文章！