[TOC] # 一 反爬思路 部分网站会通过统计某个IP或agent在单位时间内的请求次数来识别人机进行反爬 # 二 对抗措施 1. 找免费或付费的代理 2. 实例化一个代理池(IP或/agent) 3. 通过for循环或随机数调用代理IP # 三 相关代理网站 * 快代理 * 西祠代理 * www.goubanjia.com # 四 代码实现 ```python import requests import random # 1. UA和IP代理池 header_list = [ {"user-agent": "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)"}, {"user-agent": "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"}, {"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"} ] proxy_list = [ {"http": "112.115.57.20:3128"}, {'http': '121.41.171.223:3128'} ] # 2. 随机获取UA和代理IP header = random.choice(header_list) proxy = random.choice(proxy_list) url = 'http://www.baidu.com/s?ie=UTF-8&wd=ip' # 3. 设置代理 response = requests.get(url=url,headers=header,proxies=proxy) response.encoding = 'utf-8' with open('daili.html', 'wb') as fp: fp.write(response.content) ```