python开发:scrapy爬虫(七)

设置爬虫的user-agent

安装scrapy-user-agents
conda install scrapy-user-agents

setting.py文件添加

1
2
3
4
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
}

设置爬虫代理

安装scrapy-proxy-pool
conda install scrapy-proxy-pool

setting.py文件添加

1
2
3
4
5
6
7
8
PROXY_POOL_ENABLED = True

DOWNLOADER_MIDDLEWARES = {
# ...
'scrapy_proxy_pool.middlewares.ProxyPoolMiddleware': 610,
'scrapy_proxy_pool.middlewares.BanDetectionMiddleware': 620,
# ...
}

坚持原创技术分享,您的支持是我前进的动力!