设置爬虫的user-agent
安装scrapy-user-agents
conda install scrapy-user-agents
setting.py文件添加1
2
3
4DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
}
设置爬虫代理
安装scrapy-proxy-pool
conda install scrapy-proxy-pool
setting.py文件添加1
2
3
4
5
6
7
8PROXY_POOL_ENABLED = True
DOWNLOADER_MIDDLEWARES = {
# ...
'scrapy_proxy_pool.middlewares.ProxyPoolMiddleware': 610,
'scrapy_proxy_pool.middlewares.BanDetectionMiddleware': 620,
# ...
}