网络爬虫（又被称为网页蜘蛛），是一种按照一定的规则，自动的抓取万维网信息的程序或者脚本。由于抓取网页的过程会对流量造成影响，可以选择屏蔽其UA关键字，排除对流量的干扰。
| 爬虫名称（Spider name） | 关键词（Key term） | 示例UA |
|---|---|---|
| 百度爬虫 | Baiduspider | Mozilla/5.0 (compatible; Baiduspider/2.0; +http//www.baidu.com/search/spider.html |
| 谷歌爬虫 | Googlebot | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| MSN 爬虫 | MsnBot-Media | Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0 b |
| naver 爬虫 | http://naver.me/bot | Mozilla/5.0 (compatible; Yetil 1.1; +http://naver.me/bot) |
| Ping 爬虫 | pingbot | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)Ubuntu Chromium/59.0.3071.109 Chrome/59.0.3071.109 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http: //www. pingdom. com/) |
| python爬虫 | pyspider | pyspider/0.3.10-dev (http: //pyspider. org/) |
| 360 爬虫 | 360Spider | User-Agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider |
| 搜狗爬虫 | Sogou web spider | Sogou web spider/4.0（ +http: //www. sogou.com/docs/help/webmasters.htm#07) |
| monitor爬虫 | monitor-spider | Mozilla/5.0 (Windows NT 6.1;rv:17.0) Gecko/20100101 Firefox/17.0 monitor-spider |
| 监控宝爬虫 | jiankongbao | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0; JianKongBao Monitor 1.2)Gecko/20100101 Firefox/25 |
| 听云爬虫 | networkbench | Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv: 11.0;NetworkBench/8.0.1.309-5774440-2481662) like Gecko |
| OneAPM爬虫 | OneAPM FFAgent | Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0: OneAPM FFAgent)Gecko/20100101 Firefox/39.0 |
| PhantomJS | PhantomJS | Mozilla/5.0 (Unknown; Linux x86_64)AppleWebKit/538. 1 (KHTML,like Gecko)PhantomJS/2.1.1 Safari/538.1 |