服务器禁用爬虫
2022-09-26 17:05 By
"Powerless"
2642
0
3
【Nginx禁止爬虫访问的方法】
if ($http_user_agent ~* "Scrapy|Baiduspider|Curl|HttpClient|Bytespider|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser
|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSp
ider|Ezooms|^$"){
return 403;
}
如需跳转其他页面，只需要吧return 403 换成对于的地址即可，配置如下:
if ($http_user_agent ~* "Scrapy|Baiduspider|Curl|HttpClient|Bytespider|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser
|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSp
ider|Ezooms|^$") {
return 301 https://yoursite.com;
}
如需禁止特定来源用户，配置如下:
if ($http_referer ~ "baidu\.com|google\.net|bing\.com") {
return 403;
}
如需仅允许GET,HEAD和POST请求，配置如下:
#fbrbidden not GET|HEAD|POST method access
if ($request_method !~ ^(GET|HEAD|POST)$) {
return 403;
}
【Apache禁用爬虫的配置】
mod_rewrite模块确定开启的前提下，在.htaccess文件或者相应的.conf文件,添加以下内容:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (^$|FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms) [NC]
RewriteRule . - [R=403,L]
View in WeChat
Others Discussion
-
浏览器访问网站经历的步骤-Html Posted on 2018-11-28 18:48
-
TCP协议的特性 Posted on 2019-04-26 16:46
-
HTTP头中隐藏PHP版本号 Posted on 2021-01-11 16:38
-
Composer 异常 [ErrorException] Posted on 2019-11-25 17:55
-
PHP练习-移动数组内的0到最后并保持其他元素顺序不变 Posted on 2020-08-14 20:32
-
能创建多少个 TCP 连接？ Posted on 2021-08-02 16:00
-
Mysql联合索引的最左前缀匹配原则 Posted on 2018-08-25 15:00
-
PHP没你想的那么差 Posted on 2021-12-17 15:40
评 论