Leilukin
/
leilukin-tumbleblog
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add AI2Bot and Ai2Bot-Dolma

This commit is contained in:
Helen Chong 2024-08-30 13:52:46 +08:00
parent bc2a14037d
commit b08bcd7a89
2 changed files with 5 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.htaccess
View File

@ -21,6 +21,8 @@ RewriteRule ^.+$ index.php [L]
# Block bad bots
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AI2Bot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (Ai2Bot-Dolma) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (Amazonbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (Applebot-Extended) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (Bytespider) [NC,OR]

4
robots.txt
View File

@ -1,6 +1,8 @@
User-agent: *
Disallow:
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Bytespider
@ -26,6 +28,7 @@ User-agent: PetalBot
User-agent: Scrapy
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: YouBot
User-agent: anthropic-ai
User-agent: cohere-ai
@ -33,5 +36,4 @@ User-agent: facebookexternalhit
User-agent: img2dataset
User-agent: omgili
User-agent: omgilibot
User-agent: Webzio-Extended
Disallow: /