BinaryDigit/kilobytekeith2024
1
0
Fork 0
mirror of https://github.com/binarydigitlol/kilobytekeith2024.git synced 2025-06-20 17:02:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

update robots txt

Browse Source
This commit is contained in:
BinaryDigit 2024-01-29 09:56:23 -05:00
parent 5b2ec1bdff
commit 9a52eca1a5
1 changed files with 7 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
static/robots.txt
View File

@ -6,6 +6,12 @@
User-agent: anthropic-ai User-agent: anthropic-ai
Disallow: / Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/bytespider
User-agent: Bytespider
Disallow: /
# AI Data Scraper # AI Data Scraper
# https://darkvisitors.com/agents/ccbot # https://darkvisitors.com/agents/ccbot
@ -31,18 +37,9 @@ User-agent: GPTBot
Disallow: / Disallow: /
# AI Data Scraper # AI Data Scraper
# https://darkvisitors.com/agents/omgilibot # https://darkvisitors.com/agents/omgili
User-agent: omgilibot
Disallow: /
User-agent: omgili User-agent: omgili
Disallow: / Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: AdsBot-Google
Disallow: /
Sitemap: https://kilobytekeith.com/sitemap.xml Sitemap: https://kilobytekeith.com/sitemap.xml