Leilukin
/
leilukin-site
mirror of https://github.com/helenclx/leilukin-site.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
leilukin-site/robots.txt

46 lines
1.0 KiB
Plaintext
Raw Normal View History

Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: *
Build: (6300906) Remove default crawler block
2024-08-20 00:38:59 +00:00
Disallow:
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
Build: (114daaa) Add The Eleventy Bundle
2024-08-29 03:21:48 +00:00
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: Amazonbot
Build: (dfd9c0e) Replace content paragraph top margin with margin-block-start
2024-10-20 13:33:30 +00:00
User-agent: anthropic-ai
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
Build: (dfd9c0e) Replace content paragraph top margin with margin-block-start
2024-10-20 13:33:30 +00:00
User-agent: cohere-ai
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: Diffbot
User-agent: FacebookBot
Build: (dfd9c0e) Replace content paragraph top margin with margin-block-start
2024-10-20 13:33:30 +00:00
User-agent: facebookexternalhit
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
Build: (dfd9c0e) Replace content paragraph top margin with margin-block-start
2024-10-20 13:33:30 +00:00
User-agent: GPTBot
User-agent: iaskspider/2.0
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: ICC-Crawler
User-agent: ImagesiftBot
Build: (dfd9c0e) Replace content paragraph top margin with margin-block-start
2024-10-20 13:33:30 +00:00
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
Build: (f2c88f4) Remove trailing closing tag for DykeNet webring list item
2024-09-29 02:16:52 +00:00
User-agent: Kangaroo Bot
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: OAI-SearchBot
Build: (dfd9c0e) Replace content paragraph top margin with margin-block-start
2024-10-20 13:33:30 +00:00
User-agent: omgili
User-agent: omgilibot
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: Scrapy
Build: (f2c88f4) Remove trailing closing tag for DykeNet webring list item
2024-09-29 02:16:52 +00:00
User-agent: Sidetrade indexer bot
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
Build: (f93ed52) Add Bara El Shaer's fundraiser and The Palestine Academy
2024-08-23 15:29:34 +00:00
User-agent: Webzio-Extended
Build: (f627206) Add robots.txt
2024-08-19 15:59:57 +00:00
User-agent: YouBot
Build: (4d88d00) Dynamically generate bad bots to block by fetching remote list
2024-08-20 01:53:59 +00:00
Disallow: /
Build: (d083f49) Remove tooltip argument from pride button macro
2024-10-23 15:32:55 +00:00
Build: (4e8894a) Change default sitemap change frequency to weekly
2024-08-20 06:52:41 +00:00
Sitemap: https://leilukin.com/sitemap.xml