MediaWiki:Robots.txt

User-Agent: YandexBot Crawl-Delay: 2.5
 * 1) Throttle YandexBot

User-agent: bingbot Crawl-delay: 1
 * 1) Throttle BingBot

User-Agent: SemrushBot Disallow: /
 * 1) Block SemrushBot

User-agent: MJ12bot Crawl-Delay: 10
 * 1) Throttle MJ12Bot

Sitemap: https://static.miraheze.org/pgrwiki/sitemaps/sitemap.xml Sitemap: https://grayravens.com/sitemap.xml Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_0-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_2-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_4-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_6-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_8-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_10-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_14-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_200-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_400-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_1198-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_3000-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_3002-0.xml.gz Sitemap: https://grayravens.com/sitemaps/pgrwiki/sitemaps/sitemap-pgrwiki-NS_3004-0.xml.gz
 * 1) Dynamic sitemap url