About robots.txt Tester
Test robots.txt online. Fetch your site’s robots file and review disallow rules for crawlers — free robots.txt checker on Fah Swe Tools.
How it works
- Enter a site origin (https://example.com).
- Fetch robots.txt.
- Review rules.
FAQ
Is crawling polite?
Single GET with short timeout and rate limits.