crawl-stop. A test page designed to validate web crawlers' compliance with restrictions and anti-scraping measures. Features rate limiting, access controls, and crawling deterrents.

github.com/CameronWhiteside/crawl-stop

Problem

Audience

Updates

No founder updates yet. Check back soon.