About GFreeMeBot

Gluten Free Me! runs a small, low-volume crawler — User-Agent GFreeMeBot/1.0 — to build a browsable catalog of gluten-free products for shoppers. It only reads publicly published product pages and documented public catalog endpoints.

It respects each site's robots.txt (including Crawl-delay), makes requests to any one site sequentially rather than in parallel, and caps itself to a small number of pages per site per run.

If you run a site this bot visits and would rather it didn't, disallowing GFreeMeBot in your robots.txt is honored automatically on the next crawl.