Robots.txt Generator
Build a robots.txt file. Save the output at /robots.txt in your site root.
User-agent: *
Disallow: /admin
Disallow: /private
Sitemap: https://example.com/sitemap.xmlAbout the Robots.txt Generator
The Robots.txt Generator builds a syntactically correct robots.txt file for your site based on the choices you make — which directories to allow or disallow for all crawlers, per-bot overrides (Googlebot, Bingbot, GPTBot, ChatGPT-User), crawl-delay, and the Sitemap directive. Robots.txt lives at the root of your domain (`https://yoursite.com/robots.txt`) and is the first thing every search engine and crawler fetches.
A well-configured robots.txt keeps crawlers away from admin areas, staging paths, faceted search URLs and other pages you don't want indexed — while making sure your public content stays fully accessible. A misconfigured robots.txt can accidentally block your entire site from search engines (a `Disallow: /` under `User-agent: *` is a very common self-inflicted SEO wound).
In 2026, robots.txt has grown beyond just search engines — AI crawlers (GPTBot, Google-Extended, ClaudeBot, Bytespider) also respect it. Blocking these prevents your site's content from being used to train large language models, which many publishers now choose to do.
Enter the required value in the input field above (domain, IP, URL, or text depending on the tool).
Click the action button to run the check — results are computed instantly from our edge network.
Review the parsed output, key fields and any warnings shown in the result card.
Copy the result, share the page URL, or jump to a related tool from the sidebar to continue debugging.
- Per-bot rules for search engines and AI crawlers
- Sitemap directive included
- Common templates for WordPress, Shopify and custom sites
- Copy-paste ready output
Frequently asked questions
Related tools
Interactive 1xx–5xx codes & Cloudflare edge errors.
Response headers, status, TLS.
Follow the redirect chain.
TLS cert issuer, validity, SANs.
Every TLS cert ever issued for a domain (crt.sh).
Grade HSTS, CSP, X-Frame-Options A–F.