Where does robots.txt go?

At the root of your domain: yourdomain.com/robots.txt. It must be exactly there — crawlers don't look in subfolders. Upload the file you generate here to your site's root directory.

Does robots.txt control indexing?

Not directly. Disallow stops compliant crawlers from fetching a URL, but a blocked URL can still be indexed if other sites link to it. To keep a page out of search, use a noindex meta tag or header — and don't disallow it, so crawlers can actually see the noindex.

Should I list my sitemap in robots.txt?

Yes. A 'Sitemap:' line with the absolute URL helps every search engine discover your pages, and it's the one line we add by default. You can list more than one sitemap if your site is split across several.

Is it free and private?

Yes — no signup, no limits, and everything runs in your browser. Your rules and paths are never uploaded.

Free SEO Tool

Robots.txt Generator

Build a clean robots.txt in your browser. Start from a preset, set allow and disallow rules per user-agent, add your sitemap, and copy the result — or paste an existing file to validate it. No signup.

Free
No signup
Live validator
Runs in your browser

Start from a preset Sitemap URL (recommended) Crawl-delay (optional, seconds)

robots.txt

How it works

01
Start from a preset
Allow everything, block everything, WordPress, or block common AI bots — then tweak it.
02
Set your rules
Add user-agent groups with allow and disallow paths, and a crawl-delay if you need one.
03
Add your sitemap
Drop in your sitemap URL — the one directive every robots.txt should have for discovery.
04
Copy or validate
Copy or download robots.txt, or switch to Validate to check an existing file for errors.

Next step

Workflow → Schema Markup Generator Round out the technical basics — generate valid JSON-LD for your pages. Compared → Doing this by hand for every page? See our side-by-side of the leading AI SEO tools — compared on capabilities, pricing and fit, with hands-on scores landing as testing completes.

FAQ

Frequently asked questions

Where does robots.txt go?: At the root of your domain: yourdomain.com/robots.txt. It must be exactly there — crawlers don't look in subfolders. Upload the file you generate here to your site's root directory.
Does robots.txt control indexing?: Not directly. Disallow stops compliant crawlers from fetching a URL, but a blocked URL can still be indexed if other sites link to it. To keep a page out of search, use a noindex meta tag or header — and don't disallow it, so crawlers can actually see the noindex.
Should I list my sitemap in robots.txt?: Yes. A 'Sitemap:' line with the absolute URL helps every search engine discover your pages, and it's the one line we add by default. You can list more than one sitemap if your site is split across several.
Is it free and private?: Yes — no signup, no limits, and everything runs in your browser. Your rules and paths are never uploaded.