SEO workflow

Robots.txt allow and disallow rules explained

Robots.txt rules guide crawlers, but a small path mistake can block pages you meant to keep visible. Test representative URLs before deploying changes.

Test robots.txt rules

Rule basics

User-agent selects the crawler group.
Disallow blocks matching paths.
Allow can permit a more specific path.
Sitemap points crawlers to sitemap locations.

Testing workflow

Paste the rules, add a URL to test, and confirm whether the path should be allowed or blocked. Test examples from public pages, private folders, assets, and generated language paths.

Common mistakes

Blocking CSS or JavaScript can make pages harder to render. Blocking a URL in robots.txt is also different from using noindex; crawlers may still know the URL exists.

Troubleshooting

If an important page is not being crawled, check robots.txt, page-level robots tags, canonical tags, redirects, and sitemap inclusion together.

Example workflow

Start with a small representative sample instead of a full production file. Run the matching tool, inspect the result, and only then repeat the workflow with the complete content. This helps catch formatting, encoding, or data-shape problems while the input is still easy to review.

When the output looks correct, copy or download the result and keep a note of the source, date, and settings you used. For team workflows, link the relevant guide and tool together so another reviewer can reproduce the same check later.

Final review checklist

Confirm the input is safe to handle in a browser and does not contain secrets.
Run the tool with a small sample before processing a larger file.
Review the output for structure, formatting, and unexpected changes.
Use the related tools when the workflow needs validation, comparison, cleanup, or conversion.