A practical check before using the result
Before changing a rule, list two example URLs it should affect and two it should leave alone. Include a nested route and an important asset if those are relevant to your site. That small checklist gives a reviewer concrete cases to inspect after deployment. If a site builder already manages the file, find its supported settings first; maintaining competing copies in different places makes later troubleshooting unnecessarily difficult.
Start with the crawl policy your site needs
A robots.txt file is a small set of instructions for crawlers that choose to follow it. Before adding rules, decide which public pages you actually want them to request. A new product site with a home page, pricing, documentation and a few articles often needs very little configuration. Adding a long block list copied from another site can accidentally cover useful pages or assets.
This generator starts with crawling allowed. Choose custom paths when you have a specific reason to exclude a section, then enter only those paths. Use block all for a deliberate full-site restriction and review that choice before publishing. The output is a draft you control: changing an option here does not change your website.
Write a small, understandable file
For the standard file placement and syntax, consult Google’s robots.txt instructions and the Robots Exclusion Protocol. Publish a UTF-8 text file at /robots.txt on the relevant host. Rules use paths, while Sitemap lines use complete URLs. An Allow rule can describe an exception within a blocked path. Crawlers compare matching rules rather than treating the file as a simple top-to-bottom script.
For example, a team might exclude /search/ while allowing /search/help/. That is a hypothetical policy, not a recommendation for every product. Confirm that the excluded section has no valuable landing pages. Keep a separate note explaining each rule so the next person maintaining the site can understand why it exists. Remove rules that no longer correspond to real routes.
Keep crawling, indexing and access separate
A crawler instruction is not a password. Do not put sensitive path names in a public file as a substitute for protecting them. Authentication and authorization belong in the application. Equally, asking a crawler not to fetch a page does not prove that the URL will disappear from search results. Think through the outcome you need before picking the mechanism.
If your problem is a misleading search snippet, work on the page itself. The SERP preview helps you review its title and description. If you are checking an external listing, the dofollow checker inspects link attributes and page-level nofollow directives. Neither task is solved by adding arbitrary paths to your robots file.
Publish and check the real response
Download the text, review it alongside any existing rules and deploy it through your normal release process. Avoid replacing a working file without comparing the contents. Some hosting tools generate their own version, so the file you upload and the file visitors receive may differ. Open the root URL after release and verify the body, content type and HTTP response.
Check each sitemap URL separately. This generator validates syntax but does not visit the sitemap or confirm its XML. Also check the hosts you actually use: your main website, documentation host and preview environment may have different configurations. Keep a copy of the previous file in version control so an accidental restriction can be understood and corrected.
Review rules when the product changes
Make the check part of a release that changes routes, navigation or rendering. A renamed folder can leave an old exclusion harmless while a broader pattern unexpectedly affects a new section. Ask the person reviewing the release to open representative allowed and excluded URLs. Record the intended behavior in the pull request instead of relying on memory.
After a directory launch, use the referral measurement guide to inspect useful visits and actions. A tidy crawl policy supports maintenance, but it cannot promise traffic. Keep the file short, revisit it when the site changes, and assess discovery using the pages and audiences that matter to your product.
