Robots.txt is a text file located in the site’s root directory that specifies for search engines’ crawlers and spiders what website pages and files you want or don’t want them to visit. Usually, site owners strive to be noticed by search engines, but there are cases when it’s not needed: For instance, if you store sensitive data or you want to save bandwidth by not indexing excluding heavy pages with images.
A robots.txt file lives at the root of your website. So, for site YYY com, the robots.txt file lives at http:/www YYY com/robots.txt. A robots.txt file consists of one or more rules. Each rule blocks (or allows) access for a given crawler to a specified file path in that website.
Basic robots.txt examples:
Prevent the whole website from indexation by all crawlers:
Allow all web crawlers to index the whole website:
Prevent only several directories from indexation:
Prevent the website’s indexation by a specific web crawler:
Why is robots.txt important?
The Importance of a Robots.txt File for Your SEO. Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.
The robots.txt file is beneficial for keeping your search engine spiders from accessing parts files and folders in your website hosting directory that are completely unrelated to your real website content. You can select to have the search engine spiders kept out of areas that include programming that search engines cannot parse appropriately, and to keep them out of the site stats section of your website.
Using Robots.txt generator by PufferSEO
You can easily create a new or edit an existing robots.txt file for your site with PufferSEO's robots.txt generator. Use this tool to create directives with either Allow or Disallow directives (Allow is default, click to change) for User Agents (use * for all or click to select just one) for specified content on your site. Click Add directive to add the new directive to the list. To edit an existing directive, click Remove directive, and then create a new one. You can also add a link to your XML-based Sitemap file. Just type or paste the full URL for the XML Sitemap file in the XML Sitemap text box.
When done, just click Export to save your new robots.txt file. Use FTP to upload the file to the domain root of your site.