Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Robots.txt is a text file located in the site’s root directory that specifies for search engines’ crawlers and spiders what website pages and files you want or don’t want them to visit. Usually, site owners strive to be noticed by search engines, but there are cases when it’s not needed: For instance, if you store sensitive data or you want to save bandwidth by not indexing excluding heavy pages with images.

A robots.txt file lives at the root of your website. So, for site YYY com, the robots.txt file lives at http:/www YYY com/robots.txt. A robots.txt file consists of one or more rules. Each rule blocks (or allows) access for a given crawler to a specified file path in that website.

Basic robots.txt examples:

Prevent the whole website from indexation by all crawlers:
User-agent: *
 Disallow: /

Allow all web crawlers to index the whole website:
User-agent: *
 Disallow:

Prevent only several directories from indexation:
User-agent: *
Disallow: /script/

Prevent the website’s indexation by a specific web crawler:
User-agent: Bot1
Disallow: /

Why is robots.txt important?
The Importance of a Robots.txt File for Your SEO. Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.
The robots.txt file is beneficial for keeping your search engine spiders from accessing parts files and folders in your website hosting directory that are completely unrelated to your real website content. You can select to have the search engine spiders kept out of areas that include programming that search engines cannot parse appropriately, and to keep them out of the site stats section of your website.

Using Robots.txt generator by PufferSEO
You can easily create a new or edit an existing robots.txt file for your site with PufferSEO's robots.txt generator. Use this tool to create directives with either Allow or Disallow directives (Allow is default, click to change) for User Agents (use * for all or click to select just one) for specified content on your site. Click Add directive to add the new directive to the list. To edit an existing directive, click Remove directive, and then create a new one. You can also add a link to your XML-based Sitemap file. Just type or paste the full URL for the XML Sitemap file in the XML Sitemap text box.

When done, just click Export to save your new robots.txt file. Use FTP to upload the file to the domain root of your site.