How to create a robots.txt file

Instructions for creatings a robots.txt file for your website

The robots.txt file tells search engine spiders where NOT to go.
Seems odd, doesn't it? We spend time and money getting Google to find us, but then we have to create a file telling them to stay out of certain folders.
There are some valid reasons for that:

• Your website keeps articles and documents for your staff or students and you don't want them indexed by the search engines. Keep them in a folder on your web server called "articles," then configure your robots.txt file and they won't appear on search engine results pages.


• You are redesigning your website and have a folder called "redesign" on your web server and you don't want the folder indexed because the pages are just "practice" pages.


• You have images you don't want to appear in image searches, so you restrict the "images" folder through your robots.txt file

Creating a robots.txt file is easy. If you don't have an HTML editor, use Notepad that comes with Windows. Remember to save the file with the .txt extension so the spiders can read it. Upload it into the same folder as the rest of your web pages.


You can copy the template below:

# Robots.txt file created by http://www.put-your-domain-name-here.com
# For domain: http://www.put-your-domain-name-here.com

# All robots will spider the domain
User-agent: *
Disallow:

# Disallow directory /test/
User-agent: *
Disallow: /test/

# Disallow directory /picture_library/
User-agent: *
Disallow: /picture_library/


Notice how each entry begins and ends. Copy that as many times as you need for the folders you want restricted. Save as robots.txt, and upload. That's it!


Don't forget to change your robots.txt file as your website grows or you may end up inadvertently allowing the search engines to post pages you don't want on the internet.