A robots txt is a text file uploaded
to your website server to keep compliant search engine
robots from indexing particular pages of your website.
Create a robots txt file like this
User-agent: Search Engine Robot Name
Disallow: File or Directory
Ex.
User-agent: Googlebot
Disallow: /adcampaign1.html
or
User-agent: *
Disallow: /adcampaign1.html
The * is used to cover all compliant search engine robots.
Common Robot Names
Search engine robots follow links throughout
the Internet to crawl through all the websites on the
world wide web. Here's a list of top search engine robot
names:
Google -----> googlebot
Ask Jeeves -----> Teoma
Alta Vista / Yahoo / All the Web / Inktomi ----->
Slurp
Check Your Robots txt on Search Engine World's Robots.txt Validator.
About The Author: SEO Consultant
- Offers search engine optimization, link popularity
building and search engine friendly website design services.
|