Tahoe Stores.com

Some of you have asked 'How do I keep 'search engine A' from indexing pages designed for 'search engine B'. The answer is to use a robots.txt file. There are also other reasons for wanting to keep search engines from indexing some or all pages on a site. Therefore, I've put together this detailed article to show you how to do that, and to avoid common mistakes that are made all too often.

If you create different versions of essentially the same doorway page and every search engine indexes every copy of the page, then you could, in theory, get in trouble for spamming. AltaVista in particular is known to dislike duplicate or near duplicate content. Therefore, if you create pages that are too similar, you run the risk of being red-flagged.

In practice, many people don't worry about having too many duplicate pages indexed by a single search engine because they are not creating huge numbers of similar pages. In fact, I spoke to the CEO’s of three search engine positioning companies. Each said they did not use robots.txt files, although their reasons varied.

If your pages vary enough in the size and number of words, then you should not need to worry about being red-flagged. If you focus primarily on optimizing existing pages on your site that have unique content rather than building a lot of new pages that are similar, you’ll also avoid any potential problems.

Other people simply submit pages designed for a particular search engine to just the engine(s) to which the page applies. This can be the simplest method to avoid spamming the search engines. This can work IF there are no other links to that page from the rest of your site. However, if another search engine spider manages to find a link to that page, it could index it even though you never submitted the page to that engine.

Despite this, two of the consulting companies I spoke to said submitting hallway pages that pointed to just the pages built for that engine worked well for them and they’d done it successfully for years.

However, if you want to create a lot of doorway pages, targeted for a large number of engines, where many will be very similar to each other, you should consider using a robots.txt file. This file can tell the search engine spiders which pages they are not allowed to index. That way you can build pages for search engine A and tell search engine B to ignore them. The search engine's like this because it keeps them from indexing pages that don’t apply to them. Therefore, it benefits the search engines, the users that use that engine, and it keeps you from being labeled as a spammer.

I have seen some people debate whether the search engines even honor robots.txt files since this is a purely voluntary feature of the Web. However, the search engines have historically been challenged by companies - and in the courts - about indexing copyrighted materials without the permission of the copyright holder. The search engine's most prominent argument for being able to index copyrighted material without permission is that the Web site owner always has the option to exclude their indexing by creating a robots.txt file.

Therefore, it's unlikely the search engines would intentionally ignore the robots.txt or they could get themselves into unnecessary legal problems. They might in theory spider the page and then after checking the robots.txt file drop it. This may explain reports I've heard from a couple of people that claim the spider ignored their robots.txt file because they saw it opening the page in their log file. Another explanation is that the Webmaster used the wrong syntax when creating the robots.txt. Therefore, always double check your work.

I'll try to point out common errors in this article and give plenty of examples. Please don't be intimidated. It really is not as hard as it looks. There is also a method you can set it up once and then not have to mess with it ever again. I’ll explain this method near the end of this article.

To create a robots.txt file, open Window's NotePad or any other editor that can save plain ASCII .txt files. Use the following syntax to exclude a file name from a particular search engine spider:

User-agent: {SpiderNameHere}
Disallow: {FilenameHere}

Note: For the purpose of this article, the term spider and search engine may be used interchangeably.

For example, to tell Excite's spider, called ArchitextSpider to not index files called orderform.html, product1.html, and product2.html, create a robots.txt file as follows:

User-agent: ArchitextSpider
Disallow: /orderform.html
Disallow: /product1.html
Disallow: /product2.html

According to the official robots.txt specifications, the above is case-sensitive so you should spell it as 'User-agent:' rather than 'User-Agent:'. Whether this causes a problem in practice, I cannot say for certain. To be safe, keep the names in the correct case. In addition, make sure you include a forward-slash before the file name if the file is in the root directory.

The User-agent line is the identifier for the search engine you wish to target. It is like a 'code name' for the search engine's spider that goes around and indexes pages on the Web. It may be similar to the name of the search engine or it may be completely different. (I'll list the official User-agent names for the major engines later in this article).

Once you create your robots.txt, you would then upload this text file to the root directory of your Web site. Although robots.txt is a voluntary protocol, most major search engines will honor it. If you do not have your own domain name but instead use a subdirectory off of your host's domain, then your robots.txt may not be recognized in theory since standard practice is to look only at the root directory of the domain. This is just one more reason to invest in your own domain name!

You can add additional lines to exclude pages from other engines by specifying the User-Agent parameter again in the same file followed by more Disallow lines. Each disallow statement will be applied to the last User-Agent that was specified.

If you want to exclude an entire directory, use this syntax:

User-agent: ArchitextSpider
Disallow: /mydirectory/

A common mistake is to include the asterisk after the directory name to indicate that you want to exclude all files in that directory. However, the proper syntax is to NOT include any asterisks in the Disallow statement. According to the robots.txt specifications, it is implied that the above statement will disallow all files in 'mydirectory.'

To disallow a file named product.htm in the 'mydirectory' subdirectory, do this:

User-agent: ArchitextSpider
Disallow: /mydirectory/product.htm

You can exclude pages from ALL spiders with this User-agent:

User-agent: *

In the case of the User-agent line, you CAN use the asterisk as a wildcard.

To disallow all pages on your Web site for the specified spider use:

Disallow: /

To re-iterate, you use only a forward slash to indicate you want to disallow your entire site. Do NOT use an asterisk here. It's important that you use the proper syntax. If you misspell something, it may not work and you won't know it until it's too late! It is possible that certain search engines may handle common syntax variations without problems. However, this doesn't guarantee that they will all tolerate variances in the syntax. Therefore, play it safe. If at some point you do find that your syntax was wrong, don't panic. Correct the problem and then re-submit. The search engine will then re-spider the site and drop the pages that you excluded.

If you wish to include comments in your robots.txt file, you should precede them with a # sign like this:

# Here are my comments about this entry.

Each set of disallow statements should be separated by a blank line. For example, you might have something like the following to exclude different files from different spiders:

User-agent: ArchitextSpider
Disallow: /mydirectory/product.htm
Disallow: /mydirectory/product2.htm
 

User-agent: Infoseek
Disallow: /mydirectory/product3.htm
Disallow: /mydirectory/product4.htm
 

The blank line between the two groups is important to group things into 'records.'

If, on the other hand you wanted to exclude the same set of files for more than one spider, you could do something like this:

User-agent: ArchitextSpider
User-agent: Infoseek
Disallow: /mydirectory/product.htm
Disallow: /mydirectory/product2.htm

Side note about subdirectories: Some Webmasters like to organize their doorway pages into different subdirectories according to which search engine they are optimized for. However, some engines are suspected of assigning lower rankings to pages appearing in subdirectories versus the root directory of a Web site. If they perceive that those pages belong to a Web site that shares a domain with its host, they could discriminate against those pages as being potentially of a lesser quality. I asked three search engine consultants there opinion of subdirectories. The general feeling was that pages in the root directory was probably better, but they’d not seen evidence that it caused problems.

If you were still concerned about being penalized for keeping pages in subdirectories and wished to use them, you could ask your hosting service to give you 'machine names' like myproduct.mydomain.com that you could submit. The myproduct.mydomain.com URL could then be configured by your hosting service to point to your 'myproduct' subdirectory or whatever directory you desired. That way no discrimination could occur by the search engine since they would not see the subdirectory in the URL. In addition, you could include keywords in that machine name which may also improve your rankings. (Note: A machine name is normally just 'www.' prefixed at the start of your domain name. However, rather than 'www.' it could be any name you desire and it could point to any location on any physical machine).

We are often asked about the proper names for the User-agent. The name of the agent does not always correspond to the name of the search engine. Therefore, you can't just put in 'AltaVista' in the User-agent and expect AltaVista to exclude your designated pages. Don't ask me why it can't be that simple. Perhaps it's a job security plan for professional Webmasters :-)

In any case, there's a lot of confusion in newsgroup forums and on the Web about what the proper agent names should be. The confusion derives from Webmasters reading their server log files and noticing all kinds of complicated agent names being logged such as Scooter/2.0 G.R.A.B. X2.0, InfoSeek Sidewinder/0.9, or Slurp/2.0. However, the agent names listed in your log are not necessarily what you are expected to use in your robots.txt file.

The reason is very logical when you think about it. Names like InfoSeek Sidewinder/0.9 in a robots.txt are not very useful if the search engine updates their agent software and decides to start using Infoseek Sidewinder/2.0 as their new name next month. Would it make sense to expect millions of Webmasters to know this and to all update their robots.txt files to the new name? Would they expect people to update the file EVERYTIME any search engine updated their agent version number and do it precisely when the name change occurred? It's not likely.

In reality, the name that needs to appear in the robots.txt file is whatever name the search engine spider is programmed to look for. Therefore, the best source of information for this name is not your log files but the help files on the search engine itself. In theory, a search engine could look for a wide variety of name variations. However, in general they will simply look for the least common denominator such as 'Scooter' rather than 'Scooter/2.0'. If the search engine is smart they will allow you to use Scooter/2.0 too, but that is not guaranteed. Therefore, if you've already setup a robots.txt on your site, double-check the syntax and the agent names against the list below. All names are case sensitive.

Here are the User-Agent names that we have compiled. Most of these came directly from the search engine's own help files, or when not available, from other respected sources:

Search Engine:

User-Agent

AltaVista:

Scooter

Infoseek:

Infoseek

Hotbot:

Slurp

AOL:

Slurp

Excite:

ArchitextSpider

Google:

Googlebot

Goto:

Slurp

Lycos:

Lycos

MSN:

Slurp

Netscape:

Googlebot

NorthernLight:

Gulliver

WebCrawler:

ArchitextSpider

Iwon:

Slurp

Fast:

Fast

DirectHit:

Grabber

Yahoo Web Pages:

Googlebot

Looksmart Web Pages:

Slurp

very old list for demonstration purposes only

You'll notice that many of the engines use the 'Slurp' agent which is the Inktomi spider used on HotBot and other Inktomi related sites. Unfortunately, I'm not aware of a way you can exclude pages from the HotBot spider and not exclude them from all other Inktomi sites. As far as I can tell, they use the same spider to index the pages and thereby recognize only one User-agent string in the robots.txt file. (If I am wrong, please reply to this e-mail and let me know how this is done!)

The individual Inktomi sites tend to rank the pages differently, although they will often be rather similar. Normally you can create a handful of pages that will rank well on most of the Inktomi powered sites, so the duplicated content issue does not normally become a big problem with Inktomi.

If you're now scratching your head on how this all comes together in relation to the doorway pages you created, check out the two detailed examples of a robots.txt file we've put together.

The first one shows how you can disallow INDIVIDUAL files:

The second example shows how you can group your doorway pages into DIRECTORIES and disallow the entire directory.

The advantage to method #1 is that it can be more flexible for working with a small number of files already on your site, and it 'might' be a little safer. Some people believe that locating your doorways in the root directory rather than a subdirectory can give you a ranking advantage. The theory is that the search engines might discriminate against sites that don't have their own domain name, so pages submitted in a subdirectory could be perceived as sharing a domain with their host.

The disadvantage to method #1 is that if you have very many doorway pages then the size of your robots.txt file could be enormous. This runs the risk that a search engine might have problems with a robots.txt file that exceeds a certain reasonable size. It might also slow down the spider from accessing your site if it must read in an extremely large robots.txt file. Lastly, a robots.txt with a lot of entries in it could be a red-flag in itself to a search engine. This is all speculation, but it’s enough that I would avoid excluding a lot of files individually if you don’t have to.

Example method #2 is to organize your doorway pages into subdirectories for each search engine. The advantage to method #2 is that it is much easier to track your doorway pages if they are organized in separate subdirectories. In addition, the size of your robots.txt will be relatively small. You'll also not need to update the file every time you upload new doorway pages. Once the robots.txt is set up with method #2, all you have to do is upload it to the appropriate directory, submit and you're done!

So do the engines discriminate against files in subdirectories? The consultants I talked to did not think so. Based on these conversations, if you properly design a hallway page in your ROOT directory that links to the doorway pages in your subdirectory, and submit that hallway page, then you’ll be fine. This demonstrates to the engine that the pages are most likely sub-pages of the main site. In addition, it would be dangerous for the search engines to penalize pages in subdirectories since most large Web sites must organize their pages into subdirectories to avoid complete chaos. As an added precaution, you could assign machine names to subdirectories as I mentioned earlier in this article. If you have any experience, comments, or observations on this issue, please let me know by replying to this e-mail.

My conclusion: If all your pages have good content and are fairly unique, don’t worry about robots.txt files. If you focus only on optimizing existing pages on your site, don’t worry about a robots.txt. If, however, you decide you need to experiment with more than a handful of pages that are rather similar, consider making use of the robots.txt file, particularly with AltaVista. Use example method #1 if you’re only dealing with a small number of pages or special scenarios. Otherwise, organize your files into directories and use example method #2.

However, I tested it on a couple of files and it sometimes complained about things that were perfectly valid. Therefore, the service in my opinion may be too buggy to be of great use. If it points out errors in your file, refer to this article or the article below to verify that the errors it catches are real. If you know of a better syntax checker, let me know and I'll pass the information along and give you and your Web site credit for the tip. I will also try to give you credit for any other search engine marketing tips I end up using of which I was not aware!

Note: The information presented here adapted, under license agreement, from FirstPlace Software.