robots.txt file

[plainbiblet]

I just started using the robots.txt file instead of meta tags. I placed the file on my site, but the Google robot test is not finding it. How do I get it to the highest level of the root directory?

Thanks to anyone who could helpo me!

[wisemx]

First line catches all, additional lines list folders that should not be crawled.
Common Google crawlers would be: User-agent: GoogleBotYou can comment your robot text like this:
# Mark WisecarverAdditionally the asp.net robots.txt file shown here specifies a crawl delay.User-agent: *
Disallow: /Admin
Disallow: /aspnet_client
Disallow: /backup
Disallow: /bin
Disallow: /forum
Disallow: /Languages
Disallow: /Utility
Disallow: /Script
Disallow: /j
Disallow: /i/
Disallow: /vwd
Disallow: /Data
Disallow: /calypso
Disallow: /images
Disallow: /SqlScripts
Disallow: /Styles
Disallow: /IBS_Portal
Disallow: /IBS_Store
Disallow: /ibuyspy
Crawl-delay: 120

[plainbiblet]

Thanks Wisemx for your reply, but I have made the robots.txt file, that's not the problem. I've uploaded it to my site in the root directory, but when I run a testthrough Google Analysis, it is bringing back a 404 (file not found). I read on robotstxt.org that the file has to be in the "top level server directory." So I'm not sure if I have it in the right location. I'm not sure what the "top level server directory" means. It shows up when you go to my site http://plainbibletruth.org/robots.txt. I'm not sure what I'm doing wrong.


Thanks again!!

[plainbiblet]

Wisemx Disregard. I guess Google took some time before picking up my robots.txt file, because this morning it has been found.

Thanks again for your help!! You've helped me quite a bit in the past as well.

[wisemx]

Cool http://community.discountasp.net/emoticons/wink.gif