I just started using the robots.txt file instead of meta tags. I placed the file on my site, but the Google robot test is not finding it. How do I get it to the highest level of the root directory? Thanks to anyone who could helpo me!
First line catches all, additional lines list folders that should not be crawled. Common Google crawlers would be: User-agent: GoogleBotYou can comment your robot text like this: # Mark WisecarverAdditionally the asp.net robots.txt file shown here specifies a crawl delay.User-agent: * Disallow: /Admin Disallow: /aspnet_client Disallow: /backup Disallow: /bin Disallow: /forum Disallow: /Languages Disallow: /Utility Disallow: /Script Disallow: /j Disallow: /i/ Disallow: /vwd Disallow: /Data Disallow: /calypso Disallow: /images Disallow: /SqlScripts Disallow: /Styles Disallow: /IBS_Portal Disallow: /IBS_Store Disallow: /ibuyspy Crawl-delay: 120
Thanks Wisemx for your reply, but I have made the robots.txt file, that's not the problem. I've uploaded it to my site in the root directory, but when I run a testthrough Google Analysis, it is bringing back a 404 (file not found). I read on robotstxt.org that the file has to be in the "top level server directory." So I'm not sure if I have it in the right location. I'm not sure what the "top level server directory" means. It shows up when you go to my site http://plainbibletruth.org/robots.txt. I'm not sure what I'm doing wrong. Thanks again!!
Wisemx Disregard. I guess Google took some time before picking up my robots.txt file, because this morning it has been found. Thanks again for your help!! You've helped me quite a bit in the past as well.
