I have a web site which is protected by a user name and password but when I enter the domain name into the Google search engine all the web pages become accessible as references, bypassing the entry page. This does not happen with Bing or Yahoo search What is happening and how do I resolve the problem so that unauthorized users cannot access the pages of the web site?
I don't think that I am using either Windows authentication or ASP.NET authentication. The log-in is just a form which requests Username and Password and on Submit goes to an .aspx page which checks the Username and Password with a couple of simple if, then statements and if valid redirects to the Home Page. There is no database of members names so no authentication of eveyone's names
I would suggest you take a look at this article http://www.15seconds.com/Issue/020220.htm and this one too http://msdn.microsoft.com/en-us/library/xdt4thhy.aspx The first link (supported by the msdn docs) describes how to implement forms authentication with user credentials stored in the web.config along with an authorization rule that denies access to secure parts of the web application to non-authenticated users. If you get that setup correctly, no user (including Googlebot) would have direct access to web resources that require authentication. Any attempt at direct access to a secured resource will automatically redirect to the login page. After that it is a matter of implementing a robots.txt to prevent future indexing attempts by search engine bots on your secured content. You can also make requests for existing pages to be removed from Google's index using their webmaster tools, if you sign up for that.
It sounds like just the entry page is protected, so if you know the direct URL anyone can access them without a password. So if people with the Google toolbar installed login and view the pages, Google gets to know about the hidden URLs!
