Visual Studio 2008 (hide pages on Google Searches & avoid url access to folder)

Discussion in 'Visual Studio' started by Senor Engineer, May 2, 2014.

  1. Hi,

    I'm new to DiscountASP.NET. I need some help and thanks for educating me. I worked on actual web servers with Service Accounts access to the servers, however I'm trying to find info on secured data how to make it safe in DiscountASP.NET sites

    1. Our application is running on VS2010 on DiscountASP.NET, the site is not authenticated by login parameters, how do I hide the pages showing on Google Searches?
    2. If user truncates the ASPX page from and hit enter, the page lists folder directory with the active links to each folder on server. How can I restrict?
    Thanks a bunch for help,
     
  2. Thanks and any help is appreciated ! Basic questions are how can I avoid showing urls or pages on Google Searches (site is not designed for authentication - with ID and PWD). Also, when user removes ASPX page from url (at directory level) and hit enter, how do I redirect the page start page of url? Thanks a lot.
     
  3. Jmeeks

    Jmeeks DiscountASP.NET Staff

    In regards to your question about Google, the best way to prevent Google (or any major search engine) from listing specific pages in the search results is to use a "No Index" tag. Basically this is a meta tag that will tell the search engine bot that this page is not to be indexed for the search results. The tag should look like this:

    <meta name="robots" content="noindex">

    You can also do the same with the links on the page using the "No Follow" tag. You tend to find people use these two meta tags together because if you want the page not to be indexed, you normally don't want the bot following the links on the page either, which they would regardless of the "No Index" tag. If you want to add the "No Follow" you can just add it to the meta tag like this:

    <meta name="robots" content="noindex, nofollow">

    Another way to do this is to add a robots.txt file to your site. With the robots.txt file you can create a global rule set that search bots are suppose to follow when the hit your site. However you can easily block your whole site from being indexed without knowing it, so just tread with caution when looking at this option if you are only looking to block specific pages. You can view Google's write up on using a robots.txt file at https://support.google.com/webmasters/answer/156449

    Finally, if the pages are already in Google specifically, and you want them out now, you can always use the Remove URLs tool available in the Google WebMaster Tools. You would need to create a Google WebMaster Tools account to use this, but I would highly recommend it as it has a large number of valuable tools besides the Remove URLs tool. You can sign up for Google WebMaster Tools at http://www.google.com/webmasters/
     
    martino and RayH like this.
  4. Thank you so much for great help, you explanation and time really helps me a lot. So I have two options

    1. Open APSX page and add the "meta name" tag for list of pages not to appear in Google Searches (this can not resolve the problem of directory display of our site)
    2. Create robot.txt at root directory and specify what files and directories are to be blocked from Google Searches or any bot searches

    I will spend time and explore on second option. Again, thanks a bunch I really appreciate quick response. I'm sure I will learn a lot from this forum !
     
  5. martino

    martino DiscountASP.NET Staff

  6. Modification of Web.Config is also a great idea, to make it simple. I will spend time tonight and weekend and let you guys know of my test. Again, thanks to both for making me think out of box.
     
    martino likes this.
  7. I tried creating robots.txt and placed at root folder. I have the below in robots.txt. When I try in Google webmaster tool, looks like one url with "folder1" is blocked. However when I search in Google the file still showing. Probably I should wait for a day to see if robots.txt code works.

    User-agent: Googlebot
    Disallow: /*.pdf$

    User-Agent: Googlebot
    Disallow: /folder1/

    User-agent: *
    Disallow: /

    For example, I have the url below:
    http://www.someurl.com/folder1/file-name-1.pdf
    http://www.someurl.com/folder1/file-name-2.pdf
    http://www.someurl.com/folder1/file-name-3.pdf

    On web browser, If I type http://www.someurl.com/folder1/, I see three files with hyperlinks on webpage. How do I restrict ? I guess robots.txt avoids displaying Google Links (I can do test tomorrow at least 8 hours implementing), how do I restrict the directory level access? I use VS2008 on DiscountASP.NET.

    Thanks for the all the help, I made the progress. Also, for extra precaution, I started inserting the below after title end tag ( "</title>" ). Yet to publish the code (ASPXs only no code-behind) tomorrow as there are several files in our site directory.

    Pl. advise on directory level browsing restriction. Thanks again.
     
  8. I think avoiding Google Links works now (tried robots.txt). Thanks for suggestions.

    Now I'm down to the below. This is not Google Links realted .. FYI,

    For example, I have the url below:
    http://www.someurl.com/folder1/file-name-1.pdf
    http://www.someurl.com/folder1/file-name-2.pdf
    http://www.someurl.com/folder1/file-name-3.pdf

    On web browser, If I type http://www.someurl.com/folder1/ and hit enter, I see three files (file-name-1.pdf, file-name-2.pdf and file-name-3.pdf) with hyperlinks on webpage. How do I restrict directory level view on webpage ? I use VS2008 on DiscountASP.NET.
     
  9. martino

    martino DiscountASP.NET Staff

  10. Thank you for the help !!!! I truly appreciate it.
     

Share This Page