Announcement

Collapse
No announcement yet.

Bing - Homepage Excluded by Robots.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Bing - Homepage Excluded by Robots.txt

    According to Bing Webmaster tools, my robots.txt is disallowing access to my homepage as well as numerous other URLs. I can't fathom why. Also, according to bing webmaster tools, I've lost 1300 indexed pages in the last 2 days. Here are some examples of the URLs that bing reports as being restricted:

    HTML Code:
    http://www.harrysarmysurplus.net/ 
    http://www.harrysarmysurplus.net/chameleon-face-veil.html 
    http://www.harrysarmysurplus.net/nalgene-draft-3-liter-hydration-backpack-black.html
    http://www.harrysarmysurplus.net/tri-color-desert-m-65-field-jacket.html
    Here's the robots.txt content:
    User-agent: *
    Disallow: /AccountSettings.asp
    Disallow: /add_cart.asp
    Disallow: /checkout.asp
    Disallow: /crm.asp
    Disallow: /EmailaFriend.asp
    Disallow: /Email_Me_When_Back_In_Stock.asp
    Disallow: /error.asp
    Disallow: /giftregistry_home.asp
    Disallow: /login.asp
    Disallow: /myaccount.asp
    Disallow: /PhotoDetails.asp
    Disallow: /PhotoGallery.asp
    Disallow: /ProductDetails.asp
    Disallow: /recommendafriend.asp
    Disallow: /reviewhelpful.asp
    Disallow: /rssfeed.asp
    Disallow: /SearchResults.asp
    Disallow: /shipquote.asp
    Disallow: /ShoppingCart.asp
    Disallow: /ticket_new.asp
    Disallow: /view_cart.asp
    Disallow: /mobile/
    Disallow: /stats/
    Disallow: /3droi/


    Anyone have an idea of what might be wrong?
    Last edited by bzeltzer; 10-28-2011, 02:28 AM.

  • #2
    I would remove the following from your robot.txt
    Code:
    Disallow: /PhotoDetails.asp
    Disallow: /PhotoGallery.asp
    Disallow: /ProductDetails.asp
    Disallow: /recommendafriend.asp
    Elegant Weddings +
    www.elegantweddingsplus.ca
    www.elegantweddingsplus.com

    Comment


    • #3
      In the below example, I want to keep the Googlebot out of my /images/ directory but I also want to keep Yahoo!’s bot out of the /videos/ directory. In addition I want to keep ALL cooperating bots out of my /cgi/ and /tmp/ directories. As a final stipulation, I also want VodaBot (okay, I made this one up) to stay away from an image file called pointless.jpg which is in my /images/ directory.

      ------
      User-agent: Googlebot
      Disallow: /images/

      User-agent: yahoo
      Disallow: /videos/

      User-agent: *
      Disallow: /cgi/
      Dissallow: /tmp/

      User-agent: VodaBot
      Dissallow: /images/pointless.jpg
      ------

      Finally, you will note that while the fictitious VodaBot cannot access the file pointless.jpg it can access the rest of my /images/ directory ... but what if I wanted it the other way round? What if I wanted the excellently named VodaBot to NOT be able to access anything in the /images/ directory EXCEPT an image file called “meaning-of-life.jpg”? Then I would use an Allow statement in my robots.txt file.http://imagicon.info/cat/5-6/vbulletin-smile.gif

      Comment


      • #4
        Thanks for the info but does anyone know which item could be causing the hompage and examples listed above to be disallowed? I noticed that my hompage does actually appear in a bing search so maybe I should disregard their errors. I'm not sure.

        Comment


        • #5
          Originally posted by bzeltzer View Post
          Thanks for the info but does anyone know which item could be causing the hompage and examples listed above to be disallowed? I noticed that my hompage does actually appear in a bing search so maybe I should disregard their errors. I'm not sure.

          See my previous post, those are the ones that was/is blocking the bot from accessing your public pages.
          Elegant Weddings +
          www.elegantweddingsplus.ca
          www.elegantweddingsplus.com

          Comment


          • #6
            Originally posted by ElegantWeddings View Post
            See my previous post, those are the ones that was/is blocking the bot from accessing your public pages.
            Awesome. Thank you both for the information!

            Comment


            • #7
              I made the changes and bing webmaster tools is still reporting that 501 of my pages are being blocked. Some of the URLS include these. Does anyone have an idea of where I should go with this? Even though the homepage seems to be excluded from the crawl, my amount of indexed pages has gone back up to higher than it was before. Also, strangely, it says that there was a robots.txt exclusion/crawl error for my homepage yet it shows up in a search. Should I ignore this?

              Code:
              http://www.harrysarmysurplus.net/24-7-Tactical-Clothing_c_306.html
              Code:
              http://www.harrysarmysurplus.net/location-hours.html
              Code:
              http://www.harrysarmysurplus.net/

              Comment


              • #8
                robots.txt

                did you look at this

                https://support.3dcart.com/index.php...barticleid=191

                I'd open a support ticket as well

                Comment


                • #9
                  dead link

                  thanks for your reply. The link doesn't work because they updated their support portal. Do you remember the title of the article or something I can use to search for it with? I'd called support before but, as you probably know, getting a knowledgeable tech depends on the luck of the draw.

                  Comment


                  • #10
                    Remove:
                    Disallow: /ProductDetails.asp
                    David
                    David's Gifts and Things

                    Wholesale Gifts, Home Decorating, Jewelry and More

                    Quality, Selection, Value Always

                    The more you buy the more you save!

                    Comment


                    • #11
                      Originally posted by InsnWizard View Post
                      Remove:
                      Disallow: /ProductDetails.asp
                      I did that when ElegantWeddings suggested it but bing's still claiming that my homepage is blocked by robots.txt

                      Comment


                      • #12
                        Try first.
                        Remove:
                        Disallow: /mobile/

                        If Needed remove:
                        Disallow: /giftregistry_home.asp

                        You can also try to add a call for BingBot and "allow all".
                        If that cures your problem then start adding restrictions one at a time until you trigger a problem. A lot easier than chasing a ghost.
                        David
                        David's Gifts and Things

                        Wholesale Gifts, Home Decorating, Jewelry and More

                        Quality, Selection, Value Always

                        The more you buy the more you save!

                        Comment

                        Working...
                        X