Announcement

Collapse
No announcement yet.

Google Mobile Friendly & robots.txt Problem - need help

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Google Mobile Friendly & robots.txt Problem - need help

    We ran our site through the google mobile friendly test. For the most part, it's fine and mobile friendly. However we're getting an issue that we don't know how to handle.

    The message under the heading "How Googlebot sees this page" is:
    "This page uses 5 resources which are blocked by robots.txt."

    I don't care so much about 4 of them because they're various external tracking scripts. The one that DOES matter is our logo gif. Because of issues we were having with our SSL a little while back, we were advised to move our logo file to https:// That fixed the SSL problem. But now, we have THIS problem because our 3dcart robots.txt file blocks access to https by default (and rightly so I believe) and the logo is on every single page. So googlebot is having a hissy-fit about not being able to see an element that's on every single page of our site (confirmed this in webmaster tools too).

    Here is our regular HTTP robots.txt file:
    # Disallow all crawlers access to certain pages.
    User-agent: *
    Disallow: /checkout.asp
    Disallow: /add_cart.asp
    Disallow: /view_cart.asp
    Disallow: /error.asp
    Disallow: /shipquote.asp
    Disallow: /rssfeed.asp
    Disallow: /mobile/

    And here is our HTTPS robots.txt file:
    # Disallow all crawlers access to all pages. SSL
    User-agent: *
    Disallow: /

    Anyone have any "best practices" idea for how to solve this problem?

    Thanks.

  • #2
    To clarify, our site passed the mobile test (using a responsive theme by Shaun), BUT it had an error/warning as stated in my original post.

    We need to know, with respect to the message we got from the Google checker, how to account for the fact that in order to keep browsers happy about the SSL status of our site (and get the green address bar, lock symbol, or whatever), images that may display on https pages must be kept in the https section of the site.

    Comment


    • #3
      I had the same problem with several images on my site. Google kept telling me that there was no image or it was being blocked by the robots.txt, when it was not.
      Google support had no answers except generic proforma emails telling me to "contact my webmaster".

      It seems that the Googlebot will not crawl HTTPS urls. I deleted the S out of the image url and made it HTTP.
      The problem was resolved - at least for me! It took me weeks to figure this out. OK so I am a little sloooow.

      Comment


      • #4
        1) To use Google's Mobile Friendly checker, completely delete the contents of your robots files, run the checker, get your result, and the restore the contents.

        2) For HTTPS issue, read here: http://deanp.net/2011/05/identifying...curity-issues/

        a) tldr; never ever ever ever type
        Code:
        src="http:// . . . etc"
        anywhere in your store templates or email templates. Observing that HTTP content while on an HTTPS page will cause pop-ups, errors, problems, and messes.
        - Dean P. e-commerce and small business consulting

        Comment


        • #5
          DeanP You hit the nail on the head! Your article reminded us about relative URLs! That was a head-smack moment for sure, since we're very familiar with them and have used them extensively. But for some reason, when we had the original SSL issue way back and 3dcart support advised that we needed the site logo in HTTPS, we just did it without thinking it through completely and we hardcoded it to HTTPS in the header instead of using a relative URL. Just made the change, it works perfectly and there are no problems now.

          Can't thank you enough. Google's giving us gold stars now on the mobile checker. Kudos to YOU!

          Comment


          • #6
            That's good news. You're welcome. I didn't mention it but hardcoding HTTPS is fine. Be sure to set your robots file back to normal if you wiped it to clear the path for googlebot. Good luck
            - Dean P. e-commerce and small business consulting

            Comment


            • #7
              I need some clarification. I have NOT switched to an HTML5 template because too many things stopped functioning even with the same theme, but the HTML5 version. I have the mcommerce module enabled. The Google mobile friendly test was blocked by the SSl robot.txt. I removed the robots.txt and Google reported my site as "Awesome, Mobile Friendly"

              What I don't understand is whether to replace the SSL robots.txt or whether it should remain deleted for future crawls by Google checking for my mobile quality?

              Comment


              • #8
                Originally posted by JustPoppin View Post

                And here is our HTTPS robots.txt file:
                # Disallow all crawlers access to all pages. SSL
                User-agent: *
                Disallow: /

                Anyone have any "best practices" idea for how to solve this problem?

                Thanks.
                You need to delete the last line "Disallow: /", that is blocking all your HTTPS content at the top level. or, add the word "admin/" after the slash to just block your admin pages, like this:

                # Disallow all crawlers access to certain pages. SSL

                User-agent: *
                Disallow: /admin/


                Once I did that, the MCommerce plugin worked great with google and it passed the google mobile-friendly test


                Comment


                • #9
                  Regarding your logo, do not add anything to the URL. Do not use HTTPS or HTTP, simply start the URL //yourlogo.gif. The image will work for both secure and non-secure pages.
                  Joanne Barker
                  VS Web Design
                  [email protected]

                  Comment

                  Working...
                  X