Announcement

Collapse

Posting Rules

Posting Rules
This forum was created to help 3dcart users share tips on getting the best out of the service, offer knowledge and experiences about e-commerce in general, and to promote a positive environment in which our merchants can request/dispense advice regarding best practices.

The forum was not created for 3dcart users to submit requests for customer support. Any support related thread will be removed and a support ticket will be placed on your behalf. If you have technical difficulties please visit http://support.3dcart.com.

The 3dcart forum is also not intended to be a medium in which to express dissatisfaction about 3dcart's service, policies or its management and staff. Threads of this nature will also be removed from the forum. If a support issue with your 3dcart store was not resolved in a satisfactory manner; or if you feel that a particular 3dcart offering/policy is not to your liking, please let us know by emailing [email protected] so we can address your concerns directly.

Again, our purpose is to cultivate a positive learning environment for our merchants rather than platform in which to express grievances.

Very Important Rules:
* Never give away your store administration login/password in this forum.

Should you need technical help, please let 3dcart staff help.
Also, please be aware that 3dcart staff will NEVER need to ask for your 3dcart related passwords.

General Rules:
* While debate and discussion are acceptable, we will never tolerate rudeness, insulting posts, personal attacks or inflammatory posts. Our decision is final in these matters.
* Please refrain from posting meaningless threads, one word (or short) nonsensical posts, or similar postings.
* Multiple or repeated posting in order to increase your post count is not allowed.
* Advertising, spamming and trolling is not allowed. This includes using the forum email and Private message system to spam other members.
* We also do not allow posts that are sexual in nature.
* Please wait at least 24 hours before bumping posts.
* Discussion of illegal activities such as software and music piracy and other intellectual property violations are not allowed.
* Each member is allowed one login account. Registering with multiple accounts is not allowed.
* We reserve the right to remove offensive posts without notice.
* We also reserve the right to remove postings which are in violation of our specified posting rules
* Also, while these rules cover most common situations, they cannot anticipate everything. Consequently, we reserve the right to take any actions we deem appropriate to ensure these forums are not disrupted or abused in any way.
* You cannot post any affiliate or referral links, or post anything asking for a referral. Such posts will be subject to removal.
* You cannot post advertisements or notices for contests.
* We also reserve the right to ban anyone who willfully violates the forum rules, as access to our support forums are a privilege and not a right. A banned customer is still entitled to support from the support system, but will not be able to participate in the forum.
See more
See less

Robot.txt file blocking all products

Collapse
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
  • #1

    Robot.txt file blocking all products

    Yesterday I restored the default robot.txt file on my website. (It's on a Core theme.) I then added the code for google bots so that the robot.txt file looks like this:

    Sitemap: https://www.-----/sitemap.xml

    # Disallow all crawlers access to certain pages.
    User-agent: *
    Disallow: /checkout.asp
    Disallow: /add_cart.asp
    Disallow: /view_cart.asp
    Disallow: /error.asp
    Disallow: /shipquote.asp
    Disallow: /rssfeed.asp
    Disallow: /mobile/

    User-agent: Googlebot
    Disallow:

    User-agent: Googlebot-image
    Disallow:

    User-agent: AdsBot-Google
    Disallow:

    Now, my Google Merchant Center account shows that my products are disapproved because "Desktop page not crawlable due to robots.txt" and
    "Mobile page not crawlable due to robots.txt." In addition, if I use the URL Inspection tool in the Search Console, it shows "URL is not available to Google,.. It cannot be indexed.... Blocked by Robot.txt."

    What is wrong with my Robot.txt file?


    Tags: None
  • #2
    I had that problem briefly as well - in maybe an error with https vs http. Go to Google Search Console and check bot access. https://www.google.com/webmasters/tools/googlebot-fetch

    I have also found that 3dcart servers can take up to 24 hours to reflect changes with Bing and Google.

    Comment

    • #3
      Thanks. I think that maybe the new robots.txt file hasn't been updated by the servers because if I test it in Google, it shows the default robots file without the google bot code::
      Sitemap: https://www.----.com/sitemap.xml # Disallow all crawlers access to certain pages. User-agent: * Disallow: /checkout.asp Disallow: /add_cart.asp Disallow: /view_cart.asp Disallow: /error.asp Disallow: /shipquote.asp Disallow: /rssfeed.asp Disallow: /mobile/ I guess I'll wait a little longer to see if the updated robots.txt file allows my product pages to be indexed.

      Comment

      • #4
        Try clearing the site cache in general settings

        Comment

        • #5
          I've done that multiple times since yesterday. Thanks though.

          Comment

          • #6
            Your robots.txt file is located at www.site.com/robots.txt so you can check it there. If it's not updated, you should be able to edit it directly using FTP access. It's located in the top level "web" folder. There's also another file www.site.com/robots_ssl.txt that needs to be edited

            Comment

            • #7
              All is good. It just took a while for the robots.txt file to be updated. Thanks for your help.

              Comment

              Previous template Next
              Working...
              X