Announcement

Collapse
No announcement yet.

Googlebot Bandwith Usage

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Googlebot Bandwith Usage

    I have noticed a huge spike in my bandwith this month so I signed up for the active stats today -- It appears that googlebot is my biggest BW eater?? 66.249.72.198 5/18/2011 8:00:14 PM - 5/20/2011 3:49:21 PM
    Hits 1,302
    Page Views 1,255
    BW 58,557

    This is just the usage for less than 48 hours and seems VERY excessive???

    Is this the same for everyone else --- Do I need to change something?

  • #2
    Originally posted by 3dcristywisty View Post
    I have noticed a huge spike in my bandwith this month so I signed up for the active stats today -- It appears that googlebot is my biggest BW eater?? 66.249.72.198 5/18/2011 8:00:14 PM - 5/20/2011 3:49:21 PM
    Hits 1,302
    Page Views 1,255
    BW 58,557

    This is just the usage for less than 48 hours and seems VERY excessive???

    Is this the same for everyone else --- Do I need to change something?

    I had this problems since the last few days. the bot actually adds items to the cart and abandons it. updated the robots.txt to specifically for the googlebot agent to disallow add to cart but still no luck.

    User-agent: googlebot
    Disallow: /checkout.asp
    Disallow: /add_cart.asp
    Disallow: /view_cart.asp
    Disallow: /error.asp
    Disallow: /shipquote.asp
    Disallow: /rssfeed.asp
    Disallow: /mobile/
    Disallow: /admin/

    I hope someone will have a solution, support suggested the robots.txt as well,it seems google is ignoring it.
    Elegant Weddings +
    www.elegantweddingsplus.ca
    www.elegantweddingsplus.com

    Comment


    • #3
      I just went in and checked and Googlebot has eaten another 10k in bw and 400 more page views!

      I called tech support and their suggestion was to close my store temporarily??? Doesnt that kind of defeat the purpose of having a store -- If its not open then I cant sell anything to pay the 3d cart fee??

      support suggested the robots.txt as well,it seems google is ignoring it
      Tech suport did mention that if we modify the robots.txt file to disallow a certain bot and/or restrict the bot -- and it doesnt listen -- then we need to put in a support ticket so they can figure out why the robots.txt file is not working. Wonder if they will credit the extra BW fee if they find an problem with the robots.txt file?

      At this point for me the easiest thing was to restrict the ip address. Not an ideal solution... but I cant afford to pay and extra $5 every couple days for the BW the Googlebot is eating up. :mad:

      Comment


      • #4
        Does anyone else notice an increase in spider traffic? I assume its a good thing -- so my site gets indexed -- but at this point I cant afford to stay in business with 3d cart and the overages I am paying for the spider traffic!

        I even went so far as to block the ip address that were hitting my site -- including google -- just to try and slow it down a bit... but its not working?? Somehow the spiders are getting thru the IP block -- including google!

        Any suggestions would be appreciated.

        Comment


        • #5
          If you use Google's Webmaster Tools, look under Site Configuration and select settings. You will be able to set a custom crawl rate for googlebot. This may be more helpful because it is probably unwise to block googlebot altogether.
          Luxlife

          Comment


          • #6
            Thanks for the suggestion Lux. I have lowered the crawl rate... hopefully this will made a difference!

            Comment


            • #7
              Originally posted by 3dcristywisty View Post
              Thanks for the suggestion Lux. I have lowered the crawl rate... hopefully this will made a difference!
              Let me know if this fixes your issue as I would like to try this myself. Thank you!
              Jimmy

              Comment


              • #8
                Here's what we're using for our robots.txt file:

                User-agent: *
                Disallow: /AccountSettings.asp
                Disallow: /add_cart.asp
                Disallow: /checkout.asp
                Disallow: /crm.asp
                Disallow: /EmailaFriend.asp
                Disallow: /Email_Me_When_Back_In_Stock.asp
                Disallow: /error.asp
                Disallow: /giftregistry_home.asp
                Disallow: /login.asp
                Disallow: /myaccount.asp
                Disallow: /PhotoDetails.asp
                Disallow: /PhotoGallery.asp
                Disalow: /ProductDetails.asp
                Disallow: /recommendafriend.asp
                Disallow: /reviewhelpful.asp
                Disallow: /rssfeed.asp
                Disallow: /SearchResults.asp
                Disallow: /shipquote.asp
                Disallow: /ShoppingCart.asp
                Disallow: /ticket_new.asp
                Disallow: /view_cart.asp
                Disallow: /mobile/
                Disallow: /stats/
                Disallow: /3droi/

                Comment


                • #9
                  Thanks Piaf. I will make those changes as well and see if it makes a difference.

                  From what it appears the most common file that googlebot is accessing is my my image file -- which are obviously a decent file size. I started uploading compressed images via ftp once I noticed this huge bw usage but it seems that is now also contributing to my higher than normal bw. I *thought* uploading via ftp didnt affect bw BUT IT DOES!! So now I am not sure what I am going to do? The solution I have been given by 3d cart is for me to pay for a more expensive package. Ya, not fond of that idea.

                  @Jimmy, It does appear by adjusting the crawl speed that it has cut down on the number of pages and the bw usage. Now my biggest bw sucker is Comcast!!

                  Is there a way to block entire countries from accessing my site or crawling it? Like if the ip address is for a country, say like Japan, that I dont ship too, they wont be wasting my bw?
                  Last edited by 3dcristywisty; 05-28-2011, 11:22 PM.

                  Comment


                  • #10
                    3dcristywisty,
                    I have been following this thread, and I have to admit, I am a bit concerned about my new BW usage compared to my old!! Our current cart is about $30 per month but they allow 5GB for BW. We get aprox 20,000 to 30,000 "unique visitors" per month according to our current stats. So far, as we know, we have not gone over (to the best of our knowledge!?!?! -I will explain!).
                    NOTE: I will not recommend our old cart which is why we are moving - I will explain ....

                    1. To be honest, we are not sure the stats are accurate. I have not found any way to add Google Analytics to the site, so I can't compare.

                    2. No where in the system can we view our BW usgae. According to our current cart, "when we run out of BW, our store quits working until the following month"!!!! What kind of answer is that? We have an auto site monitor to check the uptime and for the past 8yrs, we have not seen any sugnificant outages that would suggest we were "cut off" for the month.

                    3. I also understand (according to the stats) that much of our traffic is visitors on the site for less than 1min - so they obviously are not using much if any BW. This "assumes" that our stats are correct!

                    I know this is asking a lot, but would you mind telling me what your "old" unique visitor count was per month compared to your "new 3DCart"? I understand if this is asking for too much info, but any insight would be greatly appreciated.

                    I *thought* uploading via ftp didnt affect bw
                    Me too. In fact I was actually corrected by someone from 3DCart that using ftp would not use BW!
                    Last edited by Barry; 06-08-2011, 12:50 PM.

                    Comment


                    • #11
                      Barry,

                      You are leaving 3dcart? Can you contact me directly with what cart you are moving to? I haven't found any that have the features of 3dcart.

                      I am also not thrilled with the bandwidth issues here. In fact for the past few months it has not been working at all on my cart so I can't know if it is even coming close to going over or not. Tech support is of no help because they don't know anything about BW usage or how it resets and when and so on.

                      My BW meter has been stuck for the past 3 weeks and here we are 8 days into the new month and still it has not reset or shown any difference. It just doesn't work.

                      Thanks,
                      Ken

                      Comment


                      • #12
                        Ken,
                        My fault, we are not leaving 3DCart! After re-reading my last post here, I realised my wording was misleading so I edited it. Sorry about the confusion AND sorry to hear that you are having problems. I'm surprised tech wasn't helpful.

                        Comment


                        • #13
                          Hi Barry,

                          Ok, Thanks. have a great day!

                          Ken

                          Comment


                          • #14
                            whenever i see a thread about bandwidth i try to get involved. we too have had issues about how it calculates bandwidth and what is actually using it and I have never had a straight answer from anyone on the forums or tech. the best suggestion is to either pay the overage or get a higher package...seems a little fishy there. also be careful not to let this thread get too hot, as in the past, other 3dcart members start posting their bandwidth problems and the thread gets closed and no answer is given..

                            Comment


                            • #15
                              I have posted heavily on these forums about bandwidth in the past and have come to one big conclusion: TURN OFF THE RSS FEEDS if you want to conserve bandwidth.

                              Since we have every product in our store eligible for the Featured Products area on our home page, that meant every product was published daily to the Featured Products RSS feed. When we cut that feed off (via our frame page, because we still use the RSS feeds for products on sale and new products) , the bandwidth savings was immense (approx 50% reduction) due to our particular scenario of having all products eligible as a Featured Product.

                              I did a lot of testing (and posting!) regarding bandwidth using our Smarter Stats data, so the information is out there. :)

                              Comment

                              Working...
                              X