Announcement

Collapse
No announcement yet.

How can a bot get to the rss feeds when they are blocked?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • How can a bot get to the rss feeds when they are blocked?

    I am pissed at the Twengabot. I had them blocked previously and shut down the .rss feeds on my site but once again Twengabot changed their IP addresses and hit me hard today using over 72,921 KB of bandwidth before I blocked their new range.

    My question is how can they get access to the .rss feeds when in Settings>General>Store Settings I have "disabled the feeds" for months?

    Does anyone know how they have accomplished this attack? Since the feeds are disabled and they have broken in and used them this is an attack and it needs to be stopped.

    How can they be stopped from this activity?

    Thanks,
    Ken

  • #2
    Ken,

    How have you been able to identify them as being the Twengabot? I don't doubt this info, just wondering how I can do the same. Our site was seriously attacked over the weekend, with a ton of spam resulting in hundreds of incomplete orders and bogus information being added to several product pages.

    To say this is frustrating is putting it lightly. :mad:
    I too would like to know how to stop this.

    Comment


    • #3
      ScribeTime,

      Using the Smarter Stats, if you look in the reports it will tell you the IP addresses of all those that have come to your site. Then do a look up online for that IP address and in this case 108.59.0.106 comes up as Twengabot.

      If you have not subscribed to the 3dcart Smarter Stats then you would need some other 3rd party site stats running to find the IP addresses accessing your site. Or you need to subsribe to the Smarter Stats.

      They have a website that looks real and they claim they are "indexing" sites all over the world so they can include products from all stores in the world into their site for use by people in the EU. This is a lie. They are criminals and that is all they are. Using our bandwidth up and forcing us to buy additional bandwidth for no reason is a crime. They will not include any ones products in their site.

      They only have a few sites listed like Amazon and ebay with a few other smaller sites. When they attacked my site months ago I checked then to see if they had any of our products or even our site listed and they did not. I blocked their IP addresses that they used at that time. I checked again today and still they do not have our products or site listed. They have no intention of adding products or other websites.

      I have blocked their new IP range and it appears they have been stopped for now. I have contacted 3dcart as well about this attack. Because when the .rss feeds are blocked through the control panel on our back end no one should be able to get access to them. But this Twengabot has broken in somehow.

      I have no problem with legitimate search engines (google, bing, yahoo, the find and so on) indexing as many pages as they want. I'll gladly pay any extra fees for excessive bandwidth from legitimate sites. But Twengabot is a well known attack site.

      Do a search on these 3dcart forums and you will see they have been at this for a long time.

      Here is the Twengabot current IP address range: 108.59.0.1 to 108.59.0.255

      Go to Settings>General>IP Security and put this IP range in and save it. This will block them from your site until they change their IP range again.

      Hope this helps.

      Comment


      • #4
        Thanks Ken for the detailed response. I do have Smarter Stats and will do a more thorough investigation as well as take your advice. I know that this kind of activity is nothing new, but it's just a shame that we as business owners or webmasters have yet another "hat" to wear because of criminals who seek to do more harm than good.

        Comment


        • #5
          Since so many of us keep getting hit by twenga, I got it at the beginning of the month, maybe 3D can apply some force to them on our behalf. Unless they just like getting the overage fee.
          David
          David's Gifts and Things

          Wholesale Gifts, Home Decorating, Jewelry and More

          Quality, Selection, Value Always

          The more you buy the more you save!

          Comment


          • #6
            Here's more for twenga.
            You can add to the robots.txt if there bot pays attention.
            User-Agent: twengabot
            Disallow: /

            User-agent: twenga2.com
            Disallow: /

            User-agent: twenga.com
            Disallow: /

            User-agent: twenga.com
            Disallow: /
            David
            David's Gifts and Things

            Wholesale Gifts, Home Decorating, Jewelry and More

            Quality, Selection, Value Always

            The more you buy the more you save!

            Comment


            • #7
              We too are seeing a big rise in bandwidth usage lately, unfortunately we don't have Smarter Stats to see what IP's are using the most bandwidth.

              QUESTION: Is there any way to track this info in GA? I have been looking but didn't see any.
              We just started using Google analytics on our new 3Dcart site about 2 weeks ago and our old site had no way to enable GA (that's a whole nuther story!) so I am a rookie with GA.

              Comment


              • #8
                As far as I know GA won't see it.

                What store level do you have?
                David
                David's Gifts and Things

                Wholesale Gifts, Home Decorating, Jewelry and More

                Quality, Selection, Value Always

                The more you buy the more you save!

                Comment


                • #9
                  As far as I know GA won't see it.
                  That's what I thought, just wanted to be sure!

                  What store level do you have?
                  Currently we are at the "Starter" level but will be moving into the next level up shortly. If I understand correctly, the Smarter Stats start with the "Pro" level or can be purchased as an add-on ($15/month) for anyone.

                  Comment


                  • #10
                    Yeah, it's included with Pro. It's worth the money when your trying to see the traffic. Most of the time I just leave countries open but it's a great tool for seeing the ip's hitting the store. It actually has some fairly useful information in it.
                    David
                    David's Gifts and Things

                    Wholesale Gifts, Home Decorating, Jewelry and More

                    Quality, Selection, Value Always

                    The more you buy the more you save!

                    Comment


                    • #11
                      Originally posted by Barry View Post
                      We too are seeing a big rise in bandwidth usage lately, unfortunately we don't have Smarter Stats to see what IP's are using the most bandwidth.
                      This large bandwidth could also be from any rss feeds you may have going. Trust me if you are not using them disable them! They will suck down however much bandwidth you have available.

                      I am at the pro level now and don't really need it for the bandwidth since I turned off those rss feeds. I think if you want to do rss it would be better to find a third party app that could do it for you. The 3dcart feeds are strictly bandwidth hogs even if you have no feeds setup!

                      If you can't shut down your feeds because you are using them you'll constantly be over your bandwidth each month and constantly upgrading to the next highest level. Each upgrade will give you about a month of no overage fees. But the second month will be over and from there on it continues to climb each month.

                      As for seeing who is wasting your bandwidth other than the rss feeds you'll need some tracking like the Smarter Stats. It is worth it when you get hit like we did with Twenga.

                      Comment


                      • #12
                        Originally posted by InsnWizard View Post
                        Here's more for twenga.
                        You can add to the robots.txt if there bot pays attention.
                        User-Agent: twengabot
                        Disallow: /

                        User-agent: twenga2.com
                        Disallow: /

                        User-agent: twenga.com
                        Disallow: /

                        User-agent: twenga.com
                        Disallow: /
                        Thanks for these. I had all but one in my Robots.txt file. From doing a search online it seems the Twenga bot does not care about robots.txt files. This has been an ongoing problem for many years. On one forum I found someone claiming they were working for Twenga and they posted claiming there was a bug that caused it to disregard robots.txt files and supposedly they fixed it but after reading the posts after they claimed to have fixed it, it was obvious that it was not fixed.

                        Twenga bot is from China. That should tell you everything you need to know about it.

                        Sure wish there was a way to attack it back!!! Even kill it! Does anyone make Raid for web bots?

                        Comment


                        • #13
                          You can attempt to file a complaint with the issuer but if they don't do anything there probably isn't much we can do except ban the ip.
                          David
                          David's Gifts and Things

                          Wholesale Gifts, Home Decorating, Jewelry and More

                          Quality, Selection, Value Always

                          The more you buy the more you save!

                          Comment


                          • #14
                            watchdr,
                            Thanks for the info .. but I do have all RSS feeds disabled (at least I think so!!). These were all disabled in the control panel. Besides doing this, are there any feed codes that we should comment out in the templates?

                            About Twenga - I have also read ALOT from others that claim this robot doesn't follow ANY rules!

                            Comment


                            • #15
                              Barry,

                              From what tech support said disabling them is all we can do. Although I have taken the rss lines of code from my templates. I don't know if this will help when a bot won't follow the rules. Banning the IP's will keep them from getting to your site. Just know that Twenga is notorious for changing their IP addresses and attacking again down the road.

                              I continue to call this an attack because in reality this is what it is even though tech support is unwilling to see it this way. Anything that uses all your bandwidth for the month in a matter of a few hours for no reason is an attack. 1) it causes excess financial expense with no benefit. 2) uses bandwidth so real honest customers can't use the site. That is an attack.

                              They don't have to 'steal' anything to be attacking. Actually they are stealing. They are stealing my bandwidth allotment and robbing me of sales that I would have had during that time they were attacking my site.

                              This is just my belief.

                              Comment

                              Working...
                              X