Announcement

Collapse
No announcement yet.

Yandex Spider Help

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Yandex Spider Help

    In SmarterStats, I noticed that the Yandex Spider is eating up my bandwidth. Is there any benefit to this spider or should it be blocked?

  • #2
    If you only sell within the United States you'd be OK blocking Yandex. I believe that duckduckgo.com is powered by Yandex, so you wouldn't show there if you block it. See if there is a way to limit requests on your site.

    Comment


    • #3
      If you have no business with buyers in Russia then Russian crawlers likely have no business using up your site's resources and bandwidth. Add to your robots.txt:


      User-agent: YandexBot
      Disallow: /

      User-agent: YandexDirect
      Disallow: /

      User-agent: YandexDirectDyn
      Disallow: /

      User-agent: YandexMedia
      Disallow: /

      User-agent: YandexImages
      Disallow: /

      User-agent: YaDirectFetcher
      Disallow: /

      User-agent: YandexBlogs
      Disallow: /

      User-agent: YandexNews
      Disallow: /

      User-agent: YandexPagechecker
      Disallow: /

      User-agent: YandexMetrika
      Disallow: /

      User-agent: YandexCalendar
      Disallow: /
      - Dean P. e-commerce and small business consulting

      Comment

      Working...
      X