Announcement

Collapse
No announcement yet.

getting crawled by bots

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • getting crawled by bots

    Hi all! I was wondering how well your sites are being crawled by google and yahoo. I know with my last cart provider I got a lot of my pages indexed. I'm just curious how it is going for 3dcart users. We've only been live for a weekend so I am nowhere near being able to tell anything. :)

  • #2
    The crawling of the pages could take a few days, to a few weeks. Example:

    http://www.google.com/search?hl=en&l...nooksandniches

    That link lists all your existing pages that where crawled by Google. If you see 99% of them are MonsterCommerce pages. There are some (just a few) Volusion pages. The only 3dC page right now i think its the home page. We need to give google a chance to spider through all, but as you can see, it can take a bit. We have 3 things you should take advantage of:

    1) Google sitemap xml builder
    2) Product page index
    3) Category page index

    2 & 3 are pages you can link of from say the very bottom of your page which link to a page that has every single product (and every single category in the category page index), so that a spider like Google can easily go through all the pages. If you don't link to these, the spidering process may take a little longer, as spiders will generally only go so deep before quitting. (Note that they will probably come back and resume next time, but this means that the job could take 2-3 rounds, instead of 1 if you give it a nicely formatted link list).

    Let's see what happens in the next month or so, i would hope all those old Monster/Volusion URLS are removed and replaced with the new ones, but that's all up to Google , hopefully it likes you and comes back to see you often.
    ----------------------------
    Gonzalo Gil
    3dCart Support
    800-828-6650 x111

    Comment


    • #3
      1) Google sitemap xml builder --- Is this automatic or do we need to do something for this to work?

      Comment


      • #4
        You just need to submit it to google.

        Go to:

        http://www.google.com/webmasters/sitemaps/login

        When asked for the sitemap XML, tell it

        http://yourdomain.com/sitemap.xml

        The xml file is created dynamically, so no need to do anything to update the file.
        ----------------------------
        Gonzalo Gil
        3dCart Support
        800-828-6650 x111

        Comment


        • #5
          Hi Gonzalo,
          I was just talking to someone about marketing and they recommended stiemaps.
          How do I create the file they ask for:

          "Create the file that we ask for (we will ask for a unique filename, such as GOOGLE11e5844324b7354e.html). The file itself can be empty. We will not read the contents of the file, only check to see that it exists. You can create this file in any text editor. Make sure this file has the exact name shown on the Verify tab. The filename is case-sensitive and must have an .html extension. Once you have created the file, upload it to your webserver to the location we specify. Generally, this is the root folder for your site.
          Once you have uploaded the file..."

          Do I just create a file in Word or something and upload to my ftp? That file name is not the one assigned to me, BTW)

          Thanks,
          Dana
          Material Pleasures

          Comment


          • #6
            You can open up notepad, leave it blank, and do File -> Save as:

            Enter the file name.

            Then FTP that file up to the root of the site (Inside the web folder).

            That's pretty much it.

            When asked whats the url to the sitemap its:

            http://www.yourURL.com/sitemap.xml
            ----------------------------
            Gonzalo Gil
            3dCart Support
            800-828-6650 x111

            Comment


            • #7
              I'm not very knowledgable about the whole seo duplicate content problems and am curious about this. Do both the asp and the html pages get spidered and if so, is there a way to prevent that from happening? Thanks for any insight here.

              Comment


              • #8
                If you have HTML mode on, every link on the store links to an HTML page, so you should not see .ASP pages in the search engines. However, if by mistake you link to an ASP page, or Google spidered your site before you turned it in to HTML, these pages may show.

                We've installed permanent redirects on asp pages for when you have HTML mode on, so next time they are spidered, they should see that the ASP page is no longer "available" and that it should look into the HTML page for the new content.

                If you have both ASP and HTML content showing on the search engines, it is something you need to fix, as Google may eventually think of it as duplicate content, and 1 out of the 2 pages will get dropped, it may be the asp or the html, and so just to be sure, should this happen, let us know and we'll take a look at what's making the spiders see .asp pages and html pages.
                ----------------------------
                Gonzalo Gil
                3dCart Support
                800-828-6650 x111

                Comment


                • #9
                  Originally posted by GonzaloGil View Post
                  We have 3 things you should take advantage of:

                  1) Google sitemap xml builder
                  2) Product page index
                  3) Category page index

                  2 & 3 are pages you can link of from say the very bottom of your page which link to a page that has every single product (and every single category in the category page index), so that a spider like Google can easily go through all the pages. If you don't link to these, the spidering process may take a little longer, as spiders will generally only go so deep before quitting.
                  I'm unclear about the product and category indexes. Can some one shed some light on how to link to them and where they are located?

                  Thanks

                  Comment


                  • #10
                    Hi MagnetMan,

                    They're automatically created as www.yourdomain.com/product_index.asp and www.yourdomain.com/category_index.asp. You can go to Settings>Design>Site Content and create a new page that is a link to the product and category indexes and put that somewhere on your site.

                    Alternately, you can insert the links in your footer or navbars. I placed links in my footer so they're out of the way but still accessible to human and search engine visitors.

                    Comment


                    • #11
                      Why is this an option? ( Asp vs html ) Is one better than the other for the search engines or would the content dictate which would be the better option. What dictates which mode we would want to use?

                      Comment


                      • #12
                        Originally posted by megan View Post
                        Hi MagnetMan,

                        They're automatically created as www.yourdomain.com/product_index.asp and www.yourdomain.com/category_index.asp. You can go to Settings>Design>Site Content and create a new page that is a link to the product and category indexes and put that somewhere on your site.

                        Alternately, you can insert the links in your footer or navbars. I placed links in my footer so they're out of the way but still accessible to human and search engine visitors.


                        HELP - - - I too am not so good at all this but know the value of spiders, bots..etc.

                        I am interested in the alternately way for the footer or navbars but do not know what to do or how to.

                        I am also understanding that the sitemap is automatic but I do not know how to make it work....help please...thanks

                        Comment


                        • #13
                          Originally posted by hddesigncenter View Post
                          HELP - - - I too am not so good at all this but know the value of spiders, bots..etc.

                          I am interested in the alternately way for the footer or navbars but do not know what to do or how to.

                          I am also understanding that the sitemap is automatic but I do not know how to make it work....help please...thanks
                          The Footer can be edited at Settings->Design->Header and Footer from the admin panel. Just edit the Global Footer from there. You can edit directly in html or click the WYSIWIG button to edit like a word document if you do not know html. Whatever you place there will be on the bottom of every page of your site. This is also where you change your logo image as well.

                          For the sitemap, you should create an account in yahoo and google to add your sitemap to their systems so they can more easily traverse your website. The sitemap for your domain is at http://www.yourdomain.com/sitemap.xml

                          jhb

                          Comment


                          • #14
                            Should we still be doing these footer links?
                            Thanks
                            Steve
                            Steve Teske
                            Director of eCommerce
                            Hogan Walker LLC
                            www.GreenFunStore.com
                            www.GreenPartStore.com
                            www.FarmFunStore.com
                            www.HoganWalker.com

                            Comment

                            Working...
                            X