getting crawled by bots

[nooksandniches]

Hi all! I was wondering how well your sites are being crawled by google and yahoo. I know with my last cart provider I got a lot of my pages indexed. I'm just curious how it is going for 3dcart users. We've only been live for a weekend so I am nowhere near being able to tell anything.

[GonzaloGil]

The crawling of the pages could take a few days, to a few weeks. Example:

http://www.google.com/search?hl=en&l... oksandniches

That link lists all your existing pages that where crawled by Google. If you see 99% of them are MonsterCommerce pages. There are some (just a few) Volusion pages. The only 3dC page right now i think its the home page. We need to give google a chance to spider through all, but as you can see, it can take a bit. We have 3 things you should take advantage of:

1) Google sitemap xml builder
2) Product page index
3) Category page index

2 & 3 are pages you can link of from say the very bottom of your page which link to a page that has every single product (and every single category in the category page index), so that a spider like Google can easily go through all the pages. If you don't link to these, the spidering process may take a little longer, as spiders will generally only go so deep before quitting. (Note that they will probably come back and resume next time, but this means that the job could take 2-3 rounds, instead of 1 if you give it a nicely formatted link list).

Let's see what happens in the next month or so, i would hope all those old Monster/Volusion URLS are removed and replaced with the new ones, but that's all up to Google , hopefully it likes you and comes back to see you often.

[Breeze]

1) Google sitemap xml builder --- Is this automatic or do we need to do something for this to work?

[GonzaloGil]

You just need to submit it to google.

Go to:

http://www.google.com/webmasters/sitemaps/login

When asked for the sitemap XML, tell it

http://yourdomain.com/sitemap.xml

The xml file is created dynamically, so no need to do anything to update the file.

[materialpleasures]

Hi Gonzalo,
I was just talking to someone about marketing and they recommended stiemaps.
How do I create the file they ask for:

"Create the file that we ask for (we will ask for a unique filename, such as GOOGLE11e5844324b7354e.html). The file itself can be empty. We will not read the contents of the file, only check to see that it exists. You can create this file in any text editor. Make sure this file has the exact name shown on the Verify tab. The filename is case-sensitive and must have an .html extension. Once you have created the file, upload it to your webserver to the location we specify. Generally, this is the root folder for your site.
Once you have uploaded the file..."

Do I just create a file in Word or something and upload to my ftp? That file name is not the one assigned to me, BTW)

Thanks,
Dana
Material Pleasures

[GonzaloGil]

You can open up notepad, leave it blank, and do File -> Save as:

Enter the file name.

Then FTP that file up to the root of the site (Inside the web folder).

That's pretty much it.

When asked whats the url to the sitemap its:

http://www.yourURL.com/sitemap.xml

[dynoJean]

I'm not very knowledgable about the whole seo duplicate content problems and am curious about this. Do both the asp and the html pages get spidered and if so, is there a way to prevent that from happening? Thanks for any insight here.

[GonzaloGil]

If you have HTML mode on, every link on the store links to an HTML page, so you should not see .ASP pages in the search engines. However, if by mistake you link to an ASP page, or Google spidered your site before you turned it in to HTML, these pages may show.

We've installed permanent redirects on asp pages for when you have HTML mode on, so next time they are spidered, they should see that the ASP page is no longer "available" and that it should look into the HTML page for the new content.

If you have both ASP and HTML content showing on the search engines, it is something you need to fix, as Google may eventually think of it as duplicate content, and 1 out of the 2 pages will get dropped, it may be the asp or the html, and so just to be sure, should this happen, let us know and we'll take a look at what's making the spiders see .asp pages and html pages.

[MagnetMan]

Originally Posted by GonzaloGil

We have 3 things you should take advantage of:

1) Google sitemap xml builder
2) Product page index
3) Category page index

2 & 3 are pages you can link of from say the very bottom of your page which link to a page that has every single product (and every single category in the category page index), so that a spider like Google can easily go through all the pages. If you don't link to these, the spidering process may take a little longer, as spiders will generally only go so deep before quitting.

I'm unclear about the product and category indexes. Can some one shed some light on how to link to them and where they are located?

Thanks

[megan]

Hi MagnetMan,

They're automatically created as www.yourdomain.com/product_index.asp and www.yourdomain.com/category_index.asp. You can go to Settings>Design>Site Content and create a new page that is a link to the product and category indexes and put that somewhere on your site.

Alternately, you can insert the links in your footer or navbars. I placed links in my footer so they're out of the way but still accessible to human and search engine visitors.