Sitemaps

In order for search engines to see the huge number of "hidden" pages that it would otherwise be unable to find (such as all of the country-to-country lookup combination's my site offers), the use of a sitemap is going to be important. The basic idea of an XML sitemap is such:

Google introduced Google Sitemaps so web developers can publish lists of links from across their sites. The basic premise is that some sites have a large number of dynamic pages that are only available through the use of forms and user entries. The sitemap files can then be used to indicate to a web crawler how such pages can be found.
Google, MSN, Yahoo and Ask now jointly support the Sitemaps protocol.

Since MSN, Yahoo, Ask, and Google use the same protocol, having a sitemap lets the four biggest search engines have the updated page information. Sitemaps do not guarantee all links will be crawled, and being crawled does not guarantee indexing. However, a sitemap is still the best insurance for getting a search engine to learn about your entire site.

XML sitemaps have replaced the older method of "submitting to search engines" by filling out a form on the search engine's submission page. Now web developers submit a sitemap directly, or wait for search engines to find it.

A couple sites explain the importance of these sitemaps from a different perspective. In my case, it's clear I need to use these sitemaps in order for search engines to find what's hidden behind the front page.

Step 1 - Build the Sitemap

Studing Google's handy XML Sitemap Overview, I came up with a rough idea of how I need to make the final XML look:

...

The code was relatively simple, simply iterating through a 2x2 matrix ouputing every combination of country-to-country while ignoring the case where the to and from country are the same. After generating the entire list (which contained over 54,000 URLs) I learned that each sitemap can have no more than 50,000 URLs. This meant I had to create a sitemap index, which then points to a number of sitemaps each containing less then the 50,000 max:


Depending on your site, the process of creating this sitemap could be rather different. As my site is very simple, there are only two types of URLs (the front page, and each of the lookup pages). There are a number of tools out there that help this process, especially if all of your pages are available to the public. If not, you'll need to generate the sitemap as I did, straight from the code.

One last piece of advice is to run the sitemap through a validator. Here is a good one.

Step 2 - Publish the sitemap

Knowing that Google will be the primary traffic driver, and since Google came up with the sitemap concept, I went straight to Google Webmaster Tools and submitted my sitemap:


You can see that Google quickly recognized the sitemap, and the number of links that it references.


Now it's a waiting game to see if all of these links get indexed (so far just 1 URL, presumably the front page, has been indexed).

The other way to let search engines know about your sitemap is to include it in your robots.txt file:

User-Agent: *
Allow: /
Sitemap: /sitemapindex

This way, Google isn't the only one that takes advantage of this hard work, and hopefully I start to see some traffic from the other engines in the near future.

Update: I've discovered the Yahoo has a place to submit your sitemap, so I've gone ahead and done that:


Update 2: MSN and Ask.com have some backdoor ways to submit your sitemap. Better then nothing:

http://api.moreover.com/ping?u=http://www.simplecountrycodes.com/sitemapindex

http://submissions.ask.com/ping?sitemap=http://www.simplecountrycodes.com/sitemapindex.xml