Questions about sitemap.xml

August 23, 2008 2:56 AM

Sitemaps seem like a huge pain in the butt.

I have been making a sitemap.xml file, using www.xml-sitemaps.com, and informing Google of its existence.

But it seems such a pain in the ass! I am adding new web pages almost every day, and remaking the sitemap file and retelling google about it adds a big extra chunk of time to the process. So I guess my questions are:

• Is a sitemap so necessary? What if I don't have one?
• What if I don't update my sitemap, and there are many new pages with interesting new content that don't get included in the sitemap for a while?
• Is there some even more automagical way to have my sitemap updated, such that as soon as a new page is created the sitemap.xml file gets updated as well?
• What question have I not asked about using sitemaps that I should?

Thanks in advance for anyone willing to share wisdom on this issue.

posted by Meatbomb to computers & internet (10 comments total)

That's a waste of your time imho.

As far as I understand it:

As long as all your pages are linked goole will follow those links and index. Sitemaps may help speed up the process. It's worthwhile for us as we have an inhouse cms and update the xml file automatically when a client adds a new section to their site. But if you are doing it manualy that seems like a pain that isn't worth your time.

Google will still index as long as it can follow links to the new content

Depends on how you create your pages... If its through an admin you could update the xml as you create a new page, maybe get an xml editor and edit the xml file directly if that's not already what you are doing.

I could be wrong in all that but I certainly don't see the sitemaps being all that. You might better spend your time optimising the content for search engines.

posted by twistedonion at 3:16 AM on August 23, 2008

Oh, and here is another thing: if I am just overwriting the file called "sitemap.xml", do I need to go tell Google? Or will it just come and look at the new one each time?

And thanks, twistedonion, it is sounding like maybe this is a potential waste of time that I can ignore form now on anyways... any other opinions?
posted by Meatbomb at 3:42 AM on August 23, 2008

If you look in the Google Webmaster tools, if I recall correctly, they will tell you the last time they checked out your sitemap file. Looks like they checked mine a few days ago and I haven't even looked at that page for months. I stopped dealing with them for my sites because

1. I was using an auto-generator that was a WordPress plugin for a while that did them automagically which was fine until it broke
2. I didn't notice there was any huge difference really. Once Google knows you have a part of the site that gets updated a lot -- like a blog -- they seem to hit it more frequently. If you notice they hit AskMe what seems like every ten minutes; new questions are on Google almost as soon as they are here.

So, if you made some huge new section it might be worth a re-write and a ping to Google, but otherwise I don't see it as being as big a deal as, say, getting like-minded sites to link to you so that you seem more "relevant" and plugged in and not just a little island of content. This is just my $.02 as the owner of a few websites, I don't know much about how Google really works except through observation.
posted by jessamyn at 6:31 AM on August 23, 2008

Sitemaps are supposed to be automatically generated off the database, really. What software are you using? Wordpress? Something like that?
posted by Leon at 6:40 AM on August 23, 2008

It can be a pain if you're publishing sites without a CMS. Many CMSs have the ability to automatically generate and update it as you add pages. In my MODx site, I have defaults in place to automatically add pages, but I can also change those defaults at the time of page creation, such as frequency, priority, if the page should appear at all, etc.
posted by juiceCake at 8:33 AM on August 23, 2008

I generate mine automatically from the CMS (as juicecake suggests). However, it would be braindead simple to generate one using your favorite scripting language- all you need to do is recursively scan your directory tree and spit out the correct XML. in Perl, this would be File::Find, although a quick trip through CPAN showed a bunch of candidates to automate this.

I saw an immediate uptick in people hitting the site through search engine keywords once I installed the sitemap. I also saw additional Yahoo and MSN traffic, as I installed a reference to the sitemap in robots.txt, and they saw it and used that for searching.

It depends on the kind of site, of course- mine is a big ecommerce / product catalog thing, so the backing data can change pretty frequently.

It's probably worth a few hours of work to automate, but not worth redoing by hand every week. If you can't script it, don't do it.
posted by jenkinsEar at 10:46 AM on August 23, 2008

Just do it from time to time. No need on every update.
posted by wackybrit at 2:12 PM on August 23, 2008

It sounds like you upload HTML pages you've created, is that right? How are your pages hosted? If you have a UNIX web host that provides command line access and lets you run scripts with cron, you should be able to use Google's sitemap generator to automate the whole process. Have it run once a day or something and it will add any new files.
posted by Good Brain at 2:44 PM on August 23, 2008

I really appreciate it when a big site, or a site with many possible audiences, has a site map. And then I want it to be purely alphabetical, not pre-sorted in any way. Quicker to guess.
posted by Riverine at 5:35 PM on August 23, 2008

Riverine: OP is talking about the sitemap.xml, kind of a different thing.

But this is absolutely why dynamic, database-based or CMS systems are better for this kind of thing.
posted by Deathalicious at 5:45 PM on August 23, 2008