Most search engines like Google send crawlers out to your site to see what pages you have created and then proceed to add them to their index. But what if they miss some pages or you make changes and they do not come back for another few months? How can you let the search engines know? The answer is through sitemaps.
What is a sitemap?
A sitemap is a file that contains the details of all the pages on your website. If you add a new page or modify an existing one all you have to do is edit your sitemap file to let the search engines know. In November 2006, Google, Yahoo, and Microsoft all agreed to support the sitemap protocol. Ask.com has now also agreed which means the four leading search engines are all on board.
How do I create a sitemap?
There are a number of ways to create a sitemap. My personal favorite is the GSiteCrawler. It is a free download that you can download to your computer. Once you run the program it will crawl your website and automatically create a sitemap for you according to the sitemap protocal (saving you tons of time and energy!). Because of all the new content we are regularly adding to my site, I run a sitemap every week. As of the writing of this post my sitemap currently contains 16,639 pages.
How do I tell the search engines about my sitemap?
You should create a link in your robots.txt file to your sitemap. All the major search engines will look through your robots.txt file so it is a quick way to let them know. Google and Yahoo have also created tools to allow you to let them know as soon as you make a sitemap change. This is not yet possible with Microsoft and Ask.com.
If you are not yet using sitemaps chances are the search engines are missing some of your pages and have an out of date record of what you have on your website. Make sure the hard work you put into creating quality website content is not wasted and tell the search engines that you have updated your pages by creating and submitting your sitemaps.
Evan Carmichael
How do you do it if your website is a blog for blogger.Com
I have no clue how to create a robot txt. file. ANy help would be great.
Hi Josh,
As far as I know you cannot create a robots.txt file on a blogspot domain. Have you considered publishing on your own domain name? You can use this feature through Blogger.
The good news is you can use the GSiteCrawler tool to create your sitemap and then instruct Google and Yahoo that it exists.
Evan Carmichael
Great article – every web site should have a Google sitemap as well as a Yahoo sitemap IMO. Thank you for sharing.
Hey Evan
So you are saying that I can still use the GSiteCrawler for Blogger. I just cant use the robot correct? Either way thanks for the help
Hi Josh – Yes, that’s correct.
I just ran the GSiteCrawler for your site and it came up with a sitemap of 145 pages of content – congrats on the work so far!
I’ve uploaded it to my server so you can see what it looks like:
http://www.evancarmichael.com/JoshH/sitemap.xml
You can do the same thing on your own by using the tool. You will just need to find a place to put the sitemap online so you can tell Google that it exists.
Evan Carmichael
Hey Evan
Thanks for the help you went beyond duty on this one. Now to just figure out how to get it uploaded to blogger and i’ll be in business. Thanks
Not a problem Josh – I don’t know if there is a way to upload a file to your blogger account but I’ll leave the sitemap up on the server for a while if you want to use it.
You may want to consider getting your own domain and hosting it off of blogspot. It will give you a lot more control over what you can do with your blog and you can still use Blogger’s interface to write the content – it will just upload to your own domain instead of blogspot. GoDaddy has some great deals from $3.99 / month.
Evan Carmichael
Hey Evan
Yeah I have been thinking about it lately. I have been using 1and1.com
I never knew about Godaddy thanks for the hook up. I may go with joshhoughton.com
thanks for all of your help.
I’m actually hosting with 1and1. When I started my site GoDaddy didn’t have dedicated servers available. 1and1 has been a good option so far. Getting your own domain gives you the flexibility to do whatever you want as well as some valuable real estate if you ever want to sell out.
Let me know as you progress!
Evan Carmichael
Yeah 1and1 has been great for my company website http://www.2ndchancehouses.com
Lots of great tools which have made it a great hosting company. I will keep you up to date and if you ever need to promote something online, just let me know.
I couldn’t keep up with Google… I would submit a sitemap and it was always 20-45 days behind. The site was still growing and it constantly showed errors. I eventually removed the site from Google’s webmaster tools. In 30 days it’s already improved in Google’s index.
I have a blog http://onepuzzleaday.blogspot.com. How can i edit Robot.txt file to allow search engines to index my site?
Same question goes to me, how to edit the blogger robot.txt so that it will index my blog(http://mycamjourney.blogspot.com).
If you’re constantly adding new material every day, is it possible to have multiple sitemaps or do you have to edit the one and resubmit to google each time? thanks.
Suman, Freddy – you don’t need to edit your robot.txt file to get Google on your site. If you’re not yet indexed, focus on creating and submitting a sitemap and then getting links to your website from other popular sites.
Traume – the best thing to do is resubmit the new sitemap on top of the old one. Google will usually pick it up pretty quickly.
Josh:
I think the blogs are very well optimized, so you don’t have to do anything.
hello, good site.