Earlier this month I wrote about Google’s Broken Link Filter and how important it is to make sure all the links on your website work. For internal links I recommended using the GSiteCrawler program and asked my blog readers to suggest a tool to check a site’s external links as I have not yet found a good option to use.

Enter blog reader Dan Chase who recommended trying out Xenu’s link Sleuth. I downloaded it last night and began running a report on my website to see how it works.

The Pros:

• The program is a free download that is easy to install and straightforward to use
• It crawls a website extremely quickly and will find all the links (internal and external) that you are listing
• It validates the links to make sure they exist and gives you a fairly easy to understand report on what is broken
• Unlike the Google Sitemaps tool it not only tells you which links are broken but also tells you on which page the broken link is listed so you can fix the problem
• It found a number of broken external links on my site (including to one of my sponsors – yikes!) that I am now able to fix

The Cons:

• Because the program works so quickly it seems to have trouble accessing pages that do exist on my site. As a result it tells me in the report that I have many internal broken links when they are actually fine
• Note: My case is not the typical one as I have almost 20,000 pages of content on my site that the program has to sift through. Because the program includes internal links it will have to analyze around 500,000 or so links!
• There is an option to only test your internal links but, as far as I can tell, no option to only test your external links (which is what I am looking for) so you end up having to test both your internal and external links

Overall:

• It is a easy to use, valuable tool to test your internal and external links
• My preference for internal links remains to be GSiteCrawler program as I find it much more reliable
• I will likely continue to use Xenu until I can find a tool that gives the option to show only external broken links and can handle the size of my website

Thanks for the suggestion Dan!

Evan Carmichael

The next in my series on Google Filters looks at the Broken Link Filter. The backbone of Google’s search engine is based on links so it is no wonder that the Broken Link Filter is given a lot of attention. A broken link is when you link to a page that does not exist. Broken links usually occur as a result of either making a mistake in the link (ie missing a letter in the url) or the page expiring (it was up and was taken down and your link is still active. There are two kinds of broken links: internal and external.
Internal Broken Links

Google adds pages from your site to its index by crawling your homepage. The homepage links to subpages which in turn link to other subpages. As Google searches through these subpages, its crawlers makes a list and adds the pages. If your pages are not properly linked or if the links are broken then Google will not add your pages to the list. You can always create a sitemap that will help you, but again sitemap software uses the same crawling technology. I wrote a few weeks ago about the GSiteCrawler to help you build a sitemap. It is a free program that will also tell you when there are broken links. I use it every week to see which of the internal links I have on my site have broken down.

External Broken Links

External broken links are when you link to outside websites and they take their page down. You could, for instance, link to a fellow blog, only to find a few weeks later that the blog has closed and your link is now invalid. I have yet to find a good tool that will tell you when your external links are broken so if you have found one, please let me know!

Why This Matters

The first reason why the Google Broken Link Filter is important is obvious – if you are linking to your own pages and the links do not work, the pages will not be added to Google’s index.

The second reason why the Google Broken Link Filter is important is not as well known. Google wants to show high quality pages in their search results. The last thing they want to have coming up #1 for important keywords is a site that has old, outdated, and broken links. If your site is full of links that do not work you will be penalized for not having a high quality website and will drop in the rankings. This is especially important for your key pages. Make sure that none of your homepage links are broken and keep an eye on any links from the high Page Rank pages that drive traffic to your website!

Evan Carmichael

Most search engines like Google send crawlers out to your site to see what pages you have created and then proceed to add them to their index. But what if they miss some pages or you make changes and they do not come back for another few months? How can you let the search engines know? The answer is through sitemaps.

What is a sitemap?
A sitemap is a file that contains the details of all the pages on your website. If you add a new page or modify an existing one all you have to do is edit your sitemap file to let the search engines know. In November 2006, Google, Yahoo, and Microsoft all agreed to support the sitemap protocol. Ask.com has now also agreed which means the four leading search engines are all on board.

How do I create a sitemap?
There are a number of ways to create a sitemap. My personal favorite is the GSiteCrawler. It is a free download that you can download to your computer. Once you run the program it will crawl your website and automatically create a sitemap for you according to the sitemap protocal (saving you tons of time and energy!). Because of all the new content we are regularly adding to my site, I run a sitemap every week. As of the writing of this post my sitemap currently contains 16,639 pages.

How do I tell the search engines about my sitemap?

You should create a link in your robots.txt file to your sitemap. All the major search engines will look through your robots.txt file so it is a quick way to let them know. Google and Yahoo have also created tools to allow you to let them know as soon as you make a sitemap change. This is not yet possible with Microsoft and Ask.com.

If you are not yet using sitemaps chances are the search engines are missing some of your pages and have an out of date record of what you have on your website. Make sure the hard work you put into creating quality website content is not wasted and tell the search engines that you have updated your pages by creating and submitting your sitemaps.

Evan Carmichael