Tools That Every Web Developer Should Use: Link Checkers

As a consultant, I am constantly asked to review and analyze websites that my clients or potential clients have to review performance and stats.  Over the years, I have found several tools known and unknown that I use in combination to get an idea of the overall health and performance of these sites.  As a developer, if you are not looking at these tools or similar tools, help yourself and get on board.

This week I am exploring link checkers and why they can be a useful tool in development.

Link Checkers

I’ve worked on sites that range from just a few pages to a few thousand pages and all of them have one thing in common; broken links.  In smaller sites, this isn’t much of an issue to test.  A tester can easily hit every page and check every link, but how do you do this if your site has a few thousand pages?

Why are broken links bad?

• Lower search engine rankings
• Frustrates users when they can’t get to the content that they want to see
• Stop web crawlers from crawling your site.

What causes links to become broken?

The most common cause of broken links is moving content around your own site.  When ever a page or image moves, all references to that address become broken.  Some CMS’s are good about automatically updating these links but others are not.

Other things that can break are links to external sources.  Linking to content on Joe Schmoe’s Web Emporium might seem like a good idea but chances are that a few years down the road, Joe’s site will get updated and rearranged.  Everything will come crashing down.  If referencing other sites is a necessity to make sure they are a big player so that the chance of the link breaking is smaller.

So broken links suck, how do I find them?

 Xenu’s Link Sleuth

This tool is a very convenient link checker that will completely crawl a website, find every link on it, and hit those links to verify those links exist.  When estimating and analyzing a website this tool is invaluable for finding out how many pages, images, JavaScript, and CSS files exist.  I use this all time to determine the number of pages a site has which helps determine how much content will need to be migrated.

After a site has been created, it is also very useful for finding broken links and dead pages because it will alert you to 404 and 500 errors.  This is very useful as smoke test to make sure deployments haven’t broken any pages.  This often the first tool I run when looking at potential clients’ sites.  It can also be impressive to users when you can hand them a list of broken links before working on it and let them know you can work on them before the project is done.

Where to get it? – Xenu’s Link Sleuth  The page itself looks a little sketchy, in my opinion, but nothing about the tool  is sketchy at all.

Broken Link Check

Some non-technical users do have issues using Xenu’s Link Checker, so I often send them to brokenlinkcheck.com to have a much nicer user interface without having to download a file.  The negative here is that they have a limit to the number of URL’s they will search.  I have also found that it does miss some links that Xenu’s Link Checker does find.

So I found all my broken links, how do I fix them?

Fixing broken links should be a priority especially if your SEO rank is very important to you.  There are several option on how to fix these broken links

• 301 Redirect Permanently – This server code physically tells web crawlers that link to this page should be updated permanently.  Search engines can then assign all of the SEO ranking from the old link to the new link  This should be done if you’ve changed the location a web resource you control has changed and you know the correct location.
• Delete the link – If the content no longer exist, you should delete the link and update your site map so that web crawlers can begin updating their indexes to remove that link to your site.
• Don’t do anything – If you don’t care about your SEO ranking, you can choose to not do anything.

Conclusion

These are just a handful of tools that I use on a frequent basis to help me get an understanding of a website’s performance.   I wanted to share these with developers just in case the thought never occurred to them.  Do you have any tools that I have not mentioned that have helped you out?  Please comment and share!  You can also contact me on Twitter or LinkedIn.