How Search Spiders Work

If you’ve no love for hairy beasties with mouth parts and more than four legs, then the word ‘spider’ may leave you a little uncomfortable. In the world of search engine optimization, a spider is more like a harmless house spider that a huge, slavering tarantula and will be happy so long as you feed it keywords and meta tags.

A spider of this sort ‘crawls’ the World Wide Web, building and maintaining lists of sites with relevant words that a search engine will find useful. This enables the search engine to find particular documents and other information.

The Web is huge, so where does a spider start its journey? According to Google, it hit the 1 trillion unique URLs milestone in 2008, so a spider has its work cut out. The common starting point will be a list of servers that see a heavy amount of traffic. It will start with a popular site, and begin to catalogue the content of its pages whilst following each link around on the site.

Google’s spiders will not only look at the significant words contained within the page, but where they can be found. A company offering reputable SEO services will know that titles and subtitles, are given special consideration, whilst words such as ‘the’, ‘an’ and ‘a’ are ignored. Where a word may have more than one meaning, synonyms and context will help the search engine determine which one is correct.

With the competition to be noticed by a search engine so great, it’s obvious to see where SEO services can play an important part of your site.