Creepy Web Crawlers

They crawl their way into your system. They sneak around without you knowing. You never feel them coming. They are evil. Click here to be frightened beyond belief.


Alright that was blown just a bit out of context, and has nothing to do with this post. I just hate spiders. Recently, I met with a co-worker to discuss how we were going to attack a specific category SEO wise. I nearly finished the entire conversation when they finally said “You keep saying spiders are on my webpage, I haven’t seen any, but there were some cobwebs behind my computer.”

From her perspective I can understand her confusion in not knowing what they do, I didn’t explain it so I was at fault here. Making robot spiders a reality just never seemed to cross my mind (and I now realized how terrifying that sounds). After shuddering at the visual of that, I explained what spider crawlers are as follows – Crawlers are bots that sweep through a website in the blink of an eye while feeding information to search engines to give your website relevance.

What Does It Do?

A web crawler scans through websites providing search engines with a high amount of information. It tracks links, content, site architecture and relevance towards your websites target. This directly influences how you are ranked on search engines, making crawlers a very important part of organic search. That being said websites are built for great user capability, not to appeal to crawlers. Optimizers make the websites more crawler friendly and help eliminate bad practices that crawlers see as a violation.

How Do They Work?
Crawlers are basically information gatherers. They are constantly surfing the web in search of new information and are extremely up to date. They gather a websites information through internal and external links, <meta> tags, <h1> tags, <title> tags, and most importantly content. They can also see how popular the websites are to gain a better understanding of relevant and user-friendly content

Do I Need Them?

Yes spiders suck, I believe both species have come to an understanding that we don’t like them, they don’t like us. Unlike those terrifying 8-legged creatures, website crawlers are a good thing. If you want to be indexed and increase your organic traffic, being crawled is a positive.

I Hate Spiders Too, Can I Step on Them?

This was actually asked to me. This was my exact response. “Yes, but if you have the technology to virtually place your foot onto a program and crush them, I would like more information for a potential investment.” In actuality, using a robots.txt file actually gives the crawlers directions for the website. If you don’t want them sneaking around there are specific instructions you can place in your robots.txt, for more information and instructions list click here.

Now that you have a better understand of crawlers, you know how search engines gather information on your website. They check tags, and content, as well as links. So be sure to have constant monitoring of your Google Webmaster tools and analytics to see if you are being penalized or have duplicate content that Google frowns upon.

Just for kicks:


If you have any more questions feel free to comment, or a great spider destroying story email me directly, I would love to hear it.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *