Time for everyone to go secure – https?

In October of 2011 the search king Google went secure.
In 2012 – 2013 Safari, Mozilla, and Bing did the same.
In January 2014 Yahoo finally joined the club to make 99% of the worlds digital information completely secure.

Well, that’s awesome! But not for SEO’s. Secure searches mean keyword data, referral data go out the door and that leaves us in the dark with a flash light trying to find our way. With Yahoo going secure they’ve taken away our batteries and now we only have an empty tool to defend ourselves. Nevertheless we will prevail!

I work for an enterprise level ecommerce website and the news of losing data comes at a large loss. It’s hard because our SEO numbers have been on the uptrend without paying much attention to Yahoo.

So why does this matter you ask?

Well ~6% of our traffic comes from Yahoo and with the new secure search, our Yahoo referral data will become direct load (according to omniture). In short direct load is someone coming directly onto our website without coming through any type of marketing channel. So that box in your web browser, someone types in the exact domain address, that equals direct load.

This means we lose 6% of our SEO traffic to direct load by the time Yahoo goes 100% secure (which is aimed at March 2014). The problem is with high level reporting direct load is such a large number that 6% of our traffic attributed to it doesn’t look like much of a dent. When you get more granular you notice a little dip in our allocated traffic which might not look to good.

It’s ok don’t panic. I’ve got a solution.

And I found it on Danny Sullivan’s post on searchengineland.com here I knew about most of the information before getting this article but there was one quote/tweet thread I read that I found very useful.

By the way, when it comes to searches that lead to secure servers, Yahoo appears to be following standard protocol and passing along full-referrers. However, as most sites are not secure sites, most publishers won’t receive this information.

Bam. There it is. By making your website secure itself (https) Yahoo will pass referral data because it follows standard secure server protocol’s. The twitter thread that explains most of it is located here.

So there you have it. search referral data becoming direct load can be offset by making your own website secure. Might even be some good PR around it (that’s a shout out to my old manager) 🙂

If you’d like to find out more feel free to contact me, or tweet me. I’m here all day folks

Creepy Web Crawlers

They crawl their way into your system. They sneak around without you knowing. You never feel them coming. They are evil. Click here to be frightened beyond belief.


Alright that was blown just a bit out of context, and has nothing to do with this post. I just hate spiders. Recently, I met with a co-worker to discuss how we were going to attack a specific category SEO wise. I nearly finished the entire conversation when they finally said “You keep saying spiders are on my webpage, I haven’t seen any, but there were some cobwebs behind my computer.”

From her perspective I can understand her confusion in not knowing what they do, I didn’t explain it so I was at fault here. Making robot spiders a reality just never seemed to cross my mind (and I now realized how terrifying that sounds). After shuddering at the visual of that, I explained what spider crawlers are as follows – Crawlers are bots that sweep through a website in the blink of an eye while feeding information to search engines to give your website relevance.

What Does It Do?

A web crawler scans through websites providing search engines with a high amount of information. It tracks links, content, site architecture and relevance towards your websites target. This directly influences how you are ranked on search engines, making crawlers a very important part of organic search. That being said websites are built for great user capability, not to appeal to crawlers. Optimizers make the websites more crawler friendly and help eliminate bad practices that crawlers see as a violation.

How Do They Work?
Crawlers are basically information gatherers. They are constantly surfing the web in search of new information and are extremely up to date. They gather a websites information through internal and external links, <meta> tags, <h1> tags, <title> tags, and most importantly content. They can also see how popular the websites are to gain a better understanding of relevant and user-friendly content

Do I Need Them?

Yes spiders suck, I believe both species have come to an understanding that we don’t like them, they don’t like us. Unlike those terrifying 8-legged creatures, website crawlers are a good thing. If you want to be indexed and increase your organic traffic, being crawled is a positive.

I Hate Spiders Too, Can I Step on Them?

This was actually asked to me. This was my exact response. “Yes, but if you have the technology to virtually place your foot onto a program and crush them, I would like more information for a potential investment.” In actuality, using a robots.txt file actually gives the crawlers directions for the website. If you don’t want them sneaking around there are specific instructions you can place in your robots.txt, for more information and instructions list click here.

Now that you have a better understand of crawlers, you know how search engines gather information on your website. They check tags, and content, as well as links. So be sure to have constant monitoring of your Google Webmaster tools and analytics to see if you are being penalized or have duplicate content that Google frowns upon.

Just for kicks:


If you have any more questions feel free to comment, or a great spider destroying story email me directly, I would love to hear it.