<<< Saturday, January 19, 2008 11:49 PM

Home

Tivo > Moxi >>>


the referer mess

Sunday,  01/20/08  10:33 AM

I started blogging a little over five years ago (!), and at that time referer logs were a really cool thing.  (BTW, yes, "referer" is misspelled; it always is, going back to the early days of the 'net when a bad speller defined the HTTP header syntax.)  It was fun to see who was linking to you, and to find interesting new sites in that way.

{
Quick digression if you don't know what this is about:  Whenever you click a link on a web page, the website which serves the new page gets passed the old "referring" page's address.  By looking at server activity logs, a website owner can find all the websites which link to his.  In my case, I can find other blogs and sites which have linked to my blog, which is fun.
}

So now, in 2008, referer logs are basically a cesspool of misinformation.  If you are ever able to find anything useful of interesting in there, good luck.  First you have all the spiders; programs which access pages on your site for indexing purposes, their referers are often faked or meaningless.  Next you have access via RSS feeds, feed readers typically don't even pass a referer.  And then you have referer spam, which you might remember from my referer spam wars, where various websites send you fake referrals in the home that you'll click through to them.  (This is a weird and despicable practice, first, the volume from such traffic has to be uselessly low, because only webmasters see the spam, and second, no webmaster is going to care about a slimy site which is advertised in this way.  But I digress.)  And finally you have just plain old garbage; various bots which don't access anything useful and which pass junk as a referer.

This is a classic tragedy of the commons, we see this all over the 'net.  In the bad old days it worked great, but people have subverted the mechanism for other purposes to the point where it barely works at all.  The legitimate stuff is till in there but the signal to noise is decreasing to the point of uselessness.  Kind of like email :(