Simon is the systems guy at the OTHER media and he makes sure log files get rolled on virgin thighs and that permissions are chmodded and kernels are compiled properly. I have no idea what he does really but I know I can’t do it. Anyway, he is always monitoring attempted break-ins or anomolies in usage for our clients and today he sent round a few figures about usage.
Total Pages Served : 51% Increase
Total Data Served : 77% Increase
That’s roughly 3 pages a second all year, all the time… some of our servers like ECB do get white hot for the ashes though. Does this mean our pages are getting “heavier”… that they contain larger images? I wonder. Our pages are a lot lighter than they used to be, being fully CSS. Does this mean our CSS files are perhaps getting larger or that visitors are more likely with a broadband connection to look at pages with rich media.
Now, at first glance, if you run a web site, you’re probably familiar with the idea that over time your web site usage will go up, month by month. I don’t think I’ve ever really seen a web server hit count go down (unless it’s broken that is). So, as web designers, we and everybody else looks at our log files and collectively we sit around congratulating ourselves, saying “We must be doing something right” because the numbers keep going up.
But, what if the increase in hits is being caused by spiders and crawlers. I’ve found that if you take easily identified spiders out of your log files you remove close to half of your visitors. As more and more companies explore web crawling as part of their business, more and more crawlers are being created. There are also more and more tools that may lexically analyze your site, find telephone numbers on your site, or do something else to your data, all of which require your site to be crawled at some point. I recently played with an online tool to create your own focussed search engine called RollYo. It’s a great tool but it’s yet another crawler trying to gather up ALL the internet again.
There are also lots of desktop tools that do a nice diagram of your site structure or analyze the keyword density of given sites. They all add to your hit count, as do RSS readers.
What if the number of users and the size of the internet is a system? What if, as more users come online, more sites get built in some sort of natural symbiosis and the increased traffic for those sites get shared amongst the increasing number of sites. This would mean that the growth of internet usage is much smaller than most of us think it is.
Sales are going up, at least that’s what we see. But maybe a large part of that increase is caused by new people on the internet hitting it being prepared to spend more, because fears about security etc. are being proven in sheer numbers to be unfounded… and the long time internet users are now prepared to spend more in more places.
I’ve recently been wondering if I could slim down our hits even more by usage behaviour. That is, if a visitor to our sites slowly looks at all our pages in a particular order, they are probably not a real user at all.
What all this means is that in many instances, our conversion rates will be much higher than we previously thought, or to put it another way, the small changes we made that resulted in small increases in conversion rates are much more significant than we previously thought. Intelligently and craftily removing lots of the crap from usage data makes your measurements a lot more sensitive and effective.
Less than scrupulous crawlers will always try to mask their true nature because they fear getting “blocked” by the site. I am now of the opinion that rather than include all data about usage in a log file, that before anything starts being logged, a visitor has to prove themselves to be a real person and not a crawler by usage, not by IP, User Agent or cookie, all of which can be faked.
Suggesting the internet is getting smaller is churlish, I know, but, given there are lots of variables, such as the number of crawlers (which may vary depending on how much somebody want your data), the number of sites (which I guess can go up as well as down) and the number of users (some of whom still haven’t heard of Google or have found it to be inferior to their little known search engine). One factor that is rarely, er, factored in is a visitors context, by that I mean, 5 years ago, I could remember URLs. If I heard of a new service or site and I really wanted to remember it, I could. Now, every site is just another site. I don’t even start to remember it because I know I’ll forget. I even hear great brand names… visit their sites, like Rollyo, and then in the course of the day forget all about them… If I forget to blog them, they’re gone.. completely. It’s like it’s Out of Blog Out Of Mind. So even if you really are a great bookshop, way better than Amazon in all ways, it is now more difficult to bring customer back, simply because they forget about you.
The internet getting bigger, in a strange way makes it more dilute… thinned down. There may 100 alternatives for any service I want, but I still only have the time and inclination to research and try 2 or 3. So my choices for starting points is being globbed together. Kind of obvious maybe. Not quite sure what I’m saying here other than size isn’t everything and that I feel that measuring an ecommerce sites figures against last years figures misses the points sometimes. It seems too crude.
So, I guess the bottom line is the bottom line again, that you should be doing all you can to ensure your sales are going up. But I’m starting to think that I am going to get a lot more creative about what I’m measuring against those sales to properly inform redesigns and SEO improvements.
