Are UU numbers worth the bother?

We’ve been having a bit of a discussion here of late about the cost/benefit ratio of providing ‘proper’ (that is, properly and accurately calculated) Unique User (UU, or sometimes called Visitor) numbers in web analytics reports.

Whilst UU numbers are useful and desirable (I don’t think anyone would argue that you can’t benefit from them at all), they come at a cost. And what’s more, the benefit they deliver can fail to be appreciated by users, even causing questions to be raised about a tool’s accuracy. So it is pertinent to ask whether it’s worth delivering UU numbers throughout your web analytics reports.

To expand, let’s take a closer look at the costs & challenges of providing UU numbers:

1. Computational cost
   To calculate a UU count for a range of data, you have to count up the number of unique user identifiers that you find in the entire data set. This is a computationally expensive thing to do. If you’re designing a web analytics platform, you can do this kind of stuff up-front and cache the results, but if you want your tool to be able to offer UU counts over custom date ranges, you’ll always hit a point where a user asks for a UU count that hasn’t been pre-cached. This will be slow to deliver, and probably annoy users in the process.

   The reason for this is because UU count numbers are not additive over a date range. That is, if you know the UU numbers for each individual day of a given week, you can’t calculate the UU count for the entire week by just adding the day numbers together. This is because of people returning during the week on different days, who would be double-counted if you just added the days up. So you have to go back to the underlying data and recalculate from scratch, which is slower.

2. Tool complexity/ungrateful users
   The real tragedy of UU numbers is that, after you go to the effort of calculating them, you then have to spend hours explaining to skeptical users why they’re important, and why the UU number for March is not simply the sum of the individual UU counts for all the days of March. I’ve lost count of the number of times I had to justify the numbers that WebAbacus was producing for unique users, as if their failure to add up was somehow a failure of the tool itself.

   The problem is exacerbated by the use of  segmentation or filtering, because then you find that (No. of users who did A) + (No. of users who did B) > (Total no. of users), because, of course, some users did both A and B.

Some low-end tools sidestep both these problems (at the expense of their credibility) by calculating daily UU numbers and then just adding them up for the weeks, months numbers etc; and by not offering any segmentation capability. So poorly educated users don’t see numbers that confuse them, and the tool doesn’t have to go to the trouble of calculating UU numbers properly. But those tools are a dying breed.

Another way around the challenges of providing UU numbers (which has more integrity than just calculating them badly) is to avoid providing them at all, and instead to convince your users that what they really need to measure is visit (or session) numbers to measure the effectiveness of online marketing.

Eric Peterson has an interesting post on his blog where he quotes an attendee at the recent E-metrics Summit, who denounces the attitude that visit-based conversion rate calculations are the best as “crap”. So there’s clearly still a lot of debate about whether visit or UU (visitor) numbers are better. I tend to agree with Eric’s assessment – that you should use both for different reasons. I’ll address this topic in more detail in a future post.