Metrics wish list

Scott over at Artificial Simplicity just posted about some metrics he wishes he could measure. I just got off a phone call with a very intelligent ex colleague of mine who was also dealing with metrics he wishes he could measure.

On the call, my esteemed ex colleague brought up the Mystery and Puzzle notion so eloquently explained in a Gladwell op-ed piece for the New Yorker. A great idea to apply to anything you want to try and understand - be it marketing metrics or conundrums in general.

To quote directly from the Galdwell piece:

The national-security expert Gregory Treverton has famously made a distinction between puzzles and mysteries. Osama bin Laden’s whereabouts are a puzzle. We can’t find him because we don’t have enough information. The key to the puzzle will probably come from someone close to bin Laden, and until we can find that source bin Laden will remain at large.

The problem of what would happen in Iraq after the toppling of Saddam Hussein was, by contrast, a mystery. It wasn’t a question that had a simple, factual answer. Mysteries require judgments and the assessment of uncertainty, and the hard part is not that we have too little information but that we have too much.

The distinction is a great one. Too often we think we're trying to solve a puzzle, when in fact there is no real solution or vital missing piece. What we have is a mystery.

I would venture that the vast, vast majority of metrics/measure we use to look at consumer behavior are mysteries or parts of mysteries. Yet, in most cases, we believe them to be solutions to puzzles. Why? Because puzzle solutions are far easier to understand. You end up with a final solution, a factual outcome, a 'result'.

I think Scott asks some great questions, but they are all mysteries - and hence tough to solve.

The Brand Utility Mystery
A mystery if their ever was one - but a great thing to try and understand. Scott describes utility as more than 'usability'. It encompasses a brand's ability to understand and anticipate. Amazon being the gold-standard.

There is no one score than can encompass this though. It's just not possible (I've seen it tried). The big problem is tacit knowledge. It's too hard to divorce what a person knows from their perceptions of how easy or useful something is. Give the same utility test to top users of Amazon and Barns and Nobel and you will get the same result. You find utility in what you use often, for the very reason that you use it often.

So what would you need to do to get to this concept? Probably multiple measures that segment out new, recent and older users (to try and control for tacit knowledge). Apply different questioning techniques to each segment. Look at adoption times for new functionality across these groups. The goal would be to build up a body of insight, not a single measure. Less sexy, less useful... but it's a mystery!

Shareability/Network/Viral Effects
Another pure mystery, and Scott treats it as such. I'd love to know this one, I'd make a million dollars. An interesting question is why is this a mystery? Why is it so hard to predict what will and won't go viral? I think one of the reasons is that network effects are significantly more important than content. But you can't control network effects - who sees it, at what time, who they send it to, who they in turn send it to, what they say about it, etc, etc. There are way more great pieces of content (or conversations) that never went viral than there are networks with no content.

Participation
Again, a great mystery. What does encourage participation in a community? A host of factors have to come into play here - networks, interest, time, branding, incentives, hype, content, functions, fun, etc. The list is long. I'd love to see someone even try and model it! If you broke it down into manageable chunks - importance of just incentives for instance - you might learn something. As a broader question though, it's unknowable.

Linking Sales and Branding
This was the conversation I had with my ex colleague. Again, this delves into the realm of mystery. It also has to be the single most common question you get from Marketers - why are my sales going south but my brand metrics are strong? (or vice-versa). The truth is that there is never a clear relationship, never a stable relationship, and it's never solvable - it's not a puzzle. I worked for one of the fastest growing phone companies in the US in recent years and saw sales skyrocketing and brand metrics tanking - it made perfect sense. But it wasn't a situation that was typical and nothing you learned from it was useful to anyone other than that specific company.

Mysteries aren't 'general'. You can't derive learnings and apply to situations that look similar. I've blogged about this before, but if you really want to understand uncertainty, read The Black Swan - possibly the best book I've ever read that deals with what is and isn't knowable, and what to do about it.

Most of Marketing deals with mysteries. Beware people that package mysteries as puzzles - a common technique used by Marketing Research companies to sell you research. Beware consultants and industry pundits that package their experience as laws.

Sphere: Related Content