coverspace - streetsmart with web analytics

Whenever you handle metrics remember: garbage in - garbage out. All analysis done based on muddy metrics is at least as muddy no matter how shiny the powerpoint armor might appear

Make sure that your metrics are right from the start - we are here to hand you the 5 most common pitfalls with metrics…and how to avoid them

Pitfall #1: Definition

Often metrics are defined in a way that nobody save a few experts understands what they are all about. Also, lots of definitions do not render enough differentiation to set one metric clearly apart from other metrics

How to define:

a) simple=good, complex=bad - say what your metric is. Do it in one sentence. add an example. then test it with non-reporting-experts - if they get it you are safe

b) say what your metric is not. look at definitions for metrics that are related/close in meaning. find at least one compelling reasion why they differ from the metric you define. include that into the definition

Pitfall #2: Syncronization

When looking at different sources metrics can sometimes have different labels (e.g. impressions, page impressions, page views) even if they really are the same thing. on the other hand we find metrics that are labeled the same but differ nonetheless (e.g. Unique Users - sometimes they include global IP-addresses, sometimes they only include IP-addresses from one country)

How to syncronize from different sources:

a) look behind the label - always check out the definition to compare. if you cannot get hold of the definition compare a set of data on ceteris paribus conditions - big differences are very suspicious

Pitfall #3: Collection

This is probably the largest pitfall - how is the data for the metric tracked? Is it logged or is it based on a panel or has it been calculated/estimated based on other data? If it is logged, what determines whether it is picked up into the data warehouse? Depending on how these questions are answered metrics with the same definition can render very different data sets

How to collect:

a) Understand how data for your metric is collected - really, really understand.

b) Get to know the limitations (if there are any) especially if data is estimated and make sure quality is still sufficient for your analysis

c) If you use data from different sources for the same metric make sure it is collected the same way. In case it is not make sure you understand the difference and the impact on your analysis

Pitfall #4: Enhancement

Logged data for metrics is almost always filtered (for good reasons) - these methods of filtering can differ significantly. If data is estimated based on a panel the methods of estimation might vary depending on the sources/reliability and size of panel etc. possibly scewing the results of the analysis you are about to do

How to enhance:

a) Make sure you understand what the filter does to the data. sense-check the estimation method for panel data. if necessary have the estimation redone.

Pitfall #5: Continuity

With changes to product environment and organizations data for metrics might get muddy over time because tracking and enhancing methods change. Also, outages and unusual events might influence the data causing big scews for a certain period of time. This makes comparison over time difficult and might render analysis that is not reliable

How to ensure continuity:

a) know about and eliminate effects that have nothing to do with performance e.g. outages, one-time events

b) know about and adjust for effects like product launches and reorganization

c) know about and adjust for changes in tracking, filtering, estimation methods

d) if you cannot do all/any of the above understand and clearly highlight the resulting limitations of your analysis

Technorati Tags: webmetrics, controlling

Share This