• by Judah Phillips , October 26, 2007

Selecting a Web Analytics tool is never an easy decision.  There is no standard, scientific way to do so -- it's a bit of an art.  The decision is full of compromises -- no one tool or fancy family of tools from one brand will be able to do everything you think or want them to be able to do.  Nor will any one tool have all the bells and whistles you want.  

Lots of resources exist for helping you select a web analytics tool and vendor - from Marketing Sherpa to CMS Watch to a whole slew of consultants.   Even with good resources and the best consultants, it's still tough to narrow down the selection and really identify what's important to your business.  The first thing I'd recommend before beginning the due diligence process is asking yourself or your boss the following (relatively) simple questions:

·    How much money can I spend?

·    What resources do I have?

·    Do I have the organizational capability and maturity to run an in-house software solution?

·    Do I prefer to eliminate overhead and technology expense by delegating control of my Web analytics technology and infrastructure to a hosted solution run out of a vendor's data center?  

·    Do I want to integrate Web analytics data with data from offline systems? If so, what systems and what methods (i.e. web services)?

You'll have a short list of potential vendors rather quickly.  I would recommend framing your vendor evaluation across these dimensions in the context of how they are relevant to your business needs and goals:
1.    Company and Technology
2.    Product and/or Services
3.    Features
4.    Vendor Organization
5.    Strategic Fit
6.    Cost

Create a matrix so that the attributes presented below are on the left axis and the companies you are selecting are on the top axis.  Fill in the cells with your custom information evaluating a vendor:


Useful attributes for beginning your evaluation of a potential company and technology for Web analytics include:

·    Company Description. Describe the company using publicly available sources.  How long has the company existed?  How solvent is it?  What do customers say about the company?

·    General Technology Description. Explain the technology and how it works. If technology uses OLAP, what happens to the confidence level and confidence interval (i.e. margin of error) when drilling down on the data?  Can I report on every dimension and attribute of available data about a segment or is the reporting limited?  How about when exporting?  

·    Product and Service Capabilities. Assess the overall ability of the vendor's technology and services organization when compared to the industry.  What percentage of the company's customers successfully deploys tags and gets complete tag coverage across every page from day one?  Or successfully transfers and correctly parses customized log files from day one?

·    Product(s) Required for Solution. List the product or products required to support the full solution.  Can I run identical queries and get identical answers across all company technologies?

·    Ease of Use. Indicate the complexity of interacting with and navigating through the interface and reports.   Assess the user experience of the GUI from usability and information architecture perspectives. Can I simply find the data I need to gain analytic momentum?

·    Product Updates and Difficulty. Indicate difficulty of product updates and general migration path for upgrades. Does taking advantage of new functionality in a release usually require upgrading the code throughout my Web site? 

·    Real-time reporting latency. Identify the delay or lag in availability of the data within the technology.  Continuous processing?  Batch?

·    Time to Implementation.  Indicate the time to deploy the baseline, out-of-the-box solution. What percentage of the company's customers have successfully tagged all site pages and/or processed logs within one month after beginning? Three month? Six months? 

·    Ease of Implementation. Indicate the difficulty level of implementing the technology. What percentage of the company's application can I use if no changes are made to the javascript page tag?

·    Data Collection Model. Identify data collection methods.  Does the company's data schema simply roll up and report "unique" counts across time periods and delete the underlying data (even if I don't buy an additional product)?  Does it cost more money to retain full, unsummarized visitor data for 12 months? 24? Longer?

·    Data Retention and Ownership. Indicate if I retain ownership of my data.  If so, for how long and at what level of granularity? For what duration does the company retain visitor data?  Is that the same across all applications (not just a data warehousing component)?

·    Integration. Identify features and methods for integration with external systems.  API? Web services? Summary extracts?  Just Excel?

·    Innovation. Indicate the level of innovation perceived by looking into the company when compared to industry competitors.  What do the analysts say?  How large is the company's engineering organization?  What percentage of overall expense does the company spend on R&D?  Partnerships?

·    Security. Identify the security model. Does the tool support integration with Active Directory or LDAP?  What is cost per seat or license?

·    Segmentation.  Identify the flexibility and ease of segmenting data.  What is the total, maximum number of segments available for use "out of the box"? How much more does it cost if I want to increase segments or filters?

More attributes exist.  More questions should be asked.  Truly understanding a Web analytics technology means asking hard questions and assessing the way a company answers those questions to frame your subsequent analysis and guide your selection.