• by Jim Sterne , February 22, 2008

Besides being fun to say and a legitimate excuse to wear my "Mind The Gap" hat purchased the last time I was caught in the rain in Covent Garden, the Feedback Loop Gap is the most pernicious problem faced by those in large companies tasked with running large, sophisticated Web sites.

I don't for a minute discount the trials and tribulations of those plagued by shrinking budgets, incredulous senior executives, or IT departments facing mergers and acquisitions. I am not turning a blind eye to the pending economic downturn. I am simply responding to what I hear from clients, audiences, blogs and online discussions.

Your Web server generates a great deal of information. It creates logs. Mountains of data are collected in your server logs every day. The assumption is that this information is generated to help you determine how well you're doing. Nice thought, but not the case.

Log files are merely the result of good, solid engineering. If you create a system that does something, it's a good idea to record what happened. That way you've got a history file to look at when things go bump in the site. Log files are, therefore, the results of a server doing its job, and not a formal effort to capture valuable business intelligence.

The business management side of large companies went to the Web-heads and asked for reports. The log files were available and were delivered. Ever since, mankind has been trying to interpret log files like astrologers, phrenologists, and readers of the I-Ching. Log files do, indeed, contain more data than, say, the configuration of the bumps on your head, but it's a matter of torturing usable information out of them.

Here then, is the data torture process practiced by multitudinous multinational corporations:

1. Capture log & page tag data. Everybody's happy to lasso the data that Web servers spit out on a daily basis. It's just a matter of file storage -- rather mundane, actually. And it gives everybody hope.

2. Cleanse the data. Some data are useful, some are not. Some are informative, some are confusing. It usually requires a couple of rounds of attempted comprehension before the dirty data (robots and spiders, internal traffic, etc.) are revealed and can be expunged.

3. Report. Nice, crisp, colorful charts and graphs adorn the walls of technical and marketing offices. "Look! We have an active, interactive Web site and people are doing things there!" These charts are pretty to look at. They are proof that people are paying attention. They are useless.

4. Analyze. At some point, one of the technical committees squints at a series of reports over a several months period and begins to wonder why some numbers are going up, some are going down, while others do not change in the slightest. They wonder what external forces are at work. They scratch their heads and come up with 17 technical explanations for the change over time of the appearance of the pie charts, bar charts and diagrams.

5. Deliver. Convinced that they have discovered something important, the technical team calls a meeting with their business compatriots and presents the findings. They expound on the statistics and the resulting graphical depictions in terms of how many people showed up when, how often, for how long, and to what end. The marketing staff are thrilled to receive actual data - hard evidence that the work they've been doing has had an effect.

The End.

Notice something missing?

Several things, in fact?

Here's what I have found only in the rarest organizations:

6. Interpret. Technical people and business people gather enough knowledge to actually understand what the charts, graphs and diagrams really mean.

7. Plan. Based upon a thorough understanding of the reported results, a team made up of information system, designers and business people map out the changes necessary to improve usability, increase customer satisfaction, and boost revenue. Armed with an understanding of site visitors' desires and frustrations, new designs, new content, and new applications are devised and developed.

8. Execute. The best options that will result in the greatest impact at the lowest investment in time or money are selected. Upper management support is secured. A timeline is created. Stakeholders buy in. The project moves forward.

9. Repeat. Capture, measure, and analyze the results of the changes and incorporate those results in the next planning cycle. Ad infinitum.

The problem facing most large companies today falls between items 5 and 6: the dreaded Feedback Loop Gap. We have the data, we have the pretty reports, but we do not have an educated team made up of technical and business people who can take the resulting statistics and turn them into meaningful plans. Feeding Web server statistics, customer satisfaction surveys, sales data, etc. back into the process of creating Web strategy and executing on Web tactics has gone missing.

We need to sufficiently educate those controlling the marketing and customer service budgets so that they can interpret the data they get. At best, they will understand the metrics reports well enough to make well-informed plans for the next step *and* instrument the execution of those plans so that the right information is gathered with each iteration: the Feedback Loop. When this process begins running like the well-oiled machine it can be, the result is continuous improvements.

We can only hope. In the meantime, Mind the Gap.