• by Eric J. Hansen , August 9, 2007

An increasing number of Web marketers are turning to multivariate testing as an effective means to determine the optimal set of content on their sites. Marketers are testing variations of everything from landing page copy to navigational structure to registration forms, and optimizing their sites around the versions that most effectively achieve the marketer's goals. For transaction-oriented sites, these goals typically include persuading users to make purchases or register for accounts; for media sites, goals include encouraging users to spend time consuming content and clicking on ads.

But when analyzing the result of a multivariate test, simply tracking the aggregate behavior of users is only the first step. For truly effective testing and optimization, Web marketers must slice their data by customer segment in order to determine not only which Web site variations were most effective, but for whom they were most effective.

In online marketing, segmentation is a method for dividing users into distinct subgroups that share similar attributes. Attributes can include behavioral segments, such as those making first-time visits to a site vs. return visits, as well as search campaign segments such as the PPC keyword or ad group that referred the user to the site.

Web marketers can glean much deeper insights into user behavior by looking at their multivariate test results in the context of specific user segments. This makes it possible, for example, to tailor the site's content around the segments that are expected to be most profitable - in essence, optimizing the site content specifically for a target audience rather than for everyone and anyone.

In one case study, a major auction site operator had a goal of increasing email newsletter subscription rates. The company ran a multivariate test on the newsletter registration form, with variations of the location of the subscription box both (both within the page as well as across sections of the site), alternate copy and text colors, and so on. Using a multivariate testing tool, the Web marketers analyzed their test results and gleaned several insights that were both delightful and surprising. First, the marketers looked at all the data in aggregate and learned that the best set of variations resulted in a 390% lift in subscription rates across all users -- a big success! But then the marketers segmented the users into two groups: first-time vs. return users. What they found was that there were two "best" sets of variations: one set was best for first-time users, and resulted in a 350% subscription rate lift. The other set was best for return users, and caused subscriptions to increase by more than 600%.

What can we learn from the example of segmenting the auction site's multivariate test results? First, that users in different segments - in this case first-time vs. return users - responded differently to Web site content. This makes sense with the following explanation: on the first visit, users are taking a cursory browse around, essentially "dabbling" but not seriously engaging any site features. On repeat visits, however, users are already familiar with the site and perhaps have come back with the intention to bid on an item, or in this case subscribe to a newsletter. The results from this test led the marketers to configure their multivariate testing tool to dynamically display the "best" newsletter registration to users based on whether they were first-time or returning.

Multivariate testing is a big leap forward in optimizing user behavior, but adding segmentation can make the results even better. When analyzing results, make a practice of segmenting user data around common attributes like search engine queries or past user behavior. You'll be pleased with how much more effective your marketing efforts can be when you have better insight into each user segment's preferences.