We don't need Multivariate Testing. We already do focus groups!

5 Comments

by MrOptimization in Basics

Yes, we've actually heard people say that. People in positions who should know better. They think focus groups make their digital spend more "customer centric" and give them a digital competitive edge.

Agencies love focus groups because they're profitable. Many marketers do focus groups because that's what they learned in school.

Those who use Multivariate Testing will always beat their focus groupie competition. Here's why...

Relics in the age of digital?

Focus groups have been standard operating procedure for consumer packaged goods companies and political candidates since the 80's.

Focus groups are great for getting opinions about what people feel when smelling a product or whether a congressman's hairdo conveys positive, neutral or sinister impressions.

Focus groups don't do digital well

Using focus groups to guide a digital optimization strategy is a mistake. Here's why:

• The Hawthorne Effect is "a form of reactivity wherein subjects alter their behavior simply in response to the fact that they're being studied." Knowing the Hawthorne Effect is subtly manipulating the results of every focus group study should be reason enough to seriously second guess the results
• If you've ever been involved in a focus group for a digital project, you know how maddening the answers can be. The group participants are all over the map. They often can't even able to articulate what they really want and why
• A focus group consider can only consider a very narrow range of potential digital elements if you want to have any hope of keeping the interviews on track and collect actionable data
• The sample size is always too small to reflect your actual web audience
• The sample size isn't just small, it's self-selected. If you have a life (and buying power) do you really want to give up hours or days to be on a focus group for a few bucks? It's nearly impossible to get sufficient demographic and psychographic diversity in focus group participants to represent the actual distribution of digital audiences
• You simply can't accurately simulate the diversity of intent an actual web audience would have as they interact with your digital properties. "Imagine you just searched on Google and landed here" is about the best you can do
• The group interview approach is fraught with data integrity problems. For example, if you've been involved in this style of focus group, you've probably seen the loudest personalities start to influence others as the sessions progress
• "We do one-on-ones so we don't have that problem!" Congratulations. There are worse things than having too much time and money, but even if you have an unlimited number of skilled researchers working simultaneously with a diverse group, you've still got all the other focus group deficiencies creeping into your results
• In both the one-on-one and group approaches, researcher bias also has a significant influence the outcome. This is especially apparent in agency-run focus groups where the agency's designers have a strong preference for a certain creative direction
• Focus group participants are often incented to please, so they give the answers they believe the interviewers want to hear
• Professionally run focus groups typically cost more than £7,000 per day. We won't even comment on those companies that try to cut costs by using their own employees as focus group participants

Focus groups just don't predict how things will work in reality. Guess who said they loved New Coke?

Page: 12