Perhaps the most written about topic in research these days is social media. It’s hard to look at a conference agenda, industry magazine, or blog without seeing someone’s opinion about using social media in research. What has been in much shorter supply is any sort of evidence of social media’s usefulness – or lack thereof – as a research tool.
I confess, I’m just as guilty as anyone when it comes to pontificating about social media in research. It’s been the subject of some of my most popular blog posts.
A big part of the debate has been whether and how social media mining might replace other qualitative methods like focus groups. After all, at first glance, an online community looks a lot like the ultimate focus group.
But is it?
Being researchers, we thought the obvious way to find out if social media mining could be used like focus groups was to conduct an experiment. So that’s what we did.
Earlier this year, we did parallel studies to compare the information we could get from a traditional focus group to what could be found mining data from online forums. We conducted 3 focus groups in Chicago. We asked people to tell us about their relationships with and opinions about their banks and credit card companies. In parallel to the focus groups, we selected 4 online personal finance forums to review.
What we found
There was a great deal of consistency between what was heard in the live focus groups and what we read on the Internet forums. The primary conclusions that any researcher would draw from each approach were the same. The alignment wasn’t perfect – but good enough to make social media mining a viable alternative to focus groups in this scenario.
Our main takeaways were the same, but we did see some differences among minor points. Many of these difference simply reflect natural differences in the make-up of the “participants.” In fact, the differences between the groups and the forums weren’t noticeably bigger than differences between the individual focus groups.
Some differences resulted from methodological differences between the two approaches. The two biggest methodological differences were:
- Interaction. We interacted with the participants in the focus groups, but not on the forums. So, we could steer the conversation in groups with probes, drawing out more detail in some cases.
- Motivation. People came to our groups because we invited them and we paid them (just like every other focus group). People came to the forums because they had something to say or were seeking information.
What does this mean?
It means that, in some cases, social media mining can be an effective substitute for traditional focus groups. This test project has shown social media mining will get essentially the same information as focus groups, but faster and more cost effectively.
I say social media mining can work “in some cases” because, like every other research methodology, it has to be applied intelligently. Using social media mining instead of focus groups can work in situations where you are trying to get a general feel for the nature and range of opinion in the market, as we were in this scenario.
Social media mining seems to be best suited to exploratory research. This is the type of qualitative you might do at the beginning of a big qual/quant project where the goal of the qualitative is to sketch the broad outlines of the situation, frame the issues, or suggest hypotheses for quantitative testing.
This approach isn’t going to work for explanatory research. Qualitative projects where the primary goal is to understand “why” requires a two-way dialog so the interviewer can probe answers. This is more often the type of qualitative you might do on its own or as a follow up to quantitative to understand context.
The end of the world as we know it?
So, should you stop hiring qualitative researchers and simply subscribe to one of the social media mining tools out there?
That would be a mistake. There isn’t an automated tool that can do this work.
The mining tools out there can help to identify forums where relevant discussions are happening, but you need a real live qualitative researcher reviewing, assembling, and interpreting what’s there. The value is in the analysis, and if the person doing it isn’t experienced, well, garbage in, garbage out.
Is that it?
There’s a whole lot more information available about this test, which will be in a white paper that will be published on this site shortly.