This was going to be a more philosophical piece about how to design a good experiment when the outcome you are trying to measure is an emotional response. To be honest I doubt I would have done more than touch briefly on the ethical dimension. However events have taken over, and Facebook are under attack for publishing the results of one such experiment they carried out in 2012.
The New York Times “Facebook Tinkers With Users’ Emotions in News Feed Experiment, Stirring Outcry”
I don’t think anyone is going to argue the case for experiments which risk doing harm to customers or take advantage of vulnerable groups. Customers also need to feel that their privacy is being protected at all times.
It is clear Brands need to develop trust with their customers. If we are to perform experiments then customers must be happy to be involved. People like to feel in control. The anger demonstrated by Facebook users I suspect is based on that feeling a lack of control. Whether we need to go through a process of informed consent or not, I’m not sure, but anyone who is involved in an experiment needs to feel in control of the process.
In the physical sciences an experiment will vary one-factor-at-a-time. There will be a theory created before hand. The effect of this variation is measured and if the results don’t match the theory, then the theory will have been falsified. Notice theories are never proven correct. Experimenters need to be cautious of false-positives both statistical and those due to confirmation bias. Good experiments need to be repeatable by other teams.
However we are talking here about building an understanding of how customers will respond to your brand experience. This is about experimentation on people. In the world of clinical trials the gold standard for experimental design is known as the Randomized Controlled Trial (RCT). Here different treatments go head-to-head. Often the trial will be blind, so the participants and the experimenters don’t know which treatment is being used in each case. When it is not possible to carry out a RCT then a less rigorous Observational study offers a good compromise in terms of quality of results vs. cost.
We can now put two strategies in this context of experimental design. The first is the A/B or the more complex multivariate testing. These tests work well for changes in UI within an ecommerce environment where impact on sales can be measured. The other is the cohort based tests favoured by the Lean Startup community. This approach allows a much richer change in factors and will also allow us to measure the effect over time. For example to see the effect on a customer who first searches for a product, before returning a few days later to make a purchase, as is common when selling flights.
However these approaches are very useful in the right circumstances both are susceptible to false-positive errors, and don’t help us measure the quality of the customer experience directly. Just a proxy measure such as purchase behaviour, which further components the false-positive error rate. I suspect this is the case with the Facebook incident. They claim some link between users click on likes or uploading messages containing certain words, with the users emotional state.
The question I’m not going to answer here is are there other options when it comes to the design of experiments?
References
gothandy 