Perception, Sampling and Crowdsourcing

In  Kurosawa’s 1950 film classic Rashomon, four people share with us gripping details about a murder and alleged rape that had taken place 3 days earlier on the road in the middle of a deserted forest. Three of them were part of the actual event (including the murdered Samurai who, thanks to a medium, manages to come back to tell us his version of the events) and the last one is a passer-by who was able to see most of the action without being seen by the others.  Not surprisingly, the stories we hear from our four characters do not match; actually, they contradict each other big time. Leaving aside the philosophical questions about truth and reality, the film poses interesting questions to the audience: Who do we believe? Who can we believe? Why are they all coming up, even the dead, with these seemingly convoluted stories? By the end, the film does provide some hints on these issues but, smartly, does not offer any solutions.

Perception surveys and sampling

Regardless, it seems clear that perception, how we individually interpret and understand specific ideas and events, plays a key role in the way we respond to external phenomena – leaving aside the issue of self-interest, which is a whole different story. Having a stake in some issue can thus also affect our perception about stuff we care about.

Enter perception surveys.  Opinion polls, which have been around in one form or another for over 100 years now, are regularly used to assess the way people, stakeholders, the public in general, perceive issues relevant to those undertaking the poll. Polls are most commonly used for upcoming electoral processes and popularity contests of individuals and causes.  While they were done in a haphazard form initially, it was only until the 1930s that polls based on random and representative samples of the population started to be used systematically to ensure accuracy of results (this is how Gallup was created, in fact). On the other hand, most private sector companies use perception surveys to complete market studies, assess demand for their products and services, and make investment decisions.

Having the right sample is thus critical in such exercises. In simple terms, a sample is a probabilistic selection of a relatively small group of individuals that represent the overall population being targeted by the survey or poll being implemented. Attempting to use the whole population for a survey can be very costly, prove difficult to manage and analyze and demand lots of human inputs. Furthermore, in most cases, it is challenging to identify every single member of the population.

Wait, hasn’t the growth of the Internet and the rapid diffusion of mobiles changed this?

ICT diffusion and crowdsourcing

One of the key benefits of crowdsourcing is its potential to systematically capture the voices of populations that had none before. With 2.7 billion Internet users and 4.5 billion mobiles users (not subscriptions, which are now close to 7 billion according to the latest ITU data), it is now relatively cost-effective to “crowdsource”  perception surveys that can also cover a great chunk of the overall population. Big data could also play a role here by facilitating the management and storage of massive amounts of data and strong computer analytics. While there is certainly some potential here, we should be aware of the emerging issues in this regard.

Nowadays, we see a lot of polls and surveys done via the Internet or smartphones. The problem with these efforts is that they do not rely on random samples. Rather, Internet or smartphone users self-select themselves as respondents and fill out online questionnaires. The data capture in this fashion is thus not representative of the overall population, especially if we consider that most people are not connected to the Internet – 4.3 billion people to be precise.

The situation is even more critical if we focus on development work and Internet penetration in developing countries. For example, recent data indicate that less than 20% of the almost 1 billion inhabitants on the African continent are connected to the Net.  If our research is thus focused on development issues (poverty, inequality, exclusion, etc.), using Internet-based panels or polls will not be the best way to go – not even if we were to select a random sample of all those connected to the Net.

What about SMS-based crowdsoucring? While penetration of basic mobiles is much deeper than that on the Internet in developing countries, SMS polls are still undertaken on an opt-in, non-random basis.  Target populations respond if they wish to do so, thus potentially biasing results towards their own specific perception about key issues.

This is a critical issue if we are indeed trying to promote stakeholders’ participation in public policymaking and identification of local, national, or global development priorities. To be really effective here, we will need to ensure that we capture all the voices that need to be involved in such processes.  And while new ICTs offer new opportunities here, they do not offer the whole solution to the issue.

Random crowdsourcing polls are, for example, feasible but need to be kept at the local level and take into account coverage limitations. Combining them with more traditional polling options is a good idea to complement the former and get a more representative sample.

Cheers, Raúl