In Kurosawa’s 1950 film classic Rashomon four people share with us gripping details about a murder and alleged rape that had taken place 3 days earlier on a road in the middle of a deserted forest. Three of them were part of the actual event (including the murdered Samurai who thanks to a medium manages to come back to tell us his version of the events) and the last one is a passer-by who was able to see most of the action without being seen by the others. Not surprisingly, the stories we hear from our 4 characters do not match at all; actually, they contradict each other big time. Leaving aside the philosophical questions about truth and reality, the film poses interesting questions to the audience: Who do we believe? Who can we believe? Why are they all coming up, even the dead, with these seemingly convoluted stories? By the end, the film does provide some hints on these issues but, smartly, does not offer any solutions.
Perception surveys and sampling
Regardless, it seems clear that perception, the way we individually interpret and understand specific ideas and events, plays a key role in the way we respond to external phenomena – leaving aside the issue of self-interest, which is a whole different story. Having a stake on some issue can thus also affect our perception about stuff we care about.
Enter perception surveys. Opinion polls, which have been around in one form or another for over 100 years now, are regularly used to assess the way people, stakeholders, the public in general, perceive issues that are relevant for those undertaking the poll. Polls are most commonly used for upcoming electoral processes and popularity contest of individuals and causes, among others. While in their beginning they were done in haphazard form, it was only until the 1930s that polls based on random and representative samples of the population started to be used in systematic fashion to ensure accuracy of results (this is how Gallup was created, in fact). On the other hand, most private sector companies use perception surveys to complete market studies, assess demand for their products and services and make on that basis investment decisions.
Having the right sample is thus critical in such exercises. In simple terms, a sample is a probabilistic selection of a relatively small group of individuals that represent the overall population being targeted by the survey or poll being implemented. Attempting to use the whole population for a survey can be very costly, prove difficult to manage and analyse and demand lots on human inputs. Furthermore, in most cases it is very difficult to identify every single member of the population.
Wait, hasn’t the growth of the Internet and the rapid diffusion of mobiles changed this?
ICT diffusion and crowdsourcing
One of the key benefits of crowdsourcing is its potential to help systematically capture the voices of populations that had none before. With 2.7 billion Internet users and 4.5 billion mobiles users (not subscriptions, which are now close to 7 billion according to latest ITU data) it is now relatively cost effective to “crowdsource” perception surveys that can also cover a great chunk of the overall population. In addition, big data could also play a role here by facilitating management and storage of massive amounts of data and strong computer analytics. While there is certainly some potential here, we should be aware of the emerging issues in this regard.
Nowadays, we see a lot of polls and surveys done via the Internet or smartphones. The problem with these efforts is that they do not rely on random samples. Rather, Internet or smartphone users self-select themselves as respondents and fill out online questionnaires. The data capture in this fashion is thus not representative of the overall population, especially if we consider that most people in the world are not connected to the Internet – 4.3 billion people to be precise.
The situation is even more critical if we focus on development work and the penetration of the Internet in developing countries. For example, recent data indicates that less than 20% of the almost 1 billion inhabitants on the African continent are connected to the Net. If our research is thus focused on development issues (poverty, inequality, exclusion, etc.) using Internet base panels or polls will not be the best way to go – not even if we were to select a random sample of all those connected to the Net.
What about SMS-based crowdsoucring? While penetration of basic mobiles is much deeper that that on the Internet in developing countries, SMS polls are still undertaken on an opt-in, non-random basis. That is, target populations respond if they wish to do so thus potentially biasing results towards their own specific perception about key issues.
This is a critical issue if we are indeed trying to promote the participation of stakeholders in public policy making and identification of local, national or global development priorities. To be really effective here, we will need to ensure that we capture all the voices that need to be involved in such processes. And while new ICTs offer new opportunities here, they do not offer the whole solution to the issue.
Random crowdsourcing polls are for example feasible but need to be kept at the local level and take into account coverage limitations. Combining them with more traditional polling option is a good idea to complement them and get a more representative sample of the population.