I am
sure many of you have conducted surveys at some point in your life. Many
students love to use survey results in their projects to substantiate their
stand or to identify trends pertaining to certain issue. In a properly designed and sampled survey,
this is certainly viable. The problem is, conducting a proper survey requires
so much resources, it is almost impossible and impractical for any students to
do so. Most students choose the easy way out by conducting online survey due to
its convenience. The end result is a brunch of numerical values fitted
forcefully and incorrectly into statistics. I have even seen an intern from a
certain national body recruiting online survey responses through Facebook.
Since then, I have serious doubt on the validity of the reports and
findings produced by that particular organization.
The
ability of a small sample in a survey to reflect the opinions of the larger
target population does not happen haphazardly. It requires proper probabilistic
sampling method. Otherwise, even simple statistics such as taking the average
of the response is wrong as it implicitly assumed that everyone in the target
population have equal probability of responding to the survey, which is certainly
untrue given the way how most students conduct their surveys.
Self-Selected Sample
With
the advent of forums, blogs and social medias, one of the most commonly used
methods for students to conduct such surveys is through these online platforms.
I mean, why wouldn't they? With the help of the countless free survey tools
available online, it is extremely easy for anyone to set up a questionnaire and
distribute it online to gather responses. Most of these tools are easy to use,
often producing beautiful graphs and charts with the click of a button and most
importantly, they are convenient.
The
main problem lies in the fact that the respondents are self-selected. These
volunteer samples are likely to have stronger opinion on the survey topic,
which compelled them to do the survey in the first place. In the case of conducting
survey through social media such as Facebook by asking your friends to fill up
the survey, it may also lead to over-representation of people who belongs to a
very similar demographic group.
As
an illustrative example, in 1993, shortly after Bill Clinton was elected as the
president of the United States, a TV station in Sacramento, California
conducted a TV poll seeking viewers’ opinion to the question: “Do you support
the president’s economic plan?” Coincidently, the result of a properly
conducted survey asking the same question was published around that time, with
the following results:
Self-Selected
Sample (TV Poll)
|
Random
Sample
|
|
Yes
|
42%
|
75%
|
No
|
58%
|
18%
|
Not Sure
|
0%
|
7%
|
As
shown in the table, the results are contradicting. In the TV poll, majority of
the viewers are not supportive of the president’s economic plan while in the
proper survey, 3 in 4 respondents are supportive of it. In addition, none of
the respondents in the TV poll was ‘Not Sure’, highlighting the tendency of
people who responded have a stronger opinion.
In
summary, survey which relies on
self-selected sample is a complete waste of time. The result is
meaningless, or worse, misleading. The only accurate information you can derive
from such survey is probably the count of number of people who bothered to
respond.
Wording
Even
for some reasons, you managed to get a proper sample, e.g. in a very simple
survey where your target population consists of only your class. There are many
other factors which need serious consideration, but are often ignored. One such
pitfall is the wording of questions used in the survey.
Building
questionnaire may seem like a simple task at first, but this couldn't be
further from the truth. The way you phrase a question and the order in which respondents respond to them can have
unintentional consequences on the resulting responds.
In
an experiment conducted by Loftus and Palmer, a video footage of an automobile
accident was shown to 2 groups of college students. The first group was then
asked the question: “About how fast were the cars going when they contacted each other?” While the second
group was asked “About how fast were the cars going when they collided with each other?” The average
responses of the 2 groups were 51.2km/h and 65.7km/h respectively. Simply
changing one word in the question led to a very different result.
The
ordering of questions can also skew the responses in the survey. For
example, if a survey asked, “How often do you dine at KFC?” and then
asked “Name the top 5 fast-food restaurant you think teenager visit most
frequently.” It is quite likely that the respondents will include KFC in the
latter question.
These
two examples are definitely not an exhaustive list of the potential pitfall one
can encounter when designing questionnaires. There are many other
considerations researchers should be aware of in order to create a good survey.
However, I have no desire to turn this blog post into an entire chapter, so I
will skip them. For those who are interested, you may read more about them
here
Characteristic of a proper survey
So
how do we conduct a proper survey? While each survey should be customized according
to the objective of the study, a good survey necessarily share the following
characteristics:
1.
Well
defined target population
Who are the people or
what are the objects you are trying to make inference on? It also allows us to
identify potential sampling frame.
2.
A
sampling frame and sampling method
Sampling frames are lists
intended to identify all elements from the target population. However, a
perfect sampling frame which contains all elements in the target population can
be prohibitively expensive or even non-existent. Therefore, we often have to make
do with frames that are imperfect which may not contain all elements in the
target population or include ineligible elements.
After obtaining the
sampling frame, we have to have a probabilistic method of selecting the sample
from the frame in order for the survey result to be statistically valid. The
simplest approach is to use Simple Random Sampling, where each element in the
frame is assigned a probability of selection and you select a sample of predetermined
size based on this probability profile.
3.
Questions
should be carefully phrased to avoid bias. Pilot study to ensure question yield
the desired information
As mentioned previously,
wordings can cause bias to the response collected. Therefore, care must be
taken to ensure the questions in the survey have minimal bias. If need be,
pilot study should be conducted to evaluate the questionnaire’s effectiveness
in eliciting the desired information.
4.
Properly
trained surveyor
If the survey requires
surveyor to collect responds from respondents through face-to-face interview or
telephone interview, adequate training should be provided in order to minimize influence
of the surveyor on the respondents.
5.
Ensuring
a good response rate
In a sample survey of humans,
it is almost impossible to obtain 100% response rate. Some respondents could
not be reached while others simply refused to be interviewed. People who tend
to respond to a survey could be very different from people who don’t. This
difference can bias the survey result. Therefore, measures should be taken to
minimize non-response.
After reading this post, I hope you will
realize that designing a proper survey requires a lot of efforts and resources.
The way you have been conducting surveys are most probably wrong and the results are nothing but a pile of meaningless numbers. As a student, you may be bounded by the syllabus to churn out ‘surveys’, but as a responsible user and consumer
of data, this stupid practice of conducting survey with no statistical basis
should be dumped into the garbage bin where it belongs.
No comments :
Post a Comment