Five Big Decisions About Your Survey

1. Defining a Target Population and a Sampling Frame

The first thing you need to conduct a survey is a list of relevant potential respondents. Usually there are two slightly different groups of interest.  The target population refers to everyone you are ultimately interested in, everyone who would be theoretically eligible to be in your study, and the sampling frame is everyone you could actually invite and would invite if you had the resources and wherewithal to do it.  It is a list of all possible practically feasible people. So first you want to think about the population of interest, the target population, and then you need to think about where you can get a list, or sampling frame, that would roughly represent that population.  Defining a target population is a conceptual problem; defining a sampling frame is a procedural problem.

Say you are interested in high school students.  Your target population might include 

• all students in your own high school
• all students in Michigan
• all students in an urban city like Lansing or in a county like Ingham
• all students who are juniors or seniors
• all students who are currently enrolled in some high school
• all students who have taken World history and are currently enrolled.

You want to define your target population according to things like what age, what level of education, what demographic range, gender, ethnicity, geographic region and so forth.  

Once you have a definition of a population, you need to figure out how you can find a list of all the people who are in it.  This could be the list of students enrolled in some specific schools, it could be a list of young adults who just obtained drivers licenses in the past year, a list of families who have high-school age students, a list of students who have appeared in court or students who have taken their SAT exams. 

Notice that none of these lists includes all young adults between the ages of 14-18.  All are tilted in some way. Even a telephone directory is not a complete list of a population.  Some families have no phones, some have only cell phones.  No list will completely cover your target population, so you need to think about which list is closest and what types of biases each one may have.

Having a list is obviously important for procedural reasons, but it is also important to think about because any specific list you use will necessarily differ in some way from your population of interest.  For instance, if you use the list of students enrolled in Williamston high school, you will miss students who have dropped out, and their views may be important to you. Or there may be errors in the records so that some are listed as sophomores when really they are juniors, or as girls when really they are boys, or as in a college prep program when really they are in a vocational program.  

Any list you use will have some errors and some discrepancies and will not be an exact accurate match to the population you conceived of.  Hence the importance of thinking about possible lists and what flaws each one might have.  You want to know how well your specific list matches your conception of the population you want to survey, and you want to be aware of ways in which it may miss the population you are envisioning.

For more on the problem of defining who you will study, see my page on sampling

2. Persuading People on the List to Participate

One you have defined the population of interest and have figured out an acceptable list of potential study participants, you have to decide how many of them you will actually contact and how many of them are likely to respond.  If your population and sampling frame are really large, you won't have the resources needed to survey all of them, so you'll need a way to sample a subset.  For instance, if you have lists of students from each of 15 high schools, you might want to sample a handful from each one, rather than surveying every student in every school.

The other side of the solicitation problem is how to persuade them to participate.  Here are some things to consider:

• Everyone these days is bombarded with stuff, and there is little incentive to respond to it all
• An ordinarysurvey will likely get something like a 10% response rate; A really good one might get 75%
• The longer the questionnaire, the less likely people are to respond
• There is some evidence that people are more likely to respond if you offer a gift of some sort, like by putting a dollar in each envelope or something
• There is also evidence that people are more likely to respond if you have a raffle and a prize that goes with it, so everyone who responds is automatically in the raffle.  Maybe an iPod or a gift certificate to an on-line music store or something else that kids would value.
• If you are asking people about a particular event, say a state test, you also need to think about WHEN they would complete the questionnaire.  Views will differ before and after, and may differ depending on how much before or how much after as well.  If you want to compare views before with views after, you will need to persuade them to respond twice you will need to think about how to schedule your survey in conjunction with the focal event you are asking about.

3. Figuring out What You Want to Learn From Them

This is the heart of the matter here, and it will take you some time to do this. Often, it takes many questions to really sort out an issue, yet the more questions you ask, the smaller the number of people who will reply.  So you need to think hard about what you really want to learn from them and ask only about those things.  There are three steps to this: (a)  figure out what you want to learn, (b) figure out what you'd have to ask them in order to learn that, and (c) figure out how to ask them that.

(a)  What you want to learn. Say you are interested in students views about MEAP tests in social studies.  You should be able to formulate a series of questions that might look something like this:

• -Do they have different views about social studies than they do about other subjects
• -Do they have different views about social studies assessment than assessment in other subjects
• -Are their views related to their college plans?
• -Are their views related to their grades in social studies

You also need to think about whether, when you ask about people's views, you mean views about taking the test itself, views about the consequences of taking the test, views about the content, views about tests in general, etc., or do you actually mean feelings like frustration, confidence, fear etc.

• (b) What you'd actually ask them will differ from the list above.  Now you are thinking about what they'd need to tell you in order for you to learn the stuff you list above.  For instance, if you want to know whether their views are related to their college plans, you need to ask about college plans and also about views.  So once you have a list of things you want to learn, you need to write an outline of the issues you would want to cover in your questionnaire.  The outline  might look something like this:
  
  - what school subjects do they like best or hate most, or maybe what are their majors and minors
• Are they thinking they are going to go to college
• Wwhich specific social studies classes have they taken so far
• Wwhat do they think about tests in general, maybe distinguishing classroom tests from MEAP from PSAT or AP or some other standardized tests they may have taken
• What do they think about the MEAP in particular

Notice that, although I've written these outline topics in the form of questions, they are not as general as the questions in the original list of what you want to learn. But they are still not in the form that you would actually ask them on your survey. The purpose of this step is to force you to think through what topics you will need to cover on the survey. If the list is long, you'll need to think about tradeoffs. Once you have this settled in your mind, you can begin to think about how to phrase questions for each topic on your list.

(c)  How to ask. The actual survey questions will look different from the outline. At this stage, we get into details of how to phrase things so you aren't biasing their responses, aren't confusing them, etc.   When you get to this stage, you need to talk with an advisor who knows about surveys, to learn nore about question formats and their relative advantages and disadvantages, how to avoid confusion with your wording, how to be efficient and so forth. A good way to begin would be to search the literature for other studies that relied on surveys and studied topics similar to your topic.

Question construction is really important. It can increase the likelihood that people will respond and that you'll learn what you hope to learn. If your respondents don't feel that the questions allow them to say what they really feel, they will become annoyed and toss the whole thing or, if they feel obligated to comply, fill it out with nonsense.

4. Coding and Cleaning your Data

Here are the types of things that will need to be done:

- Once you get questionnaires back, you will discover that the markings are not clear.  Sometimes people will check two options, sometimes none, and you will need to have some decision rules for how to handle these.

- You will need to figure out how to actually translate all of these responses to a spreadsheet. For instance, if you have yes-no questions, you might code the yes's as 1 and the no's as 0. Each question format will have some details to be sorted out as you translate pencil markings on sheets of paper into numbers on a spreadsheet

-After everything is on the spreadsheet you'll want to check it again for errors. Usually that means asking a computer program such as SPSS to present you with some histograms of each item, just to see what they look like. This helps you find, say, a erroneous score of 8 on a question that has 1-5 scale.

© Mary Kennedy, 2006