Mini-Course in Survey Methods, Spring 2009


Logistics

Instructor: Erin Sills, sills@ncsu.edu, 3112 Jordan Hall
Office hours: 3-4PM Monday and 10-11AM Thursday
Classroom: 3214 Jordan II
Time: 2:45 - 5:30PM, Fridays, January 23 - February 27
Syllabus


Team name

partners

 

Key research question(s):

1. NC non-game

Jane

Joelle

How much public funding are NC residents willing to dedicate to non-game conservation?

2. Forest cooperators

James

Anna

Among NC participants in forest landowner groups, (1) how do they engage in the groups?, (2) what is the value of participating in the groups? (3) what aspects of participation do they value least/most, and (4) are they interested in expanded participation through educational, social, and/or economic activities?

3. Richmond recreators

Megan

Joseph      

How do park users perceive & value ecological restoration within Richmond Natl Battlefield Park?

4. Zoo visitors

Ines

Susan

What do people know about (1) rabies, its symptoms, transmission and who should they report rabies cases to?, and (2) how rabies can be transmitted to their companion animals and the benefits of vaccination?  (3) Do they vaccinate their pets, how often, where?

5. Racoon     relations

6. Cacao farmers

Maria


Braden

Kathryn


Genevieve

How do Hispanic visitors differ from other visitors in the park, in terms of group composition, membership in zoo, perceptions of zoo – especially conservation & education efforts, favorite exhibits, and their involvement in special events?

What do NC consumers consider to be "eco-friendly" or "green" chocolate?

7. Land stewards

Shari

Kimberly

 What are the respondents' perceptions of stewardship (including definitions & motivation) and landowner rights (including balance of rights & responsibilities)?

8. NCMAT visitors

Kim

Yari

How do Museum Park patrons perceive and respond to restoration efforts
within the park?
    

Class Schedule

Class 1: Survey error, mode, and total design

Readings:
1. Chapter 2 and Chapter 3, section 3.6, of Biemer & Lyberg (2003) Introduction to Survey Quality (e-book)
2. Chapters 1 - 4 of "What is a survey?"
3. Supplemental: Chapter 1 of Rea & Parker (2005), on reserve in the NRL

Handouts in class included (1) set of tables and figures from Dillman and Biemer & Lyberg, and (2) a case study from Dillman.

Progress report due next week: Select two possible modes for your survey case study, and (1) develop multiple contact protocols for each mode, and (2) compare advantages and disadvantages of two modes in table format.  Either bring a hard copy to class, or submit by email attachment before class.  (Grades ranged from A- to A+ on this assignment, worth 5% of your final grade.)

Class 2: Writing questions

Readings:
1. Dillman 2007, Chapter 2
2. Supplemental: Chapter 4 of Biemer & Lyberg (2003) Introduction to Survey Quality (e-book);  Chapters 2-3 of Rea & Parker (2005), on reserve in the NRL.
See "Readings" page for many more supplemental readings.

Progress report due next week: (1) Write and/or revise one module of questionnaire (~ 12 questions).  Bring three copies to class (one for instructor, and two for team that will peer review your module).  (2) Decide on the key research question that you are addressing in the questionnaire module.  Write this as clearly and concisely as possible, preferably in the form of a specific question.  Email to Erin (sills@ncsu.edu) before class on Friday.  Note that this is not a graded assignment per se, but provides context for peer review and grading your other assignments.  (All teams received an A on this assignment, worth 5% of your final grade.)

Class 3: Pre-testing and mode-specific issues with data collection

15 minute quiz  (covering all required reading, lectures, and discussion from first two classes; weight increased to 10% of final grade)
Answer key

Note: please bring hand-out from last week to class

Readings:
1. Presser et al. 2004  (carried over from last week)
2. Tourangeau et al. 2004
3. Highly recommended supplemental reading, depending on your survey project: from Dillman (2007), on reserve in the NRL, I suggest C3 "Constructing the Questionnaire" for NC Non-Game, C6 "Mixed Mode Surveys" for Forest Cooperators, and C7 "Alternative Questionnaire Delivery" for Richmond Recreators, NCMAT Visitors, Zoo Visitors, and Raccoon Relations.  For Cacao Farmers and Land Stewards, more relevant to read Glewwe's chapter "An overview of questionnaire design for household surveys in developing countries". 
4. More supplemental: Chapters 3 and 7 of "What is a survey?"  On focus groups, see Chapter 4 in Rea & Parker (2005), on reserve in the NRL.  On data entry options, see this UNSD site.  On cognitive interviews, see Collins 2003, Knafl et al. 2007, and PPT from Odum Course on Cognitive Interviewing (and other pre-testing techniques including QAS)  - provided by Shari. 

Progress report due next week: 
(1) Conduct and write-up two cognitive interviews.  We decided to pair up the groups, starting with NC Non-Game working with Forest Cooperators, down to Land Stewards working with NCMAT Visitors.  Let's take the example of Jane, and let's say that she does her interview with James.  Before she starts the interview, she will have some questions/doubts about her questionnaire that will be her probes.  She will ask James to complete her questionnaire module (or a sub-set of questions, if the module is very long).  She can ask James to "think aloud" as he completes the questionnaire, and/or ask him the probing questions as he completes questions or after he finishes the questionnaire.  My guess is that this will be a 20 minute exercise.  Meanwhile, Joelle will take notes (and/or tape record if she prefers).  Then Joelle will interview Anna, following same procedures, while Jane takes notes.  To report the results, Joelle and Jane could write separate reports (e.g., two tables of problematic questions; how subject interpreted these questions; and whether problems were with comprehension, recall & judgment, or formatting/ reporting).  Or they could consolidate their notes into one report, distinguishing their subjects as A and B.  That is, you can choose whether to submit one combined or two separate reports (e.g., Joelle reporting on Jane's interview of James, and Jane reporting on Joelle's interview of Anna).
(2) Peer review of another team's questionnaire module (resources for this include: first page of handout from 29 Jan, QAS-04 handout from this week, PPT from Odum course that Shari provided, and perhaps this website that I recently came across: QUAID).  For this, you swapped questionnaires in class.  Each team should write-up one report, and bring two copies to class: one for the team that drafted the questionnaire, and one for me.  You can choose the format for this report - table, list of bullet points, edited version of the questionnaire with comments explaining reasons for suggested edits. 

Class 4: Sampling and non-response

Start at 2:15PM with Joe Rabiega from IRB, who will give a short presentation, distribute information, and have plenty of time for Q&A and discussion.  Be sure to look at application for administrative review before class  -  this is a great opportunity to get a head start on IRB approval.

Readings:
1. Chapter 9 of Rea & Parker (2005), on reserve in the NRL, OR Chapter 9 of Biemer & Lyberg (2003) Introduction to Survey Quality (e-book) (If you read Rea & Parker - which is more basic, less detailed introduction - note that section 9.2 of Biemer & Lyberg (2003) may also be useful for planning your pre-test.)
2. Supplemental: Chapter 3, sections 3.1 and 3.3, of Biemer & Lyberg (2003) Introduction to Survey Quality (e-book)
    Website by Trochim that provides overview of sampling (similar to Rea & Parker)
    For use of "random digit dialing" in telephone surveys, see debate in the most current issue of Survey Practice.
    For challenges of building sampling frames in developing countries, see Chapter 3, section 3.2 in "Observers are worried ... counting the people in  
    Northeast Ghana" and Chapter 4 "Random Sampling and Repeat Surveys in South India" in Devereux & Hoddinott (1993), on reserve in the NRL (and a
    fun book to read, especially relevant to cacao farmers and land stewards).

In class exercise: plan for pre-test

Progress report
due next week:  Revise your questionnaire module to reflect input from cognitive interview, peer review, and my review.  Develop a data entry template (see excel example here).  Pre-test your questionnaire module on at least 2 people per team (i.e., complete at least one interview each).  For this pre-test of the questionnaire module, you will probably use some variety of convenience or purposive sampling (distinguised from SLOP by your attempt to reach a somewhat representative group and by your recognition that you are not doing random/probability sampling!)  Bring the completed questionnaires AND data entered into the spreadsheet to class.  If you complete the data entry significantly before start of class, please also email the spreadsheet to Erin (sills@ncsu.edu).  Rules for the pre-test include: (a) do not record any identifying information about the respondent, (b) do not add/revise any questions to elicit information that may represent a risk to respondent if disclosed, and (c) do not interview minors.  When you introduce yourself, explain that the questionnaire is completely voluntary and anonymous, how many questions or minutes are involved, and that it is an assignment for one of your classes at NCSU.  (I emailed all teams acknowledging receipt of data, suggesting changes in data entry template, and indicating grade for this progress report - worth 5% of your grade.)

Please note: I will not have office hours on Monday, 16 February.  I will be back in the office for the remainder of the week.

Class 5: Working with survey data

15 minute quiz
    As with previous quiz, this one will ask about concepts rather than particular equations and will ask you to discuss pro's and con's rather than recalling which specific authors or readings posed which specific arguments.  This quiz may cover pre-testing methods, visual layout and interviewer quality, sampling and non-response - including readings, lectures, and class discussion.  If the lecture on sampling went too quickly for you, I recommend that you (a) read Rea & Parker (2005), and (b) review the PPT, linked above.  We will return to many of the same sampling and non-response issues in this week's class, from the perspective of data analysis.
    Grades ranged from 6.25 to 14 out of 15 total points.  Most did not remember interpretation of graphs describing non-response and coverage bias: in both cases, the key point is that the bias depends both on the coverage (tc = % responded or were covered) AND on the difference between non-respondents and respondents, or between those excluded and those included in the sampling frame.  I suspect that many of you got this concept but did not remember the graphs, so I have down-weighted this quiz to 5% of the final grade.

Topics: (a) Review coding (open-ended; missing & NA), (b) descriptive & exploratory statistics (checking for outliers), (c) adjusting for stratified and clustered samples, and (d) testing for non-response bias and handling missing values.

Supplemental Readings:
(1) Chapter 3, section 3.6.11, of Biemer & Lyberg (2003) Introduction to Survey Quality (e-book)
(2) P|E|A|S: Practical Exemplars and Survey Analysis (explanation and examples of how stratification and clustering affect data analysis)
(3) Presenting simple descriptive statistics from household survey data

Final report due next week: (1) Title page including your names and research questions (revised, if appropriate); (2) Final version of questionnaire module; (3) Annotations or narrative describing changes in questionnaire that you have made or are planning due to cognitive interview and pre-test; (4) Brief description of how/who you interviewed in pre-test; (5) Spreadsheet with data from pre-test (at least 12 respondents per team); (6) Descriptive statistics, tables, or graphs summarizing 5 results from pre-test relevant to your original research question; (7) Draft sampling plan for your survey, including clear statement of target population, sampling frame including any concerns with coverage bias, and proposed methods for assessing non-response bias; and (8) Draft application for administrative review by IRB. 
If you submit electronically, I would prefer to receive one document with (1), (3), (6), and (7); a second document with your questionnaire (2); an excel workbook with your data (5); and finally your IRB application (8).  If you submit on paper, please still send an electronic version of your data (i.e., the excel workbook with your data (#5).

For the IRB application, there are three options: (1) submit application already approved, with completed modification/ addendum form if relevant; (2) submit complete draft application for administrative review (request for exemption); or (3) if you not yet ready to start on your application to IRB, complete just sections 1 through 4 of the application: project summary, description of participant population, description of how you will approach participants, and description of how identifying information will be recorded.  For these sections, you should be able to draw on other components of your progress and final reports, re-packaging them appropriately for the IRB.

Class 6: Household surveys in developing countries

Team Reports (2-6 minutes to report on whatever you think would be of greatest interest to others in the class, ranging from (a) description of the interviews that you conducted and what seemed to affect people's willingness to respond, to (b) presentation of your descriptive statistics and graphics, to (c) class discussion of a problematic question identified in the pre-test)

Focus: in-person surveys (including working with enumerators, panel surveys, ethical issues)

Readings:
(1) Chapter 4 "Overview of the implementation of household surveys in developing countries" in UNSD
(2) Supplemental: See many links below.  Chapter 12 "Thinking about the Ethics of Fieldwork" in Devereux & Hoddinott (1993) on reserve in the NRL. Other chapters in Devereux & Hoddinott (1993) and Barrett and Cason (1997), both on reserve in the NRL.

Evaluation of course, self, and student partner(s).  Several of you commented that you did not like the format for the evaluation of class participation  -  thanks for your feedback and your suggestions.


Readings

What is a survey? (An introduction)

Dillman (2007) on reserve in the NRL and recent papers available from his website

Biemer and Lyberg (2003) Introduction to Survey Quality (e-book)

Hh surveys in developing countries

Grosh and Glewwe (2000) 

UNSD


Resources

Survey Practice

QUAID  (click QUAID tool, and then "sign up" and "retrieve password" to receive a password by email)

ISS

USDA Survey Instruments

Websurveys


Hh surveys in developing countries:

PEN

SANDEE

STICERD

IHSN

LSMS

Household survey data from the World Bank

Enumerator manuals:

http://go.worldbank.org/WKSVUSONI0

http://www.who.int/chp/steps/en/

http://www.cpc.unc.edu/projects/china/data/questionnaires.html

 

Ethics

Most surveys conducted by graduate students are designed to be exempt from full review by the IRB.  However, you still have to submit your survey plan and instrument for administrative review before you start any pre-testing (other than expert/peer review).  Remember that this is an absolute requirement of all research involving human subjects at NC State  -  with the exception of research exercises for educational purposes (as part of courses) that meet the requirements for exemption  -  these can be approved by instructors, and we are operating under this procedure in this course.

See WAPOR for a take on survey ethics from survey professionals.



Learn more

Odum Institute
SAPOR
Courses at NCSU: SOC711, SOC712 (not offered this year), ST432, ST715
Courses at Duke: ENV280, SOC208

Courses at UNC: Sociology, Biostatistics

Graduate programs 

Careers in survey research


More links

Subpages (2): IRB PDFs
Comments