Objectives: After completing this session, you will be able to

• Assess the rigor of research procedures and methods in the reported research litearture
• Evaluate the impact of weaknesses in procedure and methods on your willingness to apply the results of research in your professional work
• Construct items for research instruments
• Apply cognitive and statistical procedures in instrument validation
• Create cumulative and summative scores for variables in multi-item instruments

Required Readings

These are very important readings for all of your work this semester. Collins explains the purpose of cognitive interviewing as a testing procedure. Willis provides a "how to" guide. Vacha-Haase gives an overview of what reliability is and how we assess reliability. Be familiar with this information and use it in all of your work this semester. Reeb discusses several techniques that we will use in this class.

Collins, D. (2003) Pretesting survey instruments: an overview of cognitive methods. Quality of Life Research 12(3), 229-238.

Willis, G.B. (2005) Cognitive interviewing: A tool for improving questionnaire design. Sage, Thousand Oaks, pp. 273-298 (Appendices). e-reserve

Vacha-Haase, T., Henson, R.K. & Caruso, J.C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational & Psychological Measurement 62(4), 562-569.

Reeb, R.N., Folger, S.F., Langsner, S. et al. (2010) Self-efficacy in service-learning community action research: Theory, research and practice. American Journal of Community Psychology 46(3/4), 459-471.

Class Preparation

Examples of Questions You do not need to look at this prior to class, but bring it to class.

Fowler's Principles Bring this to class. Use it as a checklist when you develop your questionnaire.

Fowler, F.J., Jr. (1995) Improving survey questions. Design and evaluation. Sage, Thousand Oaks, pp. 156-165 (Appendix A). e-reserve. This is a list of "good" and "bad" ideas for questions. You do not necessarily need to read it ahead of time, but make sure you use it as you develop the questions for your first group project 1 and, of course, if you select a questionnaire as one of the instruments for your individual project..

Sue, V.M. & Ritter, L.A. (2007) Conducting on-line surveys. Sage, Thousand Oaks, pp. 88-98. e-reserve You do not have to read this for class, but do consult this document if you ever consider using an on-line survey.

Research Methods Reviews

Schuman, H. (2002) Sense and nonsense about surveys. Contexts 1(2), 40-47.

Additional Resources

Appelbaum, L.D., Lennon, M.C. & Aber, Lawrence. When effort is threatening: the influence of the belief in a just world on Americans' attitudes toward antipoverty policy. Political Psychology 27(3), 387-402.

Bahr, R. (2009). No injuries, but plent of pain? On the methodology for recording overuse symptoms in sports. British Journal of Sports Medicine 43(13), 966-972.

Beatty, P.C. & Willis, G.B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly 71(2), 287-311.

Blumenthal, M.M. (2005). Toward an open-source methodology. What we can learn from the blogosphere. Public Opinion Quarterly 69(5), 655-669.

Casady, R. & Lepkowski, J. (1993) Stratified telephone survey designs. Survey Methodology 19(1), 103-113.

Czaja, R. & Blair, J. (2005) Designing Surveys. A Guide to Decisions and Procedures. Pine Forge Press, Thousand Oaks, CA. Read pp. 59-123.

Curtin, R., Presser, S. & Singer, E. (2005). Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly 69, 87-98.

Dijkstra, W. & Ongena, Y. (2006). Question-answer sequences in survey-interviews. Quality & Quantity 40(6), 983-1011.

Fowler, F.J., Jr. (1995) Improving Survey Questions. Design and Evaluation. Sage, Thousand Oaks.

Galesic, M., Tourangeau, R. & Couper, M. (2006). Complementing random-digit-dial telephone surveys with other approaches to collecting sensitive data. American Journal of Preventive Medicine 31(5), 437-443.

Gaskell, G.D., O'Muircheartaigh, C.A. & Wright, D.B. (1994). Survey questions about the frequency of vaguely defined events: the effects of response alternatives. Public Opinion Quarterly 58, 241-254.

Gerxhani, K. (2006). "Did you pay your taxes?" How (not) to conduct tax evasion surveys in transition countries. Social Indicators Research 80 (3), 555-581.

Glasner, T. & van der Vaart, W. (2009). Applications of calendar instruments in social surveys: A review. Quality & Quantity 43(3), 333-349.

Green, R.G., Murphy, K.D. & Snyder, S.M. (2000). Should demographics be placed at the end or at the beginning of mailed questionnaires? An empirical answer to a persistent methodological question. Social Work Research 24(4), 237-240.

Guymer, E.C., Mellor, D., Luk, E.S.L. & Pearse, V. (2001) The development of a screening questionnaire for childhood cruelty to animals. Journal of Child Psychology 42(8), 1057-1063.

Hart, B., Rennison, C. & Gibson, C. (2005) Revisiting respondent "fatigue bias" in the National Crime Victimization Survey. Journal of Quantitative Criminology 21(3), 345-363.

Heath, A., Fisher, S. & Smith, S. (2005). The globalization of public opinion research. Annual Review of Political Science 8(1), 297-333.

Holland, J.L. & Christian, L.M. (2009). The influence of topic interest and interactive probing on responses to open-ended questions in web surveys. Social Science Computer Review 27(2), 196-212.

Holmanova, E. & Ziakova, K. (2008). Audit diabetes-dependent quality of life questionnaire: Usefulness in diabetes self-management education in the Slovak population. Journal of Clinical Nursing 18, 1276-1286

Jobe, J.B. (2003). Cognitive psychology and self-reports: models and methods. Quality of Life Research 12, 219-227.

Kembler, D. & Leung, D.Y.P. (2008). Establishing the validity and reliability of course evaluation questionnaires. Assessment & Evaluation in Higher Education 33(4), 341-353.

Kimball, D.C. & Kropf, M. (2005) Ballot design and unrecorded votes on paper-based ballots. Public Opinion Quarterly 69(4), 508-529.

Lacy, S. & Riffe, D. (1996) Sampling error and selecting intercoder reliability samples for nominal content categories. Journalism & Mass Communication Quarterly 73(4), 693-704.

Lepper, G. (2000). Categories in text and talk. Sage Publications, Thousand Oaks, CA. Pp. 11-29 (Practising the art of categorization analysis). Get from Dr. Swisher.

Link, M.W. (2006) Predicting the persistence and performance of newly recruited telephone interviewers. Field Methods 18(3), 305-320.

Malhotra, N. (2009). Order effects in complex and simple tasks. Public Opinion Quarterly 73(1), 180-198.

Martin, E. (2002) The effects of questionnaire design on reporting of detailed Hispanic origin in census 2000 mail questionnaires. Public Opinion Quarterly 66, 582-593.

Matthews, B.A., Baker, F. & Spillers, R.L. (2003) How true is true? Assessing socially desirable response bias. Quality & Quantity 37, 327-335.

Nichols, E. & Childs, J.H. (2009). Respondent debriefings conducted by experts: A technique for questionnaire evaluation. Field Methods, 21(2), 115-132.

Palaniappan, L.P., Wong, E.C., Shin, J.J., Moreno, M.R. & Otero-Sabogal, R. (2009). Collecting patient race/ethnicity and primary language data in ambulatory care settings: A case study in methodology. Health Services Research 44(5), 1750-1761.

Pandani, J.A., Banks, S.M. & Mongeon, J.A. (2006) Criminal justice involvement in survey respondents and nonrespondents among mental health service recipients. Best Practice in Mental Health 2(2), 23-30.

Potter, W.J. & Levine-Donnerstine, D. (1999) Rethinking validity and reliability in content analysis. Journal of Applied Communication Research 27(3), 258-284.

Pridemore, W.A., Damphousse, K.R. & Moore, R.K. (2005) Obtaining sensitive information from a wary population: A comparison of telephone and face-to-face surveys of welfare recipients in the United States. Social Science & Medicine 61, 976-984.

Peytchyev. A. (2009). Survey breakoff. Public Opinion Quarterly 73(1), 74-97.

Rattray, J. & Jones, M.C. (2005). Essential elements of questionnaire design and development. Journal of Clinical Nursing 16(2), 234-243.

Schwarz, N. (1999). Self-reports. American Psychologist 54(2), 93-105.

Smith, D.M. (2006). Why are you calling me? How study introductions change response patterns. Quality of Life Research 15(4), 621-630.

Toepel, V., Das, M. & Van Soest, A. (2009). Design of web questionnaires: The effects of the number of items per screen. Field Methods 21(2), 200-213.

Tourangeau, R. & Smith, T.W. (1996). Asking sensitive questions: The impact of data collection mode, question format and question context. Public Opinion Quarterly 60, 275-304.

Willis, G.B. (2005). Cognitive interviewing. A tool for improving questionnaire design. Sage, Thousand Oaks. See especially Ch. 2 (pp. 12-30) and Ch. 4 (pp. 42-62).

Wilson, S., Stussman, B., Maitland, A. & Nahin, R.L. (2009). Role of self-concept in answering survey questions on complementary and alternative medicine: Challenges to and strategies for improving data quality. Journal of Alternative and Complementary Medicine 15(12), 1319-1325.

Wlezien, C. (2005). On the salience of political issues: the problem with the "most important problem." Electoral Studies 24(4), 555-579.

Examples of Research Reports

Brown, S.A., Anderson, K.G., Schulte, M.T., et al. (2005) Facilitating youth self-change through school-based intervention. Addictive Behaviors 30(9), 1797-1810.

Cohen, M.M., Forte, T., duMont, J., Hyman, I. & Romans, S. (2005) Intimate partner violence among Canadian women with activity limitations. Journal of Epidemiology & Community Health 59(10), 834-839.

Faramazi, M., Esmailzadeh, S. & Mosavi, S. (2005) A comparison of abused and non-abused women's definitions of domestic violence and attitudes to acceptance of male dominance. European Journal of Obstetrics & Gynecology and Reproductive Biology 122(2), 225-231.

Ponnet, K., Vermeiren, R., Jespers, I. et al. (2005) Suicidal behaviour in adolescents: associations with parental marital status and perceived parent-adolescent relationship. Journal of Affective Disorders 59(1-3), 107-113.

Routhe, A.S., Jones, R.E. & Feldman, D.L. (2005) Using theory to understand public support for collective actions that impact the environment: alleviating water supply problems in a nonarid biome. Social Science Quarterly 86(4), 874-897.

Sargent, J.D., Beach, M.L., Adachi-Mejia, A.M. et al. (2005) Exposure to movie smoking: its relation to smoking initiation among US adolescents. Pediatrics 116(5), 1183-1191.

Shi, Z., Lien, N., Kumar, B.N. & Holmboe-Ottesen, G. (2005) Socio-demographic differences in food habits and preferences of school adolescents in Jiangsu Province, China. European Journal of Clinical Nutrition 59(12), 1439-1448.

Weitzman, E.R., Chen, Y. & Subramanian, S.V. (2005) Youth smoking risk and community patterns of alcohol availability and control: a national multilevel study. Journal of Epidemiology & Community Health 59(10), 1065-1071.