05_Measurement_Procedures

Measurement Procedures in Social Science Research

Objectives

Identify the sources of error in data collection, and especially systematic error that derives from poor instruments
Identify appropriate procedures you can use to create items and and multi-item variables
Identify appropriate procedures you can use to test the instruments you create
Assess the rigor of procedures used to validate measures in the research litearture
Assess the impact of weaknesses in procedure and methods on the applicability of research findings in your professional work

Required Materials -- I suggest you read these in the order they are listed.

Bernard, Chapter 2, pp. 27-60 -- You may want to review the materials on pp. 48-53 which focus on procedures to assess validity and reliability, but you do NOT need to re-read anything if you feel comfortable with your understanding.

Steps in Instrument Development is one of my cheat sheets. You will need to use this for the Group Project. Please read for detail. If my discussion of a topic is confusing, use one of the resources listed below about the same topic. You do NOT have to use Cronbach's alpha in the group project, but you do have to conduct an expert review and cognitive test. Therefore, you must understand what these measures are, how to conduct specific tests of reliability and validity, and how to interpret the results of such tests.

Reliability and Validity is a short (8 minutes) video about various tests for reliability and validity. These complement the video and readings last week and will help you understand my cheat sheet on techniques for creating and testing instruments. They do show you how to conduct specific statistical tests to assess reliability and validity.

Test Procedures for this Course provides instructions for conducting specific tests of instrument validity and reliability. It explains how to complete each required test. I am NOT requiring that you complete the procedures, but you do need to understand what procedures are vailable to you and how to interpret resutls. You may select these or other procedures if you so desire.

Calculating and Interpreting Cronbach's Alpha Using SPSS. 8.2 minutes -- shows you exactly how to calculate Cronbach's alpha. You have to do this for your group project. He also provides a good discussion of how to interpret the output.

Collins, D. (2003) Pretesting survey instruments: an overview of cognitive methods. Quality of Life Research 12(3), 229-238. You do not need to read this for detail now, but you will need to use this for the group project. For now, make sure you understand the basic concepts involved in testing for reliability and validity. All instruments require testing, even ones that are in widespread use, because there are differences in the ways that groups of people intepret questions. Just because it worked with middle-class people in New York does not mean it will work with poor people in New Orleans. The principles and techniques are the same for interviews, focus groups, and other methods of data collection. I suggest that you consult this reading for all assignments.

Willis, G.B. (2005) Cognitive interviewing: A tool for improving questionnaire design. Sage, Thousand Oaks, pp. 273-298 (Appendices). e-reserve. This is a step by step guide about how to conduct a cognitive test for an instrument. This is the simplest most "how to do it in five easy steps" kind of explanation I have seen. It is not highly sophisticated, but it is a good introduction to cognitive testing. Read this for basic understanding right now, but you will need to use this to design the cognitive test of the index. That will require a more complete understanding. I also encourage you to use this resource for all assignments.

Bialosiewicz, S., Murphy, K. and Berry, T. (2013) Do Our Measures Measure Up? The Critical Role of Measurement Invariance. Demonstration Session, American Evaluation Association, October 2013, Washington, D.C. Available at http://comm.eval.org/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=63758fed-a490-43f2-8862-2de0217a08b8. This is the simplest explanation of measurement invariance and how to test for it that I have seen and the body of the document is very short. They do include a printout of results when you run tests for invariance, which is helpful if you use the same stats package to run the anaysis.

Other Materials We May Discuss in Class .

Morgan, S.J., Amtmann, D., Abrahamson, D.C., Kajlich, A.J. & Hafner, B.J. (2014) Use of cognitive interviews in the development of the PLUS-M item bank. Quality of Life Research 23, 1767-1775. DOI 10.1007/s11136-013-0618-z Good resource for understanding how to use cognitive testing with scales, indices and questionnaires.

Duval, K., Marceau, P., Perusse, L. & Lacasse, Y. (2006) An overview of obesity-specific quality of life questionnaires. Obesity Reviews 7(4), 347-360. This article reviews many questionnaires commonly used in obesity-specific research. Ignore the topic and specific results. The importance of the article is that it alerts you to the effects of sloppy instrument testing -- you cannot trust an instrument just because someone used it. In this study, very few authors report adequate testing of the instruments, raising concerns about the quality of "what we think we know." Please look at this article -- it seems long, but most of it consists of very detailed tables that you do not need to consult in detail. Have a copy available during class.

Examples of Questions Examples of BAD questions. Things to avoid.

Fowler's Principles No need to read ahead of time. Use this as a checklist when you develop your index.

Additional Very Useful Resources Useful for Assignments

Berg, C.J., Nehl, E., Sterling, K., Buchanan, T. et al. (2011) The development and validation of a scale assessing individual schemas used in classifying a smoker: Implications for research and practice. Nicotine & Tobacco Research 13(12), 1257-1265. This is a good example of how to use the ideas of discriminant and convergent validity to make sure you are measuring what you should be measuring. They give examples of several kinds of statistical tests you can use to test for reliability and validity. Shows how to use demographic characteristics to test for the contextual appropriateness of an instrument.

Brod, M., Tesler, L.E. & Christensen, T.L. (2009) Qualitative research and content validity: Developing best practices based on science and experience. Quality of Life Research 18, 1263-1278. DOI 10.1007/s11136-009-9540-9

Dijkstra, W. & Ongena, Y. (2006). Question-answer sequences in survey-interviews. Quality & Quantity 40(6), 983-1011. DOI 10.1007/s11135-005-5076-4. This is a nice piece that examines why respondents do not answer questions as we "expect them to." Some good ideas you can use for all of your projects.

English, D., Bowleg, L., Rio-Gonzalez, A.M., Tschann, J.M., Agans, R.P. and Malebranche, D.J. (2017) Measuring Black men’s police-based discrimination experiences: Development and validation of the police and law enforcement (PLE) Scale. Cultural Diversity & Ethnic Minority Psychology 23(2):185-199. DOI 10.1037/cdp0000137

Galasinski, D. & Kozlowska, O. (2013) Interacting with a questionnaire: Respondents' constructions of questionnaire completion. Quality & Quantity 47(6), 3509-3520. Very good piece that takes us beyond cognitive testing to understand the processes that people use as they try to answer our questions.

Hohne, J.K., Schlosser, S. & Krebs, D. (2017) Investigating cognitive effort and response quality of question formats in web surveys using paradata. Field Methods 29(4), 365-382. DOI: 10.1177/1525822X17710640

Lescano, C.M., Hadley, W.S., Beausoleil, N.I., Brown, L.K., D'eramo, D. & Zimskind, A. (2007) A brief screening measure of adolescent risk behavior. Child Psychiatry & Human Development 37, 325-336. DOI 10.1007/s10578-006-0037-2

Li, R.M. (2011) The Importance of Common Metrics for Advancing Social Science Theory & Research: A Workshop Summary. National Academies Press, Washington, D.C. Available at http://www.nap.edu/catalog/13034/the-importance-of-common-metrics-for-advancing-social-science-theory-and-research. This is one of many books you can get for FREE from the National Academies Press. This one is relevant to all of your work this semester. It has a good discussion of constructs that I think will help you a lot with all of your projects this semester. It's a quick download and very easy to use.

National Institutes of Health, Office of Behavioral & Social Sciences Research. There are several chapters that deal with measurement. Available at http://www.esourceresearch.org/tabid/226/default.aspx

Priede, C. & Farrall, S. (2011) Comparing results from different styles of cognitive interviewing: "verbal probing" vs. "thinking aloud." International Journal of Social Research Methodology 14(4), 271-287. There are lots of specific techniques one can use in cognitive interviewing, but this article provides a good explanation of two quite distinct general approaches.

Wikman, A. (2006). Reliability, validity and true values in surveys. Social Indicators Research 78(1), 85-110. This article deals with the problem of capturing "real" values in what they call surveys. The author's concepts apply to ALL research instruments -- including interviews or focus groups (even direct observations where how YOU perceive of what you see can introduce systematic error). It deals with the problem that some measurement error arises from variability in the actual thought processes of the respondent. You need to understand this and use it in assessing validity in your projects.

Other Useful Materials -- some good choices for the voluntary (bonus point) research reviews

Beatty, P.C. & Willis, G.B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly 71(2), 287-311. Dated but still excellent.

Blair, J. & Conrad, F.G. (2011) Sample size for cognitive interview pretesting. Public Opinion Quarterly 75(4), 636-358.

Czaja, R. & Blair, J. (2005) Designing Surveys. A Guide to Decisions and Procedures. Pine Forge Press, Thousand Oaks, CA. e-reserve Read pp. 59-123.

Dijkstra, W. & Ongena, Y. (2006). Question-answer sequences in survey-interviews. Quality & Quantity 40(6), 983-1011. Dated but good - this does not change much.

Frost, D.M. (2013) The narrative construction of intimacy and affect in relationship stories: Implications for relationship quality, stability, and mental health. Journal of Social & Personal Relationships 30(3), 247-269.

Fujii, T. (2013) Development, validity and reliability of the Social Anxiety IAT. Japanese Journal of Personality 22(1), 23-36.

Glasner, T. & van der Vaart, W. (2009). Applications of calendar instruments in social surveys: A review. Quality & Quantity 43(3), 333-349.

Gise, V., Chambers, M., Valimaki, M. & Makkonen, P. (2010) A mixed-mode approach to data collection: Combining web and paper questionnaires to examine nurses' attitudes to mental illness. Journal of Advanced Nursing 66(7), 1623-1632.

Hannes, K., Heyvaert, M., Slegers, K., Vandenbrande, S. & VanNuland, M. (2015) Exploring the potential for a consolidated standard for reporting guidelines for qualitative research: An argument Delphi approach. International Journal of Qualitative Methods 14(4), 1-16.

Holland, J.L. & Christian, L.M. (2009). The influence of topic interest and interactive probing on responses to open-ended questions in web surveys. Social Science Computer Review 27(2), 196-212.

Kapikiran, S. (2012) Validity and reliability of the academic resilience scale in Turkish high school. Education 132(3), 474-483.

Kavussanu, M., Stanger, N. & Boardley, I.D. (2013) The prosocial and antisocial behavior in sport scale: Further evidence for construct validity and reliability. Journal of Sports Sciences 31(11), 1208-1221.

Kembler, D. & Leung, D.Y.P. (2008). Establishing the validity and reliability of course evaluation questionnaires. Assessment & Evaluation in Higher Education 33(4), 341-353.

Land, K.L., Oakes, W.P., Carter, E.W., Lambert, W.E. & Jankins, A.B. (2013) Initial evidence for the reliability and validity of the student risk screening scale for internalizing and externalizing behaviors at the middle school level. Assessment for Effective Intervention 39(1), 24-38.

Morgan, S., Amtmann, D., Abrahamson, D., Kajlich, A. & Hafner, B. (2014) Use of cognitive interviews in the development of the PLUS-M item bank. Quality of Life Research 23(6), 1767-1775.

Nichols, E. & Childs, J.H. (2009). Respondent debriefings conducted by experts: A technique for questionnaire evaluation. Field Methods, 21(2), 115-132.

Peytchyev. A. (2009). Survey breakoff. Public Opinion Quarterly 73(1), 74-97.

Rattray, J. & Jones, M.C. (2005). Essential elements of questionnaire design and development. Journal of Clinical Nursing 16(2), 234-243.

Rodriguez Bonzalez, M.S., Tinajero Vacas, C., Guisande Counago, M.A. & Paramo Fernandez, M.F. (2012) The student adaptation to college questionnaire (SACQ) for use with Spanish students. Ammons Scientific 111(2), 624-640.

Ryan, K. (2013) Issues of reliability in measuring intimate partner violence during courtship. Sex Roles 69(3/4), 131-148.

Sato, T., Harman, B.A., Adams, L.T., Evans, J.V. and Coolsen, M.K. (2013) The cell phone reliance scale: Validity and reliability. Individual Differences Research 11(3), 121-132.

Smith, D.M. (2006). Why are you calling me? How study introductions change response patterns. Quality of Life Research 15(4), 621-630.

Vacha-Haase, T., Henson, R.K. & Caruso, J.C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational & Psychological Measurement 62(4), 562-569.

Willis, G.B. (2005). Cognitive interviewing. A tool for improving questionnaire design. Sage, Thousand Oaks. See especially Ch. 2 (pp. 12-30) and Ch. 4 (pp. 42-62).

Willis, G.B. (2015) The practice of cross-cultural cognitive interviewing. Public Opinion Quarterly 79(S1), 359-395.

Wlezien, C. (2005). On the salience of political issues: the problem with the "most important problem." Electoral Studies 24(4), 555-579.

Yesil, R. (2012) Validity and reliability studies on the scale of the reasons for academic procrastination. Education 133(2), 259-274.