Objectives

Identify the sources of error in data collection, and especially systematic error that derives from poor instruments
Distinguish between the concepts of validity, reliability and discriminatory power with regard to research instruments as opposed to design
Explain the differences between constructs, variables, items and scores
Explain why multi-item variables are generally superior to single item variables
Assess the impact of weaknesses in procedure and methods on the applicability of research findings in your professional work

Required Materials

Bernard, Chapter 2, pp. 27-60. We will complete some of the exercises on pp. 58-59 in class.

Reliability and Validity of Measurement. This is an excerpt from Chiang, I.A., Jhangiani, R.S. and Price, P.C., Research Methods in Psychology, 2nd Canadian Edition, available at https://opentextbc.ca/researchmethods/. It covers many of the concepts discussed in my cheat sheet. There are very abbreviated explanations of the concepts of reliability and validity. They are more basic than my presentation and you may want to read these prior to tackling my cheat sheet.

Approach to Measurement This is a critical resource becaue it explains the approach I want you to take to measurement. You must understand the ideas in order to complete assignments successfully.

Morse, J.M. (2015) Critical analysis of strategies for determining rigor in qualitative inquiry. Qualitative Health Research 25(9), 1212-1222.

Writing Items

The two Fowler documents are excellent guides for the process of actually developing items (questions). Fowler is a well recognized methodologist. These are old, but the basic ideas have not changed. Use these as you develop your items for the index and later for the semi-structured interview.

Fowler, Floyd J. Jr. (2009) Survey Research Methods, 4th Edition. pp. 87-111, "Designing Questions to Be Good Measures." Sage Publications, Thousand Oaks, CA. e-reserve.

Fowler, Floyd J. Jr. (2009) Fowler, F.J., Jr. (1995) Improving survey questions. Design and evaluation. Sage, Thousand Oaks, pp. 156-165 (Appendix A). e-reserve. This is a list of "good" and "bad" ideas for questions. What to do and what NOT to do. You do not necessarily need to read it ahead of time, but make sure you use it as you develop the questions for the small group project and any other "check the box" type instrument you might make for your individual project. This is the simplest level of getting items "right" -- just the basics. Fowler also provides a long list of the basic response categories for different kinds of scalar response items (how important, how often, etc.). Use these response categories. They are standard.

Other Very Useful Materials that Will Help You Complete Assignments - NOT required reading, but DO use these as you complete assignments. They offer concrete examples of "what goes wrong" in instrument development.

Adcock, R. & Collier, D. (2001) Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review 95(3), 529-546. Adcock & Collier address the critical concepts of reliability and validity for both quantitative and qualitative measurement. As a scientific realist, I am just as concerned about reliability and validity in qualitative as in quantitative measurement and I expect you to apply all of the principles we discuss to both approaches to measurement. Unfortunately, there is relatively little attention to these concepts in the methodological literature that focuses on qualitative techniques of data collection and analysis. I encourage you to look at the examples Adcock & Collier provide of applying the concepts reliability and validity to both quantiative and qualitative research that starts on p. 538.

Li, R.M. (2011) The Importance of Common Metrics for Advancing Social Science Theory & Research: A Workshop Summary. National Academies Press, Washington, D.C. Available at http://www.nap.edu/catalog/13034/the-importance-of-common-metrics-for-advancing-social-science-theory-and-research. This is one of many books you can get for FREE from the National Academies Press. This one is relevant to all of your work this semester. It has a good discussion of constructs that I think will help you a lot with all of your projects this semester. It's a quick download and very easy to use. Several chapters deal with measurement.

Barusch, A., Gringeri, C. & George, M. (2011) Rigor in qualitative social work research: A review of strategies used in published articles. Social Work Research 35(1), 11-19.

Buckley, R. (2018) Simultaneous analysis of qualitative and quantitative social science data in conservation. Society & Natural Resources 31(7):865-870. DOI: 10.1080/08941920.2018.1446232

Druckman, D. & Donohue, W. (2020) Innovations in social science methodologies: An overview. American Behavioral Scientist 64(1):3-18. DOI 10.1177/0002764219859623

Elman, C., Kapisqewski, D. and Lupia, A. (2018) Transparent social inquiry: Implications for political science. Annual Review of Political Science 21:29-47. DOI 10.1146/annurev-polisci-091515-025429

Pelli Paiva, P.C., Neves de Piva, H., Messias de Oliveira Filho, P., Lamouanier, J.A., Ferreira e Ferreira, E., Conceicao Ferreira, R., Kawachi, I. & Zarzar, M. (2014) Development and validation of a social capital questionnaire for adolescent students (SCQ-AS). PLoS ONE 9)8):e103785. DOI: 10.1371/journal.pone.0103785

Zhang, W. and Watanabe-Galloway, S. (2014) Using mixed methods effectively in prevention science: designs, procedures, and examples. Prevention Science 15:654-662. DOI 10.1007/s11121-013-0415-5

Abegglen, S., Krieg, A., Eigenmann, H. & Greif, R. (2020) Objective structured assessment of debriefing (OSAD) in simulation-based medical education: translation and validation of the German version. PLoS One 15(12):1-12. DOI: 10.1371/journal.pone.0244816.

Barusch, A., Gringeri, C. & George, M. (2011) Rigor in qualitative social work research: A review of strategies used in published articles. Social Work Research 35(1), 11-19.

Bowling, N.A., Khazon, S., A;larcon, G.M., Blackmore, C.E., Bragg, C.B., Hoepf, M.R., Barelka, A., Kennedy, K., Wang, Q. & Li, H. (2017) Building better measures of role ambiguity and role conflict: The validation of new role stressor scales. Work & Stress 31(1):1-23.DOI: 10.1080/02678373.2017.1292563.

Buckley, R. (2018) Simultaneous analysis of qualitative and quantitative social science data in conservation. Cultural Diversity and Ethnic Minority Psychology 23(2):185-199. DOI: 10.1080/08941920.2018.1446232

Clark, K.D., Yovanoff, P. & Tate, C.U. (2017) Development and psychometric validation of a child Racial Attitudes Index. Behavioral Research 49(6):2044-2060. DOI: 10.3758/s13428-016-0841-y.

Gazzillo, F., Gorman, B., Bush, M., Silberschatz, G., Mazza, C., Faccini, F., Crisafulli, V., Alesiana, R. & deLuca, E. (2017) Reliability and validity of the Interpersonal Guilt Rating Scale-15: A new clinician-reporting tool for assessing interpersonal guilt according to control-mastery theory. Psychodynamic Psychiatry 45(3):362-384. DOI: 10.1521/pdps.2017.45.3.362.

Min, H., Matthews, R.A., Wayne, J.H., Parsons, R.E. & Barnes-Farrell, J. (2021) Psychometric evaluation of work-family conflict measures using clsasic test and item response theories. Journal of Business and Psychology 36:117-138. DOI: 10.1007/s10869-019-09656-5.

Morrison, M.A., Bishop, C.J. & Morrison, T.G. (2019) A systematic review of the psychometric properties of composite LGBT prejudice and discrimination scales. Journal of Homosexuality 66(4):549-570. DOI: 10.1080/00918369.2017.1422935.

Mosquera, M.J., Kaat, A., Ring, M., Agarwal, G., Glickson, S. & Victorson, D. (2021). Psychometric properties of a new self-report measure of medical student stress using classic and modern test theory approaches. Health and Quality of Life Outcomes (2021)19:2. DOI: /10.1186/s12955-020-01637-0

Nielsen, t., Dammeyer, J., Vang, M.L. & Makransky, G. (2018) Gender fairness in self-efficacy? A Rasch-based validity study of the Genral Academic Self-Efficacy Scale( GASE). Scandinavian Journal of Educational Research 62(5):664-681. DOI: 10.1080/00313831.2017.1306796.

Pachter, L.M., Szalacha, L.A., Bernstein, B.A. & Coll, C.G. (2010) Perceptions of Racism in Children and Youth (PRaCY): Properties of a self-report instrument for research on children's health and development. Ethnicity & Health 15(1):33-46.DOI: 10.1080/13557850903383196

Park, K., Shim, G. & Jeong, B. (2020) Validation of the traumatic antecedents questionnaire using item response theory. Brain & Behavior 10:e01870. DOI: org/10.1002/brb3.1870

Raju, D., Beck, L., Azuero, A., Azuero, C., Vance, D. & Allen, R. (2019) A comprehensive psychometric examination of the lesbian, gay, and bisexual knowledge and attitudes scalae for heterosexuals (LGB-KASH). jOURNAL OF hOMOSEXUALITY 66(8):1104-1125. DOI: 10.1080/00918369.2018.1491705

Rathwell, S. &, Young, B.W. (2016) An examination and validation of an adapted youth experience scale for university sport. Measurement in Physical Education and Exercise Science 20(4):208-219. DOI: 10.1080/1091367X.2016.1210152.

Saha, P., Karmakar, A., Chakroborty, Z., Mukherjee, A., Bhattacharya, M., Lala, S., Mukherjee, H., Dogra, A.K., Bhattacharya, J. & Mukhopadhyay, A. )(2020) Measuring academic engagement among school students: A scale construction. Indian Journal of Positive Psychology 11(3):203-209. https://iahrw.org/our-services/journals/indian-journal-of-positive-psychology/

Tawa, J. (2017) The Beliefs About Race Scale (BARS): Dimensions of racial essentialism and their psychometric properties. Cultural Diversity and Ethnic Minority Psychology 23(4):516-526. DOI: DOI: 10.1037/cdp0000151.

Vaughan, R. & Laborde, S. (2018) Psychometrics of the emotional intelligence scale in elite, amateur, and non-athletes. Measurement in Physical Educaiton & Exercise Science 22(2):177-189. DOI: 10.1080/1091367X.2017.1405811

Yesil, R. (2012) Validity and reliability studies on the scale of the reasons for academic procrastination. Education 133(2), 259-274.

Zonnevijlle, M. & Hildebrand, M. (2019) Like parent, like child? Exploring the association between early maladaptive schemas of adolescenets involved with child protective services and their parents. Child & Family Social Work 24(2):190-200. DOI: 10.1111/cfs.12602