Objectives

Required and Other Materials Like last week, this is a long list, some of it repetitive of last week. The objective is for you to have access to all of the resources you need to complete the group project. If you did not get to examine all the materials last week, this is an opportunity to catch up.

Testing for Reliability & Validity

Viswanathan, M. (2005). Measurement Error and Research Design. Sage, Thousand Oaks, CA. Pp. 1-41. e-reserve. We will be using the procedures described in this reading for your group project. Make sure you understand the basics of the procedures described. Use the information in this document in your projects -- both the group index project and your individual semester project.

Castillo-Diaz, M. & Padilla, J.L. (2013) How cognitive interviewing can provide validity evidence of the response processes to scale items. Social Indicators Research 114(3), 963-975. REQUIRED

Multi-Item Measures (Scales, Indices, Questionnaires)

DeVellis, Robert F. (2003) Scale Development. 2nd Edition. pp. 60-100, "Guidelines in Scale Development." Sage Publications, Thousand Oaks, CA. e-reserve. This chapter in DeVellis has good discussions of the topics covered in my documents Approaches to Measurement, Steps in Instrument Development, and Testing Procedures. If you found my discussions confusing, this is a good place to get some clarification. If you are comfortable with the materials I provided focus on the sections on pp. 66-101. Some sections are critical to your success in the group project. An example is the discussion entitled "Step 3: Determine the Format for Measurement" that begins on page 71. We will cover some of these topics in class, but you have to read DeVellis to be able to take a sophisticated approach to your assignments. Use this publication to guide your work on the Group Project -- consider it a requirement. It complements the discussion in Bernard very well.

Barry, A.E., Chaney, E.H., Stellefson, M.L. & Chaney, J.D. (2011) So you want to develop a survey: Practical recommendations for scale development. American Journal of Health Studies 26(2), 97-105. Barry et al. provide step by step procedures that can be very helpful in your assignments. This reading is briefer than DeVellis, but does not have all of the information you will need to do a good job on your projects. Plese look at the tables before class in particular. They are very good. Table 1, for example, provides five different kinds of scalar response options. Table 2 gives suggestions for making any research instrument "flow." Helping people with the "flow problem" is one of my main contributions when I work with people as a methodologist. Table 3 makes basic recommendations for item development and Table 4 provides an excellent list of diagnostic questions you should ask yourself about your items. These questions can also be used as the basis for much of your cognitive test for all three assignments.

Class Preparation: Bring a list of at least three questions about the REQUIRED readings that you want answered to class. We will use your questions to guide class discussion. Please think about these questions in depth and bring questions that will encourage your classmates and you to think about (1) the differences between questionnaires, indices and scales, (2) when to use each, (3) the types of scales and what they measure, (4) specific procedures involved in the development of each type of instrument.

Additional Useful Materials

Afolabi, Olukayode Ayooluwa (2017) Indigenous emotional intelligence scale: Development and validation. Psychological Thought 10(1), 138-154. doi:10.5964/psyct.v10i1.184 This is a good example and discussion of why it is important to consider context in developing research instruments and why we cannot assume that instruments that provide reliable and valid data in one context will do so in another.

Bendixen, M. & Ottesen Kennair, L.E. (2017) When less is more: Psychometric properties of Norwegian short-forms of the Ambivalent Sexism Scales (ASI and AMI) and the Illinois Rape Myth Acceptance (IRMA) Scale. Scandinavian Journal of Psychology 58, 541-554 . DOI: 10.1177/0013164416658325 There is a continual quest to develop shortened forms of "long" instruments that have demonstrated good ability to generate reliable, valid data because of the problem of non-response or drop-out during completion of lengthy instruments. This is one example of an attempt to develop a "short form."

Bentley-Edwards, K.L. & Stevenson, H.W. Jr. (2016) The multidimensionality of racial/ethnic socialization: Scale construction for the cultural and racial experiences of socialization (CARES). Journal of Child & Family Studies 25(1), 96-108. This article provides insights into how constructs can change meaning over time due to societal changes, creating a need to develop new or revise existing instruments, even those that have provided reliable, valid data in the past.

Berg, C.J., Nehl, E., Sterling, K., Buchanan, T. et al. (2011) The development and validation of a scale assessing individual schemas used in classifying a smoker: Implications for research and practice. Nicotine & Tobacco Research 13(12), 1257-1265. Ignore the topic -- smoking. Focus on the use of discriminant and convergent validity. Note that there are examples of several kinds of statistical tests you can use to test for reliability and validity. Shows how to use demographic characteristics to test for the contextual appropriateness of an instrument.

Deng, L & Chan, W. (2017) Testing the difference between reliability coefficients Alpha and Omega. Educational & Psychological Measurement 77(2), 185-203. DOI: 10.1177/0013164416658325 Focuses on the use of various measures of reliability with a good discussion of Cronbach's alpha. Dijkstra, W. & Ongena, Y. (2006). Question-answer sequences in survey-interviews. Quality & Quantity 40(6), 983-1011. DOI 10.1007/s11135-005-5076-4. This is a nice piece that examines why respondents do not answer questions as we "expect them to." Some good ideas you can use for all of your projects.

Freund, P.A., Tietjens, M. & Strauss, B. (2013) Using rating scales for the assessment of physical self-concept: Why the number of response categories matters. Measurement in Physical Education & Exercise Science 17, 249-263. DOI: 10.1080/1091367X.2013.807265 Discusses item response theory which you need to include in your discussion in Assignment 1. This is a good discussion of the issue of how many response categories to include.

Galasinski, D. & Kozlowska, O. (2013) Interacting with a questionnaire: Respondents' constructions of questionnaire completion. Quality & Quantity 47(6), 3509-3520. Very good piece that takes us beyond cognitive testing to understand the processes that people use as they try to answer our questions.

Garb, H.N., Wood, J.M. & Fiedler, E.R. (2011) A comparison of three strategies for scale construction to predict a specific behavioral outcome. Assessment 18(4), 399-411. I honestly provide you with only one of several ways of assessing the validity and reliability of scores produced by an instrument. This article compares and contrasts three ways of doing so, only the first of which I have included in my instructions for assignments. You may want to use or of the other two in your semester project. To be quite honest, I selected the internal assessment because it was "doable" in the context of a one-semester course.

Hohne, W. & Ongena, Y. (2006) Investigating cognitive effort and response quality of question formats in web surveys using paradata. Field Methods 29(4):365-382. DOI: 10.1177/1525822X17710640

Joo, Min-Ho and Dennen, Vanessa P. (2017) Measuring university students' group work contribution: Scale development and validation. Small Group Research 48(3), 288-310. DOI: 10.1177/1046496416685159 I suspect the topic may be interesting given that you are doing group work. However, I selected this reading because it provides a very detailed discussion of how to use statistical tests for validity and discriminatory power.

Kelly, P., Fitzsimons, C. & Bakeer, G. (2016) Should we reframe how we think about physical activity and sedentary behaviour measurement? Validity and reliability reconsidered. International Journal of Behavioral Nutrition and Physical Activity. 13:32. DOI 10.1186/s12966-016-0351-4

Pelli Paiva, P.C., Neves de Paiva, H., Messias de Oliveira Filho, P., Lamouanier, J.A., Ferreira e Ferreira, E., Conceicao Ferreira, R., Kawachi, I. & Zarzar, M. (2014) Development and validation of a social capital questionnaire for adolescent students (SCQ-AS). PloS ONE 9:(8):e103785. DOI: 10.1371/journal.pone.0103785

Priede, C. & Farrall, S. (2011) Comparing results from different styles of cognitive interviewing: "verbal probing" vs. "thinking aloud." International Journal of Social Research Methodology 14(4), 271-287. There are lots of specific techniques one can use in cognitive interviewing, but this article provides a good explanation of two quite distinct general approaches.

Revilla, M.A., Saris, W.E. & Krosnick, J.A. (2014) Choosing the number of categories in agree-disagree scales. Sociological Methods & Research 43(1), 73-97. DOI: 10.1177/0049124113509605 This article discusses some of the issues involved in the "Likert-type response" approach to measurement. I personally find this approach cumbersome and overused and the approach is criticized by many for the high intellectual demand it places on respondents. This reading specifically addresses how many response categories, which has to do with the intellectual demand issue.

Saylor, R. (2013) Concepts, measures, and measuring well: An alternative outlook. Sociological Methods & Research 42(3):354-391. DOI: 10.1177/0049124113500476. A nice analysis of the failure to consider the first key steps in measurement when we focus on making sure we are measuring the right things.

Xu, H. & Tracey, T.J.G. (2017) Use of multi-group confirmatory factor analysis in examining measurement invariance in counseling psychology research. European Journal of Counselling Psychology. 6(1):75-82. DOI:10.5964/ejcop.v6i1.120

Zhang, W. & Watanabe-Galloway. (2014) Using mixed methods effectively in prevention science: Designs, procedures and examples. Prevention Science 14:654-662. DOI 10.1007/s11121-013-0415-5