Objectives: After completing this module, you will be able to:

• Formulate research and evaluation questions that are appropriate for true experiments and quasi-experimental designs;
• Select an appropriate type of true or quasi-experiment (Solomon four-group, randomized complete block, etc.) design to create unambiguous answers to research questions and to evaluate the effects of programmatic interventions like training programs;
• Develop sampling and analytic strategies appropriate to the specific design selected;
• Interpret the results of true and quasi-experiments and use them to reach warranted conclusions;
• Understand the weaknesses of quasi-experiments and employ a variety of principles and techniques, such as sample matching, multiple control groups, multiple pre-testing, and others to increase the internal and external validity of conclusions drawn from quasi-experiments; and
• Assess the degree to which you can apply the conclusions drawn from true and quasi-experiments to your own work.

Class Preparation

Topic 4: Making Sense of it all. I know this is a lot of material I have tried to provide a lot of background so that you can succeed on Assignment 4. The last few links here are more summative in nature and shold help you succeed in creating your own experimental design. Please look at these prior to class, but you do not have to "study" them. DO use them when you complete Assignment 4.

I put the Learning Guide here (last instead of first) this week so that you can use it as a review sheet.

This very brief discussion of the logic of data analysis for experiments should help you with Assignment 4. Focus on the logic of it, not the details. Data Analysis in Experiments

Threats to Validity This is the obligatory (I feel) list of potential threats to validity in research. PLEASE DO NOT treat this as a checklist. The only way one can use this is to think carefully about one's design decisions and figure out how to reduce or eliminate threats. You CANNOT ELIMINATE ALL OF THEM. The point is to be aware that they exist, think about which (if any) are "serious" threats in a particular study and design, and address them to the best of your ability. If you start using them as a checklist, you will quickly find NO research is "good enough." I do NOT encourage you to go down that path.

The questions on the learning guide this week really require that you synthesize a lot of what you have learned so far in the semester. I most strongly encourage you to look at the guide and see if you can answer the questions. If not, raise the point in class. If one person has trouble, others usually do as well. Learning Guide: True and Quasi-Experiments

We will use these materials in class.

This document discusses ways to improve validity and reliability of the findings of quasi-experiments.

The materials below will be very helpful in completing Assignment 4 and now is a great time to share materials. I strongly encourage you to select one of these materials for a submission to the Sharing Materials discussion board. You should have received an e-mail through Canvas when I opened that discussion board. See the instructions there. Note that no two people can report on the same material -- first come gets his/her choice.

I highly recommend these materials for Sharing.

D'Onofrio, B.M., Lahey, B.B., Turkheimer, E. & Lichtenstein, P. (2013) Critical need for family-based, quasi-experimental designs in integrating genetic and social science research. American Journal of Public Health 103(S1), S46-S55. DOI: 10.2105/AJPH.2013.301252.

Fletcher, J.M. & Conley, D. (2013) The challenge of causal inference in gene-environment interaction research: Leveraging research designs from the social sciences. American Journal of Public Health 103(S1), S42-S45 (supplement). DOI: 10.2105/AJPH.2013.301290.

Lager, A.C.J. & Torssander, J. (2012) Causal effect of education on mortality in a quasi-experiment on 1.2 million Swedes. Proceedings of the National Academy of Sciences of the United States of America. 109(22), 8461-8466. https://doi.org/10.1073/pnas.1105839109

Shadish, W.R., Cook, T.D. & Campbell, D.T. (2002) Quasi-experimental designs that either lack a control group or lack pretest observations on the outcome. Pp. 103-134 in Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Wadsworth, Belmont, CA. e-reserve

Thayer, B.A. (2012) The role of group research designs to evaluate social work practice. Quasi-Experimental Research Designs. Oxford University Press, pp. 77-106. e-reserve

Travers, J.C., Cook, B.G., Therrien, W.J. & Coyne, M.D. (2016) Replication research and special education. Remedial and Special Education 37(4), 195-204. DOI: 10.1177/0741932516648462

White, H. (2013) An introduction to the Use of randomised control trials to evaluate development interventions. Journal of Development Effectiveness 5(1), 30-49. DOI 10.1080/19439342.2013.764652

Other Recommended Readings

Gersten, R., Baker, S. & Lloyd, J.W. (2000) Designing high-quality research in special education: group experimental design Journal of Special Education 34(1), 2-18.

Greeno, C.G. (2001) The skeleton: What underlies treatment research? Family Process 40(3), 361-363.

Greeno, C.G. (2001) The classical experimental design. Family Process 40(4), 495-499.

Rumrill, P.D., Jr. & Bellini, J.L. (1999) The logic of experimental design. J. Vocational Rehabilitation. 13, 65-70.

Solomon, B.R., Draine, P., DeMoya, J. & Wickrema, R. (1998) D esign-based evaluations: Process studies, experiments and quasi-experiments. Scandanavian Journal of Social Welfare 7, 126-131.

Additional Materials. These are examples of research reports using both true and quasi-experiments. They contain good discussions of how the experiments were designed and, in most cases, of how the data were analyzed. You can get ideas from these about how experiments can be used in research and practice and how to analyze the data. All of these can be used as reference for your assignment. To figure out if the study will help you, go to Academic Search Premier. Look for it and then check out the keywords. These materials are also excellent selections for sharing materials.

Becher, E.H., McGuire, J.K., McCann, E.M., Powell, S., Cronin, S.E. & Deenanath, V. (2018) Extension-based divorce educaiton: A quasi-experimental design study of the parents forever program. Journal of Divorce & Remarriage 59(8), 633-652. DOI: 10.1080/10502556.2018.1466256.

Bonetti, D., Eccles, M., Johnston, M., Steen, N., Grimshaw, J., Baker, R., Walker, A. & Pitts, N. (2005) Guiding the design and selection of interventions to influence the implementation of evidence-based practice: An experimental simulation of a complex intervention trial. Social Science & Medicine 60(9), 2135-2147.

Bravo, G. & Squazzoni, F. (2013) Exit, punishment and rewards in commons dilemmas: An experimental study. PLoS ONE 8(8), 1-6.

Butterfield, R., Park, E.R., Puleo, E., Mertens, A., Gritz, E.R., Frederick, L. & Emmons, K. (2004). Multiple risk behaviors among smokers in the childhood cancer survivors study cohort. Psycho-Oncology 13(9), 619-630.

Byiers, B.J., Reichle, J. & Symons, F.J. (2012) Single-subject experimental design for evidence-based practice. American Journal of Speech-Language Pathology 21(4), 397-414.

Carter, H., Drury, J., Amiot, R., Rubin, G.J. & Williams, R. (2014) Effective responder communication improves efficiency and psychological outcomes in a mass decontamination field experiment: Implications for public behavior in the event of a chemical incident. PLoS ONE 9(3), 1-12.

Cason, T.N., Saijo, T., Yamato, T. & Yokotani, K. (2004) Non-excludable public good experiments. Games & Economic Behavior 49(1), 81-102.

Chirumbolo, A., Manneteti, L., Pierro, A., Areni, A. & Kruglanski, A.W. (2005) Motivated closed-mindedness and creativity in small groups. Small Group Research 36(1), 59-82.

Class, Q., D'Onofrio, B., Singh, A., Ganiban, J. et al. (2012) Current parental depression and offspring perceived self-competence: A quasi-experimental examination. Behavior Genetics 42(5), 787-797.

deLuse, S.R., & Braver, S.L. (2015) A rigorous quasi-experimental design to evaluate the causal effect of a mandatory divorce education program. Family Court Review 53(1), 66-78.

Downs, J.S., Murray, P.J., Bruine de Bruin, W., Penrose, J., Palmgren, C. & Fischhoff, B. (2004). Interactive video behavioral intervention to reduce adolescent females' STD risk: a randomized controlled trial. Social Science & Medicine 59(8), 1561-1573.

Gallagher, H.M., Rabian, B.A. & McCloskey, M.S. (2004). A brief group cognitive-behavioral intervention for social phobia in childhood. Journal of Anxiety Disorders 18 (4), 459-479.

Geers, A.L. & Lassiter, G.D. (2005) Affective assimilation and contrast: effects of expectations and prior stimulus exposure. Basic & Applied Social Psychology 27(2), 143-155.

Gottlieb, D., Vigoda-Gadot, E. & Haim, A. (2013) Encouraging ecological behaviors among students by using the ecological footprint as an educational tool: A quasi-experimental design in a public high school in the city of Haifa. Environmental Education Research 19(6), 844-863.

Haapanen, R. and Britton, L. (2002). Drug Testing for Youthful Offenders on Parole: An Experimental Evaluation. Criminology and Public Policy 1(2): 217-244.

Hagglund, P. (2006) Job-search assistance using the internet: Experiences from a Swedish randomised experiment. International Journal of Manpower 27(5), 434-451.

Hagglund, P. (2014) Experimental evidence from active placement efforts among unemployed in Sweden. Evaluation Review 38(3), 191-216.

Haslum, M.N. (2007) What kind of evidence do we need for evaluating therapeutic interventions? Dyslexia 13, 234-239.

Herrmann, D. S. and McWhirter, J. J. (2003). Anger and Aggression Management in Young Adolescents: An Experimental Validation of the SCARE Program. Education and Treatment of Children 26(3): 273-302.

Hilbert, A. & Tuschen-Caffier, B. (2004). Body image interventions in cognitive-behavioural therapy of binge-eating disorder: a component analysis. Behaviour Research & Therapy 42(11), 1325-1340.

Hobolt, S., Tilley, J., and Wittrock, J. (2013) Listening to the government: How information shapes responsibility attributions. Political Behavior 35(1), 153-174.

Hoffman, S.G., Moscovitch, D.A., Kim, H. & Taylor, A.N. (2004). Changes in self-perception during treatment of social phobia. Journal of Consulting & Clinical Psychology 72(4), 588-597.

Hong, Ki Won (2004) Product evaluation bias under minimal group situations. The Social Science Journal 41(4), 667-673.

Hutchings, V.L., Walton Jr., H. & Benjamin, A. (2010) The impact of explicit racial cues on gender differences in support for Confederate symbols and partisanship. Journal of Politics 72(4), 1174-1188.

Jacob, R.T. (2011) An experiment to test the feasibility and quality of a web-based questionnaire of teachers. Education Review 35(1), 40-70.

Jakobsson, N. & Lindholm, H. (2014) Ethnic preferences in internet dating: A field experiment. Marriage & Family Review 50(4), 307-317.

Jung, D. I. (2001). Transformational and Transactional Leadership and Their Effects on Creativity in Groups. Creativity Research Journal 13(2): 185-195.

Krause, M.S. & Howard K.I. (2003) What random assignment does and does not do. Journal of Clinical Psychology 59(7), 751-766.

Lager, A.C.J. & Torssander, J. (2012) Causal effect of education on mortality in a quasi-experiment on 1.2 million Swedes. Proceedings of the National Academy of Sciences of the United States of America 109(22), 8461-8466.

Lai, E.S.Y., Kwok, C.L., Wong, P.W.C. et al. (2016) The effectiveness and sustainability of a universal school based programme for preventing depression in Chinese adolescents: A follow-up study using quasi-experimental design. PLoS ONE 11(2), 1-20.

Lavine, H., Lodge, M. & Freitas, K. (2005) Threat, authoritarianism, and selective exposure to information. Political Psychology 26(2), 219-256.

Mason, M.F., Tatkow, E.P. & Macrae, C.N. (2005) The look of love: gaze shifts and person perception. Psychological Science 16(3), 236-248.

Mauss, I.B., Wilhelm, F.H. & Groos, J.J. (2004). Is there less to social anxiety than meets the eye? Emotion experience, expression, and bodily responding. Cognition & Emotion 18(5), 631-663.

McGraw, A.P. & Tetlock, P.E. (2005) Taboo trade-offs, relational framing, and the acceptability of exchanges. Journal of Consumer Psychology 15(1), 2-16.

Nesdale, D., Durkin, K., Maass, A. & Griffiths, J. (2005) Threat, group identification, and children's ethnic prejudice. Social Development 14(2), 189-207.

Nolan, A. (2011) An extension in eligibility for free primary care and avoidable hospitalisations: A natural experiment. Social Science & Medicine 73(7), 978-985.

Norstrom, T. (2005) Saturday opening of alcohol retail shops in Sweden: An experiment in two phases. Addiction 100(6), 767-776.

Page, K.M. & Vella-Brodrick, D.A. (2013) The Working for Wellness program: RCT of an employee well-being intervention. Journal of Happiness Studies 14(3), 1007-1031.

Power, R., Khalfin, R., Nozhkina, N. & Kanarsky, I.A. (2004). An evaluation of harm reduction interventions targeting injecting drug users in Sverdlovsk Oblast, Russia. International Journal of Drug Policy 15(4), 305-311.

Reeves, A. & de Vries, R. (2016) Does media coverage influence public attitudes towards welfare recipients? The impact of the 2011 English riots. British Journal of Sociology 67(2), 281-306.

Riemersma, I., van Santvoort, F., Janssens, J.M.A.M. et al. (2015) "You are okay": A support and educational program for children with mild intellectual disability and their parents with a mental illness: Study protocol of a quasi-experimental design. BMC Psychiatry 15, 1-9.

Rodebaugh, T.L. (2004) I might look OK, but I'm still doubtful, anxious, and avoidant: The mixed effects of enhanced video feedback on social anxiety symptoms. Behaviour Research and Therapy 42(12), 1435-1451.

Rodgers, J., Herrema, R., Freeston, M. & Honey, E. Towards a treatment for intolerance of uncertainty for autistic adults: A single case experimental design study. Journal of Autism and Developmental Disorders. 48(8), 2832-2845 DOI: 10.1007/s10803-018-3550-9.

Scheepers, D. & Ellemers, N. (2005) When the pressure is up: the assessment of social identity threat in low and high status groups. Journal of Experimental Social Psychology 41(2), 192-200.

Shirom, A., Vinokur, A. & Price, R. (2008) Self-efficacy as a moderator of the effects of job-search workshops on re-employment: A field experiment. Journal of Applied Social Psychology 38(7), 1778-1804.

Skalicka, V., Belsky, J., Stenseng, F. & Wichstrom, L. (2015) Preschool-age problem behavior and teacher-child conflict in school: Direct and moderation effects by preschool organization. Child Development 86(3), 955-964.

Thomaes, S., Bushman, B.J., Orobio de Castro, B. et al. (2009) Reducing narcissistic aggression by buttressing self-esteem: An experimental field study. Psychological Science 20(12), 1536-1542.

Tsvetkova, M. & Macy, M.W. (2014) The social contagion of generosity. PLoS ONE 9(2), 1-9.

Tucker, T., Fry, C.L., Lintzeris, N., Baldwin, S., Ritter, A., Donath, S. & Whelan, G. (2004). Randomized controlled trial of a brief behavioural intervention for reducing hepatitis C virus risk practices among injecting drug users. Addiction 99(9), 1157-1167.

VanZomeren, M., Spears, R., Fischer, A. & Fischer, A.H. (2004) Put your money where your mouth is! Explaining collective action tendencies through group-based anger and group efficacy. Journal of Personality and Social Psychology 87(5), 649-665.

Vescio, T.K., Gervais, S.J. & Snyder, M. (2005) Power and the creation of patronizing environments: the stereotyp-based behaviors of the powerful and their effects on female performance in masculine domains. Journal of Personality and Social Psychology 88(4)-658-672.

Verhofstadt, L., Buysse, A., Ickes, W., DeClercq, A. & Peene, O.J. (2005) Conflict and support interactions in marriage: an analysis of couples' interactive behavior and on-line cognition. Personal Relationships 12(1), 23-43.

Waldzus, S., Mummendey, A. & Wenzel, M. (2005) When "different" means "worse": In-group prototypicality in changing intergroup contexts. Journal of Experimental Social Psychology 41(1), 76-83.