Objectives: After completing this module, you will be able to:

• Formulate research and evaluation questions that are appropriate for true experiments and quasi-experimental designs;
• Select an appropriate type of true or quasi-experiment (Solomon four-group, randomized complete block, etc.) design to create unambiguous answers to research questions and to evaluate the effects of programmatic interventions like training programs;
• Develop sampling and analytic strategies appropriate to the specific design selected;
• Interpret the results of true and quasi-experiments and use them to reach warranted conclusions;
• Understand the weaknesses of quasi-experiments and employ a variety of principles and techniques, such as sample matching, multiple control groups, multiple pre-testing, and others to increase the internal and external validity of conclusions drawn from quasi-experiments; and
• Assess the degree to which you can apply the conclusions drawn from true and quasi-experiments to your own work.

Class Preparation

Topic 1: The Case for More Use of Experiments in the Social Sciences

I do think we should use experiments much more than we do. Experiments (whether true or quasi) are the only designs that allow us to determine if an intervention has any discernible direct effect on the outcome of an intervention. To the extent that we fail to provide experimental evidence that a theory, when applied correctly, produces the predicted outcome, we have only relatively weak correlational evience to support our arguments regarding the efficacy of the interventions we recommend. We need evidence that generates strong internal and external validity -- evidence that what we tell people to do will actually result in the predicted outcome. Until we can provide such evidence, getting social science concepts into widespread application will be very difficult. There has, unfortunately, developed a litany about applied versus theoretical research, or even more destructive fairly virulent discussion about the value of practice-versus-research, with some proponents of the importance of practice arguing that research is irrelevant to practice. I am hoping that the discussion by Green and Glasgow will bring some balance to this discussion in our class. A lot of the discussion revolves around the external validity of research results -- whether any practitioner can use them or not. The authors deal with this in detail on pp. 136-141. I expect you to share your responses to the questions I pose below based on these required readings.

Gorard textbook, Chapter 9, Evaluating Controlled Interventions, pp. 123-141

Green, L.W. & Glasgow, R.E. (2006) Evaluating the relevance, generalization and applicability of research: Issues in external validation and translation methodology. Evaluation & the Health Professions 29(1): 126-153 DOI: 10.1177/0163278705284445 Focus on pages 136-142. Note that the PRECEDE-PROCEED model discussed in this article forms the theoretical basis for the Haiti Extension Experiment.

My Questions

Q1. What is a controlled experiment?

Q2. Why is control important in tests of the efficacy of an intervention?

Q3. Why is random assignment to treatment and comparison gropus (often called control groups) critical to experimental research?

Q4. Based on your answers, what are the requirements for a RCT (randomized controlled trial)?

Topic 2: SOOOO Many KINDS of Experiments. Its bewildering! Use the materials listed below to complete Assignment 4. This is a very important assignment becuase Assignment 5 builds directly on Assignment. A poor design in Assignment 4 will make successful completion of Assignment 5 very difficult. Think of Assignment 4 as playing the same role as Part 1 did on Assignments 2 and 3. If you understand the basics of setting up an experiment, Assignment 5 will be relatively easy, just as Part 2 of Assignments 2 and 3 were relatively easy after successfully completing Assignment Part 1 in each.

There are three ways of dividing up experiments into "types" -- in addition to the controlled, randomized and randomized control types. They are: (1) how participants are assigned to treatment and control groups; (2) who implemented the experiment; and (3) specific design features such as number of factors, number of comparison groups, and number of instances on which data are collected, as well as other features.

How people are assigned to treatments and control. We are covering two general categories of designs with controls and interventions -- true experiments and quasi-experiments. These names make it sound like there are "real" experiments (true experiments) and "phony" experiments (quasi-experiments). This is not the case. Both true and quasi-experiments randomly assign participants to treatment and control groups. The difference between true and quasi-experiment depends only on whether participants are assigned to treatment vs. control as individuals or as groups. In true-experiments, individual research units (birds, bears, atoms, people) are assigned to the treatment and control groups. In quasi-experiments, pre-existing groups of participants are randomly assigned to treatments and control groups. We use quasi-experiments when people "come in groups" and cannot realistically be assigned individually. Often, the group is one that is voluntarily formed by people -- membership in a sport club or in a farmer organization (see the Haiti Experiment) or some other formal or informal group. In many cases, the groups are created by some authority -- grade school students come in groups based on home room. The school would have to rearrange every class to randomly assign individual students to a home room designated as control or treatment. That would cause total havoc. Therefore one assigns home rooms to treatment versus control. In Haiti pretty much everyone belongs to a farmer group. We could have assigned individuals to treatment -- but that would actually have been a BAD idea. We can discuss why that is true in class. Quasi-experiments are by far the most common design used in evaluating interventions, such as educational programs. You do NOT need to look at these two slide shows ahead of time. We will cover them in class. If you find this confusing, the Greeno article "Major alternatives to the classic experimental design" is excellent. It is easy to read and gives good examples.

Slide Show -- Requirements of True Experiments

Slide Show -- Quasi-Experiments

Who implemented the experiment. This ranges from YOU, the researcher, to some government agency or researcher who was working on the same project independently of you who assigns to treatment/control, some other researcher whose data you can acquire (you don't even know him/her) and who may have had a totally different question from yours (leading to metanalysis in some cases) or any organization that is involved in "putting some people into treatments and others not" even if the organization did not think of this as assignment to treatment and control groups. The basic point is that somebody did in fact assign people to treatment(s) and others not (control) -- that is the essence of the experimental design. Naturally occuring comparison groups (men vs. women, adolescents vs. emerging adults, organic vs. conventional strawberry producers, kids in foster care vs. with birth parents, and on and on) are NOT treatment and control groups. They are comparison groups and are often the focus of cross-sectional or longitudinal designs, but these natural groups are not experimental groups.

Specific Design Features. My cheat sheet (linked below) includes examples of these types of features. All of these decisions have important effects on the conclusions you can draw from an experiment. In the "classic" experiment, data for the outcome variable are collected prior to implementing the experiment and again at the end of the experiment (in its simplest form). The difference in the outcome variable score (like weight loss) is tested for the treatment and control group. If the difference is significant, we conclude that the treatment had an effect. In fact, some experiments have no data for "before treatment." One whole group based on who implemented the experiment -- the experments conducted for a completely different reason with a different outcome under study -- are an example. The lack of pre-implementation data has numerous effects on internal and external validity. Simply put, without knowing the score for a variable prior to "doing something" to people, you have to take additional precautions to know whether the scores changed because of the treatment. Multiple data collections over time (repeated measures) are common. These are often used to test the persistence of an effect. Does the effect last after the controlled conditions are gone or do people revert to their prior behavior, for example. We desperately need more experiments with multiple-post intervention data points to know whether the changes we see are truly lasting or not.

Useful Materials for Assignments 4-5. Use these as much as you can.

Types of True Experiments (my cheat sheet)

Data Analysis in Experiments

Strengthening Quasi-Experiments

Threats to Validity in Experiments

Learning Guide Experimental Designs

Additional Materials. These are examples of research reports using both true and quasi-experiments. They contain good discussions of how the experiments were designed and, in most cases, of how the data were analyzed. You can get ideas from these about how experiments can be used in research and practice and how to analyze the data. All of these can be used as reference for your assignment. To figure out if the study will help you, go to Academic Search Premier. Look for it and then check out the keywords. These are excellent articles to use as the basis for Sharing Materials. You can earn up to 25 points for each annotation (review) you post to the Sharing Materials discussion board. Select materials that you use for Assignment 4. These materials will be useful for that assignment.

Becher, E.H., McGuire, J.K., McCann, E.M., Powell, S., Cronin, S.E. & Deenanath, V. (2018) Extension-based divorce education: A quasi-experimental design study of the parents forever program. Journal of Divorce & Remarriage 59(8), 633-652. DOI: 10.1080/10502556.2018.1466256.

Bonetti, D., Eccles, M., Johnston, M., Steen, N., Grimshaw, J., Baker, R., Walker, A. & Pitts, N. (2005) Guiding the design and selection of interventions to influence the implementation of evidence-based practice: An experimental simulation of a complex intervention trial. Social Science & Medicine 60(9), 2135-2147.

Bravo, G. & Squazzoni, F. (2013) Exit, punishment and rewards in commons dilemmas: An experimental study. PLoS ONE 8(8), 1-6.

Butterfield, R., Park, E.R., Puleo, E., Mertens, A., Gritz, E.R., Frederick, L. & Emmons, K. (2004). Multiple risk behaviors among smokers in the childhood cancer survivors study cohort. Psycho-Oncology 13(9), 619-630.

Byiers, B.J., Reichle, J. & Symons, F.J. (2012) Single-subject experimental design for evidence-based practice. American Journal of Speech-Language Pathology 21(4), 397-414.

Carter, H., Drury, J., Amiot, R., Rubin, G.J. & Williams, R. (2014) Effective responder communication improves efficiency and psychological outcomes in a mass decontamination field experiment: Implications for public behavior in the event of a chemical incident. PLoS ONE 9(3), 1-12.

Cason, T.N., Saijo, T., Yamato, T. & Yokotani, K. (2004) Non-excludable public good experiments. Games & Economic Behavior 49(1), 81-102.

Chirumbolo, A., Manneteti, L., Pierro, A., Areni, A. & Kruglanski, A.W. (2005) Motivated closed-mindedness and creativity in small groups. Small Group Research 36(1), 59-82.

Class, Q., D'Onofrio, B., Singh, A., Ganiban, J. et al. (2012) Current parental depression and offspring perceived self-competence: A quasi-experimental examination. Behavior Genetics 42(5), 787-797.

deLuse, S.R., & Braver, S.L. (2015) A rigorous quasi-experimental design to evaluate the causal effect of a mandatory divorce education program. Family Court Review 53(1), 66-78.

Downs, J.S., Murray, P.J., Bruine de Bruin, W., Penrose, J., Palmgren, C. & Fischhoff, B. (2004). Interactive video behavioral intervention to reduce adolescent females' STD risk: a randomized controlled trial. Social Science & Medicine 59(8), 1561-1573.

Gallagher, H.M., Rabian, B.A. & McCloskey, M.S. (2004). A brief group cognitive-behavioral intervention for social phobia in childhood. Journal of Anxiety Disorders 18 (4), 459-479.

Geers, A.L. & Lassiter, G.D. (2005) Affective assimilation and contrast: effects of expectations and prior stimulus exposure. Basic & Applied Social Psychology 27(2), 143-155.

Gottlieb, D., Vigoda-Gadot, E. & Haim, A. (2013) Encouraging ecological behaviors among students by using the ecological footprint as an educational tool: A quasi-experimental design in a public high school in the city of Haifa. Environmental Education Research 19(6), 844-863.

Haapanen, R. and Britton, L. (2002). Drug Testing for Youthful Offenders on Parole: An Experimental Evaluation. Criminology and Public Policy 1(2): 217-244.

Hagglund, P. (2006) Job-search assistance using the internet: Experiences from a Swedish randomised experiment. International Journal of Manpower 27(5), 434-451.

Hagglund, P. (2014) Experimental evidence from active placement efforts among unemployed in Sweden. Evaluation Review 38(3), 191-216.

Haslum, M.N. (2007) What kind of evidence do we need for evaluating therapeutic interventions? Dyslexia 13, 234-239.

Herrmann, D. S. and McWhirter, J. J. (2003). Anger and Aggression Management in Young Adolescents: An Experimental Validation of the SCARE Program. Education and Treatment of Children 26(3): 273-302.

Hilbert, A. & Tuschen-Caffier, B. (2004). Body image interventions in cognitive-behavioural therapy of binge-eating disorder: a component analysis. Behaviour Research & Therapy 42(11), 1325-1340.

Hobolt, S., Tilley, J., and Wittrock, J. (2013) Listening to the government: How information shapes responsibility attributions. Political Behavior 35(1), 153-174.

Hoffman, S.G., Moscovitch, D.A., Kim, H. & Taylor, A.N. (2004). Changes in self-perception during treatment of social phobia. Journal of Consulting & Clinical Psychology 72(4), 588-597.

Hong, Ki Won (2004) Product evaluation bias under minimal group situations. The Social Science Journal 41(4), 667-673.

Hutchings, V.L., Walton Jr., H. & Benjamin, A. (2010) The impact of explicit racial cues on gender differences in support for Confederate symbols and partisanship. Journal of Politics 72(4), 1174-1188.

Jacob, R.T. (2011) An experiment to test the feasibility and quality of a web-based questionnaire of teachers. Education Review 35(1), 40-70.

Jakobsson, N. & Lindholm, H. (2014) Ethnic preferences in internet dating: A field experiment. Marriage & Family Review 50(4), 307-317.

Jung, D. I. (2001). Transformational and Transactional Leadership and Their Effects on Creativity in Groups. Creativity Research Journal 13(2): 185-195.

Krause, M.S. & Howard K.I. (2003) What random assignment does and does not do. Journal of Clinical Psychology 59(7), 751-766.

Lager, A.C.J. & Torssander, J. (2012) Causal effect of education on mortality in a quasi-experiment on 1.2 million Swedes. Proceedings of the National Academy of Sciences of the United States of America 109(22), 8461-8466.

Lai, E.S.Y., Kwok, C.L., Wong, P.W.C. et al. (2016) The effectiveness and sustainability of a universal school based programme for preventing depression in Chinese adolescents: A follow-up study using quasi-experimental design. PLoS ONE 11(2), 1-20.

Lavine, H., Lodge, M. & Freitas, K. (2005) Threat, authoritarianism, and selective exposure to information. Political Psychology 26(2), 219-256.

Mason, M.F., Tatkow, E.P. & Macrae, C.N. (2005) The look of love: gaze shifts and person perception. Psychological Science 16(3), 236-248.

Mauss, I.B., Wilhelm, F.H. & Groos, J.J. (2004). Is there less to social anxiety than meets the eye? Emotion experience, expression, and bodily responding. Cognition & Emotion 18(5), 631-663.

McGraw, A.P. & Tetlock, P.E. (2005) Taboo trade-offs, relational framing, and the acceptability of exchanges. Journal of Consumer Psychology 15(1), 2-16.

Nesdale, D., Durkin, K., Maass, A. & Griffiths, J. (2005) Threat, group identification, and children's ethnic prejudice. Social Development 14(2), 189-207.

Nolan, A. (2011) An extension in eligibility for free primary care and avoidable hospitalisations: A natural experiment. Social Science & Medicine 73(7), 978-985.

Norstrom, T. (2005) Saturday opening of alcohol retail shops in Sweden: An experiment in two phases. Addiction 100(6), 767-776.

Page, K.M. & Vella-Brodrick, D.A. (2013) The Working for Wellness program: RCT of an employee well-being intervention. Journal of Happiness Studies 14(3), 1007-1031.

Power, R., Khalfin, R., Nozhkina, N. & Kanarsky, I.A. (2004). An evaluation of harm reduction interventions targeting injecting drug users in Sverdlovsk Oblast, Russia. International Journal of Drug Policy 15(4), 305-311.

Reeves, A. & de Vries, R. (2016) Does media coverage influence public attitudes towards welfare recipients? The impact of the 2011 English riots. British Journal of Sociology 67(2), 281-306.

Riemersma, I., van Santvoort, F., Janssens, J.M.A.M. et al. (2015) "You are okay": A support and educational program for children with mild intellectual disability and their parents with a mental illness: Study protocol of a quasi-experimental design. BMC Psychiatry 15, 1-9.

Rodebaugh, T.L. (2004) I might look OK, but I'm still doubtful, anxious, and avoidant: The mixed effects of enhanced video feedback on social anxiety symptoms. Behaviour Research and Therapy 42(12), 1435-1451.

Rodgers, J., Herrema, R., Freeston, M. & Honey, E. Towards a treatment for intolerance of uncertainty for autistic adults: A single case experimental design study. Journal of Autism and Developmental Disorders. 48(8), 2832-2845 DOI: 10.1007/s10803-018-3550-9.

Scheepers, D. & Ellemers, N. (2005) When the pressure is up: the assessment of social identity threat in low and high status groups. Journal of Experimental Social Psychology 41(2), 192-200.

Shirom, A., Vinokur, A. & Price, R. (2008) Self-efficacy as a moderator of the effects of job-search workshops on re-employment: A field experiment. Journal of Applied Social Psychology 38(7), 1778-1804.

Skalicka, V., Belsky, J., Stenseng, F. & Wichstrom, L. (2015) Preschool-age problem behavior and teacher-child conflict in school: Direct and moderation effects by preschool organization. Child Development 86(3), 955-964.

Thomaes, S., Bushman, B.J., Orobio de Castro, B. et al. (2009) Reducing narcissistic aggression by buttressing self-esteem: An experimental field study. Psychological Science 20(12), 1536-1542.

Tsvetkova, M. & Macy, M.W. (2014) The social contagion of generosity. PLoS ONE 9(2), 1-9.

Tucker, T., Fry, C.L., Lintzeris, N., Baldwin, S., Ritter, A., Donath, S. & Whelan, G. (2004). Randomized controlled trial of a brief behavioural intervention for reducing hepatitis C virus risk practices among injecting drug users. Addiction 99(9), 1157-1167.

VanZomeren, M., Spears, R., Fischer, A. & Fischer, A.H. (2004) Put your money where your mouth is! Explaining collective action tendencies through group-based anger and group efficacy. Journal of Personality and Social Psychology 87(5), 649-665.

Vescio, T.K., Gervais, S.J. & Snyder, M. (2005) Power and the creation of patronizing environments: the stereotyp-based behaviors of the powerful and their effects on female performance in masculine domains. Journal of Personality and Social Psychology 88(4)-658-672.

Verhofstadt, L., Buysse, A., Ickes, W., DeClercq, A. & Peene, O.J. (2005) Conflict and support interactions in marriage: an analysis of couples' interactive behavior and on-line cognition. Personal Relationships 12(1), 23-43.

Waldzus, S., Mummendey, A. & Wenzel, M. (2005) When "different" means "worse": In-group prototypicality in changing intergroup contexts. Journal of Experimental Social Psychology 41(1), 76-83.