• Formulate research and evaluation questions that are appropriate for true experiments and quasi-experimental designs;
• Select an appropriate type of true or quasi-experiment (Solomon four-group, randomized complete block, etc.) design to create unambiguous answers to research questions and to evaluate the effects of programmatic interventions like training programs;
• Develop sampling and analytic strategies appropriate to the specific design selected;
• Interpret the results of true and quasi-experiments and use them to reach warranted conclusions;
• Understand the weaknesses of quasi-experiments and employ a variety of principles and techniques, such as sample matching, multiple control groups, multiple pre-testing, and others to increase the internal and external validity of conclusions drawn from quasi-experiment; and
• Assess the degree to which you can apply the conclusions drawn from true and quasi-experiments to your own work.

Materials

Topic 1: Social scientists don't do experiments. Or DO they? SHOULD they?

Do we DO experiments?Actually, I suspect that there is more use of experiments in social science than most people think. Based on my many years of experience with this class and as a "practicing social scientist", I find that most students and even faculty members think that social scienitst (except psychologists) just don't "do" experiments. It is true that social scientists use experimental designs much less than scientists in the biological, earth, or physical sciences. Nonetheless, social scientists actually do use experiments. And I think we will be doing more experiments because donors are starting to demand that we produce the same kind of evidence to support the claims we make about what our research reveals that all the other scientists provide -- e.g., the people who pay for the research are setting new standards. I do not know why people think that experiments aren't "good" for social science and/or that there is no way for a social scientist to conduct an experiment. This video about the Oregon Healthcare Experiment gives one example of a social science experiment. I strongly encourage you to look at some of the other videos in the Causal Inference Bootcamp: Your Guide to Experiments at Duke University's Social Science Research Institute. Video 1 (Controlled Experiments), Video 2 (Randomized Experiments), Video 15 (Randomized Controlled Trials), and Video 21 (The Two Kinds of Natural Experiments) will be especially helpful with Assignment 4. This Duke University site is a very good place to get quick, clear discussions of specific topics -- efficient and effective -- and will probably be very helpful to you when you work on Assignment 3.

SHOULD we do experiments?

I do think we should use experiments much more than we do. Experiments (whether true or quasi) are the only designs that allow us to determine if an intervention has any discernible effect. To the extent that we fail to provide experimental evidence that a theory, when applied correctly, actually produces the predicted (wanted, desired) outcome, we have only relatively weak correlational evience to support our arguments to all sorts of interventions. We need internally and externally valid evidence that what we tell people to do will actually result in the outcome predicted. Until we can provide such evidence, getting social science concepts into widespread application will be very difficult. There has, unfortunately, developed a litany about applied versus theoretical research, or even more destructive fairly virulent discussion about the value of practice-versus-research, with some proponents of the importance of practice arguing that research is irrelevant to practice. I am hoping that the discussion by Green and Glasgow will bring some balance to this discussion in our class. A lot of the discussion revolves around the eternal validity of research results -- whether any practitioner can use them or not. The authors deal with this in detail on pp. 136-141. My own current research focuses on how to create "end users driven" research -- without tossing out experimental designs.

Green, L.W. & Glasgow, R.E. (2006) Evaluating the relevance, generalization and applicability of research: Issues in external validation and translation methodology. Evaluation & the Health Professions 29(1): 126-153 DOI: 10.1177/0163278705284445 Pages 136-142 (REQ). Note that the PRECEDE-PROCEED model discussed in this article forms the theoretical basis for the Haiti Extension Experiment.

Topic 2: Isn't it impossible for social scientists to conduct experiments?

The short answer is NO. Some social sciences, psychology and economics are examples, have long used experimental research designs. They are also relatively common in education. At a more basic level, if we cannot use this fundamental tool of science, we would have to question whether we should call what we do science. I say this because experiments are the only group of research designs that can provide evidence of a direct cause and effect relationship where the cause is a treatment (an intervention, a program) that we put into place and the effect is the outcome that it produces. If we cannot establish basic causal effects, the claims we make about causality (that the work we do produces desired outcomes) is very weak.

Chapter 4 in Bernard is absolutely critical reading (REQ) for this passage into the mysteries of actual research designs. BUT before you start to read the chapter, go to p. 124 of Bernard's book and read his summary of key points. I would suggest copying them and pasting them into a document for your own reference (and very helpful in Assignment 3) and that you write down any questions you have or any ideas that each summary point raises for you. This is how you USE, CITE, and REFERENCE the research design literature.

Topic 3: SOOOO Many KINDS of Experiments. Its bewildering!

There are three ways of dividing up experiments into "types" based on: (1) how people are assigned to treatment and control; (2) who implemented the experiment; and (3) specific design features, including number of factors, number of comparison groups, number of instances and others.Start your exploration with this short video about types of designs. It lays out the three main groupings very well -- and in 5 minutes.

How people are assigned to treatments and control. We are covering two general categories of designs with controls and interventions -- true experiments and quasi-experiments. True experiments provide stronger evidence of a direct cause and effect relationship between one or more treatments (interventions) and outcomes. Use true experiments if you can when you need to know whether an intervention "works". However, everything Bernard says about "experiments" also apply to "quasi-experiments". The difference has to do with random assignment (not random selection, which is NOT a requirement of experiments of any type). In true-experiments, individual research units (birds, bears, atoms, people) are assigned to the treatment and control groups. In quasi-experiments, pre-existing groups of participants are randomly assigned to treatments and control groups. We use them when people "come in groups" and cannot realistically be assigned individually. Often, the group is one that is voluntarily formed by people -- membership in a sport club or in a farmer organization (see the Haiti Experiment) or some other formal group. In many cases, the groups are created by some authority -- grade school students come in groups based on home room, people in prison are assigned to the jail where they are incarcerated, people under medical treatment are in different hospitals. In Haiti, for example, farmer organizations are a very prevalent and important aspect of rural life. Pretty much "everyone" belongs to a farmer group. We could have assigned individuals to treatment -- but that would actually have been a BAD idea, and we will discuss why that is true in class. Quasi-experiments are by far the most common design used in evaluating interventions, such as educational programs. You will almost undoubtedly have to use a quasi-experiment at some point in your professional career and some of you will have to use a quasi-experiment for Assignment 3. You do NOT need to look at these two slide shows ahead of time. We will cover them in class. If you find this confusing, the Greeno article "Major alternatives to the classic experimental design" is excellent. It is easy to read and gives good examples.

Slide Show -- Requirements of True Experiments

Slide Show -- Quasi-Experiments

Who implemented the experiment. This ranges from YOU, the researcher, to some government agency or researcher who was working on the same project independently of you who assigns to treatment/control, some other researcher whose data you can acquire (you don't even know him/her) and who may have had a totally different question from yours (leading to metanalysis in some cases) or any organization that is involved in "putting some people into treatments and others not" even if the organization did not think of this as assignment to treatment and control groups. The basic point is that somebody did in fact assign people to treatment(s) and others not (control) -- that is the essence of the experimental design. Naturally occuring comparison groups (men vs. women, adolescents vs. emerging adults, organic vs. conventional strawberry producers, kids in foster care vs. with birth parents, and on and on) are NOT treatment and control groups. They are comparison groups and are often the focus of cross-sectional or longitudinal designs, but these natural groups are not experimental groups. In experiments -- somebody did something to someone. The Duke site video Two Kinds of Natural Experiments explains these differences well. " You may want to watch Justifying As-If Randomization as well -- I suspect you will need this.

Specific Design Features. Bernard has a good discussion of many of these ways of setting up experimental designs on pp. 99-110. My cheat sheet (linked below) includes examples as well. All of these decisions have important effects on the conclusions you can draw from an experiment. In the "classic" experiment, data for the outcome variable are collected prior to implementing the experiment and again at the end of the experiment (in its simplest form). The difference in the outcome variable score (like weight loss) is tested for the treatment and control group. If the difference is significant, we conclude that the treatment had an effect. In fact, some experiments have no data for "before treatment." One whole group based on who implemented the experiment -- the experments conducted for a completely different reason with a different outcome understudy -- are an example. The lack of pre-implementation data has numerous effects on internal and external validity (which Bernard discusses). Simply put, without knowing the score for a variable prior to "doing something" to people, you have to think hard about how to know whether the scores changed or not due to any intervention. Multiple data collections over time (repeated measures) are common. These are often used to test the persistence of an effect. Does the effect last after the controlled conditions are gone or do people revert to their prior behavior, for example. We desperately need more experiments with multiple-post intervention data points to know whether the changes we see are truly lasting or not.

Types of Experiments (my cheat sheet)

Topic 4: Making Sense of it all. I know this is a lot of material I have tried to provide a lot of background so that you can succeed on Assignment 3. The last few links here are more summative in nature and shold help you succeed in creating your own experimental design. Please look at these prior to class, but you do not have to "study" them. DO use them when you complete Assignment 3.

This very brief discussion of the logic of data analysis for experiments should help you with Assignment 3. Focus on the logic of it, not the details. Data Analysis in Experiments (REQ - MOSTLY IN CLASS, BUT LOOK AT IT AHEAD OF TIME)

Threats to Validity This is the obligatory (I feel) list of potential threats to validity in research. PLEASE DO NOT treat this as a checklist. The only way one can use this is to think carefully about one's design decisions and figure out how to reduce or eliminate threats. You CANNOT ELIMINATE ALL OF THEM. The point is to be aware that they exist, think about which (if any) are "serious" threats in a particular study and design, and address them to the best of your ability. If you start using them as a checklist, you will quickly find NO research is "good enough." I do NOT encourage you to go down that path.

The questions on the learning guide this week really require that you synthesize a lot of what you have learned so far in the semester. I most strongly encourage you to look at the guide and see if you can answer the questions. If not, raise the point in class. If one person has trouble, others usually do as well. Learning Guide: True and Quasi-Experiments (REQ)

Recommended Readings

D'Onofrio, B.M., Lahey, B.B., Turkheimer, E. & Lichtenstein, P. (2013) Critical need for family-based, quasi-experimental designs in integrating genetic and social science research. American Journal of Public Health 103(S1), S46-S55. DOI: 10.2105/AJPH.2013.301252.

Fletcher, J.M. & Conley, D. (2013) The challenge of causal inference in gene-environment interaction researchy: Leveraging research designs from the social sciences. American Journal of Public Health 103(S1), S42-S45 (supplement). DOI: 10.2105/AJPH.2013.301290.

Gersten, R., Baker, S. & Lloyd, J.W. (2000) Designing high-quality research in special education: group experimental design Journal of Special Education 34(1), 2-18.

Greeno, C.G. (2001) The skeleton: What underlies treatment research? Family Process 40(3), 361-363.

Greeno, C.G. (2001) The classical experimental design. Family Process 40(4), 495-499.

Lager, A.C.J. & Torssander, J. (2012) Causal effect of education on mortality in a quasi-experiment on 1.2 million Swedes. Proceedings of the National Academy of Sciences of the United States of America. 109(22), 8461-8466. https://doi.org/10.1073/pnas.1105839109

Rumrill, P.D., Jr. & Bellini, J.L. (1999) The logic of experimental design. J. Vocational Rehabilitation. 13, 65-70.

Shadish, W.R., Cook, T.D. & Campbell, D.T. (2002) Quasi-experimental designs that either lack a control group or lack pretest observations on the outcome. Pp. 103-134 in Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Wadsworth, Belmont, CA. e-reserve

Solomon, B.R., Draine, P., DeMoya, J. & Wickrema, R. (1998) D esign-based evaluations: Process studies, experiments and quasi-experiments. Scandanavian Journal of Social Welfare 7, 126-131.

Thayer, B.A. (2012) The role of group research designs to evaluate social work practice. Quasi-Experimental Research Designs. Oxford University Press, pp. 77-106. e-reserve

Travers, J.C., Cook, B.G., Therrien, W.J. & Coyne, M.D. (2016) Replication research and special education. Remedial and Special Education 37(4), 195-204. DOI: 10.1177/0741932516648462

White, H. (2013) An introduction to the Use of randomised control trials to evaluate development interventions. Journal of Development Effectiveness 5(1), 30-49. DOI 10.1080/19439342.2013.764652

Advance Preparation -- have all of these available in class, electronic or hard copy

Extension Model Experiment for Haiti -- examine before class. I prepared this experimental design using the flow chart for constructing designs that you use in this class.

Flow Chart for Constructing Designs

Statistics

Additional Materials. These are examples of research reports using both true and quasi-experiments. They contain good discussions of how the experiments were designed and, in most cases, of how the data were analyzed. You can get ideas from these about how experiments can be used in research and practice and how to analyze the data. All of these can be used as reference for your assignment. To figure out if the study will help you, go to Academic Search Premier. Look for it and then check out the keywords.

Becher, E.H., McGuire, J.K., McCann, E.M., Powell, S., Cronin, S.E. & Deenanath, V. (2018) Extension-based divorce educaiton: A quasi-experimental design study of the parents forever program. Journal of Divorce & Remarriage 59(8), 633-652. DOI: 10.1080/10502556.2018.1466256.

Bonetti, D., Eccles, M., Johnston, M., Steen, N., Grimshaw, J., Baker, R., Walker, A. & Pitts, N. (2005) Guiding the design and selection of interventions to influence the implementation of evidence-based practice: An experimental simulation of a complex intervention trial. Social Science & Medicine 60(9), 2135-2147.

Bravo, G. & Squazzoni, F. (2013) Exit, punishment and rewards in commons dilemmas: An experimental study. PLoS ONE 8(8), 1-6.

Butterfield, R., Park, E.R., Puleo, E., Mertens, A., Gritz, E.R., Frederick, L. & Emmons, K. (2004). Multiple risk behaviors among smokers in the childhood cancer survivors study cohort. Psycho-Oncology 13(9), 619-630.

Byiers, B.J., Reichle, J. & Symons, F.J. (2012) Single-subject experimental design for evidence-based practice. American Journal of Speech-Language Pathology 21(4), 397-414.

Carter, H., Drury, J., Amiot, R., Rubin, G.J. & Williams, R. (2014) Effective responder communication improves efficiency and psychological outcomes in a mass decontamination field experiment: Implications for public behavior in the event of a chemical incident. PLoS ONE 9(3), 1-12.

Cason, T.N., Saijo, T., Yamato, T. & Yokotani, K. (2004) Non-excludable public good experiments. Games & Economic Behavior 49(1), 81-102.

Chirumbolo, A., Manneteti, L., Pierro, A., Areni, A. & Kruglanski, A.W. (2005) Motivated closed-mindedness and creativity in small groups. Small Group Research 36(1), 59-82.

Class, Q., D'Onofrio, B., Singh, A., Ganiban, J. et al. (2012) Current parental depression and offspring perceived self-competence: A quasi-experimental examination. Behavior Genetics 42(5), 787-797.

deLuse, S.R., & Braver, S.L. (2015) A rigorous quasi-experimental design to evaluate the causal effect of a mandatory divorce education program. Family Court Review 53(1), 66-78.

Downs, J.S., Murray, P.J., Bruine de Bruin, W., Penrose, J., Palmgren, C. & Fischhoff, B. (2004). Interactive video behavioral intervention to reduce adolescent females' STD risk: a randomized controlled trial. Social Science & Medicine 59(8), 1561-1573.

Gallagher, H.M., Rabian, B.A. & McCloskey, M.S. (2004). A brief group cognitive-behavioral intervention for social phobia in childhood. Journal of Anxiety Disorders 18 (4), 459-479.

Geers, A.L. & Lassiter, G.D. (2005) Affective assimilation and contrast: effects of expectations and prior stimulus exposure. Basic & Applied Social Psychology 27(2), 143-155.

Gottlieb, D., Vigoda-Gadot, E. & Haim, A. (2013) Encouraging ecological behaviors among students by using the ecological footprint as an educational tool: A quasi-experimental design in a public high school in the city of Haifa. Environmental Education Research 19(6), 844-863.

Haapanen, R. and Britton, L. (2002). Drug Testing for Youthful Offenders on Parole: An Experimental Evaluation. Criminology and Public Policy 1(2): 217-244.

Hagglund, P. (2006) Job-search assistance using the internet: Experiences from a Swedish randomised experiment. International Journal of Manpower 27(5), 434-451.

Hagglund, P. (2014) Experimental evidence from active placement efforts among unemployed in Sweden. Evaluation Review 38(3), 191-216.

Haslum, M.N. (2007) What kind of evidence do we need for evaluating therapeutic interventions? Dyslexia 13, 234-239.

Herrmann, D. S. and McWhirter, J. J. (2003). Anger and Aggression Management in Young Adolescents: An Experimental Validation of the SCARE Program. Education and Treatment of Children 26(3): 273-302.

Hilbert, A. & Tuschen-Caffier, B. (2004). Body image interventions in cognitive-behavioural therapy of binge-eating disorder: a component analysis. Behaviour Research & Therapy 42(11), 1325-1340.

Hobolt, S., Tilley, J., and Wittrock, J. (2013) Listening to the government: How information shapes responsibility attributions. Political Behavior 35(1), 153-174.

Hoffman, S.G., Moscovitch, D.A., Kim, H. & Taylor, A.N. (2004). Changes in self-perception during treatment of social phobia. Journal of Consulting & Clinical Psychology 72(4), 588-597.

Hong, Ki Won (2004) Product evaluation bias under minimal group situations. The Social Science Journal 41(4), 667-673.

Hutchings, V.L., Walton Jr., H. & Benjamin, A. (2010) The impact of explicit racial cues on gender differences in support for Confederate symbols and partisanship. Journal of Politics 72(4), 1174-1188.

Jacob, R.T. (2011) An experiment to test the feasibility and quality of a web-based questionnaire of teachers. Education Review 35(1), 40-70.

Jakobsson, N. & Lindholm, H. (2014) Ethnic preferences in internet dating: A field experiment. Marriage & Family Review 50(4), 307-317.

Jung, D. I. (2001). Transformational and Transactional Leadership and Their Effects on Creativity in Groups. Creativity Research Journal 13(2): 185-195.

Krause, M.S. & Howard K.I. (2003) What random assignment does and does not do. Journal of Clinical Psychology 59(7), 751-766.

Lager, A.C.J. & Torssander, J. (2012) Causal effect of education on mortality in a quasi-experiment on 1.2 million Swedes. Proceedings of the National Academy of Sciences of the United States of America 109(22), 8461-8466.

Lai, E.S.Y., Kwok, C.L., Wong, P.W.C. et al. (2016) The effectiveness and sustainability of a universal school based programme for preventing depression in Chinese adolescents: A follow-up study using quasi-experimental design. PLoS ONE 11(2), 1-20.

Lavine, H., Lodge, M. & Freitas, K. (2005) Threat, authoritarianism, and selective exposure to information. Political Psychology 26(2), 219-256.

Mason, M.F., Tatkow, E.P. & Macrae, C.N. (2005) The look of love: gaze shifts and person perception. Psychological Science 16(3), 236-248.

Mauss, I.B., Wilhelm, F.H. & Groos, J.J. (2004). Is there less to social anxiety than meets the eye? Emotion experience, expression, and bodily responding. Cognition & Emotion 18(5), 631-663.

McGraw, A.P. & Tetlock, P.E. (2005) Taboo trade-offs, relational framing, and the acceptability of exchanges. Journal of Consumer Psychology 15(1), 2-16.

Nesdale, D., Durkin, K., Maass, A. & Griffiths, J. (2005) Threat, group identification, and children's ethnic prejudice. Social Development 14(2), 189-207.

Nolan, A. (2011) An extension in eligibility for free primary care and avoidable hospitalisations: A natural experiment. Social Science & Medicine 73(7), 978-985.

Norstrom, T. (2005) Saturday opening of alcohol retail shops in Sweden: An experiment in two phases. Addiction 100(6), 767-776.

Page, K.M. & Vella-Brodrick, D.A. (2013) The Working for Wellness program: RCT of an employee well-being intervention. Journal of Happiness Studies 14(3), 1007-1031.

Power, R., Khalfin, R., Nozhkina, N. & Kanarsky, I.A. (2004). An evaluation of harm reduction interventions targeting injecting drug users in Sverdlovsk Oblast, Russia. International Journal of Drug Policy 15(4), 305-311.

Reeves, A. & de Vries, R. (2016) Does media coverage influence public attitudes towards welfare recipients? The impact of the 2011 English riots. British Journal of Sociology 67(2), 281-306.

Riemersma, I., van Santvoort, F., Janssens, J.M.A.M. et al. (2015) "You are okay": A support and educational program for children with mild intellectual disability and their parents with a mental illness: Study protocol of a quasi-experimental design. BMC Psychiatry 15, 1-9.

Rodebaugh, T.L. (2004) I might look OK, but I'm still doubtful, anxious, and avoidant: The mixed effects of enhanced video feedback on social anxiety symptoms. Behaviour Research and Therapy 42(12), 1435-1451.

Rodgers, J., Herrema, R., Freeston, M. & Honey, E. Towards a treatment for intolerance of uncertainty for autistic adults: A single case experimental design study. Journal of Autism and Developmental Disorders. 48(8), 2832-2845 DOI: 10.1007/s10803-018-3550-9.

Scheepers, D. & Ellemers, N. (2005) When the pressure is up: the assessment of social identity threat in low and high status groups. Journal of Experimental Social Psychology 41(2), 192-200.

Shirom, A., Vinokur, A. & Price, R. (2008) Self-efficacy as a moderator of the effects of job-search workshops on re-employment: A field experiment. Journal of Applied Social Psychology 38(7), 1778-1804.

Skalicka, V., Belsky, J., Stenseng, F. & Wichstrom, L. (2015) Preschool-age problem behavior and teacher-child conflict in school: Direct and moderation effects by preschool organization. Child Development 86(3), 955-964.

Thomaes, S., Bushman, B.J., Orobio de Castro, B. et al. (2009) Reducing narcissistic aggression by buttressing self-esteem: An experimental field study. Psychological Science 20(12), 1536-1542.

Tsvetkova, M. & Macy, M.W. (2014) The social contagion of generosity. PLoS ONE 9(2), 1-9.

Tucker, T., Fry, C.L., Lintzeris, N., Baldwin, S., Ritter, A., Donath, S. & Whelan, G. (2004). Randomized controlled trial of a brief behavioural intervention for reducing hepatitis C virus risk practices among injecting drug users. Addiction 99(9), 1157-1167.

VanZomeren, M., Spears, R., Fischer, A. & Fischer, A.H. (2004) Put your money where your mouth is! Explaining collective action tendencies through group-based anger and group efficacy. Journal of Personality and Social Psychology 87(5), 649-665.

Vescio, T.K., Gervais, S.J. & Snyder, M. (2005) Power and the creation of patronizing environments: the stereotyp-based behaviors of the powerful and their effects on female performance in masculine domains. Journal of Personality and Social Psychology 88(4)-658-672.

Verhofstadt, L., Buysse, A., Ickes, W., DeClercq, A. & Peene, O.J. (2005) Conflict and support interactions in marriage: an analysis of couples' interactive behavior and on-line cognition. Personal Relationships 12(1), 23-43.

Waldzus, S., Mummendey, A. & Wenzel, M. (2005) When "different" means "worse": In-group prototypicality in changing intergroup contexts. Journal of Experimental Social Psychology 41(1), 76-83.