Quasi-experiments

quasi-experiments

a form of research that resembles a true experiment, but does not allow you to draw as firm conclusions about causality typically done in real-life settings --> high external validity

differences between laboratory and real-life (natural) settings

control: higher in lab settings - random assignment, holding constant, balancing

external validity: higher in natural settings

goals: basic (lab) vs. applied (natural) research

consequences: far-reaching impact is greater for natural settings

TRUE EXPERIMENTS (a review)

True experiments lead to an unambiguous outcome regarding what caused an event

• intervention or treatment is implemented (IV is manipulated)

• appropriate comparison to evaluate the IV (simplest involves an experimental and control group)

• control via arranging experimental conditions, assigning subjects, choosing DVs, manipulation of IV, counterbalancing and holding constant potential confounds

Obstacles to conducting true experiments in natural settings - threats to internal validity controlled by experiments

In a true experiment, we can attribute the differences between the experimental and the control group to the manipulation of the IV when everything else is held constant and/or balanced appropriately.

Without an appropriate comparison or in situations where internal validity is not attained, the following may become potential explanations for the differences:

1) History - difference as a function of events
Examples: changes in staff or a new diet may change the results; therapy might improve behavior that might be attributed to new policies used in an institutional setting; a cataclysmic event can affect the outcome (Sept. 11 disaster, war, earthquake)

2) Maturation - difference as a function of time
Example: unpopular children at the beginning of the school year are put in a special program to help them learn to get along with others -- by the end of the year, they are doing much better

3) Testing - difference as a function of retesting
Example: improvement on a second test may come from the treatment or from retesting

4) Instrumentation - difference as a function of changes/differences in measurement -- observers might become more or less sensitive, fatigued, better at interviewing, etc.; even machines are subject to instrumentation error (e.g., a flaw in collecting data in one condition, but not another)

5) Statistical Regression - difference as a function of regression to the mean [most often found when extreme scores are used]; Example: slow learners as assessed by a test are given remedial work and show improvement

6) Selection - difference as a function of pre-existing group differences
Example: one group may be smarter, abused more, more vulnerable to stress, etc. than the other group

7) Subject Mortality - difference as a function of subject loss -- loss of subjects can change the groups so that they are no longer comparable

8) Interactions with SELECTION - differences as a function of differential change between the groups (must have at least two levels of the IV present)

Selection X History would be the case where only one group experienced an event not experienced by the other or when the event has a different effect on the groups

Selection X Maturation would occur when one group is changing at a different pace than another

Selection X Instrumentation would occur when one group is measured differently than another

threats that even true experiments might not eliminate quasi-experiments Note: a quasi-experimental design is NOT the same as a natural groups design; in a quasi-experimental design, the group that one is "preassigned to" represents one level of a treatment variable — an IV that under ideal conditions could be a true, manipulated IV quasi-experimental designs become especially useful when random assignment • is not possible,
• may restrict external validity (e.g., field experiments), or
• realistically cannot occur
Without random assignment, the groups cannot be assumed to be equivalent; you must establish that. Perhaps one of the best ways to establish equivalency is 1) to do a pretest on the groups, then
2) introduce the treatment to the "experimental" group and finally,
3) give a posttest
Quasi-Experimental Designs

non-equivalent control group design

• used when we strongly suspect that groups are not equivalent at the onset

• often used in education settings (or should be used) and for evaluation research

• in this design, a pre- and posttest is administered

outlined as: (O = observation, X = treatment, ----- = no random assignment)

O1   X1    O2
------------
O1           O2

controls for

• selection (pre and posttest)

• history, maturation, testing, instrumentation, and regression (control group)

does not take care of other threats especially interactions with selection

possible outcomes:

1) increasing treatment effect I outcome - flat control, treatment group initially higher and increases (no crossover)

threats: local history, selection X maturation, no difference is likely
2) increasing treatment and control (no crossover) [rich get richer] -
threat: selection X maturation
3) Increasing treatment effect II outcome: flat control (better at pre and posttest), increasing treatment (no crossover)
threat: regression, selection X history
4) crossover effect: flat control, increasing treatment with crossover
threats are greatly reduced (unlikely that statistical regression, selection X maturation, or testing has occurred)
simple interrupted time-series design • a repeated measures design with only one group

• several pre and posttest measures which give us more info about normal fluctuations

pretests                                   posttests
O1   O2   O3   O4   X   O5   O6   O7   O8   ...ON

several possible outcomes, but let’s examine 5:

1) true, long lasting effect - easiest to tell that an effect has occurred, discontinuity

2) chance variation - seemingly discontinuous, but much fluctuation

3) temporary effect - discontinuity with treatment, but effect dissipates

4) lag effect - same as (1), but discontinuity lags behind treatment introduction

5) maturation effect - increasing slope

controls for: • maturation (before the event)

• testing (can be examined)

However, we cannot overrule some threats to validity - especially history and instrumentation (if changes coincide with the presentation of the treatment)

time series with non-equivalent control group

one might refine the time-series design by including a control group, as in

pretests                                      posttests
O1   O2   O3   O4   X1   O5   O6   O7   O8   ...ON
----------------------------------
O1   O2   O3   O4          O5   O6   O7   O8   ...ON

In this way, we can somewhat take care of the history threat, but selection X history remains a threat
Example: introducing a new training technique to athletes during the middle of a season (e.g., imagery)

Program Evaluation

goals: to assess the effectiveness of human service organizations and to provide feedback to the agency being evaluated

can use descriptive, correlational, experimental or quasi-experimental methods to examine the needs, processes, outcomes, and/or efficiency of programs and services

types:

applied research

there are several difficulties with meaningful evaluation; some include