a form of research that resembles a true experiment, but does not allow you to draw as firm conclusions about causality typically done in real-life settings --> high external validity

differences between laboratory and real-life (natural) settings

control: higher in lab settings - random assignment, holding constant, balancing

external validity: higher in natural settings

goals: basic (lab) vs. applied (natural) research

consequences: far-reaching impact is greater for natural settings


True experiments lead to an unambiguous outcome regarding what caused an event

• intervention or treatment is implemented (IV is manipulated)

• appropriate comparison to evaluate the IV (simplest involves an experimental and control group)

• control via arranging experimental conditions, assigning subjects, choosing DVs, manipulation of IV, counterbalancing and holding constant potential confounds

Obstacles to conducting true experiments in natural settings - threats to internal validity controlled by experiments

In a true experiment, we can attribute the differences between the experimental and the control group to the manipulation of the IV when everything else is held constant and/or balanced appropriately.

Without an appropriate comparison or in situations where internal validity is not attained, the following may become potential explanations for the differences:

1) History - difference as a function of events
Examples: changes in staff or a new diet may change the results; therapy might improve behavior that might be attributed to new policies used in an institutional setting; a cataclysmic event can affect the outcome (Sept. 11 disaster, war, earthquake)

2) Maturation - difference as a function of time
Example: unpopular children at the beginning of the school year are put in a special program to help them learn to get along with others -- by the end of the year, they are doing much better

3) Testing - difference as a function of retesting
Example: improvement on a second test may come from the treatment or from retesting

4) Instrumentation - difference as a function of changes/differences in measurement -- observers might become more or less sensitive, fatigued, better at interviewing, etc.; even machines are subject to instrumentation error (e.g., a flaw in collecting data in one condition, but not another)

5) Statistical Regression - difference as a function of regression to the mean [most often found when extreme scores are used]; Example: slow learners as assessed by a test are given remedial work and show improvement

6) Selection - difference as a function of pre-existing group differences
Example: one group may be smarter, abused more, more vulnerable to stress, etc. than the other group

7) Subject Mortality - difference as a function of subject loss -- loss of subjects can change the groups so that they are no longer comparable

8) Interactions with SELECTION - differences as a function of differential change between the groups (must have at least two levels of the IV present)

Selection X History would be the case where only one group experienced an event not experienced by the other or when the event has a different effect on the groups

Selection X Maturation would occur when one group is changing at a different pace than another

Selection X Instrumentation would occur when one group is measured differently than another

threats that even true experiments might not eliminate quasi-experiments Note: a quasi-experimental design is NOT the same as a natural groups design; in a quasi-experimental design, the group that one is "preassigned to" represents one level of a treatment variable — an IV that under ideal conditions could be a true, manipulated IV quasi-experimental designs become especially useful when random assignment • is not possible,
• may restrict external validity (e.g., field experiments), or
• realistically cannot occur
Without random assignment, the groups cannot be assumed to be equivalent; you must establish that. Perhaps one of the best ways to establish equivalency is 1) to do a pretest on the groups, then
2) introduce the treatment to the "experimental" group and finally,
3) give a posttest
Quasi-Experimental Designs

non-equivalent control group design

• used when we strongly suspect that groups are not equivalent at the onset

• often used in education settings (or should be used) and for evaluation research

• in this design, a pre- and posttest is administered

outlined as: (O = observation, X = treatment, ----- = no random assignment)

O1   X1    O2
O1           O2

controls for

• selection (pre and posttest)

• history, maturation, testing, instrumentation, and regression (control group)

does not take care of other threats especially interactions with selection

possible outcomes:

1) increasing treatment effect I outcome - flat control, treatment group initially higher and increases (no crossover)

threats: local history, selection X maturation, no difference is likely
2) increasing treatment and control (no crossover) [rich get richer] -
threat: selection X maturation
3) Increasing treatment effect II outcome: flat control (better at pre and posttest), increasing treatment (no crossover)
threat: regression, selection X history
4) crossover effect: flat control, increasing treatment with crossover
threats are greatly reduced (unlikely that statistical regression, selection X maturation, or testing has occurred)
simple interrupted time-series design • a repeated measures design with only one group

• several pre and posttest measures which give us more info about normal fluctuations

pretests                                   posttests
O1   O2   O3   O4   X   O5   O6   O7   O8   ...ON

several possible outcomes, but let’s examine 5:

1) true, long lasting effect - easiest to tell that an effect has occurred, discontinuity

2) chance variation - seemingly discontinuous, but much fluctuation

3) temporary effect - discontinuity with treatment, but effect dissipates

4) lag effect - same as (1), but discontinuity lags behind treatment introduction

5) maturation effect - increasing slope

controls for: • maturation (before the event)

• testing (can be examined)

However, we cannot overrule some threats to validity - especially history and instrumentation (if changes coincide with the presentation of the treatment)

time series with non-equivalent control group

one might refine the time-series design by including a control group, as in

pretests                                      posttests
O1   O2   O3   O4   X1   O5   O6   O7   O8   ...ON
O1   O2   O3   O4          O5   O6   O7   O8   ...ON

In this way, we can somewhat take care of the history threat, but selection X history remains a threat
Example: introducing a new training technique to athletes during the middle of a season (e.g., imagery)

Program Evaluation

goals: to assess the effectiveness of human service organizations and to provide feedback to the agency being evaluated

can use descriptive, correlational, experimental or quasi-experimental methods to examine the needs, processes, outcomes, and/or efficiency of programs and services


applied research

there are several difficulties with meaningful evaluation; some include