What the Words Mean
Help for Understanding SBR From the
Software & Information Industry Association
To appreciate the true impact of scientifically based research on education, it may be useful first to understand the meaning of the various terms that the law employs as part of its definition of SBR. The following explication is from a Software & Information Industry Association publication titled "Scientifically Based Research: A Guide for Education Publishers and Developers."
To begin, the word scientific itself warrants mention. Scientific means "relating to, or employing the methodology of science," according to The American Heritage Dictionary of the English Language, Fourth Edition. Further, the methodology of science is "a continual process" applied over time (Committee on Scientific Principles for Education Research 2002). In other words, a single experiment, no matter how rigorous, d'es not constitute an indisputable body of scientific evidence. Paragraph A of the No Child Left Behind Act Section 9101 provides a top-level summary of the requirements of SBR. It describes SBR as "the application of rigorous, systematic and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs." Explanations of these key words follow. The order of presentation differs from the order in paragraph A for the sake of clarity.
Relevance
Relevance is addressed first because it determines the extent to which the other criteria for scientific research can be met. Relevance has two aspects: importance and alignment.
- Importance. Research is relevant when it addresses questions of theoretical and practical importance or need to the fields of education practice or policy, particularly with regard to student learning. Important research questions and approaches link to and flow from relevant theory and theory-based research.
- Alignment. To be relevant, not only must research study questions be important, but research procedures, analyses, and reported claims and conclusions must be of high quality and in alignment with the research questions and with each other. The research questions must guide the research design and analyses. In addition, conclusions and claims must be supported by the data and tie back to the research question(s).
Rigor
Rigor literally means strict, firm or rigid, and applies to the quality of the research. In research terms, it means:
- Selecting an appropriate research design, research instruments and procedures for addressing the research questions at hand.
- Implementing the design and procedures according to practices acceptable to the scientific community with the same or highly similar results.
Systematic
Systematic is a characteristic of the research as it is conducted that makes research logical, organized and methodical. Systematic study allows researchers to answer "what" questions such as what intervention or combination of interventions, applied to what degrees of intensity, contribute to certain differences in learning. Systematic approaches to research constitute one of the greatest challenges to conducting research in real-world settings such as schools.
Objective
Objective research is independent research conducted by a credible third party using rigorous and systematic methodology. Alternatives include research conducted by a company itself that includes 1) up-front design consultation that builds in safeguards against potential bias, and 2) follow-up, third-party review or audit. For research to be objective, two equally competent researchers, given the same data, must arrive independently at the same conclusions.
Reliable
Reliability is about consistency. The more reliable the research instruments and the more reliably the research procedures are applied, the greater the probability that:
- The same researcher can repeat the study with the same or highly similar results.
- Another researcher can replicate the study's methodology and obtain the same or similar results.
Reliability and its cousin, replicability, are important because one experiment or study by itself d'es not constitute an irrefutable body of evidence.
Valid
While reliability concerns consistency, validity is about accuracy. Imagine a target. Reliable marksmen would consistently hit the same spot on the target, but this spot might not be the bull's-eye. A marksman who shoots with validity would hit the center of the target. A reliable and valid marksman would hit the center of the target consistently. In research, we must be concerned with the validity of research instruments, procedures and findings. And validity must always be interpreted in terms of purpose. For example, if a test is designed to measure the extent to which a student has mastered a particular curriculum but the results are used to determine how well the student compares to students who may have been taught other curricula, then the findings of the comparison are not valid. The strength of a research design for meeting these criteria revolves around the extent to which the study rules out "alternative" explanations of the study's outcomes (i.e., explanations other than the treatment intervention).
Experimental and Quasi-Experimental Designs
Neither NCLB nor other credible documents such as the National Research Council report "Scientific Research in Education" (2002) claim that experiments and quasi-experiments are the only approaches to scientifically based research. The point of NCLB's emphasis on experimental and quasi-experimental designs is that these are the specific designs that can demonstrate cause-and-effect relationships. Clearly, other designs are appropriate, or even better suited, to addressing certain research questions. Which design, or combination of designs, is most appropriate for addressing a particular issue will depend on a number of factors, including the goals of the research, the extent to which the study is part of a larger body of research, and the claims that the researchers wish to be able to make on the basis of the research data.
The hallmark of both experimental and quasi-experimental designs is that they employ experimental and comparison groups. An experimental group is a group in a research study that receives the "treatment" or intervention. In order to determine whether or not there is an experimental effect, experiments and quasi-experiments require a basis for comparison. Sometimes a control group is used (i.e., a group that d'es not receive an experimental treatment). In medical trials, the control group d'es not receive the treatment drug. In education, a control group usually consists of a group of students taught by a traditional method rather than by the "experimental" method.
While experimental and quasi-experimental designs must include at least one comparison group, they may or may not include a control group. ... The hallmark, then, of experiments and quasi-experiments is that there be a group that can be used for comparison purposes, for example, between two or more experimental treatments, or between one or more experimental treatments and a control treatment.
The main difference between experiments and quasi-experiments is that in experiments, study participants are randomly selected from the population to which results of the study are to be generalized and/or randomly assigned to experimental and comparison groups, but not necessarily both. [In contrast,] quasi-experiments do not employ randomization. Random selection is not haphazard or arbitrary. It follows a specified procedure using a table of random numbers or a computer program for random selection.
Random Assignment
Random assignment refers to the use of a random numbers table or computer program to assign members of the sample, whether randomly selected or not, to experimental or comparison groups. The process of random assignment helps ensure that any differences between the experimental and comparison groups, before the start of the experiment, are due to chance alone (i.e., due to random sampling fluctuations).
When both random selection and random assignment are not feasible, researchers must determine which is more important to the claims they want to be able to make on the basis of the results. The U.S. Department of Education's Institute of Education Sciences, which oversees the What Works Clearinghouse, currently places emphasis on random assignment, the rationale being that random selection is seldom possible in the real world of education practice.
T.H.E. Journal readers receive a discount on SIIA's "Scientifically Based Research: A Guide for Education Publishers and Developers" by logging on to www.siia.net/store/describe/thes-sbr.html.
Reference
Committee on Scientific Principles for Education Research. 2002. Scientific Research in Education. National Academy Press: Washington, D.C.
This article originally appeared in the 02/01/2004 issue of THE Journal.