Educators regularly make decisions about what to do to improve student motivation and achievement. They use data, research, and evaluation to inform their decisions.
- Use of data: Educators are familiar with the idea of basing school improvement plans on the results of prior years’ student test results, and monitoring trends in the data over time. With the increasing sophistication of state and district databases, educators have access to important information about student progress over time.
- Use of research: Educators are also aware of the need to examine available research on the effectiveness of programs they are considering, particularly Scientifically-Based Research (SBR) studies that compare an intervention’s impact on students or teachers to a “business as usual” group.
- Use of evaluation: Finally, leaders realize that important improvement initiatives should be evaluated to find out whether they worked as intended, had the expected results, and how the plans of action could be improved.
One term that stresses the need to make more informed decisions using all these sources is “evidence-based decision-making.”
Evidence-based decision-making is a term that is being used across many professions which are trying to apply research and evaluation to their decision-making about clients. For example, the Oncology Nursing Society has produced “Evidence-Based Education Guidelines” for its members (who are practitioners) to use in presenting at meetings. The implication of the guidelines is that nurses should be able to articulate the “evidence base” for the nursing practices they recommend to others.
Evidence-based practice means delivering services to students (clients) in ways that integrate the best available evidence from data, research, and evaluation; professional wisdom gained from experience; and contextual knowledge of the particular classroom, school, district, or state that might impact the design or implementation.
One way of thinking about evidence-based decision-making is to visualize it as a cycle. The cycle helps in thinking about how decisions can be strengthened at every step in an improvement cycle. Each circle (stage in the cycle) can be considered separately, with revisions and improvements within that circle taking place continually. Although the circles show a progression in decision making reflecting the typical phases in planning, implementing, and evaluating an initiative, they can also provide a means for reflecting on which areas need more attention after improvement initiatives are underway. For example, oftentimes in the course of implementing a new initiative, the articulation of outcomes becomes clearer over time and the logic model developed at the beginning of the planning phase needs to be revised as implementation gets underway.
Evidence-Based Decision-Making (EBDM) Cycle
Examine Data to Understand Need
Entering into the cycle of evidence-based decision-making, educators use data to better understand the need or issue. In many cases, these data are student achievement data. For example, educators may examine math test scores and notice that students are performing particularly poorly on graphing and problem-solving skills. In other cases, educators may look at data that alert them to problems such as dropout rates, suspension rates, or problems with attendance. To supplement achievement data, decision makers are increasingly interested in measures of students’ emotional, behavioral, and cognitive engagement with their schools. Some states make data from teacher working conditions surveys available to all schools so that they can take teachers’ perceptions into account in their improvement planning.
Search for Information
The next stage of the EBDM Cycle involves searching for information about experiences of others related to the problem and what interventions address the problem. Searches might include: 1) a search for interventions related to the issue and 2) a search of the published literature on the issue.
If a particular set of studies is identified that might be relevant to decision making about an intervention or program of interest, it can be useful to establish a protocol for how the team might review and discuss the information identified. As potential interventions or strategies that might address the needs within the context of the district or school are identified, it will be important to also examine the quality of the research for the interventions/strategies. There are several resources that provide such information for educators (see What Works Clearinghouse web site).
One kind of research that educators need to inform their thinking about the potential effectiveness of various programs, strategies, and interventions is Scientifically Based Research (SBR). The most rigorous studies of program effectiveness will use random assignment which compares an intervention’s impact on students or teachers to that of a control group that didn’t get the intervention. Such experimental studies of the effectiveness of interventions (with random assignment to the treatment and control groups) are costly and difficult to do, and often take several years to complete. Thus, not many of the programs or strategies considered will have this level of evidence.
Consider Contextual Factors
As educators examine studies, they may find some interventions that have been found to be effective in settings different from theirs. Alternately, they may find that there are no high quality studies on a specific topic. As a result, they have to consider the results they find from research and literature in light of their professional wisdom. Educators may ask themselves questions such as:
- Based on my experience, will this intervention work with my students or my teachers?
- Is this intervention supported by what my experience tells me?
- When I have implemented similar interventions in the past, what has happened?
Incorporating professional wisdom into the EBDM Cycle recognizes and values the experiences that individuals have.
After investigating the research and informing the research with professional wisdom, educators must work within the reality of their setting. Questions that educators should consider as part of this process might be:
- What resources, particularly in terms of money and time, do I have?
- How will the staff/other individuals involved feel about this?
- What political constraints (school board, parents, and community) might affect the decision?
Being able to articulate the change that is desired as a result of implementing an intervention or strategy is a crucial early step in being able to assess later on whether the intervention has been successful. Teams that are able to articulate the desired outcomes will then have a better understanding of what will be required to implement an intervention effectively.
Developing a logic model is an effective process teams can use to help articulate outcomes of an intervention, as well as, for identifying strategies and resources (or inputs) that need to be put into place to achieve the stated outcomes. When developed by a team, a logic model can improve coherence around implementation and can be used as a tool to communicate with stakeholders to obtain buy-in.
Evidence-based decision-making is more than just looking for experimental research conducted on interventions. The absence of definitive findings from research and the limited number of studies conducted means that educators also need to use data and evaluations to help them understand what is working, or not working, as interventions are implemented in their states, districts, and schools. As a result, no matter what program, intervention or action is selected, the leader should adopt an experimental attitude and “evaluate” how well the action taken was implemented. It could be that the program was proven to work in a particular research setting but when teachers don’t have the level of support provided (as in the research study), it won’t work as well in a particular school. Any new program, policy, strategy, etc. that requires a significant investment of time, or resources, or that has potential impact on students should probably be piloted before it is used on a large scale. Evaluation is the collection of data that informs a decision-maker’s next steps in the particular local context. Evaluation is key to any planning process. This applies even at the level of the classroom in that a teacher’s lesson plans should have an “evaluation” component to them (e.g., at the end of the lesson, what is the current status of student learning and what needs to happen the next day as a result).
Monitoring the implementation of a program provides key information about how the intervention needs to be adapted in a specific educational environment. The final question, however, is: “Did we get the impact we wanted?” As a result, the EBDM Cycle requires considering the outcomes of the decision. What actually happened as a result of implementing this program or policy? Did student achievement increase as expected? Has teacher attendance improved? Did the dropout rates decline? Are parents spending more time volunteering at school? Whatever the desired outcomes are, educators need to examine the results of their decision.
Because this is a cycle, evidence-based decision-making does not end with evaluating outcomes. Instead, it recognizes that evaluating outcomes gives data needed to make the next decision. If the intervention worked, do we have the outcomes we want now? How we may modify the program to increase its impact? If the program did not work, do we need to work on implementation, revise the program, or toss it and start anew? The EBDM Cycle never ends; it continues in a spiral-like way, providing educators with more and more information and increasing the quality of the decision-making over time.
Included throughout the EBDM Cycle is the need for ongoing reflection. This reflection allows educators to evaluate the process itself and determine how the process is meeting their needs or how it needs to be modified to better meet their needs.