Data+Driven+Decision+Making

Type in the content of your page here. include component="comments" page="Data Driven Decision Making" limit="10"

First Things First: Demystifying Data Analysis
** To improve student achievement results, use data to focus on a few simple, specific goals. **  Mike Schmoker I recently sat with a district administrator eager to understand her district's achievement results. Pages of data and statistical breakdowns covered the table. Looking somewhat helpless, she threw up her hands and asked me, "What do I do with all this?" Many educators could empathize with this administrator. The experts' tendency to complicate the use and analysis of student achievement data often ensures that few educators avail themselves of data's simple, transparent power. The effective use of data depends on simplicity and economy. First things first: Which data, well analyzed, can help us improve teaching and learning? We should always start by considering the needs of teachers, whose use of data has the most direct impact on student performance. Data can give them the answer to two important questions: · ** How many students are succeeding in the subjects I teach? ** The answers to these two questions set the stage for targeted, collaborative efforts that can pay immediate dividends in achievement gains.
 * ** Within those subjects, what are the areas of strength or weakness? **

Focusing Efforts
Answering the first question enables grade-level or subject-area teams of practitioners to establish high-leverage annual improvement goals—for example, moving the percentage of students passing a math or writing assessment from a baseline of 67 percent in 2003 to 72 percent in 2004. Abundant research and school evidence suggest that setting such goals may be the most significant act in the entire school improvement process, greatly increasing the odds of success (Little, 1987; McGonagill, 1992; Rosenholtz, 1991; Schmoker, 1999, 2001). If we take pains to keep the goals simple and to avoid setting too many of them, they focus the attention and energies of everyone involved (Chang, Labovitz, & Rosansky, 1992; Drucker, 1992; Joyce, Wolf, & Calhoun, 1993). Such goals are quite different from the multiple, vague, ambiguous goal statements that populate many school improvement plans.

Turning Weakness into Strength
After the teacher team has set a goal, it can turn to the next important question: Within the identified subject or course, where do we need to direct our collective attention and expertise? In other words, where do the greatest number of students struggle or fail within the larger domains? For example, in English and language arts, students may have scored low in writing essays or in comprehending the main ideas in paragraphs. In mathematics, they may be weak in measurement or in number sense. Every state or standardized assessment provides data on areas of strength and weakness, at least in certain core subjects. Data from district or school assessments, even gradebooks, can meaningfully supplement the large-scale assessments. After team members identify strengths and weaknesses, they can begin the real work of instructional improvement: the collaborative effort to share, produce, test, and refine lessons and strategies targeted to areas of low performance, where more effective instruction can make the greatest difference for students.

So What's the Problem?
Despite the importance of the two questions previously cited, practitioners can rarely answer them. For years, during which **//data//**and **//goals//** have been education by-words, I have asked hundreds of teachers whether they know their goals for that academic year and which of the subjects they teach have the lowest scores. The vast majority of teachers don't know. Even fewer can answer the question: What are the low-scoring areas within a subject or course you teach? Nor could I. As a middle and high school English teacher, I hadn't the foggiest notion about these data—from state assessments or from my own records. This is the equivalent of a mechanic not knowing which part of the car needs repair. Why don't most schools provide teachers with data reports that address these two central questions? Perhaps the straightforward improvement scheme described here seems too simple to us, addicted as we are to elaborate, complex programs and plans (Schmoker, 2002; Stigler & Hiebert, 1999).

Over-Analysis and Overload
The most important school improvement processes do not require sophisticated data analysis or special expertise. Teachers themselves can easily learn to conduct the analyses that will have the most significant impact on teaching and achievement. The extended, district-level analyses and correlational studies some districts conduct can be fascinating stuff; they can even reveal opportunities for improvement. But they can also divert us from the primary purpose of analyzing data: improving instruction to achieve greater student success. Over-analysis can contribute to overload—the propensity to create long, detailed, "comprehensive" improvement plans and documents that few read or remember. Because we gather so much data and because they reveal so many opportunities for improvement, we set too many goals and launch too many initiatives, overtaxing our teachers and our systems (Fullan, 1996; Fullan & Stiegelbauer, 1991).

Formative Assessment Data and Short-Term Results
A simple template for a focused improvement plan with annual goals for improving students' state assessment scores would go a long way toward solving the overload problem (Schmoker, 2001), and would enable teams of professional educators to establish their own improvement priorities, simply and quickly, for the students they teach and for those in similar grades, courses, or subject areas. Using the goals that they have established, teachers can meet regularly to improve their lessons and assess their progress using another important source: formative assessment data. Gathered every few weeks or at each grading period, formative data enable the team to gauge levels of success and to adjust their instructional efforts accordingly. Formative, collectively administered assessments allow teams to capture and celebrate short-term results, which are essential to success in any sphere (Collins, 2001; Kouzes & Posner, 1995; Schaffer, 1988). Even conventional classroom assessment data work for us here, but with a twist. We don't just record these data to assign grades each period; we now look at how many students succeeded on that quiz, that interpretive paragraph, or that applied math assessment, and we ask ourselves why. Teacher teams can now "assess to learn"—to improve their instruction (Stiggins, 2002). A legion of researchers from education and industry have demonstrated that instructional improvement depends on just such simple, data-driven formats—teams identifying and addressing areas of difficulty and then developing, critiquing, testing, and upgrading efforts in light of ongoing results (Collins, 2001; Darling-Hammond, 1997; DuFour, 2002; Fullan, 2000; Reeves, 2000; Schaffer, 1988; Senge, 1990; Wiggins, 1994). It all starts with the simplest kind of data analysis—with the foundation we have when all teachers know their goals and the specific areas where students most need help.

What About Other Data?
In right measure, other useful data can aid improvement. For instance, data on achievement differences among socio-economic groups, on students reading below grade level, and on teacher, student, and parent perceptions can all guide improvement. But data analysis shouldn't result in overload and fragmentation; it shouldn't prevent teams of teachers from setting and knowing their own goals and from staying focused on key areas for improvement. Instead of overloading teachers, let's give them the data they need to conduct powerful, focused analyses and to generate a sustained stream of results for students.

Realizing the Promise of Standards-Based Education
To avoid curricular chaos, educators must be judicious about the standards they assess. The standards movement is arguably a major force in education today, and some researchers assert that the significance of the standards campaign will be huge. Undoubtedly, historians will identify the last decade of this century as the time when a concentrated press for national education standards emerged (Glaser & Linn, 1993, p. xiii). But will the standards movement endure? And if it does, will it contribute significantly to higher achievement? We believe it will—but only if we rein in its most excessive tendencies. Those tendencies can be seen in the nature and length of state and professional standards documents—and in their unintended consequences.
 * Mike Schmoker and Robert J. Marzano **

The Promise of the Standards Movement
Make no mistake: The success of any organization is contingent upon clear, commonly defined goals. A well-articulated focus unleashes individual and collective energy. And a common focus clarifies understanding, accelerates communication, and promotes persistence and collective purpose (Rosenholtz, 1991). This is the stuff of improvement. The promise of standards can be seen in places like · ** Frederick County, Maryland, where the number of students reaching well-defined and commonly assessed standards rose dramatically, lifting them from the middle to the highest tier in **** Maryland **** schools. Local assessments were deliberately aligned with standards as they were embedded in the state assessments. ** · ** Fort ** ** Logan ** ** Elementary School **** in **** Denver ****, **** Colorado **** , where scores rose significantly when teams of teachers analyzed weaknesses in performance relative to grade-level standards. Each team reviewed test data and developed strategies for helping students learn in identified areas of difficulty. ** · ** Lake Havasu City ****, **** Arizona **** , where teams of Title I teachers identified, defined, and focused instruction on common reading skills. Once teachers had a shared language about which skills to concentrate on, they improved strategies and systems to improve instructional quality and consistency. As a result, the number of students reading at or above grade level rose from 20 to 35 percent in just one year. ** · ** Glendale Union High School District near Phoenix, Arizona, where teams of teachers have increased student performance for almost every course offered. All district teachers—whether they teach algebra, **** U.S. **** history, biology, or senior English—are teaching to the same year-end assessments developed by subject-area teams. The same coordination is happening at **** Adlai ** ** Stevenson ** ** High School **** in **** Lincolnshire ****, **** Illinois **** , where teacher teams continue to set measurable achievement records on every kind of assessment. ** · ** Amphitheater ** ** High School **** in **** Tucson ****, **** Arizona **** , where teacher Bill Bendt routinely helps exceptional numbers of students pass advanced placement tests by carefully focusing instruction on the standards made explicit by the AP exam. ** How did they get these results? Interestingly, not by focusing on standards contained in state or professional documents. Their efforts preceded those documents. Nonetheless, in each case, **//teachers knew exactly what students needed to learn, what to teach to, where to improve, and what to work on with colleagues.//** Clear, common learning standards—manageable in number—promote better results. They are essential to focus and to coherence. If this is true, then educators face two important questions: (1) Do we already have sufficiently clear standards? and (2) Are state and professional standards documents truly helping us achieve the focus and the coherence that are vital to success? In too many cases, the answer to both questions is no.

Don't We Already Have Standards?
Curiously, standards in most districts are often similar. We have curriculums, scope, and sequence for each grade level, course, and subject area. But the perception of a common, coherent program of teaching and learning is a delusion. One of us once sat with a curriculum coordinator, poring through a dense curriculum notebook of the district's grade-by-grade "learner outcomes." The document was years in the making. Nonetheless, when the coordinator was asked what influence the curriculum was having on instruction, she was candid enough to reply "probably none." Consultant and author Heidi Hayes Jacobs likes to say that curriculum guides are "well-intended fictions." Her conclusion is that the current system actually encourages teachers to simply teach what they like to teach. It is time to admit that at the ground level, where teachers teach and students learn, there is not coherence, but chaos. The chief problem is that there is simply too much to teach—arguably two to three times too much (Schmidt, McKnight, & Raizen, 1996)—and too many options for what can be taught (Rosenholtz, 1991). There are enormous differences in what teachers teach in the same subject at the same grade level in the same school. Even when common, highly structured textbooks are used as the basis for a curriculum, teachers make independent and idiosyncratic decisions regarding what should be emphasized, what should be added, and what should be deleted (see, for example, Doyle, 1992). Such practices create huge holes in the continuum of content to which students are exposed. In **//The Learning Gap,//** researchers Stevenson and Stigler (1992, p. 140) observe that teachers are "daunted by the length of most textbooks." In a system that does little or nothing to help them coordinate priorities, they are forced to select or to omit different topics haphazardly. This only adds to the prevailing chaos.

Standards and School Improvement
The implications of this chaos go to the heart of school improvement. Researcher Susan Rosenholtz found that The hallmark of any successful organization is a shared sense among its members about what they are trying to accomplish. Agreed-upon goals and ways to attain them enhance the organization's capacity **//for rational planning and action.//** (1991, p. 13; our emphasis) For this reason, she was dismayed to find that schools were unique among organizations in lacking common goals and that the goals of teaching were "multiple, shifting and frequently disputed" (p. 13). This state of chaos was the rationale for the standards movement—and the most visible and influential manifestations are the state and professional standards documents. Yet these documents themselves have contributed to the very problems they were intended to solve.

The Perils of Standards-Based Education
"Less is more" we keep telling ourselves. Students learn more when we teach less—but teach it well (Dempster, 1993). Nowhere is this principle more obviously violated than in the standards documents. The official documents generated by 49 states and the professional subject-area organizations have had unintended consequences. Commentator Ronald Wolk has found some of them not only to be written in language that is "absurd" but also to contain such quantity that it would take a 10-hour teaching day to cover the material in them (1998). Because it is easier to add and enlarge than to reduce and refine, we are caught in the snare of having honored (perhaps for political reasons) far too many suggestions for inclusion in the standards documents. We have often failed to place hard but practical limits on the number and the nature of the standards. The result? Bloated and poorly written standards that almost no one can realistically teach to or ever hope to adequately assess. We are making the same mistakes with these documents that we made with our district curriculums. In the case of standards, quantity is not quality. The irony of the Third International Mathematics and Science Study (TIMSS) shouts at us: Although U.S. mathematics textbooks attempt to address 175 percent more topics than do German textbooks and 350 percent more topics than do Japanese textbooks, both German and Japanese students significantly outperform U.S. students in mathematics. Similarly, although U.S. science textbooks attempt to cover 930 percent more topics than do German textbooks and 433 percent more topics than do Japanese textbooks, both German and Japanese students significantly outperform U.S. students in science achievement as well (Schmidt, McKnight, & Raizen, 1996). Clearly, U.S. schools would benefit from decreasing the amount of content they try to cover. And teacher morale and self-efficacy improve when we confidently lay out a more manageable number of essential topics to be taught and assessed in greater depth.

Getting Standards Right
Too many of the state standards documents, informed as they are by the professional subject-area standards, have frustrated rather than helped our attempt to provide common focus and clarity for teachers and students. The good news is this: Clear, intelligible standards are a pillar of higher achievement. **//Aligned with appropriate assessments,//** they can help us realize the dream of learning for all. They are the heart of the infrastructure for school improvement (Rosenholtz, 1991; Fullan & Stiegelbauer, 1991).

The Standards-Driven School
Consider a school where teachers know exactly what essential skills and knowledge students should learn that year and where they know that their colleagues are teaching to the same manageable standards. Because of this, their fellow teachers can collaborate with them on lessons and units. This in turn leads to a living bank of proven, standards-referenced instructional materials—lessons, units, and assessments perfected through action research. Both new and veteran teachers can peruse these targeted materials, learning from and adding to the richness of the faculty's repertoire. Because of these rich resources, new and struggling teachers achieve confidence and competence much more rapidly, and experienced teachers have a sense of making a meaningful, ongoing contribution to their craft while being renewed by instructional ideas that are engaging for students. Proven methods, practices, and lessons aligned with established standards become the center of the professional dialogue. Results on local, state, and formative assessments get better and better. Such an alignment leads inevitably to better short- and long-term results on local and state assessments as well as on norm-referenced, alternative, and criterion-referenced assessments. To create this infrastructure in schools, we can take a few concrete steps: · **// Start with the standards that are assessed. // Be circumspect about standards that are not assessed. After thoroughly reviewing the state standards documents, we believe that many of them never will be thoroughly assessed. Many of the existing standards that educators are working manically to "cover" will disappear because of their own irrelevancy and imprecision. Expending organized effort on every standard is senseless because many of them will turn out to be ephemeral. Start by focusing teaching on the standards actually contained in current state norm-referenced or criterion-referenced assessments. ** · **// Beyond state assessments, add judiciously to the list of standards you will teach and assess. // For Michael Fullan, "assessment is the coherence-maker" in school improvement (1998, personal communication). Because of the limitations of state and norm-referenced tests, we must develop local and district standards and assessments that take us beyond them. Districts should review the standards documents, but then exercise severe discipline in prioritizing on the basis of what students will most need if they are to become reflective thinkers, competent workers, and responsible citizens. For every grade or level, pilot your new standards and assessments while asking the question, Are the standards clear, relevant, and not so numerous that they sacrifice depth over breadth? Don't be afraid to do a rough account-ing of time for teaching topics. ** · **// Do not add more topics than can be taught and assessed reasonably and effectively. // A key to developing science assessments in **** Lake Havasu City **** was following open discussions with fast, fair rank-ordering procedures that used weighted voting to quickly establish priority standards. Because we can expect educators to differ in philosophy and priority, every school employee could benefit from training in the use of these simple decision-making tools. ** Standards—when we get them right—will give us the results we want. But this will require hard-headed, disciplined effort. The lesson of TIMSS should considerably diminish the perceived risk of downsizing the curriculum. The very nature of organizations argues that we succeed when all parties are rowing in the same direction. We will realize the promise of school reform when we establish standards and expectations for reaching them that are clear, not confusing; essential, not exhaustive. The result will be a new coherence and a shared focus that could be the most propitious step we can take toward educating all students well. Curriculum-based reform, which aligns curriculum with content and performance standards, is sweeping education systems. But what makes curriculum-based reform effective? The Mid-continent Regional Educational Laboratory (McREL) is heading a series of studies to survey the implementation of this reform approach and its impact on student achievement. McREL researchers have identified four state-level components for successful curriculum-based reform: an ongoing standards review, a professional development plan, an assessment program, and an accountability system. Although 80 percent of states reported that they impose sanctions when school or district assessment results are low, only 55 percent of states reported that assessment is tightly aligned to standards. And more than 45 states require that all students meet standards and participate in standards-based assessment projects. The reports "Curriculum Reform: What State Officials Say Works" and "Taking Stock of States' Curriculum-Based Reform Efforts" are available from McREL, Curriculum, Learning and Instruction Project, 2550 South Parker Rd., Ste. 500, Aurora, CO 80014-1678 (Web site: www.mcrel.org). ||
 * As state assessments develop, real priorities become clear. And we must learn all we can about how to teach to these priorities most effectively. Teachers in **** Colorado ****, now that they know the reading and writing standards through their experience with the state assessments, are responding in a positive and coordinated fashion. Many schools, like **** Bessemer **** Elementary in **** Pueblo **** , which has an 80 percent minority population, have realized dramatic gains. At **** Bessemer **** , from 1997 to 1998, the number of students performing at or above the standard in reading rose from 12 percent to 64 percent. In writing, they went from 2 percent to 48 percent. Weekly standards-based team meetings made the difference. **
 * State and standardized assessments do not measure everything we deem important, but success on such tests in this age of accountability is vital. Strong standardized scores earn us the trust of our communities as we begin to demonstrate measurable progress on local criterion-referenced and alternative assessments. In districts where improvement on formal, public assessments is of the essence, we should assemble clear lists of the standards and proficiencies that the assessments will require of students. District offices and regional consortiums must take the initiative here: They must assemble representative teams of teachers to develop—and provide every teacher with—a precise, manageable list of the essential, assessed standards. **
 * Every school year, the full faculty should conduct a review of assessment results. Teams of teachers should identify the most pronounced patterns of student weakness, then seek absolute clarity on the nature of these problems. Through staff development and regular, professional collaboration, teachers should focus on these areas, while monitoring progress regularly. **
 * Adlai ** ** Stevenson ** ** High School **** has achieved world-class results in this way. **** Glendale ** ** Union ** ** High ** ** School District **** has done a masterful job of successfully concentrating on norm-referenced tests while implementing a coherent system of formative and end-of-course alternative assessments for high school courses. These assessments require students to do investigative science and to write analyses about social and historical issues—all according to clear standards and criteria. These common, teacher-made assessments embody and clarify precisely those thinking and reasoning standards that norm-referenced tests don't adequately assess. The result is an education that ensures a level of both breadth and substance that goes far beyond what is now required of the average high school graduate. **
 * Perhaps the best time to develop such standards-based assessments is summer. Such work doesn't always require enormous amounts of time or resources. In **** Lake Havasu City ****, **** Arizona **** , educators developed common K–12 assessments in almost every subject area for about $25,000 over a two-year period. They took only four days to prioritize core science standards and generate common K–12 assessments. **
 * The tendency toward overload is strong in schools—and crippling to improvement efforts (Fullan & Hargreaves, 1996). A district we know has received high praise for showcase work by developing grade-by-grade benchmarks for the state standards. For 4th grade, educators developed 210 items to be taught in math, but 125 of these //were also to be taught in six to eight other grades.// In another district, in another state, there are only 17 items for 4th grade math, and they're written in language that is clear to parents and teachers. **
 * At the local and state levels, we must demand that economy and clarity inform all standards and that they be meaningfully—not just rhetorically—aligned with assessments. Every teacher deserves a clear, manageable, grade-by-grade set of standards and learning benchmarks that make sense and allow a reasonable measure of autonomy. Anything less is frustrating, inhumane, and counterproductive. **
 * === McREL Researches Curriculum-Based Reform ===