About the Consortium
Research Plan
Consortium Members
Advisory Panel
Summary of Findings
Technical Report
Press Releases


The Evaluation Plan's Design and Key Research Questions

The evaluation design recommended is guided by six principles that emerged from conversations with state-level policy makers, superintendents, principals, teachers, and representatives of research organizations and professional groups.

  • A single, integrated evaluation is preferable to a set of studies on topics of concern. Assessing CSR through disconnected projects would probably fail to provide a meaningful picture of the whole initiative or capture the relationships among the multiple actions needed to implement the program.
  • It is essential that the study be comprehensive, addressing all relevant issues, from resource allocation to student achievement.
  • The evaluation should provide information to improve implementation as well as to determine whether the initiative ultimately succeeds. The initial formative phase will produce information to aid ongoing decision-making at all levels of the system. State leaders and educators emphasized the value of ongoing feedback and information about the status of CSR implementation, problems encountered at all levels, their resolutions, and innovative practices associated with class size reduction.
  • The summative evaluation should answer questions about the relationship of reduced size classes to student achievement and to educational practices throughout the system.
  • A longitudinal approach to the evaluation is essential because CSR will take years to be fully implemented. An immediate snapshot study will not adequately reflect the changes that occur as the program matures, while a summative approach alone would not reveal mid-course corrections that may be needed.
  • The evaluation needs to be rigorous and objective so the findings will be credible to both supporters and skeptics.

The underlying conceptual model that should guide the evaluation is shown in Figure 1. It has built into it the connections among the seven major issues identified above. The model begins with an examination of how district and school policies might have been affected by the Stateís CSR initiative and how these policies relate to resource allocation, other ongoing reforms, parental involvement and support for the program, and to teacher quality and training. These factors in turn are assumed to relate to classroom practices, which in turn are assumed to relate to student outcomes.

Each of the various components in the model give rise to a set of evaluation questions which will be elaborated in the following paragraphs. However, the overall guiding questions are simple:

  • How can the reform be implemented most effectively?
  • What are its effects on students?
  • What factors account for its success or failure?
  • What is the relationship between the programís benefits and costs?

We now turn to a discussion of each of the seven components and the research questions associated with each which the evaluation should address.

State, District, and School Policy Making

Participation in the class size reduction program is voluntary. Districts have choices about the number of grade levels and schools that will reduce class size to 20, and these decisions can change annually. A number of factors influence these decisions, with the local context playing as great a role as the state guidelines. Policy making at the state, district, and school levels to understand how beliefs, concerns, and context influence decisions regarding class size reduction will be considered. For example, the match between state, district, and teacher expectations for the goals of CSR may influence its success. The degree to which existing policies and practices are aligned with CSR is also likely to have an impact on the initiativeís success. Consequently, how CSR implementation is affected when local policies (e.g., school and district literacy initiatives, professional development activities, and standards and assessments) complement the goals of CSR, should be investigated. Collective bargaining agreements between teachers and school systems are another part of the local policy context that should be examined.

Key research questions that should be examined include:

  • What goals do state policy makers and district and school administrators have for CSR, and what are their concerns regarding implementation? How do differences in expectations and/or concerns affect their actions?
  • How are implementation decisions made about which grade levels or classrooms participate in each year? How does participation in these decisions vary across districts and why?
  • Do policy makers share a common set of expectations about how CSR can influence student learning? Are these expectations different from those of teachers?
  • Which educational policies, regulations, and labor agreements facilitate or impede the effective implementation of CSR?

Resource Allocation

Preliminary studies of FY-97 CSR implementation have raised a number of questions about the reallocation of resources within and among schools and districts (Office of the Legislative Analyst, 1996; Blattner et al., 1997). In the first year, the average cost per student of CSR exceeded the state allocation of $650 per pupil by 21 percent, or $140 per student. Research suggests that this funding shortfall may have affected the relative distribution of funding between elementary and secondary schools as well as the distribution of resources within schools between the targeted early primary grades and subsequent grades. FY-98 funding increases alleviated this shortfall, on average. However, some districts are still experiencing surpluses and others, deficits, through this program. It is important that the evaluation examine the impact of implementing this program in these two types of districts. What has been the effect on the allocation of space and facilities in districts and what other programs and/or educational activities have been affected by CSR classroom demands?

Key research questions that should be examined include:

  • What is the impact of the CSR program on equality of funding for education across districts? Across schools? Across subpopulations of students such as LEP students, minority and low-income students, and those with special needs?
  • How does CSR affect district revenues and expenditures? Within districts, how does the initiative affect school spending levels on operations and facilities?
  • How does CSR affect the distribution of resourcesófunding, space, and materialsó across primary and secondary schools and across programs within districts? What strategies have schools used to make space for new classrooms? What tradeoffs were made and why? How permanent are they?
  • Within schools, how does CSR affect resource allocation across grades, instructional support services (e.g., libraries, media centers, counseling) and instructional programs (e.g., art, P.E., music)?

Integration with Other Reforms

The CSR initiative is recent, large, and potentially of great significance, but it is not the only reform initiative involving California schools. Other ongoing reform efforts include state initiatives on literacy, professional development, and charter schools; district initiatives such as LEARN in Los Angeles; privately funded initiatives like the Annenberg Challenge, which supports the Bay Area School Reform Collaborative (BASRC) in the San Francisco area and the Los Angeles Metropolitan Project (LAMP), Accelerated Schools, and the Coalition of Essential Schools; as well as a variety of local technology, science, language proficiency, and arts efforts.

CSR has the potential to enhance these existing reform initiatives or to detract from them. On the one hand, a reform initiative which integrates school literacy and professional development can be focused on how literacy may be improved by new small class instruction practices. However, at the same time class size reduction may distract schools from the pursuit of other school reform initiatives. For example, some elementary science labs have been closed to provide additional classroom space for CSR. Similarly, initiatives to strengthen the middle school curriculum may lose momentum because of the concentrated effort needed to implement CSR. Schools undergoing restructuring may find that the attention focused on class size reduction displaces efforts to bring about systemic change.

Key research questions that should be examined include:

  • Is CSR integrated with the district's master planning efforts, or does it exist independently of other initiatives the district/school may wish to pursue? Does CSR serve as a catalyst to enhance coordination of existing reform efforts, or does it provide a diversion from more systematic efforts?
  • How does CSR interact with large categorical programs like special education and Title I? How does CSR affect programs serving limited English proficient students? Special education students?
  • What is the nature of other reform efforts in the district/school at the time CSR is introduced? How are other school reform efforts affected by the introduction of CSR? Are resources (dollars, time, and people) redirected from other reform efforts to assist in CSR implementation? Does CSR affect changes in staff assignments at the district or school levels?
  • Do CSR implementation approaches differ by district/school characteristics? For example, do low-revenue districts implement CSR differently than high-revenue districts?

Teacher Quality, Assignment, and Training

Human resources are an essential part of CSR. The effectiveness of the reform will depend in large measure on the quality of the teachers in the system, choices about which teachers are assigned to smaller classes, and the preparation teachers receive for these classes. There are reasons to be concerned on all fronts. First, the demand in the first year of the program alone vastly exceeded the supply. The Legislative Analystís Office (1997) reports that 30 percent of the new hires were not credentialed, 24 percent were granted emergency permits, and 6 percent were enrolled in university programs but not credentialed. Only 14 percent of the new hires had more than five years of teaching experience. Overall, it appears as though underqualified teachers may be entering California in large numbers.

The evaluation should track the assignment of these new teachers, as well as of those already in the system. As noted previously, some districts report that teachers at higher grade levels and special education teachers are requesting transfers to smaller classrooms, leaving less well prepared teachers to tackle these demanding assignments. There have also been reports of highly qualified teachers transferring from urban districts to suburban districts as new positions are created. Therefore it is important that the evaluation also investigate professional development programs used to train teachers and to examine whether the support being provided is adequate to ensure that effective teaching is occurring in small classes. Finally, it is important that the evaluation examine whether and how CSR has affected teachersí attitudes toward their job and their engagement in it.

Key research questions that should be examined include:

  • How is CSR affecting the recruitment and assignment of teachers across districts, schools, grades, and special programs? How does collective bargaining influence this process?
  • What are the qualifications and experience of teachers assigned to smaller classes? What is happening to the qualifications of teachers in classrooms with high concentrations of limited English proficient students, minority students, and students with special needs?
  • What professional development activities and support are provided for teachers assigned to smaller classrooms? How do these activities differ across categories of teachers (e.g., noncredentialed, newly credentialed, and experienced but new to primary grades)? What type of training do teachers assigned to smaller classes receive with regard to language instruction strategies for limited English proficient students?
  • How does CSR affect teacher satisfaction and attitudes toward teaching and students? How do the attitudes of teachers in smaller classes affect studentsí learning opportunities and potential?

Classroom Practices

Little is known about which classroom practices are most effective when class size is reduced. Some advocates of smaller classes argue that reducing class size enables teachers to have more individual contact with students. Smaller classes also reduce teachersí burdens associated with discipline, paperwork, and other noninstructional duties and free them to devote more class time to teaching. From this perspective, the key advantage of class size reduction is that it permits more activities and interactions to occurómore contact, more feedback, and more exposure to curriculum.

Other proponents argue that the principal advantage of class size reduction is that it permits different activities and interactions to occur. According to this argument, the reduction in noninstructional demands coupled with a better knowledge of studentsí individual needs allows teachers to engage in different kinds of interactions, including student-centered learning in which the teacher acts as facilitator rather than dispenser of knowledge, extended project-based learning, increased emphasis on higher-order skills such as problem solving, and richer literacy experiences. In this view, class size reduction permits changes in the nature of teacher-student interactions and in the content of the curriculum.

It is important that the evaluation of CSR examine whether and how it has affected the rate and/or the nature of the activities and interactions teachers have with the children in their classrooms.

Key research questions that should be examined include:

  • What changes have and are occurring in teaching practices as a result of CSR, including changes in emphasis or coverage of different topics, methods of instruction, and the range of learning experiences?
  • What types of language instruction strategies/models are used in CSR classrooms for limited English proficient children in CSR classrooms? Do changes in instructional practices differ across districts, classrooms, and categories of students, including LEP, minority, and special education students?
  • What changes occur in the availability and allocation of instructional support personnel and other resources (e.g., staff development, curriculum guidance, and opportunity for collaboration with other teachers)?

Parental Involvement

Many educators believe that parent involvement will improve childrenís educational success. Certainly, the available research supports the belief that parents matter. At the elementary school level, research has demonstrated an association between parent involvement and fewer behavioral problems (Comer, 1984), lower dropout rates (NCES, 1992), higher student achievement (Muller, 1993; Stevenson and Baker, 1987; Reynolds, 1992; Kohl, 1994; Klimes-Dougan et al., 1992), and childrenís perceived level of competence (Wagner and Phillips, 1992).

Some proponents expect CSR to have a positive effect on parent involvement, although the mechanisms that would facilitate such an effect are far from clear. It might be that parents whose children are in smaller classes view the district as being more concerned about children and about their child. Hence, they may feel their support is less essential. On the other hand, parents may feel less intimidated about "bothering" a teacher who has fewer students, believing that she will have more time for them and their concerns. In fact, teachers may actually have more time for parents, and may more actively seek their involvement at both the classroom and school levels.

The evaluation of CSR should examine whether teachers have more time for parents and whether parents are in fact spending more time with the teachers and participating more in the education of their children. It should also examine parentsí satisfaction with the school and whether their attitudes about the quality of their childrenís have changed.

Key research questions hat should be examined include:

  • To what extent have parents been involved in decisions about grade participation, reallocation of resources and space, and the assignment of students to classrooms at the district or school level?
  • Has the range or intensity of parent involvement programs and efforts at the school declined or increased as a result of CSR? Has the amount and nature of parent involvement in the schools changed as a result of CSR?
  • Do class assignments and activities assume increased or decreased amounts of parent participation? Do parents feel more welcome in their childrenís classrooms? Do they have more parent-teacher conferences?
  • Do CSR parents believe that their children are receiving a better education (e.g., more individualized attention)? How does the initiative affect their behavior toward their children? Does CSR affect parentsí satisfaction with the teacher, the school, or the district?

Student Outcomes

The primary motivation for reducing class size is to improve student learning. The main criterion for academic achievement in the evaluation should be performance on the new standardized reading and mathematics tests (STAR) recently adopted by the State. However, because many standardized tests measure mainly comprehension, it is important that the evaluation examine studentsí oral reading ability as well.

For English language learners whose primary language is Spanish, Spanish versions of standardized tests adopted by selected districts should be used. In addition, it is recommended that the evaluation measure Limited English Proficient studentsí reading readiness. Standardized reading tests may not be sensitive enough to capture gains in English language development that have occurred with the introduction of reduced class size. More specifically, while some students may not be able to read per se, they may have developed language skills, such as word recognition, that would show up on a special reading readiness assessment.

Studentsí engagement with schooling, as measured by attendance, promotion/retention, homework completion, and frequency of disciplinary actions may also change with the introduction of reduced class size and therefore should be assessed as part of the evaluation of student outcomes. Changes in referrals and transition rates of students into and out of special education, bilingual education, or other programs should also be measured, along with teachersí views about long-term improvements in studentsí readiness for a new grade.

Key research questions that should be examined include:

  • Has student achievement in reading and math improved since CSR began? Have promotion rates to the next grade changed as a function of CSR? Do next grade teachers perceive improvements in studentsí preparation to master grade-level material?
  • Has there been an increase or decrease in transition rates into or out of special programs (e.g., sheltered English programs, resource classes, reading interventions)?
  • Has reading readiness improved for ESL students with the introduction of CSR?
  • Are students more engaged in school (in terms of attendance, behavior, and homework completion)?
  • Do any of the relationships between class size and student outcomes vary on the basis of school, teacher, classroom practices, and/or student characteristics (e.g., do limited English proficient students benefit more than English proficient students)?
  • Are changes in classroom practices associated with changes in studentsí educational outcomes?

 

Next Section: Methodology