About the Consortium
Research Plan
Consortium Members
Advisory Panel
Summary of Findings
Technical Report
Press Releases


 

The Year Two Workplan for the Evaluation of
California’s Class Size Reduction Programs

Background

Summary of Results of the Year 1 Evaluation

The Year 2 Evaluation Plan

Research Design

Deliverables

Click here if you would like to view the pdf version of The Year Two Workplan for the Evaluation of California’s Class Size Reduction Programs.

 

Background

In 1996, California enacted SB 1777, providing a substantial incentive for districts to reduce their class sizes from an average of roughly 30 students per teacher to 20 or fewer students per classroom. With the signing of the bill, in 1996-97 districts were provided with nearly $1 billion in education funds to reduce class size in grades K–3; the expenditure was roughly $1.5 billion in the second year and will continue at this level or higher through the lifetime of the program. This represents what is arguably the single largest investment any state has made in an education program.

The relation of class size to student performance, upon which the Class Size Reduction (CSR) program is predicated, has been studied for over 30 years. Earlier findings regarding the efficacy of CSR are mixed, but a recent set of high-profile statewide CSR studies, especially the Tennessee STAR study, has tipped the policy scales firmly in favor of smaller classes. Such findings provided encouragement for California to invest in CSR. In the eyes of its many supporters, California’s CSR program had the potential to reverse years of decline and serve as a model for other states. In a state with one of the lowest per-pupil expenditures in the country, the injection of new funds into early grades was seen as cause for celebration. However, CSR had never been implemented at anywhere near the scale of the California initiative. Nor had CSR been implemented in a state with near the cultural diversity of California. Furthermore, the demands that CSR placed on districts to find space and hire new teachers was high given existing shortages that existed prior to the introduction of CSR. Finally, CSR was introduced at a time when the proportion of children for whom English is their second language was at an all-time high in California. CSR was presented with a sizable challenge in trying to meet the language instruction and other special needs of many of the state’s schoolchildren.

In order to assess the overall effectiveness of the CSR initiative, as well as answer questions about how and where the program is most likely to succeed, the state mandated a comprehensive, multi-year review of CSR. In response to this mandate, a team of representatives from research and policy organizations in California responded to the RFP to evaluate the impact of this potentially far-reaching, but expensive program. The CSR Research Consortium, led by the American Institutes for Research (AIR) and the RAND Corporation, also includes Policy Analysis for California Education (PACE), WestEd and EdSource.

Shortly after SB 1777 was signed into law, representatives from a group of research and policy organizations in California formed a consortium that met to discuss the importance of planning for an evaluation of the impact of the law. The consortium set out to gain expertise in all aspects of class size reduction, and especially the application of CSR in California. To that end, funds were obtained from several prominent California foundations to plan for an evaluation based on discussions of class size reduction with California state-level policy-makers, school superintendents, principals, teachers, and representatives from educational research organizations and professional education groups. The State of California, in recognition of the Consortium’s leadership in studying class-size reduction in the state, commissioned it to develop a formal plan for evaluating the California CSR Initiative. The plan developed by this group, which PACE led, was approved and adopted in October 1997 by the State Board of Education.

From its earliest meetings, the Consortium agreed that the evaluation of California’s Class Size Reduction (CSR) program needed to be comprehensive. The evaluation should not only consider the effect of class size reduction on student achievement, but should also examine the impact of the reform on all aspects of the educational system. From the outset, the Consortium believed that some of the most important effects of CSR might not directly relate to student achievement.

The conceptual framework in the Consortium’s plan for the evaluation was approved by the California State Board of Education in the spring of 1998. A Request for Proposals was then issued; the state awarded the contract to conduct the evaluation to the American Institutes for Research (AIR) on behalf of the Consortium.

The Consortium felt that it was important to maintain the integrity of the plan it had developed to the greatest degree possible, so representatives of the Consortium approached foundations seeking funding for those parts of the plan not included in the state’s contract. In addition, given the late issuance of the state’s RFP, foundation funds were obtained to fund surveys of districts, schools, teachers, and parents.

The model the Consortium developed is shown below in Figure 1.

The model begins with a need to understand the role of policymaking on a district’s decision about whether to implement CSR or not, and if so, when. The model includes not only the passage of SB 1777 but also the larger state and district policies within which the CSR existed. At the district and school levels, CSR implementation was assumed to be influenced by the existing reform milieu as well as by the physical and human resources available. CSR occurred at a time when other educational reforms were either being put in place or were already in place, so it was important to consider the extent to which CSR either complemented or competed with other, ongoing or developing reform efforts./P>

Attention is also paid in the model to the resources that were available in the districts as they planned implementation. At his level, it was especially important to examine the impact of the implementation of CSR on the overall quality of California’s teaching corps, as well as which districts hired the newest, least experienced, least well-credentialed teachers.

The next factor to be considered was parents. What impact did CSR have on their interest and involvement in their children’s education? Also of central importance was to ascertain what effect CSR had on classroom instructional practices, if any.

 

Figure 1 - The Impact of the California Class Size Reduction Initiative

 

 

Finally, the model focuses on whether being in a larger versus a smaller classroom is related to student achievement. Achievement effects were roughly twice as large for minority and inner city students as for other students in the Tennessee STAR study. Given the many differences between the two programs as well as the differences in the context in which the programs were implemented, would California realize the same benefits, especially for its minority and low-income students as were realized in Tennessee?

Limited funds for the Year 1 evaluation, combined with the fact that the data for the project were not available until late February, meant that not all of the relationships implied by Model 1could be explored. As an example, funds were not available to collect in-depth data about district policies, and on-going education reforms – data that would have enriched the analyses we did examining these issues. Instead we were limited to the use of survey data for these analyses. We also did not have the funds to collect data at the district level from a set of districts that have linked school, teacher and student data. As a result, no analyses below the school level were possible. Nor did we have funds to collect data from districts that have high percentages of Spanish-speaking students and also have and continue to administer an achievement test in Spanish (e.g., SABE) in addition to the SAT-9 test. These data would have allowed us to compare the achievement of Spanish-speaking ELL students on both STAR and a Spanish achievement test in large versus small classes in those districts.

The analyses completed for the Year 1 evaluation were also affected by the nature of the data available. As pointed out above, it was not possible to carry out analyses at the classroom/teacher level because the State does not have a data system that integrates schools with classroom/teacher with individual outcome data. As a result, the school was the lowest level at which analyses could be conducted.

Finally, there were cases where data were collected but we did not have the funds with which to analyze them. For example, we collected rich observational data on instructional practices in both large and small third grade classrooms using videotapes, albeit in a small number of classrooms, but we did not have the funds to analyze the data. As documented below, other interesting questions could have been addressed with the quantitative data we have, but we had neither time nor funds to do so. To the degree that funds are available, we will pursue these questions in Year 2 of the evaluation.

 

Summary of Results of the Year 1 Evaluation

The Consortium recently released the results of its Year 1 evaluation along with a discussion of implications for California’s CSR program as well as for elementary education policy in California more generally. Some of the key findings include:

Implementation: Rates and Costs. Implementation of CSR occurred very quickly. By the end of its second year, 86 percent or over 1.6 million California K–3 students were in classes of 20 or fewer students. Schools varied in their ability to implement CSR, however. Schools with high percentages of low-income, minority, or English Language Learner (ELL) students were significantly less likely to implement CSR program than were schools with smaller percentages of disadvantaged students. The principal reasons given for not implementing CSR faster were lack of classroom space and the financial wherewithal to do so. Because of slower implementation rates, districts with high percentages of low-income, minority, or ELL students also received fewer total CSR dollars. And because of the funding formula California used, CSR created surplus funds for some districts, others broke even, but in most cases CSR created a deficit. The main ways in which districts with funding deficits were able to implement CSR was by postponing building maintenance and cutting administrative services. Finally, although CSR slowed down implementation of other district or school reforms in many cases, principals and superintendents also were likely to report that CSR had contributed to their reform efforts by boosting teacher enthusiasm for them and by bringing in teachers with new and stimulating ideas.

CSR and Teacher Qualifications. During the first two years of CSR implementation, the number of K–3 teachers increased by over 38 percent as some 23,500 teachers were added. The new workforce in 1997–98 was slightly more male and Hispanic than the one in 1995–96. The huge influx of additional teachers reduced the overall experience and education level of K–3 teachers; there was also a large increase, over 11 percentage points, in the number of not fully credentialed K–3 teachers. The drops in K–3 teacher education and credentialing levels were not spread evenly across schools; comparatively greater drops in teacher education and credentialing levels were seen in schools serving more low-income, ELL, Hispanic, or minority students, as well as in schools that were large and urban. While all schools had significant increases in the numbers of teachers with Cross Cultural Language and Academic Development (CLAD) or Bilingual Cross Cultural Language and Academic Development (BCLAD) credentials, those schools with the highest proportion of ELL students received proportionately fewer teachers with these credentials.

CSR and Instructional Practices. There were few differences in instructional practices between non-reduced and reduced size third-grade classes.1 For example, teachers in reduced size classes did not spend significantly more time during regular lessons working individually with students, which is one way that smaller classes might promote achievement. Similarly, there were no differences in curriculum content and very few differences in the frequency of teachers’ use of specific instructional strategies or student activities in either language arts or mathematics. However, there were a few differences in classroom practices in third grade that are worth noting. When specifically asked about students who needed help with reading skills, teachers in reduced size classes reported that they spent more time giving sustained attention (five or more continuous minutes) to these students than did teachers in non-reduced size classes. Teachers in reduced size classes also reported spending more time addressing individual students’ personal concerns and less time disciplining students than did teachers in non-reduced size classes.

CSR and Parental Involvement and Satisfaction. Parental contact with teachers and parental satisfaction with schools were higher among parents of third-grade students in reduced size classes. Among parents who responded to our survey,2 a somewhat larger percentage of parents with children in reduced size classes initiated contacts with teachers (74%, compared with 69% in non-reduced size classes) and were contacted by teachers (85%, compared with 81% in non-reduced size classes) during the 1997–98 school year. Parents of students in reduced size classes rated the overall quality of their children’s education as very good; parents of students in non-reduced size classes rated it as good to very good. Furthermore, the former group rated all aspects of educational quality higher than did the latter group.

CSR and Academic Achievement. Because of rapid implementation of the reform, it only was possible to compare achievement in reduced and non-reduced size classes in third grade. The analyses show that participation in CSR was associated with a small positive difference in the achievement of third-grade students. This difference can be expressed in a number of equivalent ways: as an effect size of about 0.06 to 0.1 standard deviation units, as a 2–4 percentage point gain in the percentage of students scoring above the 50th national percentile rank, or as an average gain of about 2–4 scale points on the SAT-9 test. The differences were similar in size for reading, mathematics, and language, but smaller for spelling. Furthermore, the differences were similar regardless of students’ income level, English language fluency, or minority status.

 

The Year 2 Evaluation Plan

The comprehensive plan guiding the CSR evaluation shown in Figure 1 identifies a number of major research issues. In Year 2 these issues will continue to form the core of our evaluative efforts.

The plan has been broken down by source of funding; activities to be completed are listed under the funding source upon which their completion depends. Activities fall into one of three categories: those to be carried out with CDE funds, those supported by in-hand non-CDE funds, and those for which completion is contingent upon securing additional outside funding.

Work to be completed with CDE funds

The following is a description of the Year 2 activities that will be carried out using CDE funds:

Updating Extant Data Files. During Year Two of the CSR Evaluation, we will build on the extensive collection of databases assembled during Year One. We will contact the appropriate CDE division to update the files supplied to us last year with 1998-99 data. The following categories of data will be acquired and developed into analyzable files for use by Consortium members.

CBEDS: During Year One, we downloaded all available DBase files posted on the Education Demographics Office website (summary information from the CDIF, SIF, PAIF, Language Census, and AFDC reports). We combined the data into a set of school- and district-level analysis files for the school years 1990-91 through 1997-98. During Year Two, we will download all new data files posted on the Ed Demo site and create school and district files for the 1998-99 school year. Some files have already become available; others (e.g., Language Census data) are expected late in summer 1999.

CBEDS-PAIF: The Consortium will also work with CDE to obtain two kinds of data from the Professional Assignment Information Form (PAIF) data collection. First, we will request a copy of the full 1998-99 PAIF data file (a raw file accompanied by a tape layout. Second, we will request an update of the "linkable" PAIF files that CBEDS staff provided to us last year. These files include special identification codes for each respondent that allow individual’s data from different PAIF files to be combined, but do not allow the individual to be identified, as would be possible if the person’s actual ID code (e.g., a Social Security number) were used as the linking variable. These linked files will support analyses of teachers’ career trajectories over four school years—1995-96 through 1998-99. Studies of teacher flows during the CSR implementation years were not conducted during Year One, but will be a major focus of Year Two research. We will work with CDE to ensure that in addition to the actual data, we acquire a clear understanding of how the files were generated and what their limitations may be.

CASEMIS: We will contact the Special Education Division to request a copy of the December 1998 CASEMIS file to update our current holdings (December counts for the years 1994 through 1997).

STAR: We will download the available school-level scores for the Spring 1999 administration of the (augmented) SAT/9 that have been posted on CDE’s website. In addition, we will work with our contract monitor to obtain student-level data files from CDE. The Consortium’s Year One achievement results were all based on student-level data from the 1998 SAT/9 because the background variables included on the student records supported analyses that built in statistical controls for students’ demographic characteristics and identified students who were in reduced-size classrooms.

J-7 Data: We will contact the Facilities Division to request copies of the 1997-98 J-7 Facilities data files as well as Operations and Facilities files for 1998-99. These data will allow us to continue tracking the flow of CSR funds to the districts during the second and third years of program implementation.

Commission on Teacher Credentialing (CTC) data: In Spring 1999, we acquired Excel spreadsheets from the CTC that summarize the numbers of emergency permits and credential waivers issued to California’s school districts for each of the years 1994 through 1997. These files became available too late to be analyzed during Year One. During Year Two, however, we will generate analysis files for these data and request copies of the corresponding Excel files for 1998. Since we propose to focus on issues of teacher quality during Year Two, these data will be of particular importance to us this year.

Collection of Survey Data. During the spring of the Year Two of the evaluation, we will develop and administer a third wave of surveys to our CSR sample districts, schools, and teachers, using the same sampling frame that was used in the 1998 and 1999 surveys. These data will be added to the survey data file constructed for the 1998 data and to the soon-to-be added 1999 survey data. This will provide us with three years of teacher survey data and two years (1998 and 2000) of survey data from districts and schools. Since the survey data will be collected in the spring of 2000, only the data collected from the 1998 and 1999 surveys will be analyzed and reported on in the Year 2 report.

Examination of CSR Implementation: Rates and Costs. As we continue to examine the continued implementation of CSR in California schools we will pay particular attention to the following issues:

  • What percentage of students by grade level were in reduced class sizes in each of the first three years of the CSR reform?
  • How has the percentage of teachers working in small classes changed over the first three years of implementation of CSR? Does the character of changes over time vary as a function of school characteristics such as percentage of low-income students 3 and percentage minority students, and whether the school is in an urban, suburban, or rural area?
  • Because of the differential ability of districts to implement CSR, we will continue to examine how CSR revenues available have varied by type of district (e.g., districts serving different percentages of low-income students or different percentages of minority students)?

Examination of CSR and Student Achievement. The most critical question regarding the CSR initiative is whether, and to what degree, CSR affects student performance. The Year 1 Evaluation Report described small achievement gains associated with reduced class size. In our Year 2 evaluation we will examine the association of achievement gains to CSR now that the reform has been in place for three successive years? We will use the cross-sectional design used in the Year 1 evaluation in Year 2 as well in an attempt to replicate the Year 1 findings. However, in addition, since there will be two years of STAR data, we will also be able to implement a dose-response model for the first time. Successive classes of 4th graders will have had increasing exposure to CSR. We will use this method to examine the effects of any exposure to CSR, and to also assess how the amount and timing of CSR exposure influences student outcomes. We will also examine the relationship of CSR to achievement for categories of special needs students (e.g., percent low income, minority, etc.) to determine whether as the CSR program matures whether these students gain more from CSR than other students, as was found in the Tennessee STAR study.

Examination of CSR and Teacher Characteristics. In the years prior to the CSR initiative, California enrollment surged, exceeding the national average growth rate from 1988 on, and creating an increased demand for qualified teachers. Implementation of CSR required many new teachers to fill new K–3 classrooms. This heavy demand for teachers raised important issues about the ability of schools and districts to find additional qualified teachers. The Year 1 Evaluation Report showed that teacher education and credentialing levels decreased significantly in the first two years of CSR implementation. Furthermore, these drops were not spread evenly across schools; comparatively greater drops were seen in schools serving more low-income, English language learner (ELL), Hispanic, or minority students, as well as in schools that were large and urban. We will focus on the following questions regarding teacher characteristics in Year 2:

  • Has the K–3 workforce continued to change during the third year of CSR implementation?
  • Has the gap in teacher qualifications between schools with high versus low percentages of special needs students continued to widen?

CSR and Changes in Instructional Practices. One of the hopes associated with the implementation of CSR was that once the constraints of larger class size had been lifted, teachers would be able to change their practices in ways that would benefit the instruction of their students. Analysis of changes in instructional practices using two years worth of data should allow us to begin to understand whether and how CSR has resulted in benefits to the education of California’s K-3 students.

Examination of CSR and Special Populations. An important issue to consider in evaluating CSR is how the implementation of class size reduction affects special needs students, such Limited English Proficiency (LEP) students, special education students, low-income and minority students. As indicated above, our analyses of the achievement and teacher qualification data will be done taking into account the percent of such students at the school or district level. We will also do more focused analyses on the flow of special education teachers in order to determine the degree to which CSR has siphoned off teachers with these credentials into regular classrooms. Finally, we will use the CASEMIS data to determine whether the percentage of students identified as needing an IEP has increased or decreased across the three years that CSR has been in place.

Examination of Promising Practices at the District and School Levels. One of the important issues to examine is the set of challenges that districts and schools faced in the implementation of CSR and how they overcame them. Doing so allows for the development of a set of "best practices" for districts that have not yet implemented CSR as well as for other states that are considering class-size reduction.

It had been our plan to investigate the relative importance of class size and teacher qualifications on student achievement. However, the data available from the California Department of Education do not allow the linking of the STAR data with the PAIF data at the classroom level. As a result, it is not possible to do convincing analyses of this important question. Therefore, in a separate letter, we are requesting an amendment to the contract that would support the collection of such data from a set of districts where the teacher data can be or has been linked to the student-level STAR data. These data would allow us to assess the relative roles of class size and teacher qualifications on student achievement.

Work to be completed with non-CDE funds in-hand

Examination of Classroom Practices. Although much attention has been given to the impact of class size on achievement, "the question of why these effects are realized remains largely unanswered" (Finn and Achilles, 1990). Presumably, differences in teaching practices in non-reduced and reduced size classrooms account for some of the observed differences in student outcomes. In the absence of a comprehensive theory, the scholarly and popular literature on class size offer starting points for thinking about teaching in non-reduced and reduced size classrooms. These notions are worthy of investigation in the absence of strong theory. One of continuing efforts of the CSR evaluation is to work toward an understanding of how curriculum and teaching practices differ in non-reduced and reduced size classes as a result of the implementation of CSR and to identify other factors that may contribute to these differences.

During the second year of evaluation, the classroom practices research will continue to focus on the following three questions:

  1. How prevalent are selected classroom practices in second, third and fourth grades in California?
  2. How do classroom practices differ between regular and reduced-size third grade classes?
  3. Are teacher background factors associated with classroom practices (e.g., teacher education, certification, professional development, and experience)?

Work to be completed with additional outside funding if obtained

The research activities described thus far will provide basic answers to the questions posed by the state regarding the efficacy of the CSR reform. However, the examination of more extensive issues regarding CSR will be contingent on the generation of outside support. Following is a list of activities that will be completed to the extent that outside funding is secured; activities are listed by research question addressed.

CSR and Student Achievement. In the Year 1 Evaluation Report we reported that gains in student achievement were associated with being in small classes, and this was true regardless of student demographics. However, we had neither funds nor time to examine the relative impact of class size on achievement compared other instructionally-relevant variables that might be related to achievement as well, i.e., teacher qualifications and instructional practices. Therefore, if outside funding is secured, we will address the following two achievement-related questions.

  1. Effects of CSR and Teacher Qualifications on Student Achievement: The Year 1 Evaluation Report showed that CSR had brought about a decline in the average level of teacher experience and qualifications. Furthermore, these changes were most pronounced in large, urban schools as well as those schools serving high percentages of low income, ELL, and ethnic minority students. In this substudy we would examine the relative impact of CSR versus teacher qualifications on achievement. It will be especially important to look for interaction effects in conducting these analyses. Doing so will allow an examination of whether credentials regardless of class size or whether schools with poorly qualified teachers are related to low achievement regardless of class size. If the data permit, we will also examine this question as a function of the percent of special needs students in the school. Doing so, will allow us to determine whether special needs students show larger achievement gains than other students when in small classes with fully credentialled teachers.
  2. Effects of CSR and Instructional Practices on Student Achievement: CSR supporters point out that smaller classes provide teachers with a better environment for teaching. However, findings from the Year 1 Evaluation Report suggested few differences in instructional practices between teachers in reduced and non-reduced classes. The initial analyses were aggregated over schools and classrooms; as such, they may not have sufficient sensitivity to identify practices related to improved achievement. By employing a finer grain of analysis in examining instructional practices and student achievement, we stand to gain more insight into how CSR may affect student achievement through changes in instructional practice. It is important to note that data currently collected for other components of the evaluation can be used for school-level analyses; thus, no additional data need be collected at this level. However, in order to address this question at the classroom level, additional data will need to be collected to link instructional practices and achievement of students within specific classrooms.

Effects of CSR on Teacher Qualifications in Grades 4-6. The Early Evaluation Report described changes in the qualifications K-3 workforce; overall experience and credentials of K-3 teachers has declined since the beginning of CSR. It is likely that changes have occurred in other divisions of the teacher workforce as well. If additional outside funds are obtained, one of the issues that would examine is whether and how CSR has affected the qualifications of teachers in grades 4-6.

CSR, Teacher Qualifications and Instructional Practices. One of the goals of the CSR evaluation is to learn what practices have been most efficacious for student learning. Once obtained, this information can be shared with California's education community to improve educational practice statewide. In order for this goal to be realized, extensive research must be completed to gain an understanding of how CSR enhances teacher practice in the classroom. Additional funding will allow us to examine the following two issues with regard to CSR and instructional practice:

  1. Effects of CSR and teacher qualifications on instructional practices: We have noted that the statewide implementation of CSR has been associated with a decline in teacher qualifications. How has instructional practice changed in light of the change in the teacher workforce?
  2. Microanalyses of instructional practices in large versus small classrooms: Multiple studies have detailed the merits of reducing class size; however, little insight has been provided into the link between this type of reform and increased student achievement. This trend demonstrates the elusive nature of the mechanism by which reducing class size may bring about positive student outcomes. As with any educational reform, multiple factors likely become intricately linked to give rise to the observed phenomena. By conducting microanalyses at the individual classroom level, we stand to gain insight into the complicated nature of instructional practice as influenced by CSR.

Analysis of Teacher Flows. As discussed immediately above, it is clear that the K-3 workforce has undergone significant changes since the implementation of CSR; additionally, we know that these changes are related to the income level and minority status of students within schools. The mechanisms by which these changes have occurred are not well understood, however. The level of teacher education and credentialing across schools may be changing due to flow of new teachers into schools, transfer of current teachers across schools, or the return of experienced teachers after having left the classroom. Therefore we propose to undertake a longitudinal analysis of teacher flows in Year 2 of the study. These analyses may provide greater insight into the mechanisms that give rise to school-level changes in teacher qualifications.

CSR Implementation and Resource Reallocation. The Year 1 Evaluation Report noted that many schools implementing CSR faced a deficit due to incurring costs greater than those covered by the CSR per-pupil revenue distribution. Such deficits led many of those districts to divert funds from needed areas such as building maintenance, administrative services, or student programs. Data supporting these conclusions includes that from CDE as well as that which has been collected via survey. While such data provides information regarding rates of implementation and reallocation decisions, it falls short of providing a detailed picture of the challenges of CSR implementation and ramifications of resource reallocation faced at the school level. Such insights are more readily available at the service of methods designed for the case analyses that would be used for this in-depth study.

Additional Dissemination. One of the goals of the CSR evaluation is to conduct activities that inform the educational policy community as well as the professional workforce. With the current CDE funding, the only Year 2 report that we will able to produce is the required Year 2 technical report. With additional outside funding we will be able to more widely disseminate the results of Year 2 evaluation. We would do so by providing special reports of the various aspects of CSR, some written for lay and other for professional audiences. In addition, these funds would be used to release the results at a press conference similar to which was done in Sacramento in Year 1.

 

Research Design

In this section we describe the various approaches and data sets that will be used to analyze the various research questions outlined above. Parallel to the presentation of the research questions, we break out the design issues by work to be completed with CDE funds, work to be completed with non-CDE funds in-hand, and work to be completed if outside funding is obtained.

For work to be completed with CDE funds

Research Design for Examination of CSR Implementation. The implementation issues listed above will be addressed using a combination of state data and survey information collected in conjunction with this study. State data include CSR implementation rates determined from both the California Department of Education (CDE) CSR web site and the California Basic Education Data System (CBEDS), which includes the Professional Information Assignment Form (PAIF) data set. Allocations by school district and descriptive information on California school districts will also come from CBEDS. Special education enrollments in grades K–3 over time will be assessed using the California Special Education Management Information System (CASEMIS). The survey information needed to address implementation issues will come primarily from the teacher questionnaires collected in 1999.

The analysis procedures involve an examination of CSR revenues and survey responses cross-classified by type of school or district (i.e., by such school and district characteristics as percentage of low-income students, percentage of minority students, and whether students reside in an urban, suburban, or rural area). Where appropriate, results will be evaluated for statistical significance at the .05 level or less with F-, t-, and chi-square tests.

Research Design for Measuring Achievement: The Dose-Response Model. Our overall analytic approach is a "dose-response" model in which we test whether greater exposure to class size reduction leads to greater achievement (controlling for student demographic characteristics and other achievement-related school factors). Our analysis plan takes advantage of the incremental implementation of class size reduction in California. More specifically, we propose to examine STAR scores for successive cohorts of second, third, and fourth graders. If CSR has its intended effect at a given grade level (e.g., Grade 2), we should see gains in achievement at that grade level over time. The best test of this "dose-response" approach will involve comparing successive cohorts of fourth grade students. With this approach, it will be possible to look at the cumulative effects of exposure to CSR in Grades 1-3.

To appreciate the logic behind our "dose-response" plan, it may be helpful to first review how CSR has been implemented over the first few years. The CSR program included approximately 51% of the students in grades K-3 in 1996-97. In 1997-98 the number of CSR-participating students increased to roughly 85%. We estimate that in 1999 over 90% of the K-3 students in California will be in reduced-size classes, with roughly 98% by the year 2000.

The dose-response model capitalizes on this natural variation in implementation to compare groups of students with different exposure to CSR. The following discussion illustrates how this is done. Students who were in fourth grade in the spring of 1998 can be divided into two groups: those with no exposure to CSR and those with one year of exposure (during third grade). Students in the third grade form three groups with zero, one, or two years of exposure to CSR. Furthermore, the third graders with one year of exposure can be further subdivided into subgroups based on the grade at which exposure occurred (second grade or third grade). Second grade students also form three groups with zero, one, or two years of exposure, and those with one year of exposure to CSR can be further subdivided based on time of exposure (first or second grade).

This distribution of levels of exposure to CSR enables us to use the STAR data set to test a number of hypotheses. We will conduct three tests of the primary hypothesis that achievement is related to exposure to CSR. We can compare fourth grade students with no exposure to CSR to those with one-year of exposure. Similarly, we can compare third-grade students with one-, two- or three-years of CSR and second grade students with one- or two-years of CSR. We can also test the sustaining effects of prior exposure by comparing the difference between fourth grade students with zero and one year of exposure to the difference between third grade students with the same pattern of exposure. (These differences confound exposure and cohort effects that cannot be fully disentangled.) The data also permit us to test the hypothesis that earlier exposure is more potent than later exposure. There are two sets of third-grade students who have a single year of exposure to CSR, one set with exposure in second grade and the other with exposure in third grade. Similarly, there are two sets of second grade students with one year of exposure, either in first grade or second grade.

As time since CSR's passage increases, the number of students with extended exposure (two or three years) increases and the number of students with low-exposure declines. This limits some of the partial exposure comparisons. By comparing data from 1999 to that from 1998 we can test for implementation effects, i.e., whether teachers have become more effective as they gain more experience with CSR for students with the same amount or type of exposure. For example, we can compare students at the same grade level with the same level of exposure in two successive years. This is possible with second-grade students and third-grade students who have two years of exposure. Again, these cross-year comparisons are not perfect tests because implementation differences are confounded with cohort differences, but they will shed some light on whether teachers become more effective over time.

The dose-response model serves as our main technique for examining the relationship between CSR and achievement. Furthermore, it will be the analysis of successive cohorts of fourth-grade students from 1998 (when only about 15% were exposed to CSR) through 2000 (when over 90% will have been exposed to CSR) which will be most informative. Dummy variable regression analysis controlling for student and school characteristics, such as teacher experience, will be the main analytic technique. Dummy variable analysis will also allow us to examine interaction effects (e.g., having a teacher without a degree, teaching LEP students, in an urban district). Finally, to the degree that the data allow, we will construct dummy variables to capture the effects of different patterns of exposure to CSR (e.g. the effects of a single year of exposure at Grade 2 versus Grade 3 versus exposure at both Grades 2 and 3). In addition we will construct a dummy variable to capture the total amount of exposure to CSR.

While we will not be able to draw firm causal conclusions from these analyses, we will be able to determine whether results are consistent with hypotheses about where we expect to find CSR most or least effective. Although we have access to STAR data at the student level, these data do not include classroom-level identifiers; thus, analyses described above will necessarily be conducted at the school level.

Research Design for Monitoring Teacher Recruitment, Assignment and Qualifications. These analyses will examine whether school-level inequities noted during the first two years of CSR have persisted, increased, or decreased. Furthermore, we will try to characterize the change in these inequities by school-level demographic dimensions (e.g. SES, %LEP). As was done in the Year 1 report, we will use the PAIF data to examine whether there have been changes in the teacher qualifications. In particular we will compare the overall distributions of teacher qualifications (e.g., experience, degree, and credentials) for 1995-96 through 1998-99. In addition, we will again cross-classify our indicators of teacher qualifications by school characteristics (obtained from the CBEDS data set) to determine whether the gap in such qualifications has continued to widen as a function of school characteristics. We will also use these same data sets to examine whether changes in the ratio of special education teachers to the number of children with IEPs as well as the ratio of bilingual education teachers to the number of ELL students has increased or decreased in the third year of CSR.

Research Design for CSR and Instructional Practices. Survey responses from the 1998 and 1999 teacher surveys will be used to estimate changes in teachers’ instructional practices, such as changes in approach to curriculum content or changes in the amount and kind of instructional and parental support. Initial analyses will focus on estimates of the prevalence and distribution of these changes. Next, the same contextual factors will be used for cross-tabulations and analyses of these changes by subgroup, using similar statistical techniques. This step will tell us if trends are consistent for different types of districts, schools, and students. We will use t-tests to identify characteristics and outcomes for which there was significant change over time.

Research Design for Special Population. STAR data achievement analyses described above (for the general population of students) will be replicated for subsamples of students according to special needs, e.g. by ethnicity, poverty, and English language proficiency.

We will also do more focused analyses on the flow of special education teachers in order to determine the degree to which CSR has siphoned off teachers with these credentials into regular classrooms. Some data on this topic are available from the Year 1 district level surveys. In addition, in 1999, CDE prepared a dataset that includes three years of data on K-3 teachers from the CBEDS-PAIF files. CDE linked teachers' records across the three years and then scrambled their ID numbers so individuals could not be identified. We will use these data to investigate the rate at which teachers with special education credentials moved to regular classrooms during the first two years of CSR. Finally, we will use the CASEMIS for 1995-96 through 1998-99 to determine whether the percentage of students identified as needing an IEP has increased with the implementation of CRS and whether those rates have increased, decreased, or remained the same across the three years that CSR has been in place.

Research Design for Identifying Promising Implementation Practices at the District and School Levels. Following on the advice of the Advisory Group and CDE staff, twelve districts were chosen in the spring of 1999 for in-depth interviews about the problems and challenges they faced in implementing CSR. They were also asked what steps or actions were undertaken to overcome these challenges. These data will be analyzed and presented in a separate chapter on promising implementation practices in our Year 2 report.

Work to be completed with non-CDE funds in-hand

Research Design for Analysis of Instructional Practices. Data to answer the questions listed above comes from three sources: surveys of representative samples of California teachers conducted in 1997-98 and 1998-99, in-depth case studies of sixteen California third grade teachers, and the STAR testing program. The case studies included classroom observations, teacher interviews, logs of instructional behaviors and videotapes of lessons in mathematics and language arts. The survey data will be analyzed to describe classroom practices statewide, and to explore links between teacher background factors and their classroom behaviors. The case study data will be the primary source of information for looking at relationships between class size and classroom practices. The analysis of field notes from the case study observations will be done using NUDIST, a software tool for examining textual data. Classroom logs and artifacts will be tabulated and analyzed using traditional quantitative methods. Most of this work is supported by a grant from the US Department of Education’s Office of Educational Research and Improvement (OERI). At the present time we have not raised funds to analyze the instructional videotapes.

Work to be completed with additional outside funding

Research Design for CSR and Factors Related to Student Achievement. The effects of CSR and teacher qualifications on student achievement would be examined at the school level with our linked PAIF-STAR data file. Because the state’s data sets do not allow for the linkage of teachers with student STAR data, these analyses can only be carried out by grade at the school level using this data set. The analyses will be done by cross-classifying teacher qualifications indicators (credentials, experience, and degree) with whether classrooms have been reduced or not at a given grade level in the school and looking at average STAR scores for various combinations of qualifications and class size. If there is a sufficient number of cases, we will add school characteristics (e.g., percent of low-income students) as another cross-classification variable. All analyses will be replicated using multiple regression as well, where school and teacher characteristics will be captured with dummy variables. Tests for interactions will be done for both types of analyses.

Research Design for Analysis of Grade 4-6 Teacher Qualifications. Above we described the research design to examine the effects of CSR on characteristics of the K-3 teacher workforce. A similar analytical framework will be employed to examine changes in the characteristics of teachers of grades 4-6 since the implementation of CSR. In particular, cross-classifications with be done using data on teacher qualifications form the PAIF data set.

Research Design for the analysis of CSR, Teacher Qualifications and Instructional Practices. Regression analyses will be used provide insight into the effect of CSR and changes in teacher qualifications on classroom instructional practices drawing on the 1998 and 1999 teacher surveys. Teachers for whom the greatest changes occurred (with respect to different outcomes) will be identified and analyses will be conducted to try to identify factors associated with these changes.

Research Design for Analysis of Teacher Flows. In 1999, CDE prepared a dataset that includes three years of data on K-3 teachers from the CBEDS-PAIF files. CDE linked teachers' records across the three years and then scrambled their ID numbers so individuals could not be identified. We will use these data to investigate the movement of teachers between schools and districts during the first two years of the class size reduction initiative. The analysis will reveal the extent to which CSR was associated with changes in the mobility of teachers, whether teacher mobility was associated with school demographic characteristics, and how mobility of teachers was related to their preparation and experience. We also hope to obtain a supplemental dataset that adds an additional year of data and also includes all teachers in grades K-6, so we can investigate the transfer of teachers across grade levels as well as between schools.

Research Design for CSR and Instructional Practices: Classroom Microanalyses. In order to gain insight into the complicated nature of CSR's effect upon instructional practice, we will conduct microanalyses at the individual classroom level. After selecting a sample of large and small classes, we will use ethnographic methods to gather data. This data will be used to determine the nature of instructional practice within the small context, which will be compared to that within the large class context.

Field notes collected for the instructional practices ethnographies will be entered into a computer-based program such as NUDIST or Ethnograph for organizing text data. The data will be coded to correspond to research questions and variables of interest. As a group, the study team will compare results, clarify definitions, and identify any missing categories. To illustrate relationships derived from the analysis of qualitative data, visual displays such as matrices, and narrative tables will be developed.

CSR Implementation and Resource Reallocation. We will employ methodological techniques from the qualitative paradigm in order to gain a comprehensive depiction of individual instances of CSR implementation and corresponding resource reallocations. After securing a sample of districts willing to serve as sites for this analysis, we will conduct extensive case studies at each site; activities will include interviews of school personnel, analysis of school records and budgets, and examination of implementation documentation. This data will be analyzed with qualitative techniques to arrive at a picture of the type of resource reallocation decisions faced by institutions implementing.

 

Deliverables

There will be two major deliverables in Year 2 – monthly reports and a Year 2 evaluation Report. The monthly reports will provide information on work carried out for the various tasks outlined above which are being supported by CDE funds. The reports will indicate any problems we have incurred in the previous month. They will be delivered to CDE within 30 days of the completion of a given month’s work. The Year 2 technical report will be delivered to CDE by June 30, 2000. It will contain analyses and write-ups not only for the analyses of CDE-research funded questions, but for the examination of research questions funded by non-CDE sources as well. Policy implications drawn from the findings will be presented in the report as well. Prior to completion of the final draft of the Year 2 report, we will call a meeting of the Advisory Group to review it.

Finally, the Co-PIs will attend quarterly meeting with the project monitor in Sacramento in order to provide detailed information about the progress of the project.

 

1 The investigation of classroom practices focused on teachers in third grade. < back to text >

2 Completed surveys were received from 1,075 parents of third-grade students (slightly greater than half of those who were mailed surveys). < back to text >

3 Students are referred to as low-income in this report if state records classify them as receiving public assistance in the form of Aid to Families with Dependent Children (AFDC) or its successor in California, CalWORKS. < back to text >