Research Reports
Testing the Effectiveness of Professional Development in Academic Vocabulary on Observed Teaching Practice and Academic Vocabulary Knowledge of English Learners in 8th Grade Social Studies Classrooms
Effective Teachers and Effective Teaching – Goal 3 Replication Study
(2024)
The Reliability and Consequential Validity of Two Teacher-Administered Student Mathematics Diagnostic Assessment
(September 30, 2020)
Several school districts in Georgia use two teacher-administered diagnostic assessments of student knowledge of mathematics as part of their multi-tiered system of support in grades K–8: the Global Strategy Stage (GloSS; New Zealand Ministry of Education, 2012) and the Individual Knowledge Assessment of Number (IKAN; New Zealand Ministry of Education, 2011), which comes in two formats (Counting Interview and Written Assessment). However, little is known about whether two teachers using the same assessment to assess the same student on two occasions within a short period of time assign the same Stage Score (interassessor reliability) or about how useful the teachers found the assessments (consequential validity).
Rather than relying on occasional testimonials from the field, decisions about using diagnostic assessments across the state should be based on psychometric data from an external source. Districts not currently using the GloSS and IKAN assessments have indicated an interest in using them, if they are proven to be reliable and valid diagnostic assessments, to assess students’ understanding of mathematics and determine appropriate levels of instruction and intervention. This study found adequate interassessor reliability for the GloSS and for the IKAN Counting Interview but not for the IKAN Written Assessment.
The IKAN Written Assessment requires additional attention to improve training so that reliability can be established. Teachers indicated that they found the screening data from the GloSS and IKAN assessments more useful for guiding decisions about student instruction and intervention than the screening data currently employed. Although teachers in the study’s focus groups expressed strong support for both assessments, teachers reported in the study’s survey that the GloSS is more useful than the IKAN because it assesses students’ solution strategies, unlike most other mathematics assessments. Teachers also expressed some criticisms of both assessments; for example, they believed that the GloSS should include vocabulary familiar to students and that the IKAN Written Assessment should be untimed.
​
What is the evidence base to support reading interventions for improving student outcomes in grades 1–3?
(April 2017)
The goal of this report is to provide administrators, school psychologists, counselors, special educators, and reading specialists with a summary and analysis of the evidence that supports the use of reading interventions in grades 1–3. The review was limited to studies of Tier 2 interventions, those designed to provide preventive services to students at risk for reading difficulties. The initial literature search identified 1,813 articles and reports. After screening them for relevance and conducting a detailed What Works Clearinghouse (WWC) analysis of the rigor of the study designs, the review team determined that only 23 effectiveness studies met WWC evidence standards (Version 3.0). Of those, 22 resulted in either significant, positive, or potentially positive impacts in at least one area of reading. None produced negative outcomes. Twelve of the 13 grade 1 interventions and all seven interventions for grades 2 and 3 produced positive or potentially positive effects. Effects were strongest and most consistent in the area of word and pseudoword reading. Several also produced effects in reading comprehension and passage reading fluency. Reading vocabulary was rarely assessed. Both individually administered and small-group interventions resulted in positive or potentially positive outcomes, although especially in grade 1, more of the interventions were one-on-one. In all cases, the interventionist received some training prior to implementing the intervention. However, these studies differed from common school practice in that implementation was carefully monitored in virtually all instances and coaching or feedback was provided. It is unclear how generalizable these findings are when the typical amount of ongoing support for interventionists is far more limited in practice.
​
Impact of the Developing Mathematical Ideas professional development program on grade 4 students’ and teachers’ understanding of fractions
(March 2017)
The purpose of this study was to assess the impact of the Developing Mathematical Ideas (DMI) professional development program on grade 4 teachers’ in-depth knowledge of fractions as well as their students’ understanding and proficiency with fractions. The study was conducted during the 2014/15 school year. A total of 84 schools from eight school districts in three states (Florida, Georgia, and South Carolina) agreed to participate. Participants included 264 grade 4 teachers and their 4,204 students. The study utilized the “gold standard” methodology involving random assignment of schools to either DMI or the control condition. Teachers in the DMI condition participated in 24 hours of professional development on fractions during fall 2014. They attended eight 3-hour sessions conducted over four days (two 3-hour sessions per day; one day per month). DMI did not demonstrate any impact on student knowledge of fractions. Students of DMI teachers performed at almost the same level as those taught by control teachers; the difference was not statistically significant. The impact of the DMI on teachers’ knowledge of fractions was inconclusive. DMI teachers performed slightly better than teachers who did not participate in DMI, but the result was not statistically significant. It was, however, close to the threshold of statistical significance (p = .051).
​
The study identified and screened 910 research studies in a comprehensive literature search for effectiveness studies of math professional development approaches. Of these 910 studies, 643 examined professional development approaches related to math in grades K–12 and were conducted in the United States. Of the 643 studies, 32 focused primarily on math professional development provided to teachers and used a research design for examining effectiveness. Five of those were determined to have met WWC evidence standards (version 2.1) either with or without reservations. And of those five, only two found positive effects on students' math proficiency. Thus, there is very limited causal evidence to guide districts and schools in selecting a math professional development approach or to support developers' claims about their approaches.
​