State Releases Data On ISTEP+ Testing Problem
July 28, 2016 at 4:25 p.m.
By David [email protected]
According to a press release issued by the Indiana Department of Education, among other things, the report shows because of the efforts of teachers, administrators, students and parents, as well as the actions taken by Ritz, the average negative statewide impact on scores was not measurable. However, this does not mitigate the effect the interruptions had on students, parents and teachers throughout Indiana.
At this time, the exact impact of interruptions at the individual, classroom and teacher level cannot be ascertained.
“First, I want to acknowledge the extraordinary efforts of Indiana students, parents, teachers, administrators and the employees of the Department of Education,” said Ritz. “Because of their dedication and hard work, the impact of these interruptions was limited. However, let me be clear, the problems with the ISTEP+ contractor were absolutely unacceptable. Every student deserves the opportunity to take a fair and uninterrupted assessment.
“I have spent the last several months talking with Hoosiers about the impact these interruptions had in the classroom. Although Dr. Hill’s report found that the statewide average score was not affected by the interruptions, there is no doubt that thousands of Hoosier students were affected. As Dr. Hill stated in his report, ‘We cannot know definitively how students would have scored this spring if the interruptions had not happened.’ Because of this, I have given local schools the flexibility they need to minimize the effect these tests have on various matters, such as teacher evaluation and compensation. I have also instructed CTB McGraw-Hill to conduct enhanced stress and load testing to ensure that their servers are fully prepared for next year’s test and ensure that this never happens again,” she continued.
The IDOE is conducting an ongoing negotiation regarding settlement with CTB McGraw-Hill. Next steps for the IDOE include processing student reports to be available online to parents and students, and calculating A-F accountability results.
An interactive map showing the ISTEP+ interruptions by school corporation can be found by clicking online at http://www.stats.indiana.edu/maptools/ISTEPinterruptions.html
According to the map, Warsaw Community Schools had 634 students interrupted (20 percent). By content area, 573 were interrupted during the math portion of the test, 44 in English, 12 in science and 22 in social studies. Some students may have been interrupted in more than one content area.
Tippecanoe Valley had 97 students (11 percent) interrupted. Of those, 70 were interrupted during the math portion, 11 in English, six in science and 14 in social studies.
Manchester Community Schools had 178 students (25 percent) interrupted. That included 96 in math, 66 in English, 18 in science and four in social studies.
There were 190 students (14 percent) interrupted at Wawasee, including 176 in math, nine in English, six in science and one in social studies.
Whitko Community Schools had 139 students interrupted (19 percent). That included 136 in math, two in English, one in science and three in social studies.
Triton School Corp. did not test students online, according to the map.
The Indiana Statewide Testing for Educational Progress—Plus is Indiana’s statewide testing program. Students in public and nonpublic schools in third through eighth grade take this test. There are substantial consequences for test results at all levels in the public schools, including teachers.
Indiana has been transitioning the administration of the test from paper-and-pencil to on-line testing since 2009. This past spring, approximately 95 percent of the students took the test online, an increase from 71 percent the previous year.
Testing began this year April 29, according to the release. Starting at about 10:30 that morning, students throughout Indiana experienced interruptions during their testing. It was quickly discovered that the interruptions were caused by a memory issue on the CTB/McGraw-Hill servers. Because CTB’s immediate efforts to resolve the situation were unsuccessful, their technology engineers worked to isolate the source of the issues and made necessary adjustments to return to normal status as soon as possible. Based on these interruptions, Ritz extended the testing window by two days to May 14.
On the second day of testing, at around 11:15 a.m., a different memory issue on CTB/McGraw-Hill’s servers caused additional widespread interruptions for Indiana students. Students again experienced the issues seen on April 29, but in greater volume. In response, CTB determined that the ISTEP+ Online system had to be “cut over” to the disaster recovery site. While the system remained accessible, this “cut over” caused interruptions for almost all students who were active in the system.
Also, as the system was moved from the regular to the disaster recovery servers, not all of the student responses were immediately accessible to students when they logged back into that test session. All of the student responses had been saved, but they were not immediately available due to the system issues. Based on the severity of the interruptions and a recommendation from CTB, Ritz requested that students should complete their current test session and then schools should suspend online testing for the rest of the day. Ritz asked that schools reduce their online testing to 50 percent of their planned testing load for the following day. Also, Ritz extended the online testing window three additional days, through May 17.
On May 1, online testing resumed at 50 percent of planned capacity. Students using CTB’s system experienced no further widespread interruptions. As a precautionary measure, Ritz asked schools to continue to reduce online testing to 50 percent of their planned testing load for the following day. On May 2, Ritz once again asked schools to reduce online testing to 50 percent of their planned testing load for one more day as a precautionary measure. On May 3, Ritz conducted three conference calls with Indiana superintendents. On May 6, she directed schools to resume online testing at 100 percent of their capacity. Online testing was completed May 17.
On May 24, the IDOE provided schools with a list of students that CTB indicated had interrupted testing sessions. The IDOE gave that list to local schools so that they could check the list against their records and add any students they determined were impacted by the interruptions but missed by CTB.
On that same day, the IDOE also issued a request for qualifications to three national companies experienced in validating test results. From that process, the National Center for the Improvement of Educational Assessment was awarded a contract to investigate the impact the interruptions had on ISTEP+ test scores.
The report states there are two sources of data available about the interruptions, according to the release. The first comes from the records of CTB. As students completed the test, data were captured about the timing of all events. As a result, the CTB data can, for example, tell how much time a student spent on the test before an interruption occurred, how many items were presented to the student before the interruption, and how long it was before the student answered another question.
In addition to the CTB data, local school systems were provided with the opportunity to identify additional students who were interrupted – or affected by interruptions, in the judgment of the local person completing the form. These data were collected by providing local school systems a list of the students identified by CTB as having been interrupted and allowing them to append additional students to the file. In contrast to the detail of the CTB data, the local appends identified only the test (mathematics, English/language arts, science, social studies) for which a student had been affected.
According to the release, there were significant numbers of interruptions at all grades, but grades 3-6 had a higher proportion of interruptions than grades 7 and 8. This may be simply a function of the time of day that testing started. The substantial majority of interruptions occurred during Sessions 1 and 2 (when students were taking the mathematics test) than during the later sessions. Of course, it is possible that a student who was interrupted during Session 1 was affected for the remainder of the testing.
Non-public school students had approximately the same proportion of interruptions as public school students, although this trend varied from grade to grade. Non-public school students make up about 7.5 percent of the tested population, and had slightly less than 8 percent of the interruptions, totaled across the grades. Their percentage ranged from a high of 12 percent at grade 7 down to 6 percent at grade 3.
Once students were interrupted, there was a range of time before they restarted the test. Sometimes, the length of that delay was a function of the responsiveness of the system; at other times, it was due to a school decision to stop the administration for students for a period of time and have them restart the test at a later time. When students restarted, they sometimes had to redo the last item they had been working on before the interruption occurred, but for the vast majority of students, this was the extent of lost data. However, there were 600 students (440 in math and 160 in ELA) whose data was not “restored” when they logged back in. These students ended up with two sets of responses to their interrupted session and if any of their answers were different (and either one was correct), they were given credit for the correct answer.
For public school students, the most common delay was for a day or more, although that was less than a majority of the interruptions. For students delayed less than a day, the most common delay was for five minutes or more, but less than 15. Students in non-public schools had more of a tendency to restart the test the same day they were interrupted, with the most common delay being five to 15 minutes for them, too. A total of 734 observations (less than 1 percent) could not have their delay coded because their end-of-interruption time was not recorded on the interruptions file.
There were a total of 117,379 interruptions. Some students were interrupted more than once. A total of 79,442 students were interrupted, which is about one-sixth of the total population, according to the release. The IDOE cautions that just because a student was not reported as interrupted, that does not mean the student was unaffected by the interruptions. The interruption of one student in a room could conceivably have an effect on other students in that same room.
While a strong majority of the interruptions are in math, the interruptions during the ELA test total about one-fourth of all the interruptions.[[In-content Ad]]
According to a press release issued by the Indiana Department of Education, among other things, the report shows because of the efforts of teachers, administrators, students and parents, as well as the actions taken by Ritz, the average negative statewide impact on scores was not measurable. However, this does not mitigate the effect the interruptions had on students, parents and teachers throughout Indiana.
At this time, the exact impact of interruptions at the individual, classroom and teacher level cannot be ascertained.
“First, I want to acknowledge the extraordinary efforts of Indiana students, parents, teachers, administrators and the employees of the Department of Education,” said Ritz. “Because of their dedication and hard work, the impact of these interruptions was limited. However, let me be clear, the problems with the ISTEP+ contractor were absolutely unacceptable. Every student deserves the opportunity to take a fair and uninterrupted assessment.
“I have spent the last several months talking with Hoosiers about the impact these interruptions had in the classroom. Although Dr. Hill’s report found that the statewide average score was not affected by the interruptions, there is no doubt that thousands of Hoosier students were affected. As Dr. Hill stated in his report, ‘We cannot know definitively how students would have scored this spring if the interruptions had not happened.’ Because of this, I have given local schools the flexibility they need to minimize the effect these tests have on various matters, such as teacher evaluation and compensation. I have also instructed CTB McGraw-Hill to conduct enhanced stress and load testing to ensure that their servers are fully prepared for next year’s test and ensure that this never happens again,” she continued.
The IDOE is conducting an ongoing negotiation regarding settlement with CTB McGraw-Hill. Next steps for the IDOE include processing student reports to be available online to parents and students, and calculating A-F accountability results.
An interactive map showing the ISTEP+ interruptions by school corporation can be found by clicking online at http://www.stats.indiana.edu/maptools/ISTEPinterruptions.html
According to the map, Warsaw Community Schools had 634 students interrupted (20 percent). By content area, 573 were interrupted during the math portion of the test, 44 in English, 12 in science and 22 in social studies. Some students may have been interrupted in more than one content area.
Tippecanoe Valley had 97 students (11 percent) interrupted. Of those, 70 were interrupted during the math portion, 11 in English, six in science and 14 in social studies.
Manchester Community Schools had 178 students (25 percent) interrupted. That included 96 in math, 66 in English, 18 in science and four in social studies.
There were 190 students (14 percent) interrupted at Wawasee, including 176 in math, nine in English, six in science and one in social studies.
Whitko Community Schools had 139 students interrupted (19 percent). That included 136 in math, two in English, one in science and three in social studies.
Triton School Corp. did not test students online, according to the map.
The Indiana Statewide Testing for Educational Progress—Plus is Indiana’s statewide testing program. Students in public and nonpublic schools in third through eighth grade take this test. There are substantial consequences for test results at all levels in the public schools, including teachers.
Indiana has been transitioning the administration of the test from paper-and-pencil to on-line testing since 2009. This past spring, approximately 95 percent of the students took the test online, an increase from 71 percent the previous year.
Testing began this year April 29, according to the release. Starting at about 10:30 that morning, students throughout Indiana experienced interruptions during their testing. It was quickly discovered that the interruptions were caused by a memory issue on the CTB/McGraw-Hill servers. Because CTB’s immediate efforts to resolve the situation were unsuccessful, their technology engineers worked to isolate the source of the issues and made necessary adjustments to return to normal status as soon as possible. Based on these interruptions, Ritz extended the testing window by two days to May 14.
On the second day of testing, at around 11:15 a.m., a different memory issue on CTB/McGraw-Hill’s servers caused additional widespread interruptions for Indiana students. Students again experienced the issues seen on April 29, but in greater volume. In response, CTB determined that the ISTEP+ Online system had to be “cut over” to the disaster recovery site. While the system remained accessible, this “cut over” caused interruptions for almost all students who were active in the system.
Also, as the system was moved from the regular to the disaster recovery servers, not all of the student responses were immediately accessible to students when they logged back into that test session. All of the student responses had been saved, but they were not immediately available due to the system issues. Based on the severity of the interruptions and a recommendation from CTB, Ritz requested that students should complete their current test session and then schools should suspend online testing for the rest of the day. Ritz asked that schools reduce their online testing to 50 percent of their planned testing load for the following day. Also, Ritz extended the online testing window three additional days, through May 17.
On May 1, online testing resumed at 50 percent of planned capacity. Students using CTB’s system experienced no further widespread interruptions. As a precautionary measure, Ritz asked schools to continue to reduce online testing to 50 percent of their planned testing load for the following day. On May 2, Ritz once again asked schools to reduce online testing to 50 percent of their planned testing load for one more day as a precautionary measure. On May 3, Ritz conducted three conference calls with Indiana superintendents. On May 6, she directed schools to resume online testing at 100 percent of their capacity. Online testing was completed May 17.
On May 24, the IDOE provided schools with a list of students that CTB indicated had interrupted testing sessions. The IDOE gave that list to local schools so that they could check the list against their records and add any students they determined were impacted by the interruptions but missed by CTB.
On that same day, the IDOE also issued a request for qualifications to three national companies experienced in validating test results. From that process, the National Center for the Improvement of Educational Assessment was awarded a contract to investigate the impact the interruptions had on ISTEP+ test scores.
The report states there are two sources of data available about the interruptions, according to the release. The first comes from the records of CTB. As students completed the test, data were captured about the timing of all events. As a result, the CTB data can, for example, tell how much time a student spent on the test before an interruption occurred, how many items were presented to the student before the interruption, and how long it was before the student answered another question.
In addition to the CTB data, local school systems were provided with the opportunity to identify additional students who were interrupted – or affected by interruptions, in the judgment of the local person completing the form. These data were collected by providing local school systems a list of the students identified by CTB as having been interrupted and allowing them to append additional students to the file. In contrast to the detail of the CTB data, the local appends identified only the test (mathematics, English/language arts, science, social studies) for which a student had been affected.
According to the release, there were significant numbers of interruptions at all grades, but grades 3-6 had a higher proportion of interruptions than grades 7 and 8. This may be simply a function of the time of day that testing started. The substantial majority of interruptions occurred during Sessions 1 and 2 (when students were taking the mathematics test) than during the later sessions. Of course, it is possible that a student who was interrupted during Session 1 was affected for the remainder of the testing.
Non-public school students had approximately the same proportion of interruptions as public school students, although this trend varied from grade to grade. Non-public school students make up about 7.5 percent of the tested population, and had slightly less than 8 percent of the interruptions, totaled across the grades. Their percentage ranged from a high of 12 percent at grade 7 down to 6 percent at grade 3.
Once students were interrupted, there was a range of time before they restarted the test. Sometimes, the length of that delay was a function of the responsiveness of the system; at other times, it was due to a school decision to stop the administration for students for a period of time and have them restart the test at a later time. When students restarted, they sometimes had to redo the last item they had been working on before the interruption occurred, but for the vast majority of students, this was the extent of lost data. However, there were 600 students (440 in math and 160 in ELA) whose data was not “restored” when they logged back in. These students ended up with two sets of responses to their interrupted session and if any of their answers were different (and either one was correct), they were given credit for the correct answer.
For public school students, the most common delay was for a day or more, although that was less than a majority of the interruptions. For students delayed less than a day, the most common delay was for five minutes or more, but less than 15. Students in non-public schools had more of a tendency to restart the test the same day they were interrupted, with the most common delay being five to 15 minutes for them, too. A total of 734 observations (less than 1 percent) could not have their delay coded because their end-of-interruption time was not recorded on the interruptions file.
There were a total of 117,379 interruptions. Some students were interrupted more than once. A total of 79,442 students were interrupted, which is about one-sixth of the total population, according to the release. The IDOE cautions that just because a student was not reported as interrupted, that does not mean the student was unaffected by the interruptions. The interruption of one student in a room could conceivably have an effect on other students in that same room.
While a strong majority of the interruptions are in math, the interruptions during the ELA test total about one-fourth of all the interruptions.[[In-content Ad]]
Have a news tip? Email [email protected] or Call/Text 360-922-3092