The Principal, the Test Scores, and the Investigation That Wasn’t

What happens when a principal pushes for answers to questions about an unprecedented drop in high-stakes standardized test scores?

Wouldn’t you think that district officials would take every measure possible to ensure the scores reflected valid measurements of students learning and teachers teaching?

What if they don’t?

What if the answers are never found, yet the heavy burden of failing Adequate Yearly Progress (AYP) remains?

When the principal of an elementary school in Hoover opened results of the spring 2012 Alabama Reading and Mathematics Test Plus (ARMT+) in July 2012, she knew something had to be wrong. The results were nowhere near what pre-tests had indicated those results should be.

Without raising alarm among her faculty, knowing the school might be slapped with a “failing AYP” label, the principal received permission to pursue an investigation into what could possibly have happened to those test scores.

She enlisted help along the way. The district’s elementary curriculum director, the assistant superintendent of instruction, the assistant superintendent, the superintendent…all were involved in one way or another.

A statistician with experience in state education assessments helped draft a letter to send to a state assessment official and the state superintendent in order to make certain the right questions were asked when the investigation began.

A preliminary investigation confirmed the principal’s assertions that a timing issue that resulted in students not having enough time to complete the test might be the cause of the drop.

And then the investigation abruptly ended, in spite of the principal’s objections.

Parents and teachers were never told of the investigation nor of the preliminary findings. They couldn’t be told by the principal, as she was transferred to the Central Office in the midst of the investigation. The principal retired the following year.

For district officials to choose to hang the failing label on a school—affecting the teachers, the students, and the community—rather than search for answers is not a story I ever expected to tell.

This is the first time this story is being told, and it is a difficult story to tell.

The Principal and Why the Story Isn’t About Her

When Robin Litaker was named principal of Trace Crossings prior to the start of the 2010-2011 school year, she knew that improving test scores was not only a priority for district officials, but also very attainable given the talent and resources available in Hoover City Schools.

Overall standardized test scores for students at Trace Crossings, a K-4 school, had been consistently among the lowest of Hoover’s elementary schools since the time that test scores became a topic of conversation. Parents living in the fine homes surrounding the school were increasingly choosing private school for their elementary-aged children. Litaker knew the task that lay ahead could prove to be difficult.

District leaders acknowledged to Litaker that turning long-standing testing trends at Trace Crossings would take three to five years.

And so the work began.

The first year’s test results revealed a modest increase. The second year’s results revealed something different.

To be clear, this is not a story about Robin Litaker. This is not even a story about Trace Crossings Elementary School.

This is a story about a principal who pursued every avenue to find out what caused that unprecedented drop in test scores.

This is a story of the brick walls that principal encountered along the way.

And how the answers still haven’t been found more than two years later.

The Scores

Here’s a visual representing the dive that the fourth grade math scores took in the spring of 2012.

When Litaker first saw those scores in July 2012, she knew something was wrong. Within thirty minutes, she had a conversation with Superintendent Andy Craig about her concerns. Litaker’s concerns were downplayed by Craig, who assured her everything would be okay. Litaker felt certain her school would be slapped with the “failing” AYP label when determinations were released the following month.

She called a meeting with district officials to determine next steps. According to Litaker, that initial meeting included Craig, Assistant Superintendent Carol Barber, Assistant Superintendent of Instruction Ron Dodson, and Elementary Curriculum Director Deborah Camp. Math instruction was discussed, and a plan was devised to bring more resources to the school. Walkthroughs were planned once school started to provide feedback for teachers on instructional practices.

The following week, Litaker emailed the fourth grade math teachers and the assistant principal to ask for help reviewing individual student test scores to determine by how much those scores had fallen from the third grade to the fourth grade. The group met at the school to hand-calculate that difference in students’ scores.

According to Alabama State Department of Education (ALSDE) documents, 83 of Trace Crossings’ 111 third graders had reached proficiency (scoring at a Level III or IV) in math during the 2010-2011 school year. Only 57 of the 106 fourth graders had scored at Level III or IV during the 2011-2012 school year. Thirteen of the third graders who had scored proficient in the previous year no longer were enrolled at Trace Crossings.

Expectations based on pre-testing were that 83 students would reach proficiency level, although they were not the same 83 from a year earlier.

During that meeting, the teachers revealed to the principal that none of the fourth grade students had completed the math portion of the ARMT+ during testing. Litaker asked why no one had informed her of this on the day of testing. The teachers said they didn’t want to upset her.

At that point, Litaker recalled that one teacher had told her on the day of testing that none of the students in her class had completed the math portion of the test. Litaker filed a Test Irregularity Report, required by the ALSDE whenever something unusual happens during testing. But because only one teacher had reported a problem with completing the test, nothing more was done at the time.

Had the principal known at the time that none of the students completed the test, the ALSDE might have been able to provide guidance and support.

When asked what the remedy could have been had timing been an issue, Rebecca Mims, the current Coordinator of Student Assessment for the ALSDE, told me that if an irregularity due to mistiming was reported, depending on the outcome of the investigation, the scores could be invalidated.

Mims added that failing to report an irregularity is a test security violation. “If a staff member knows that a testing irregularity occurred, whether or not that teacher was involved in the testing irregularity, that staff member must report that irregularity,” Mims said.

No consequences for failure to report the testing irregularities were pursued by the principal. There were already climate and morale issues that were causing problems among teachers, and she believed the best course of action was to determine the cause for the drop in scores.

Though we don’t hear of it often, occasionally standardized test scores may be questioned due to mistakes made during administration of the test. Just last August, the superintendent of the Healdsburg Unified School District in California raised the question of whether problems with test administration could have caused a precipitous decline in their test scores.

The Statistician’s Report

Litaker and the teachers hand-calculated the difference in each student’s score from third grade to fourth grade.

Litaker had received permission from Craig to consult with Dr. Scott Snyder, a statistician from a local university with experience working with state education assessments. The following week, Litaker and Camp, who was now district official placed in charge of the investigation, shared the scores (no personally identifiable information was shared, only numbers) with Snyder. The principal reported to Craig and others that after reviewing the scores over a period of nearly three hours, Snyder reached the conclusion that the severe drop was not able to be explained statistically and that fourth grade students could have stayed home all year and not have had their math scores drop that much from third to fourth grade.

It was agreed that Superintendent Craig would ask the ALSDE to conduct an investigation. Camp was charged with drafting the letter and enlisted the statistician to help her with the wording.

Failing AYP

As expected, a few days before school began, Litaker was notified that the school failed AYP, specifically because not enough students in the Black subgroup and not enough students in the Poverty subgroup had reached proficiency in fourth grade math.

Litaker first told her teachers, then her Parent-Teacher Organization (PTO) board about the failure. She told them that she felt something might be wrong with the scores, but for them not to worry, and that she would take care of it. She was concerned about what effect failing AYP might have on teachers and parents and was compelled to continue the investigation.

Trace Crossings was one of three Hoover schools that failed AYP due to the spring 2012 test results. The other two schools that failed AYP were Brock’s Gap Intermediate and Berry Middle School, both due to failing to meet reading proficiency goals.

The Newspaper Article

At the end of August, when interviewed by al.com about Trace Crossings failing AYP, Dodson focused his remarks on math instruction, stating “We’re proud of the prin­cipal’s and faculty’s reac­tion to this news and the manner in which they’ve embraced the goal of im­proving the math instruc­tion.”

Multiple paths for improving math instruction were underway at Trace Crossings.

The Letter That Wasn’t Sent

On October 10, 2012, Camp emailed the letter she had drafted (with the statistician’s help) to Louise White, the superintendent’s administrative assistant, instructing White to print the letter on official letterhead and send hardcopies and electronic copies to Gloria Turner, Director of Assessment and Accountability at the ALSDE, and to State Superintendent Dr. Tommy Bice.

Craig, Dodson, Barber and Litaker were copied on the email.

The letter contained the specific request that the “student-completed test forms for Trace Crossings Elementary School’s fourth grade math for the 2011-2012 academic year be examined for inconsistencies such as erasures, multiple responses, and incomplete forms.” Here is the letter.

On October 12, 2012, White responded to Camp’s email, stating that the letter had been “scanned and posted to Dr. Bice and Dr. Turner.”

Shortly after the letter was believed to have been sent, Dodson took over the investigation.

Camp and Litaker were both under the impression that the letter had been sent and a full investigation was underway.

An Open Records request to the ALSDE I initiated in September 2013 revealed that the letter had never been received by Turner or Bice.

Instead, according to Dodson, he contacted Turner and “explained the situation to see if there was something else we could do or should have done.” He added that Turner offered to re-score a sample of answer sheets to confirm “there was not a mechanical error in the initial scoring process.”

Litaker submitted to Dodson the names of ten students whose scores were examples of the most egregious score drops from third to fourth grade.

Dodson explains what happened next in an email response to me: “I eventually received a call from a member of Dr. Turner’s staff who reported that the re-score indicated that all of the student answer forms had been scored correctly. She also shared that it did appear that those ten students did not finish the test.”

Dodson reported those results via email to Craig, Barber, Litaker and Camp on December 13, 2012, stating:

“We will receive an official written report from Dr. Turner in January, but I just heard the verbal response to our request for a state inquiry into the unprecedented 4^th grade math score drop at TCES (Trace Crossings Elementary School) last spring. Dr. Turner requested that we submit the names of the ten most egregious examples of student score drops. Robin selected those ten names which I submitted, and the state hand-scored those student answer sheets.

The results are that all ten student answer sheets were scored correctly, so the state finds no evidence of a scoring error. The person who led the review did tell me that none of those ten students completed part 1 of the test.”

No promise of any further investigation was made. Litaker responded to the group less than half an hour later that the question of whether there was a problem with timing and administration of the test now rose to a new level of concern. Litaker waited on the full written report Dodson said was forthcoming.

Dodson has not yet answered my questions asking what happened to the letter and why he did not follow the statistician’s recommended method of investigation which would have examined the answer sheets for “inconsistencies, such as erasures, multiple responses and incomplete forms”.

I found no record of Turner having promised him a written report would follow.

In June, I asked Bice to respond to claims that Hoover school officials had not adequately investigated circumstances surrounding the spring 2012 math score drop.

In a response to me dated June 20, 2014, Bice outlined the steps Hoover officials had taken. Item number five of the response stated the following:

“The results of the ALSDE’s analysis revealed that none of the students completed Part 1 of the assessment, which caused a significant decrease in their test scores. These results were reported back to Hoover City Schools.”

It appeared that the next move to be made, if any, was Hoover’s.

The Principal Gets Moved

While this investigation was ongoing, in mid-November of 2012, Litaker was moved out of Trace Crossings into the Central Office. Assistant Superintendent Carol Barber stepped into the Principal’s position. An email Barber wrote about Litaker’s transfer was obtained by al.com and reported here.

Alabama Education Association representative Dana Clement, in an interview with al.com shortly after Litaker was moved, shared what she had heard from teachers at the school about the climate within the school.

On Monday, I asked Clement if she had been made aware of the investigation into the AYP failure during the spring 2012. In an email response, Clement stated that she was only made aware of it through my inquiry.

The Push for More Answers

In an email to Craig at the end of December 2012, Litaker reiterated her strong belief that a full investigation was necessary to heal the Trace Crossings community.

Craig gave her his support and provided permission to pursue the investigation with Dodson. With renewed energy, Litaker asked Dodson what the next steps would be.

A month passed with no action.

On February 4, 2013, writing in an email to Dodson and Craig, Litaker stated “I am trying to remedy something that is obviously very wrong.  —- I think the teachers, students and the community deserve better than to believe it was an achievement issue….”

After a short response from Dodson, and in a second email that followed minutes later, Litaker stated:

“I am trying to be a voice for those students who will have those scores in their files….and for the teachers….Who I know worked very hard — and for that community that needs to believe in their school….

I know that it is February…. This has been going since July 3^rd…..This is my last effort….”

After that exchange, Dodson told Litaker he would speak with Turner again to determine the next steps and what could be done.

For the next few weeks, Litaker regularly asked Dodson where the investigation stood. Dodson said he hadn’t had a chance to speak with Turner but would do so.

During what became the final inquiry to Dodson during the last week of February, Dodson told Litaker that it was too late to file an AYP appeal.

Litaker asked if she could contact the ALSDE herself. Dodson advised her not to do so.

The Shutdown of the Investigation

At that point, the investigation ended. Litaker turned in her retirement papers in April 2013.

When asked why he ended the investigation at that point, Dodson said that even if the state had ruled the responses were invalid, the district would not have had its failing AYP status removed. He added that when Bice performed his investigation, that Bice assured Hoover officials that they had “responded appropriately to this situation in his judgment”.

It was clear that Dodson believed that the district would receive no benefit from investigating the drop any further.

But what about the students, the teachers, the community?

Might they benefit from having the failing AYP label removed?

Might they benefit from a thorough investigation?

I asked Dodson whether parents were notified that none of the students in the re-scored sample had completed the test and that the ALSDE had concluded that was responsible for the decline in test scores. [I asked Dodson nearly 20 more questions in a follow up email, to which he has not yet responded.]

I spoke with a few parents whose children were enrolled at Trace Crossings in the spring of 2012, and they said they had no idea that an investigation was ever conducted. They were never told that the low math scores could have been due to a timing issue. Parents were reluctant to speak with me, and none were willing to go on the record.

I asked Dodson if any notation had been made in the students’ permanent records indicating that those scores may be invalid.

That question, along with many others, remain unanswered.

Open Records requests relevant to this investigation have not been filled by Hoover City School officials. Requests for interviews with district personnel have been ignored.

While I have not attempted to speak with the teachers whose math scores were in question, only two of the five teachers whose classes were affected remain at Trace Crossings.

Litaker retired from the district December 31, 2013. Camp had retired during the summer of 2013.

The Rebound and the Residual Effects

Test results from the spring of 2013 were released late in the year. As expected, results rebounded almost back to the 2011 levels.

There was no joyous celebration. No district press release. No pats on the back for the teachers and community.

Barber just started her second full school year as Principal at Trace Crossings.

Superintendent Craig has recently begun sharing initial proposals to rezone students in eight of Hoover’s ten elementary schools. One of the reasons given for the need to rezone is the under-utilization of Trace Crossings Elementary School. The building has housed as many as 1,000 students, but enrollment for the 2013-2014 school year was 488 students. The student body was split into two schools in 2004 and rose to a peak of 619 students in 2010-2011. Current enrollment reflects a 20% decline since that year.

Do test scores have anything to do with that under utilization? Have test scores influenced movement away from Trace Crossings?

Emotions were high at the August board of education meeting and it was obvious that test scores were not far from most folks’ minds.

One parent expressed her concern that property values in her neighborhood would be affected if they were rezoned away from their current school and into the Trace Crossings zone. “We have a lot of very concerned and upset people in our neighborhood.”

At that same meeting, the President of Trace Crossings PTO stepped to the podium and did her part to let people know that while test scores did drop a couple of years ago, they have since improved.

For the principal, Bice’s affirmation that the drop in test scores was caused by students not completing part 1 of the test is helpful. She acknowledges, though, that unless the community learns of the investigation and the preliminary findings, the failing AYP label is still there.

The residual effects of that label have been painfully apparent in community discussions of rezoning.

Craig, Dodson, and Barber have had many opportunities to share the results of the ALSDE’s investigation but have not done so.

I can’t tell you whether timing was the problem or whether math instruction was really the issue. No one can at this point. Because the investigation was never completed.

According to sources at the ALSDE, those answer sheets are still being stored and could be reviewed if district officials chose to do so.

The final question that I asked Dodson in the email sent nearly two weeks ago was this: Would you be willing now to request a full investigation into the 2011-2012 test results at Trace Crossings Elementary School using the remedy that the statistician recommended? I’m still waiting on a reply.

In this document, created for Trace Crossings’ accreditation purposes in January of this year, school officials made the following statement:

Repairing trust and respect for school personnel, programs, and practices continues to be a major focus for school personnel and parents/community.

Could that trust and respect be repaired if school officials re-opened the investigation to find out, once and for all, what really happened?

More Data If You’d Like to See It

Here’s some more test score stuff to poke through if you’re interested. As you review it, remember that while test scores are the only objective measure communities have by which to measure the success of their neighborhood school, it is only one measure.

The Full ARMT Tables

The Changes in 2011-2012

Stacked Subgroups by Year

One Subgroup Over Time