Gerald Carlson’s heart sank when he received word several years ago that a controversial statistical analysis had declared his program one of Louisiana’s weakest in preparing educators to teach English language arts.
“We thought we had a good program,” said Carlson, dean of the education school at the University of Louisiana’s Lafayette campus. “We were shocked when we saw the results.” Carlson has since worked with his staff to revamp the school’s curriculum: adding a new class in reading and English language arts, requiring more writing across the board, and beefing up professional development for the university’s professors.
He is confident that when the latest results come out this spring — the first to be released in over a year and a half — the university will fare well.
Ratings of teacher-training programs, said the University of Washington’s Charles Peck, are “kind of like having a fire alarm go off in your house, but not knowing where the fire is.”
Scores of teacher-training programs across the country could face similar scrutiny in coming years. Following Louisiana’s lead, policy makers in a growing number of states are evaluating programs based on the test scores of the students their graduates teach. So far, eight states have policies requiring them to do a similar analysis, most of them adopted in the last few years, according to the National Council on Teacher Quality.
“This is a policy movement that’s sweeping the country,” said Charles Peck, a professor of special education and teacher education at the University of Washington’s College of Education.
Related efforts to evaluate individual teachers based on student test scores have sparked a flurry of publicity — and led to a major lawsuit. But efforts to evaluate teacher-preparation programs, including well-established university-based education schools and less traditional programs like Teach For America, have gone comparatively unnoticed and un-scrutinized.
As other states climb aboard the teacher-accountability bandwagon, Louisiana’s experience speaks to the promise and peril of the new approach. Some programs, like Carlson’s, have used the data to make significant changes. “We should be doing this analysis anyway, whether we’re forced to or not,” Carlson said.
But the data only capture institutions with a large enough cohort of graduates (25 in a given subject area) for the results to be statistically meaningful. As a result, not a single New Orleans-based teacher-training program — including those at the University of New Orleans, Xavier University, Southern University at New Orleans, and Our Lady of Holy Cross — is consistently included in the study. In recent years, only UNO has had a large enough cohort, but just in its undergraduate social studies and math-certification programs.
The New Orleans institutions still go through traditional accreditation processes, in which a team of experts visits the campus and gauges program quality by interviewing students, staff and course content. But notwithstanding Louisiana’s growing reputation as a state that holds its education schools accountable, the quality of many local institutions remains comparatively unknown.
Others worry that even when the cohort size is large enough the data can be overly simplistic and at times misleading. For instance, do low reading scores recorded years after a group of teachers enters the classroom mean their training program had a bad curriculum or weak instructors? Or did the program admit weaker candidates from the start or perhaps send them off to schools with less supportive principals?
“It’s kind of like having a fire alarm go off in your house, but not knowing where the fire is,” Peck said.
Digging deeper into the data
The concern voiced by Peck and others prompted state officials to break down the data for each training program in a more detailed way, said Jeanne Burns, the Louisiana Board of Regents’ associate commissioner for teacher and leadership initiatives. Over time, training programs could see how they fared with low-income students, for instance, or on which specific skills their graduates fell short.
“Our goal has always been to use the data to help our campuses improve,” Burns said.
“I assumed it would not be as long a journey from seeing results to figuring out what to do with them, but I consider the fact that we are even talking about it huge progress.” — George Noell, the Louisiana State University researcher who designed the evaluation system
Starting in 2003, the Louisiana studies tracked student performance in grades four through nine, tying it back to the teacher-preparation programs. By using what’s known as a “value-added” analysis, researchers homed in on the amount of growth seen in individual students, no matter their starting point. They then compared the overall student growth in the classrooms of recent graduates of different training programs to the growth achieved by veteran educators. No new results have been issued in over a year because state officials have been working to change the system to align it more closely to the state’s comparatively new evaluation process for individual teachers.
Louisiana officials requested the analysis more than a decade ago partly in the hope that it would weed out the state’s weakest teacher-training programs. If programs consistently rate low, the state can shut them down. But so far, no program has been forced to close as a direct result of the new system.
“I don’t view it so much as an accountability mechanism because none have been shut down,” said Tim Daly, president of the national organization TNTP (formerly The New Teacher Project), which trains many of New Orleans’ Teach For America instructors, and whose organization has fared fairly well in the studies. But he says the analysis has improved transparency, including for prospective teachers shopping for a training program.
It has also prompted self-examination and improvements at a few programs, including not just the University of Louisiana at Lafayette but also the Louisiana Resource Center for Educators, a private teacher-certification program based in Baton Rouge.
The initial round of results showed that the Resource Center was “doing a lousy job of teaching reading,” said Nancy Roberts, the executive director. But the initial data did not provide any information about which of the center’s graduates posted the weakest results, which aspects of reading their students had failed to master (phonemic awareness? overall comprehension? something else?), or which grade levels struggled most.
“It was like shooting darts in the dark,” Roberts said. “But we decided to roll up our sleeves.”
After looking at the more detailed data, the Resource Center began to tailor its literacy instruction more precisely, based on the age group the graduate would be teaching. It also increased the amount of overall class time spent on reading and literacy, from less than 20 percent to about a third.
George Noell, the Louisiana State University researcher who designed the evaluation system, said he is pleased that some of the lowest-performing programs are making improvements, although “across the whole spectrum, it’s not as clear the data have helped people as much as I hoped.”
“I assumed it would not be as long a journey from seeing results to figuring out what to do with them,” he said. “But I consider the fact that we are even talking about it huge progress.”
The selectivity factor
Some experts say that determining how much of a young teacher’s success or failure can be tied to his or her training program is like asking which came first, the chicken or the egg.
If a program’s graduates post weak results, it could be that the program is “doing a fabulous job but they are not selective enough in terms of who is admitted,” said Kate Walsh, president of the National Council on Teacher Quality. She pointed out that many of the programs that have scored best in Tennessee and Louisiana evaluations also have highly competitive admissions.
“One of the dangers of single rankings is that we let school districts off the hook for providing career-long support and development.” — Brian Beabout, assistant professor at the University of New Orleans
Daly disagreed that the selectivity of a program significantly affects its graduates’ performance in the classroom. “While selectivity may play a small role, what we do to train teachers plays a big role,” he said.
Katrina Miller, the director of Tennessee’s federal First to the Top grant, said she considers selectivity something training programs can control. “You are choosing who to admit,” she said. “Selectivity, since it’s chosen by the program, is part of the training.”
Some programs are far choosier than others, however. And it’s easier for Teach For America — to which more than 10 percent of all Ivy League seniors apply — to be more selective than your average state school.
That said, in addition to revamping part of its curriculum after that initial evaluation, the teacher program at the University of Louisiana at Lafayette has also managed to raise its admission standards modestly, Carlson said. The university now requires a 2.5 cumulative GPA (which includes all of a candidate’s past grades) instead of a 2.5 adjusted GPA (which only counts the higher of two grades if a student had to repeat a course). “That’s eliminated a lot of people,” Carlson said.
Brian Beabout, a former New Orleans public school teacher who now works as an assistant professor at the University of New Orleans, said he worries that a teacher’s success in his first two years on the job says more about the quality of his K-12 education, which spans 13 years, than the quality of his preparation program, which can be as short as six weeks. And as time passes, teachers sink or swim largely based on how much support they get on the job, he said.
“One of the dangers of single rankings is that we let school districts off the hook for providing career-long support and development,” he said.
Missing the small programs
Perhaps the greatest limitation of Louisiana’s data-centered analysis is that it fails to encompass most of the state’s small teacher-training schools, a mixture of private and public university programs. All told, at least seven of the state’s programs did not have large enough enrollments for any results to be reported in 2010-2011. Fourteen programs had at least some results released.
In New Orleans, many of the programs were already small before the dual blows of Katrina and state budget cuts further reduced their size. In 2000, 172 students completed the University of New Orleans’ teacher certification programs, compared to 104 in 2009. The drop-off was even more dramatic at Southern University at New Orleans: 87 graduates in 2000 compared to 12 in 2009. UNO’s young graduates have performed fairly well in social studies and even better in math, but those were the only two areas where the cohort size was large enough to be captured in the study.
Burns said even New Orleans’ smaller programs, like SUNO, eventually will be captured in the value-added analysis. That’s because the state plans to wrap together multiple years of results from different institutions to achieve meaningful sample sizes for statistical evaluation. Meanwhile, the smaller institutions have been held accountable through national accreditation processes and a requirement that obliged all teacher-training programs to undergo re-approval over the past 10 years.
Few advocate including programs so small that their value-added results aren’t statistically meaningful. But some experts say Louisiana should develop a more qualitative component of its evaluation that can capture small programs — and provide a more multi-dimensional look at larger ones.
Larger institutions can be left with a false sense of superiority if they post high value-added scores in states where the overall quality of teacher-preparation programs is weak, Walsh said.
“We need multiple measures to see if a program is effective,” she said. “Our view is that performance by these institutions is not good across the board and even the highest performing institution is not something you want to hold up as a model.”
This story was produced by The Hechinger Report, a nonprofit, nonpartisan education-news outlet based at Teachers College, Columbia University.