What the economists found when they turned back to the rankings is that New York City parents choose schools that enroll a lot of high-achievers. While schools like that tend to be effective, they aren’t always—and just how effective they are varies, writes GAIL CORNWALL on the Atlantic. Read on:
A person trying to choose their next set of wheels might see that car A made it farther than car B in a road test and assume it gets better gas mileage. But that’s only true if the two tanks are filled with the same substance. Putting high-octane gas in one and water in the other, for example, provides little useful information about which car makes the most of its fuel. A new working paper titled “Do Parents Value School Effectiveness?” suggests that parents similarly opt for schools with the most impressive graduates rather than figuring out which ones actually teach best. The study joins a body of research looking critically at what it means for a school to be successful.
Take the work of Erin Pahlke, for example. The assistant professor of psychology at Whitman College saw research showing that girls who attend school only with other girls tend to do better in math and science. The trick, she said, is that those studies didn’t analyze “differences in the students coming into the schools.” As it turns out, those who end up in same-sex schools tend to be wealthier, start out with more skills, and have parents who are more proactive than students who attend co-ed institutions. In a 2014 meta-analysis, Pahlke and her colleagues reviewed the studies and found when examining schools with the same type of students and same level of resources—rather than “comparing [those at] the public co-ed school to [their counterparts at] the fancy private school that’s single-sex down the road”—there isn’t any difference in how the students perform academically. Single-sex schooling also hasn’t been shown to offer a bump in girls’ attitudes toward math and science or change how they think about themselves. In other words, it often looks like single-sex schools are doing a better job educating kids, but they aren't. It's just that their graduates are people who were going to do well at any school. They’re running on high-octane gas.
So too are high schools widely thought to be “life-changing”—the elite ones that students must test into. In a 2014 Econometrica paper titled “The Elite Illusion,” the economists Atila Abdulkadiroğlu, Joshua Angrist, and Parag Pathak wrote that while students who attend extremely competitive public schools like Stuyvesant High School in New York City clearly excel, that may not mean the schools provide an education that’s superior to their less competitive counterparts. The researchers looked at a group of borderline kids, the last few eighth-graders who made the cut-off to go to an elite school and the first few who didn't; that meant there was little if any academic difference between them when they started their freshman year. If a school like Stuyvesant were more effective—that is, taught more material and produced better outcomes—than the less competitive public school, the economists would expect to see a difference in how those kids performed academically four years later. But when the researchers analyzed indicators of success, such as AP exam scores and state standardized tests, they saw no difference between the borderline kids who got to attend Stuyvesant and the borderline ones who didn’t. And yet, said Pathak, a professor of microeconomics at MIT, “these are massively oversubscribed schools. People would give an arm and a leg to send their child to a school like Stuyvesant.”
That raises the question: Are parents able to figure out which schools are doing the best job? The new working paper—published by the National Bureau of Economic Research (NBER) and authored by Abdulkadiroğlu, Pathak, Jonathan Schellenberg, and Christopher Walters—discusses data from the New York City Department of Education, which enrolls around 90,000 ninth-graders every year at more than 400 high schools. For better or worse, the city’s high-school system doesn’t rely on an automatic process where students head to the school down the block; instead, eighth-graders submit a ranked list of up to 12 high schools they’d like to attend, some of which, like Stuyvesant, require a test. That system equipped the researchers with quite a bit of information about which schools parents and their children are choosing. Their task was to use other data about the students and schools to figure out what drives those choices.
Do parents prefer a school nearby? Yes, they reliably gave higher rankings to schools located in their own borough than they did to those elsewhere. But what about choosing between two nearby schools? Here, things get trickier. The economists had a pretty good measure of the type of gas in each tank: the eighth-grade test scores of schools’ incoming students. They also wanted to figure out fuel efficiency—how well each school helps kids advance academically regardless of where they start out their freshman year. In order to do so, they looked for similar students—ones who shared the same gender and race, lived in the same neighborhood, and got the same eighth-grade test scores—who went to different high schools. The researchers identified many of these “matched pairs” and looked at follow-up data similar to that used for the Stuyvesant study: scores on state tests, PSAT scores, high-school graduation records, and college-enrollment information. Then they asked if the kids who went to school A did better at these things than did their essentially identical counterparts at school B; if so, they labeled school A more “effective” than school B.
What the economists found when they turned back to the rankings is that New York City parents choose schools that enroll a lot of high-achievers. While schools like that tend to be effective, they aren’t always—and just how effective they are varies. Yet parents didn’t seem to pick up on that: When school A and school B had the same type of student body but school A was much more effective, parents didn’t apply to it more often or rank it more highly.
The free-market rationale for school-choice programs ... “depends on [having sufficient] information,” and this study suggests parents don’t.
If these results hold up under scrutiny by other economists (the paper will soon be peer-reviewed), they could prove problematic for school-choice advocates such as U.S. Education Secretary Betsy DeVos. To see why, it’s worth recalling a 1955 paper in which Nobel Prize-winning economist Milton Friedman proposed applying the principles of the free market to education. When it comes to goods like, say, cereal, buyers decide how valuable that good is, which in turn determines its fate. If I eat Crispy Grain O’s and like it, I buy it again. If I eat it and hate it, or my neighbor hates it and tells me, it doesn’t get bought, and that cereal company goes belly-up. In economics jargon, “demand-side pressure” manages the cereal market, ensuring that the tasty ones buyers demand continue to grace the aisles while those that taste like cardboard disappear. Friedman said a similar system would produce better schools. Let parents choose, the theory goes, and schools that are doing a good job of educating kids will thrive, while the bad ones will have to close their doors for lack of interest.
But the free-market rationale for school-choice programs, Pathak noted, “depends on [having sufficient] information,” and this study suggests parents don’t. What’s more, Pathak and his colleagues worry that, absent sufficient information for parents, choice-based education systems “penalize schools that enroll low-achievers rather than schools that offer poor instruction” and give school leaders a perverse incentive to focus on “making sure your school’s got the best kids” rather than improving school quality. That, Pathak said, undermines “one of the narratives for why choice-based reform may lead to [more effective] schooling.”
“We have to go where the data leads us.”
Ultimately, research on the merits of school-choice programs is extremely mixed, and not everyone who reads the latest NBER paper shares its authors’ concerns. Paul Peterson, who directs the Program on Education Policy and Governance at Harvard University, said the data Pathak and his colleagues report show that having high-achieving students is actually a good proxy for school quality. Most of the time, these two things are happening in tandem, he said. What about when the researchers identify school A and school B that have comparable students but a different level of effectiveness? Peterson’s not so sure they can actually tease those two things—student body composition and school quality—apart. “I’m not saying they did the wrong thing,” Peterson said, “I’m just saying we don’t know…. This paper has yet to be peer reviewed, so we can't be sure.” As for the argument that administrators are given the incentive to invest in marketing and screening rather than school quality, he said: “They just made that up. There’s nothing in the study that compels that conclusion.”
Pathak defended the study’s methodology and conclusions. “I certainly can understand why such a finding might be uncomfortable for school-choice advocates,” he said, “but we have to go where the data leads us.” He acknowledged that the analysis is limited in its heavy reliance on test scores as an indicator of student and school success and in its inability to measure every factor that goes into the parental decision-making process. For example, he and his co-authors didn’t track safety metrics, nor did they gauge how happy schools make their students.
That said, none of these objections calls into question a central implication of the study: School-choice systems would benefit from parents having more information. “If you were in New York City trying to figure out what high school to send your kid to, where would you figure this stuff out?” Pathak asked. Parents might rely on playground chatter, advice from middle-school guidance counselors, schools’ brochures, accountability reports put together by administrators, “school finder” apps, or websites like GreatSchools.org—but none of these sources seems to be leading parents to put weight on school effectiveness. Should data such as that contained in the new NBER study be distributed to families with schoolchildren?
Jack Schneider, an assistant professor of education at College of the Holy Cross and the author of Beyond Test Scores: A Better Way to Measure School Quality, said doing so would still promote “a culture in the United States of talking about school quality … as if there are only a few good schools, and that’s largely because there can only be a few winners in terms of who comes out on top in the battle of the test scores.” What’s needed, he said, is for parents to understand that test scores track so closely with a person’s background—their parents’ education level and income—that they say little about the quality of a school. “And if we were more honest about that,” Schneider said, “I think that would do a great deal to get parents searching for better information”—information on things like the relationships between teachers and students, how students interact with each other, and “the degree to which students are engaged and happy to be there.”
A person trying to choose their next set of wheels might see that car A made it farther than car B in a road test and assume it gets better gas mileage. But that’s only true if the two tanks are filled with the same substance. Putting high-octane gas in one and water in the other, for example, provides little useful information about which car makes the most of its fuel. A new working paper titled “Do Parents Value School Effectiveness?” suggests that parents similarly opt for schools with the most impressive graduates rather than figuring out which ones actually teach best. The study joins a body of research looking critically at what it means for a school to be successful.
Take the work of Erin Pahlke, for example. The assistant professor of psychology at Whitman College saw research showing that girls who attend school only with other girls tend to do better in math and science. The trick, she said, is that those studies didn’t analyze “differences in the students coming into the schools.” As it turns out, those who end up in same-sex schools tend to be wealthier, start out with more skills, and have parents who are more proactive than students who attend co-ed institutions. In a 2014 meta-analysis, Pahlke and her colleagues reviewed the studies and found when examining schools with the same type of students and same level of resources—rather than “comparing [those at] the public co-ed school to [their counterparts at] the fancy private school that’s single-sex down the road”—there isn’t any difference in how the students perform academically. Single-sex schooling also hasn’t been shown to offer a bump in girls’ attitudes toward math and science or change how they think about themselves. In other words, it often looks like single-sex schools are doing a better job educating kids, but they aren't. It's just that their graduates are people who were going to do well at any school. They’re running on high-octane gas.
So too are high schools widely thought to be “life-changing”—the elite ones that students must test into. In a 2014 Econometrica paper titled “The Elite Illusion,” the economists Atila Abdulkadiroğlu, Joshua Angrist, and Parag Pathak wrote that while students who attend extremely competitive public schools like Stuyvesant High School in New York City clearly excel, that may not mean the schools provide an education that’s superior to their less competitive counterparts. The researchers looked at a group of borderline kids, the last few eighth-graders who made the cut-off to go to an elite school and the first few who didn't; that meant there was little if any academic difference between them when they started their freshman year. If a school like Stuyvesant were more effective—that is, taught more material and produced better outcomes—than the less competitive public school, the economists would expect to see a difference in how those kids performed academically four years later. But when the researchers analyzed indicators of success, such as AP exam scores and state standardized tests, they saw no difference between the borderline kids who got to attend Stuyvesant and the borderline ones who didn’t. And yet, said Pathak, a professor of microeconomics at MIT, “these are massively oversubscribed schools. People would give an arm and a leg to send their child to a school like Stuyvesant.”
That raises the question: Are parents able to figure out which schools are doing the best job? The new working paper—published by the National Bureau of Economic Research (NBER) and authored by Abdulkadiroğlu, Pathak, Jonathan Schellenberg, and Christopher Walters—discusses data from the New York City Department of Education, which enrolls around 90,000 ninth-graders every year at more than 400 high schools. For better or worse, the city’s high-school system doesn’t rely on an automatic process where students head to the school down the block; instead, eighth-graders submit a ranked list of up to 12 high schools they’d like to attend, some of which, like Stuyvesant, require a test. That system equipped the researchers with quite a bit of information about which schools parents and their children are choosing. Their task was to use other data about the students and schools to figure out what drives those choices.
Do parents prefer a school nearby? Yes, they reliably gave higher rankings to schools located in their own borough than they did to those elsewhere. But what about choosing between two nearby schools? Here, things get trickier. The economists had a pretty good measure of the type of gas in each tank: the eighth-grade test scores of schools’ incoming students. They also wanted to figure out fuel efficiency—how well each school helps kids advance academically regardless of where they start out their freshman year. In order to do so, they looked for similar students—ones who shared the same gender and race, lived in the same neighborhood, and got the same eighth-grade test scores—who went to different high schools. The researchers identified many of these “matched pairs” and looked at follow-up data similar to that used for the Stuyvesant study: scores on state tests, PSAT scores, high-school graduation records, and college-enrollment information. Then they asked if the kids who went to school A did better at these things than did their essentially identical counterparts at school B; if so, they labeled school A more “effective” than school B.
What the economists found when they turned back to the rankings is that New York City parents choose schools that enroll a lot of high-achievers. While schools like that tend to be effective, they aren’t always—and just how effective they are varies. Yet parents didn’t seem to pick up on that: When school A and school B had the same type of student body but school A was much more effective, parents didn’t apply to it more often or rank it more highly.
The free-market rationale for school-choice programs ... “depends on [having sufficient] information,” and this study suggests parents don’t.
If these results hold up under scrutiny by other economists (the paper will soon be peer-reviewed), they could prove problematic for school-choice advocates such as U.S. Education Secretary Betsy DeVos. To see why, it’s worth recalling a 1955 paper in which Nobel Prize-winning economist Milton Friedman proposed applying the principles of the free market to education. When it comes to goods like, say, cereal, buyers decide how valuable that good is, which in turn determines its fate. If I eat Crispy Grain O’s and like it, I buy it again. If I eat it and hate it, or my neighbor hates it and tells me, it doesn’t get bought, and that cereal company goes belly-up. In economics jargon, “demand-side pressure” manages the cereal market, ensuring that the tasty ones buyers demand continue to grace the aisles while those that taste like cardboard disappear. Friedman said a similar system would produce better schools. Let parents choose, the theory goes, and schools that are doing a good job of educating kids will thrive, while the bad ones will have to close their doors for lack of interest.
But the free-market rationale for school-choice programs, Pathak noted, “depends on [having sufficient] information,” and this study suggests parents don’t. What’s more, Pathak and his colleagues worry that, absent sufficient information for parents, choice-based education systems “penalize schools that enroll low-achievers rather than schools that offer poor instruction” and give school leaders a perverse incentive to focus on “making sure your school’s got the best kids” rather than improving school quality. That, Pathak said, undermines “one of the narratives for why choice-based reform may lead to [more effective] schooling.”
“We have to go where the data leads us.”
Ultimately, research on the merits of school-choice programs is extremely mixed, and not everyone who reads the latest NBER paper shares its authors’ concerns. Paul Peterson, who directs the Program on Education Policy and Governance at Harvard University, said the data Pathak and his colleagues report show that having high-achieving students is actually a good proxy for school quality. Most of the time, these two things are happening in tandem, he said. What about when the researchers identify school A and school B that have comparable students but a different level of effectiveness? Peterson’s not so sure they can actually tease those two things—student body composition and school quality—apart. “I’m not saying they did the wrong thing,” Peterson said, “I’m just saying we don’t know…. This paper has yet to be peer reviewed, so we can't be sure.” As for the argument that administrators are given the incentive to invest in marketing and screening rather than school quality, he said: “They just made that up. There’s nothing in the study that compels that conclusion.”
Pathak defended the study’s methodology and conclusions. “I certainly can understand why such a finding might be uncomfortable for school-choice advocates,” he said, “but we have to go where the data leads us.” He acknowledged that the analysis is limited in its heavy reliance on test scores as an indicator of student and school success and in its inability to measure every factor that goes into the parental decision-making process. For example, he and his co-authors didn’t track safety metrics, nor did they gauge how happy schools make their students.
That said, none of these objections calls into question a central implication of the study: School-choice systems would benefit from parents having more information. “If you were in New York City trying to figure out what high school to send your kid to, where would you figure this stuff out?” Pathak asked. Parents might rely on playground chatter, advice from middle-school guidance counselors, schools’ brochures, accountability reports put together by administrators, “school finder” apps, or websites like GreatSchools.org—but none of these sources seems to be leading parents to put weight on school effectiveness. Should data such as that contained in the new NBER study be distributed to families with schoolchildren?
Jack Schneider, an assistant professor of education at College of the Holy Cross and the author of Beyond Test Scores: A Better Way to Measure School Quality, said doing so would still promote “a culture in the United States of talking about school quality … as if there are only a few good schools, and that’s largely because there can only be a few winners in terms of who comes out on top in the battle of the test scores.” What’s needed, he said, is for parents to understand that test scores track so closely with a person’s background—their parents’ education level and income—that they say little about the quality of a school. “And if we were more honest about that,” Schneider said, “I think that would do a great deal to get parents searching for better information”—information on things like the relationships between teachers and students, how students interact with each other, and “the degree to which students are engaged and happy to be there.”
No comments:
Post a Comment