Last summer, the editors of Car and Driver conducted a comparison test of three sports cars, the Lotus Evora, the Chevrolet Corvette Grand Sport, and the Porsche Cayman S. The cars were taken on an extended run through mountain passes in Southern California, and from there to a race track north of Los Angeles, for precise measurements of performance and handling. The results of the road tests were then tabulated according to a twenty-one-variable, two-hundred-and-thirty-five-point rating system, based on four categories: vehicle (driver comfort, styling, fit and finish, etc.); power train (transmission, engine, and fuel economy); chassis (steering, brakes, ride, and handling); and “fun to drive.” The magazine concluded, “The range of these three cars’ driving personalities is as various as the pajama sizes of Papa Bear, Mama Bear, and Baby Bear, but a clear winner emerged nonetheless.” This was the final tally:
Car and Driver is one of the most influential editorial voices in the automotive world. When it says that it likes one car better than another, consumers and carmakers take notice. Yet when you inspect the magazine’s tabulations it is hard to figure out why Car and Driver was so sure that the Cayman is better than the Corvette and the Evora. The trouble starts with the fact that the ranking methodology Car and Driver used was essentially the same one it uses for all the vehicles it tests—from S.U.V.s to economy sedans. It’s not set up for sports cars. Exterior styling, for example, counts for four per cent of the total score. Has anyone buying a sports car ever placed so little value on how it looks? Similarly, the categories of “fun to drive” and “chassis”—which cover the subjective experience of driving the car—count for only eighty-five points out of the total of two hundred and thirty-five. That may make sense for S.U.V. buyers. But, for people interested in Porsches and Corvettes and Lotuses, the subjective experience of driving is surely what matters most. In other words, in trying to come up with a ranking that is heterogeneous—a methodology that is broad enough to cover all vehicles—Car and Driver ended up with a system that is absurdly ill-suited to some vehicles. Suppose that Car and Driver decided to tailor its grading system just to sports cars. Clearly, styling and the driving experience ought to count for much more. So let’s make exterior styling worth twenty-five per cent, the driving experience worth fifty per cent, and the balance of the criteria worth twenty-five per cent. The final tally now looks like this:
So which is the best car?
Car and Driver’s ambition to grade every car in the world according to the same methodology would be fine if it limited itself to a single dimension. A heterogeneous ranking system works if it focusses just on, say, how much fun a car is to drive, or how good-looking it is, or how beautifully it handles. The magazine’s ambition to create a comprehensive ranking system—one that considered cars along twenty-one variables, each weighted according to a secret sauce cooked up by the editors—would also be fine, as long as the cars being compared were truly similar. It’s only when one car is thirteen thousand dollars more than another that juggling twenty-one variables starts to break down, because you’re faced with the impossible task of deciding how much a difference of that degree ought to matter. A ranking can be heterogeneous, in other words, as long as it doesn’t try to be too comprehensive. And it can be comprehensive as long as it doesn’t try to measure things that are heterogeneous. But it’s an act of real audacity when a ranking system tries to be comprehensive and heterogeneous—which is the first thing to keep in mind in any consideration of U.S. News & World Report’s annual “Best Colleges” guide.
The U.S. News rankings are run by Robert Morse, whose six-person team operates out of a small red brick office building in the Georgetown neighborhood of Washington, D.C. Morse is a middle-aged man with gray hair who looks like the prototypical Beltway wonk: rumpled, self-effacing, mildly preppy and sensibly shoed. His office is piled high with the statistical detritus of more than two decades of data collection. When he took on his current job, in the mid-nineteen-eighties, the college guide was little more than an item of service journalism tucked away inside U.S. News magazine. Now the weekly print magazine is defunct, but the rankings have taken on a life of their own. In the month that the 2011 rankings came out, the U.S. News Web site recorded more than ten million visitors. U.S. News has added rankings of graduate programs, law schools, business schools, medical schools, and hospitals—and Morse has become the dean of a burgeoning international rankings industry.
“In the early years, the thing that’s happening now would not have been imaginable,” Morse says. “This idea of using the rankings as a benchmark, college presidents setting a goal of ‘We’re going to rise in the U.S. News ranking,’ as proof of their management, or as proof that they’re a better school, that they’re a good president. That wasn’t on anybody’s radar. It was just for consumers.”Over the years, Morse’s methodology has steadily evolved. In its current form, it relies on seven weighted variables:
This ranking system looks a great deal like the Car and Driver methodology. It is heterogeneous. It doesn’t just compare U.C. Irvine, the University of Washington, the University of Texas-Austin, the University of Wisconsin-Madison, Penn State, and the University of Illinois, Urbana-Champaign—all public institutions of roughly the same size. It aims to compare Penn State—a very large, public, land-grant university with a low tuition and an economically diverse student body, set in a rural valley in central Pennsylvania and famous for its football team—with Yeshiva University, a small, expensive, private Jewish university whose undergraduate program is set on two campuses in Manhattan (one in midtown, for the women, and one far uptown, for the men) and is definitely not famous for its football team.
The system is also comprehensive. It doesn’t simply compare schools along one dimension—the test scores of incoming freshmen, say, or academic reputation. An algorithm takes a slate of statistics on each college and transforms them into a single score: it tells us that Penn State is a better school than Yeshiva by one point. It is easy to see why the U.S. News rankings are so popular. A single score allows us to judge between entities (like Yeshiva and Penn State) that otherwise would be impossible to compare. At no point, however, do the college guides acknowledge the extraordinary difficulty of the task they have set themselves. A comprehensive, heterogeneous ranking system was a stretch for Car and Driver—and all it did was rank inanimate objects operated by a single person. The Penn State campus at University Park is a complex institution with dozens of schools and departments, four thousand faculty members, and forty-five thousand students. How on earth does anyone propose to assign a number to something like that?The first difficulty with rankings is that it can be surprisingly hard to measure the variable you want to rank—even in cases where that variable seems perfectly objective. Consider an extreme example: suicide. Here is a ranking of suicides per hundred thousand people, by country:
This list looks straightforward. Yet no self-respecting epidemiologist would look at it and conclude that Belarus has the worst suicide rate in the world, and that Hungary belongs in the top ten. Measuring suicide is just too tricky. It requires someone to make a surmise about the intentions of the deceased at the time of death. In some cases, that’s easy. Maybe the victim jumped off the Golden Gate Bridge, or left a note. In most cases, though, there’s ambiguity, and different coroners and different cultures vary widely in the way they choose to interpret that ambiguity. In certain places, cause of death is determined by the police, who some believe are more likely to call an ambiguous suicide an accident. In other places, the decision is made by a physician, who may be less likely to do so. In some cultures, suicide is considered so shameful that coroners shy away from that determination, even when it’s obvious. A suicide might be called a suicide, a homicide, an accident, or left undetermined. David Phillips, a sociologist at the University of California-San Diego, has argued persuasively that a significant percentage of single-car crashes are probably suicides, and criminologists suggest that a good percentage of civilians killed by police officers are actually cases of “suicide by cop”—instances where someone deliberately provoked deadly force. The reported suicide rate, then, is almost certainly less than the actual suicide rate. But no one knows whether the relationship between those two numbers is the same in every country. And no one knows whether the proxies that we use to estimate the real suicide rate are any good.
“Many, many people who commit suicide by poison have something else wrong with them—let’s say the person has cancer—and the death of this person might be listed as primarily associated with cancer, rather than with deliberate poisoning,” Phillips says. “Any suicides in that category would be undetectable. Or it is frequently noted that Orthodox Jews have a low recorded suicide rate, as do Catholics. Well, it could be because they have this very solid community and proscriptions against suicide, or because they are unusually embarrassed by suicide and more willing to hide it. The simple answer is nobody knows whether suicide rankings are real.”
The U.S. News rankings suffer from a serious case of the suicide problem. There’s no direct way to measure the quality of an institution—how well a college manages to inform, inspire, and challenge its students. So the U.S. News algorithm relies instead on proxies for quality—and the proxies for educational quality turn out to be flimsy at best.Take the category of “faculty resources,” which counts for twenty per cent of an institution’s score. “Research shows that the more satisfied students are about their contact with professors,” the College Guide’s explanation of the category begins, “the more they will learn and the more likely it is they will graduate.” That’s true. According to educational researchers, arguably the most important variable in a successful college education is a vague but crucial concept called student “engagement”—that is, the extent to which students immerse themselves in the intellectual and social life of their college—and a major component of engagement is the quality of a student’s contacts with faculty. As with suicide, the disagreement isn’t about what we want to measure. So what proxies does U.S. News use to measure this elusive dimension of engagement? The explanation goes on:
The reputation score that serves as the most important variable in the U.S. News methodology—accounting for 22.5 per cent of a college’s final score—isn’t any better. Every year, the magazine sends a survey to the country’s university and college presidents, provosts, and admissions deans (along with a sampling of high-school guidance counsellors) asking them to grade all the schools in their category on a scale of one to five. Those at national universities, for example, are asked to rank all two hundred and sixty-one other national universities—and Morse says that the typical respondent grades about half of the schools in his or her category. But it’s far from clear how any one individual could have insight into that many institutions. In an article published recently in the Annals of Internal Medicine, Ashwini Sehgal analyzed U.S. News’s “Best Hospitals” rankings, which also rely heavily on reputation ratings generated by professional peers. Sehgal put together a list of objective criteria of performance—such as a hospital’s mortality rates for various surgical procedures, patient-safety rates, nursing-staffing levels, and key technologies. Then he checked to see how well those measures of performance matched each hospital’s reputation rating. The answer, he discovered, was that they didn’t. Having good outcomes doesn’t translate into being admired by other doctors. Why, after all, should a gastroenterologist at the Ochsner Medical Center, in New Orleans, have any specific insight into the performance of the gastroenterology department at Mass General, in Boston, or even, for that matter, have anything more than an anecdotal impression of the gastroenterology department down the road at some hospital in Baton Rouge?
VIDEO FROM THE NEW YORKER
Poaching an Egg in Red Wine, the French Way
Some years ago, similarly, a former chief justice of the Michigan supreme court, Thomas Brennan, sent a questionnaire to a hundred or so of his fellow-lawyers, asking them to rank a list of ten law schools in order of quality. “They included a good sample of the big names. Harvard. Yale. University of Michigan. And some lesser-known schools. John Marshall. Thomas Cooley,” Brennan wrote. “As I recall, they ranked Penn State’s law school right about in the middle of the pack. Maybe fifth among the ten schools listed. Of course, Penn State doesn’t have a law school.”
Those lawyers put Penn State in the middle of the pack, even though every fact they thought they knew about Penn State’s law school was an illusion, because in their minds Penn State is a middle-of-the-pack brand. (Penn State does have a law school today, by the way.) Sound judgments of educational quality have to be based on specific, hard-to-observe features. But reputational ratings are simply inferences from broad, readily observable features of an institution’s identity, such as its history, its prominence in the media, or the elegance of its architecture. They are prejudices.
And where do these kinds of reputational prejudices come from? According to Michael Bastedo, an educational sociologist at the University of Michigan who has published widely on the U.S. News methodology, “rankings drive reputation.” In other words, when U.S. News asks a university president to perform the impossible task of assessing the relative merits of dozens of institutions he knows nothing about, he relies on the only source of detailed information at his disposal that assesses the relative merits of dozens of institutions he knows nothing about: U.S. News. A school like Penn State, then, can do little to improve its position. To go higher than forty-seventh, it needs a better reputation score, and to get a better reputation score it needs to be higher than forty-seventh. The U.S. News ratings are a self-fulfilling prophecy.
Bastedo, incidentally, says that reputation ratings can sometimes work very well. It makes sense, for example, to ask professors within a field to rate others in their field: they read one another’s work, attend the same conferences, and hire one another’s graduate students, so they have real knowledge on which to base an opinion. Reputation scores can work for one-dimensional rankings, created by people with specialized knowledge. For instance, the Wall Street Journal has ranked colleges according to the opinions of corporate recruiters. Those opinions are more than a proxy. To the extent that people chose one college over another to enhance their prospects in the corporate job markets, the reputation rankings of corporate recruiters are of direct relevance. The No. 1 school in the Wall Street Journal’s corporate recruiter’s ranking, by the way, is Penn State.
For several years, Jeffrey Stake, a professor at the Indiana University law school, has run a Web site called the Ranking Game. It contains a spreadsheet loaded with statistics on every law school in the country, and allows users to pick their own criteria, assign their own weights, and construct any ranking system they want.
Stake’s intention is to demonstrate just how subjective rankings are, to show how determinations of “quality” turn on relatively arbitrary judgments about how much different variables should be weighted. For example, his site makes it easy to mimic the U.S. News rankings. All you have to do is give equal weight to “academic reputation,” “LSAT scores at the 75th percentile,” “student-faculty ratio,” and “faculty law-review publishing,” and you get a list of élite schools which looks similar to the U.S News law-school rankings:
There’s something missing from that list of variables, of course: it doesn’t include price. That is one of the most distinctive features of the U.S. News methodology. Both its college rankings and its law-school rankings reward schools for devoting lots of financial resources to educating their students, but not for being affordable. Why? Morse admitted that there was no formal reason for that position. It was just a feeling. “We’re not saying that we’re measuring educational outcomes,” he explained. “We’re not saying we’re social scientists, or we’re subjecting our rankings to some peer-review process. We’re just saying we’ve made this judgment. We’re saying we’ve interviewed a lot of experts, we’ve developed these academic indicators, and we think these measures measure quality schools.”
As answers go, that’s up there with the parental “Because I said so.” But Morse is simply being honest. If we don’t understand what the right proxies for college quality are, let alone how to represent those proxies in a comprehensive, heterogeneous grading system, then our rankings are inherently arbitrary. All Morse was saying was that, on the question of price, he comes down on the Car and Driver side of things, not on the Consumer Reports side. U.S. News thinks that schools that spend a lot of money on their students are nicer than those that don’t, and that this niceness ought to be factored into the equation of desirability. Plenty of Americans agree: the campus of Vanderbilt University or Williams College is filled with students whose families are largely indifferent to the price their school charges but keenly interested in the flower beds and the spacious suites and the architecturally distinguished lecture halls those high prices make possible. Of course, given that the rising cost of college has become a significant social problem in the United States in recent years, you can make a strong case that a school ought to be rewarded for being affordable. So suppose we go back to Stake’s ranking game, and re-rank law schools based on student-faculty ratio, L.S.A.T. scores at the seventy-fifth percentile, faculty publishing, and price, all weighted equally. The list now looks like this:
Welcome to the big time, Alabama!
The U.S. News rankings turn out to be full of these kinds of implicit ideological choices. One common statistic used to evaluate colleges, for example, is called “graduation rate performance,” which compares a school’s actual graduation rate with its predicted graduation rate given the socioeconomic status and the test scores of its incoming freshman class. It is a measure of the school’s efficacy: it quantifies the impact of a school’s culture and teachers and institutional support mechanisms. Tulane, given the qualifications of the students that it admits, ought to have a graduation rate of eighty-seven per cent; its actual 2009 graduation rate was seventy-three per cent. That shortfall suggests that something is amiss at Tulane.
Another common statistic for measuring college quality is “student selectivity.” This reflects variables such as how many of a college’s freshmen were in the top ten per cent of their high-school class, how high their S.A.T. scores were, and what percentage of applicants a college admits. Selectivity quantifies how accomplished students are when they first arrive on campus.
Each of these statistics matters, but for very different reasons. As a society, we probably care more about efficacy: America’s future depends on colleges that make sure the students they admit leave with an education and a degree. If you are a bright high-school senior and you’re thinking about your own future, though, you may well care more about selectivity, because that relates to the prestige of your degree.
But no institution can excel at both. The national university that ranks No. 1 in selectivity is Yale. A crucial part of what it considers its educational function is to assemble the most gifted group of freshmen it can. Because it maximizes selectivity, though, Yale will never do well on an efficacy scale. Its freshmen are so accomplished that they have a predicted graduation rate of ninety-six per cent: the highest Yale’s efficacy score could be is plus four. (It’s actually plus two.) Of the top fifty national universities in the “Best Colleges” ranking, the least selective school is Penn State. Penn State sees its educational function as serving a wide range of students. That gives it the opportunity to excel at efficacy—and it does so brilliantly. Penn State’s freshmen have an expected graduation rate of seventy-three per cent and an actual graduation rate of eighty-five per cent, for a score of plus twelve: no other school in the U.S. News top fifty comes close.
There is no right answer to how much weight a ranking system should give to these two competing values. It’s a matter of which educational model you value more—and here, once again, U.S. News makes its position clear. It gives twice as much weight to selectivity as it does to efficacy. It favors the Yale model over the Penn State model, which means that the Yales of the world will always succeed at the U.S. News rankings because the U.S. News system is designed to reward Yale-ness. By contrast, to the extent that Penn State succeeds at doing a better job of being Penn State—of attracting a diverse group of students and educating them capably—it will only do worse. Rankings are not benign. They enshrine very particular ideologies, and, at a time when American higher education is facing a crisis of accessibility and affordability, we have adopted a de-facto standard of college quality that is uninterested in both of those factors. And why? Because a group of magazine analysts in an office building in Washington, D.C., decided twenty years ago to value selectivity over efficacy, to use proxies that scarcely relate to what they’re meant to be proxies for, and to pretend that they can compare a large, diverse, low-cost land-grant university in rural Pennsylvania with a small, expensive, private Jewish university on two campuses in Manhattan.
“If you look at the top twenty schools every year, forever, they are all wealthy private universities,” Graham Spanier, the president of Penn State, told me. “Do you mean that even the most prestigious public universities in the United States, and you can take your pick of what you think they are—Berkeley, U.C.L.A., University of Michigan, University of Wisconsin, Illinois, Penn State, U.N.C.—do you mean to say that not one of those is in the top tier of institutions? It doesn’t really make sense, until you drill down into the rankings, and what do you find? What I find more than anything else is a measure of wealth: institutional wealth, how big is your endowment, what percentage of alumni are donating each year, what are your faculty salaries, how much are you spending per student. Penn State may very well be the most popular university in America—we get a hundred and fifteen thousand applications a year for admission. We serve a lot of people. Nearly a third of them are the first people in their entire family network to come to college. We have seventy-six per cent of our students receiving financial aid. There is no possibility that we could do anything here at this university to get ourselves into the top ten or twenty or thirty—except if some donor gave us billions of dollars.”In the fall of 1913, the prominent American geographer Ellsworth Huntington sent a letter to two hundred and thirteen scholars from twenty-seven countries. “May I ask your cooperation in the preparation of a map showing the distribution of the higher elements of civilization throughout the world?” Huntington began, and he continued:
Each contributor was given a list of a hundred and eighty-five of the world’s regions—ranging from the Amur district of Siberia to the Kalahari Desert—with instructions to give each region a score of one to ten. The scores would then be summed and converted to a scale of one to a hundred. The rules were strict. The past could not be considered: Greece could not be given credit for its ancient glories. “If two races inhabit a given region,” Huntington specified further, “both must be considered, and the rank of the region must depend upon the average of the two.” The reputation of immigrants could be used toward the score of their country of origin, but only those of the first generation. And size and commercial significance should be held constant: the Scots should not suffer relative to, say, the English, just because they were less populous. Huntington’s respondents took on the task with the utmost seriousness. “One appreciates what a big world this is and how little one knows about it when he attempts such a task as you have set,” a respondent wrote back to Huntington. “It is a most excellent means of taking the conceit out of one.” England and Wales and the North Atlantic states of America scored a perfect hundred, with central and northwestern Germany and New England coming in at ninety-nine.
Huntington then requested from the twenty-five of his correspondents who were Americans an in-depth ranking of the constituent regions of the United States. This time, he proposed a six-point scale. Southern Alaska, in this second reckoning, was last, at 1.5, followed by Arizona and New Mexico, at 1.6. The winners: Massachusetts, at 6.0, followed by Connecticut, Rhode Island, and New York, at 5.8. The citadel of American civilization was New England and New York, Huntington concluded, in his magisterial 1915 work “Civilization and Climate.” In case you are wondering, Ellsworth Huntington was a professor of geography at Yale, in New Haven, Connecticut. “Civilization and Climate” was published by Yale University Press, and the book’s appendix contains a list of Huntington’s American correspondents, of which the following bear special mention:
“In spite of several attempts I was unable to obtain any contributor in the states west of Minnesota or south of the Ohio River,” Huntington explains, as if it were a side issue. It isn’t, of course—not then and not now. Who comes out on top, in any ranking system, is really about who is doing the ranking. ♦
No comments:
Post a Comment