Douglas C. Bennett, President, Earlham College

Questions about the U.S. News and World Report rankings of colleges and universities and about our relationship to those rankings are a sideshow to the serious issues that should steadily concern us about higher education today. The most serious issues are access and quality and the relationship between those two: how do we provide access to post-secondary education for all Americans, and how do we assure that this education is of high quality? The rankings are worth our attention only insofar as they bear on these questions, and they do bear on these questions in many ways, distracting us from the real, hard work needed to improve access and distorting understandings of quality. Let us all keep access and quality in our minds as we discuss the rankings.

A good place to begin is with how the rankings and our active participation in them appear to those on our campuses who perform institutional research for our colleges. We all have very competent, very professional institutional researchers, but I’m not sure we’ve listened as attentively as we should to what they have to say on the matter. Most of us are members of HEDS, the Higher Education Data Sharing Project, and the HEDS listserv is one of the most important sites for conversations among institutional researchers. (I’m a participant on that listserv.) Periodically they discuss the U. S. News and World Report rankings, and they have nothing but anger at and professional scorn for these rankings.

Their postings detail in extraordinary detail problems with the U. S. News rankings that undermine nearly every aspect of its methodology: They question:

Rather than detail their methodological concerns (because that would take hours), let me just quote some of their summary judgments. I won’t quote anyone by name since I didn’t ask them for permission, but by and large these quotations are from institutional researchers whose Presidents are members of the Annapolis Group, some of whom are sitting in this room.

• "What made U.S. News notable and damaging was the conceit that it could come up with a single ranking scheme that identifies the best college. That is a demonstrably false, and also damaging, idea."

• "My objection has to do with measurement validity: do the rankings measure what they purport to measure – what matters in higher education."

• "I think that the U.S. News rankings are a good candidate for the IgNobel prizes, for research that cannot or should not be repeated."

• "Forgive me, but I’m a child of the 1960s, and this ranking stuff seems to cry out for a revolution. It seems none of us are satisfied with the lame science this process represents."

• "My goal before I leave this seat is to show with data that nobody really cares in the big picture about the rankings for more than five minutes after they are released. Then talk our President into being the next Reed. Wish me luck."

If you ask your institutional research person to tell you candidly whether the rankings have any professional integrity about them, they will tell you no. I doubt you can find even one institutional researcher across all of our colleges and universities who believes, as a professional judgment, that the rankings satisfy a minimum threshold of acceptable research practice.

If you ask your institutional research person to tell you candidly whether your college should actively participate in the rankings, almost all of them would tell you no. They hate the rankings—not because their college doesn’t rank well but because the rankings enterprise is beneath a level of minimum professional integrity.

And yet, U.S. News and World Report continues to claim that they are producing the best judgments about college quality available today. They claim there is professional support for what they do. How can they? In the main, because we Presidents and we Deans fill out the annual reputation survey that asks us to rate the ‘quality’ of 220 baccalaureate institutions. Our participation is a telling token of our professional support.

Look at this in another light: what has academic research of higher education had to say about the U.S. News and World Report rankings, or about the general approach to assessing ‘quality’ that the magazine takes, which these researchers refer to as a "resources/reputation" approach since it focuses on the resources a college has at its disposal and the reputation it enjoys.

Since the U.S. News and World Report rankings began, there have been a number of studies addressing whether the rankings have any validity as a measure of educational quality. Each and every one of the academic studies shows not even the palest correlation between various measures of educational quality or student learning on the one hand, and the U.S. News and World Report rankings on the other.

• Several researchers, for example have demonstrated that despite all the components that go into the U.S. News and World Report rankings, you can closely replicate the rankings simply by using one variable: the average SAT scores of incoming students.

• Several studies have shown that there is no correlation between an institution’s financial resources and student learning. (Financial resources measured in a variety of ways are a major component of in the rankings.)

• A 2004 study by George Kuh and Ernest Pascarella showed that there is no correlation between an institution’s admissions selectivity and student learning. (Admissions selectivity is a component in the rankings.)

• A 1997 evaluation of the U.S. News rankings by NORC, the National Opinion Research Center, concluded that "the principal weakness of the current approach is that the weights used to combine the various measures into an overall rating lack any defensible empirical or theoretical basis." This study was commissioned by U.S. News itself and released by the Washington Monthly (NORC, 1997). It also detailed several other serious methodological problems with the rankings.

• .A 2003 study by Gary Pike of 15 AAU institutions looking at correlations of U.S. News and World Report rankings and National Survey of Student Engagement benchmarks concludes "first and foremost, this study indicates that the quality of a student’s education is not synonymous with the resources and reputation of an institution. In fact, educational quality seems to have little to do with resources and reputation" (Pike, 2003).

• A 2001 article by Ernest Pascarella on "Identifying Excellence in Undergraduate Education" reviews a number of other studies and concludes that "the resources/reputation approach, which uses proxy variables of questionable validity, and the alumni outcomes approach, which may simply reflect institutional recruitment practices, are fundamentally flawed as methodologies for identifying institutional excellence" (Pascarella, 2001). It is noteworthy that Pascarella also concludes that more promising approaches to assessing quality are indeed at hand. | I do not believe it is possible to find a single academic paper by a reputable higher education scholar who has anything good to say about the U.S. News and World Report rankings as a measure of educational quality. Not one.

I doubt I am telling you anything new, but we are not yet acting as if we have fully digested these judgments of institutional researchers and higher education scholars. What we have to talk about is what we do about the failures of U.S. News and World Report rankings to meet minimum standards of acceptable research.

Let me add a few more voices to the mix. Recently, 25 of Canada’s leading higher education institutions decided to opt out of participation in annual rankings produced by Maclean’s, a Canadian equivalent of U.S. News and World Report. We should listen to them, too.

A year ago, David Naylor, the President of the University of Toronto wrote "As academics, we devote our careers to ensuring people make important decisions on the basis of good data, analyzed with discipline. But Canadian universities have been complicit, en masse, in supporting a ranking system that has little scientific merit because it reduces everything to a meaningless, average score" (Naylor, 2006).

Last month, Indira Samarasekera, the President of the University of Alberta, wrote that "Canadian universities are listening with great interest as the call to boycott U.S. News and World Report rankings continues to increase in volume among our colleagues to the south," and noted that "it is time to question these third party rankings that are actually market driven, designed to sell particular issues of a publication" (Samarasekera, 2007) These Canadian university Presidents have said no to further active participation in the creation of rankings. And they’ve said no on the basis of affirming their professional responsibilities as educators and scholars.

This isn’t about what U.S. News and World Report does. This is about what we do. In the United States (and increasingly around the world) the media rank everything: golf courses, retirement communities, beaches, blue jeans, beer. I don’t doubt that, whatever we do, U.S. News and World Report will go on ranking colleges and universities. U.S. News is in the magazine business, and rankings of all kinds sell magazines. We’re in the business of education and research, and our behavior should comport with recognized professional obligations of educators and researchers.

That is, the question is whether we, as education and research professionals, will actively cooperate with U.S. News and World Report (and other publications that rank colleges and universities), thus lending our professional weight and credibility to the exercise.

The process of admissions to college should itself be an educational process in the best and widest sense. Our behavior in this process, as professionals, should be accountable to the professional norms of educators and scholars: at a minimum, integrity and transparency in the use of data, conscientious attention to issues of validity and reliability, refusal to simplify if that simplification distorts in important ways.

These are the values of the Education Conservancy, an organization created recently and committed to improving college admission processes for students, colleges and high schools. A dozen of us, now grown to about three dozen, have signed a letter making two simple commitments:

• Not to fill out the reputational survey.

• Not to trumpet or advertise our resultant place in the rankings. The full text of the letter is on the Education Conservancy website here. We ask you to join us in making these commitments.

The media have described this initiative as a boycott, but that’s not really accurate. In addition to asking us to participate in the reputational survey, U.S. News and World Report asks each college to answer 600 questions. Of those, 424 are data elements that are in IPEDS or the Common Data Set. All of us already make all of these data elements publicly available. Many of us make available to the public high quality data that answer many of the remaining 176 questions. We are providing a great deal of high quality, useful data. U.S. News is free to make use of these data. So is Princeton Review, or Washington Monthly, or the U.S. Secretary of Education, or (most importantly) prospective students and their parents, teachers and counselors.

The problem we need to recognize is that we (colleges and universities) are not making these data available in an easy-to-find, user-friendly manner. As we solve that problem, U.S. News and World Report will not be the best answer. Broadly speaking, there is a further commitment we should make.|

We should feel an obligation to make public a variety of kinds of high quality data about the characteristics and functioning of our institutions. We should commit ourselves to making these easily available: easy to find and easy to use.

Several higher education organizations are already developing templates for making such data available: NASULGC and AASCU among the public universities, NAICU among the independent colleges and universities, and others, too. We should cooperate with these efforts, but we should work together as liberal arts colleges to find a format that is especially well-suited to our missions. We will likely want to include data elements that other kinds of colleges and universities will not.

Let me close by briefly (too briefly) enumerating some of the principles that our approach should follow.

1. The data we include should be collected and reported according to common, clear professional standards that assure the integrity, reliability and validity of what is reported.

2. Because we are independent institutions (that is, not-for-profit and not governmentcontrolled), there should not be a single, mandatory template or provider for this information. Of course accrediting agencies may want to mandate some data elements as a condition of accreditation, and federal or state governments may want to mandate some as a condition of students’ receiving financial aid from federal or state sources.

3. Among the information we include should be professional information about student learning. (Note that the U.S. News and World Report rankings include no data whatsoever on student learning.) Because of the variety of missions among colleges and universities, we should not insist on any one measure of student learning. Colleges and universities will (and should) use different measures, and probably each should use multiple measures, but information on student learning should be included. In the National Survey of Student Engagement (NSSE) and the Collegiate Learning Assessment (CLA) we have two quite different, excellent approaches to assessing student learning. We need more.

4. The information should not be displayed as an ordinal ranking. We may, on the other hand, want to find a way for prospective students (or others) to focus in a comparative way on the kinds of information they think most relevant. These are principles of professional responsibility. They also bear on the primary issues of access and quality that I mentioned at the beginning.

To refuse to produce rankings serves the issue of access. We need every person in this country to pursue post-secondary education. The more we focus on trying to identify the "few, best" institutions, the more we miscommunicate about college admissions. Institutions have different missions. Different students will thrive at different colleges.

The idea of "one, best" college is nonsense, and that’s the notion that rankings promote. And to insist on including information about student learning serves the issue of quality. As education professionals, we need to focus prospective students on finding an institution where they are likely to learn. We need to provide them information that helps them make that choice, not focus their attention on who is most prestigious.

Several years ago, Reed College took a courageous, solitary stand on the U.S. News rankings (For its reasons, see Diver, 2005). Since then, many of us have ceased filling out the reputational survey. Now it is time for all of us to cease providing active assistance to the U.S. News rankings, recognizing its weaknesses, and instead focus our energies on provision of relevant information about our colleges and universities in an easy-to-find, easy-to-use manner.