Q&A: Colleges measure up
We spoke with Mike Cogan, associate vice president for records and institutional effectiveness, University of St. Thomas, about outcome measurement efforts in higher education. It’s a perennial theme in the higher ed media, and even general publications, brought on in part by the publication last year of “Academically Adrift.” And the question of just how to gauge the results of what happens within higher education institutions, at for-profits as well as more traditional nonprofit and public colleges, continues to garner the attention of policymakers.
Q: Is there one indicator to use to know if a college is doing a good job?
“The key word in your question is ‘indicator.’ Many people are looking for the ‘one’ measure or ranking that will tell the whole story. Of course, this is impossible. Like anything in life it is never just one thing. The National Center for Education Statistics and U.S. News may provide good information, but their data does not adequately explain what is really happening at a college or university. A more holistic approach using multiple indicators is necessary. At St. Thomas, we have addressed outcomes using a variety of indicators to include student engagement, where students are going after graduation and graduation outcomes to name a few. By doing this, we are able to accurately determine whether we are doing a good job or not.”
Q: What do families of prospective students want to know about these days?
Parents and students have become much more consumer-oriented in the past 10 years when selecting a college. Families have many questions relating to cost, degree attainment, availability of majors, job placement, safety, values-based learning, etc. The same questions have been asked by government agencies, legislators, employers and so on. When higher education institutions have addressed these questions, the results assist them in making informed decisions. For example, students considering St. Thomas may want to know the likelihood of getting into medical school once they finish their bachelor’s degree. To answer this question, we worked with an agency that tracks medical school test scores and admission rates to identify how St. Thomas students compare nationally. This information not only answered students’ questions, but also provided important insight for faculty and administrators as they plan for the future.
Q: Do colleges use data internally, to improve what they do?
Absolutely. In fact, this is the primary reason for the Office of Institutional Effectiveness at St. Thomas. We measure the productivity of the institution — are we delivering the outcome we promised people we would deliver? The only way to do this is through integrated planning which requires that quantitative and qualitative data be used to inform decision-making. So if we say students are going to be exposed to an engaged campus at St. Thomas, then we need the tools to understand whether we are accomplishing this goal or not. Every four years our institution participates in the National Survey of Student Engagement which provides us with comparative and longitudinal information. This helps us identify strengths and opportunities for improvement. This is just one of many examples where we measure an outcome to assure our limited resources are being employed in an efficient and meaningful manner.
Q: How is the demand for outcomes information from colleges changing?
I’m a firm believer that the increased national and local focus on higher education is not going away. As such, we must continue to explain what we do and how we do it. The only way this will work is to continuously pay attention to where we have been, where we currently are, and where we plan to be in the future. What I hear most is that people are looking for a one-stop source for information. I’d say we’re not there yet and I am not sure we will ever get there. It’s difficult to agree on definitions and acceptable outcomes in your own institution. Spread this over 4,000 institutions and the task seems daunting, if not impossible. Rather, I would argue that successful institutions will identify the outcomes that are most important to the families, employers and community they serve — and work to assure this information is accurate, available and transparent. This will result in a better fit between student and the institution. Outcomes should improve based on this approach.
Q: Does outcome measurement work differently for graduate and undergraduate education?
The measures people see in the media — such as standardized test scores, cost of attendance, admissions rates and graduation outcomes — are undergraduate in nature. The same compliance requirements do not apply to graduate level education. Seven years ago, St. Thomas made a concerted effort to provide the same or similar measures for all of the graduate programs at the institution. Although this seems straightforward, the lack of federally defined graduate level measures required us to create our own definitions and outcomes. This is helpful to our institution but one could imagine the wide variation among the 1,000 plus institutions offering graduate level work in the US. Graduate outcomes are very important but they will not have the same effectiveness as undergraduate outcomes until there is consistency among the various measures.”
Q: Why does outcome measurement matter?
“The recent recession has forever changed the landscape of higher education. With the continued rise of tuition costs and student loan debt surpassing $1 trillion — the industry will never be the same. Colleges and universities recognizing this shift will develop and use outcomes to make the informed decisions necessary to thrive in this environment. Additionally, I believe we as a society will no longer accept the status quo. As an institutional effectiveness professional, I see this as an opportunity for us to increase the quality and effectiveness of the teaching and learning at our institutions — which will ultimately lead to a more engaged and productive society. As we prepare for this new reality, I am reminded of the Thomas Fuller quote, “Govern thy life and thoughts as if the whole world were to see the one, and read the other.”