Contact: Monty Jones, (512) 499-4363

Date: May 13, 2003

Testimony by Charles Miller

Written Testimony

To the U.S. House Committee on Education and the Workforce

May 13, 2003 - 2:00 p.m.

Rayburn House Office Building, Room 2175

Watch the archived webcast at http://edworkforce.house.gov/hearings/hrgarchive.htm (click the link under May 13, 2003)

1. Current Higher Education Accountability Practices

Multiple stakeholders. Higher education institutions are accountable to students, parents, and taxpayers for the education provided and the costs of providing it through state appropriations, tuition, and financial aid. Ohio Governor Bob Taft said in his inaugural speech in January 2003, "We need to ensure that the financial sacrifices parents and students are making for college today are rewarding them with the results they deserve…"

A long-term issue. Accountability in higher education has been an increasingly significant national issue over the past decade, spurred by rising costs of college, disappointing retention and graduation rates, employer concerns that graduates do not have the knowledge and skills expected in the workplace, and questions about the learning and value that higher education provides to students.

Higher education is already accountable. Higher education associations, accrediting bodies, research centers, and federal and state governments all promote higher education accountability. Institutions produce voluminous accountability information, reporting data to the Department of Education's Integrated Postsecondary Education Data System, to regional, national, and specialized accreditation associations, and to state legislatures and/or higher education coordinating boards. As American Association for Higher Education President Yolanda T. Moses wrote recently "Colleges and universities already comply with numerous accountability regulations, including voluntary programmatic and institutional accrediting processes."

Current accountability information is not effectively communicated. Yolanda Moses contends, however, that "while current quality measures in place on campuses may be working they are poorly understood by a wider public, and therefore institutions should work more closely with various stakeholders to ensure that these methods become more widely understood." (1)

Current accountability systems are fragmented. The quantity of information actually serves to obscure what we really need to know. Today, higher education accountability systems are characterized by fragmentation, frustration, and lack of utility. Professor Joseph C. Burke of the Rockefeller Institute at SUNY Albany recently reported that 44 states have some combination of higher education performance reporting, budgeting, and/or funding. (2)

Current accountability systems are not highly useful. Professor Burke warns policy makers about an ongoing weakness in existing accountability systems: "The big problem… is not just … flawed format, huge size, and excessive indicators, but the failure of state, system, and campus policy makers to use them in planning and decision making." (3)

What is needed: building alignment and a culture of evidence. The challenge is to get the right information to the right people, and to align accountability systems so that institutional, state level, and national systems use the same information. We need useful information, to build a "culture of evidence," not to add to the burden of red tape that institutions of higher education currently bear.

2. Overview of Proposals to Improve Accountability Systems

Agreement on what we generally need to know. A number of research centers, nonprofit education associations, and federal and state governments have proposed various approaches to improve higher education accountability. There is some agreement about the broad areas about which we need to know more, or at least communicate better: affordability, accessibility, retention and graduation rates, student learning outcomes, and post-graduation employment.

Lack of consensus on framework or standards. Diversity of institutional mission, students, and decentralized governance are unique strengths of American higher education. (4) But, this variability presents a major issue: with no single national curriculum, and funding streams more complex than in K-12 systems, it is difficult to prescribe uniform standards for all of postsecondary education.

Structure of Accountability Systems. Paul Lingenfelter, of the State Higher Education Executive Officers group, recently made these suggestions on effective higher education accountability systems: (5)

• Focus on a few clear, significant, measurable goals
• Determine why existing practice is not achieving goals
• Monitor progress publicly
• Focus on improving performance rather than punishing failure
• Employ both intrinsic and extrinsic incentives for individuals who produce results
• Build capacity
• Involve everybody and use multiple tools
• Invest in results

Jane Wellman, writing for the National Governors Association, has suggested key elements that should be included in statewide higher education accountability strategies. (6) These elements could inform a national system, as well.

• Establish goals in relation to statewide plans
• Focus on total state support for higher education
• Ensure comparability, simplicity, and visibility
• Include institution-specific information
• Track students
• Cultivate broad support for statewide systems
• Recognize the difference between K-12 and higher education

Report cards. Responding to information now available in the Center for Public Policy and Higher Education's Measuring Up report cards, many institutions have instituted report cards. Some have suggested that these become a requirement. There is significant overlap among the proposals. For example, Professor Joseph Burke suggests that indicators should be selected that would allow "measuring up and down," useful at the state, institution, and even department level. (7)

• Funding - state appropriations per FTE
• Affordability - Tuition and fees, less financial aid, as % of median family income
• College-school collaboration - pass rates on teacher certification examines; % of freshmen with college preparatory curriculum in high school
• Participation - rate of higher education going as % of high school graduates
• Articulation - transfer rates between 2- and 4-year institutions
• Completion - graduation rates
• Degree attainment
• Job placements of graduates
• Sponsored research - dollar volume
• Student development - alumni survey on the knowledge and skills developed in college

The Career College Association has proposed a required institutional report card, including: (8)

• Institutional mission
• Student demographics
• Student/faculty ratio
• Instructional expenditure per student
• Services provided to students
• Graduation rates
• 1st, 2nd, year retention rates
• Transfer rates into or from the institution
• Post-graduation employment success
• Licensure examination pass rates
• Student and alumni satisfaction
• Employer satisfaction with graduates

These elements can be part of system- or institution-specific accountability systems, like the one The University of Texas System is launching this year. Its comprehensive accountability system will include measures like those above, within a framework that will evaluate performance around five critical mission-related areas: 1) Student Access and Success; 2) Teaching, Research, and Health Care Excellence; 3) Collaboration with and Service to Communities; 4) Organizational Efficiency and Productivity; and 5) Aggregate System Performance.

3. Accountability for Quality

Employers want consistent skills, including good verbal and written communication skills, honesty and integrity, teamwork skills, interpersonal skills, and a strong work ethic. Also included on lists of what employers seek from college graduates are:

• Ability to learn, take initiative, decision making, teamwork, motivational fit, ability to thrive in a diverse environment;
• Strategic thinking, flexibility, initiative;
• Portable skills: writing, technology literacy, qualitative analysis, scientific literacy, oral communication, critical thinking.

Learning assessment is a critical issue. It is being examined seriously by several national research groups. (9) As the National Research Council recommends in Knowing What Students Know,

"Policy makers are urged to recognize the limitations of current assessments, and to support the development of new systems of multiple assessments that would improve their ability to make decisions about education programs…" (10)

The NRC also recommends that

"Funding should be provided for a major program of research, guided by a synthesis of cognitive and measurement principles, focused on the design of assessments that yield more valid and fair inferences about student achievement." (11)

Setting a research agenda. Stanford's National Center for Postsecondary Improvement proposed a set of research questions to guide the improvement of higher education in the 21st century. (12) Improving educational quality and institutional performance is one of the three key priorities articulated in the NCPI report, Beyond Dead Reckoning. It calls for the development of a "culture of evidence" - to create quality measures, collect data on outcomes, and use the information to redesign practices to improve quality (p. 12). This study suggests a focus on such questions as:

• What are the attributes of a culture of evidence in a higher education institution? How are these qualities cultivated?
• How are external accountability measures aligned with internal quality improvement processes? What policies motivate institutions to define and apply measures of performance that are relevant to public purposes as well as to institutions?
• What policies, incentives, and resources support institutions and their faculty to develop better measures and instruments of student learning - and use them in their teaching?

Value-Added Assessment. The Rand Corporation's Council for Aid to Education is in the midst of National Value Added Assessment Initiative. This is a long-term project to develop a way to assess the quality of undergraduate education in the U.S. (13) The purpose is to create system that will show how institutions add value to their students. The Rand authors caution that "it is simply not sufficient to import from K-12 or industry the rhetoric of assessment and efficiency….An assessment system cannot be handed down to higher education from above; it must be a faculty- and institution-driven initiative.

Need for a Conceptual Framework. The current "frenzy" lacks a "coherent conceptual framework that would align assessment with the valued outcomes of higher education," according to Richard Shavelson and Leta Huang of Stanford University. (14) For these authors, testing is a risky approach because "the tests quickly become proxies for the goals we really value." They suggest several tactical principles:

• Assess personal, social, and civic abilities as well as cognitive ones.
• Encourage real dialogue and greater agreement on the content of the assessments.
• Recognize that what we test and make public will greatly influence what is taught and what is learned.
• Achieve clarity in the debate about what to assess through the use of a conceptual framework.
• Develop multiple and varied assessments.
• Distribute meaningful feedback on assessment to all stakeholders.

Role of Testing. Testing can play an important role in the assessment of student learning. More is being learned about how testing affects quality in K-12 education, as part of the No Child Left Behind movement. (15) Some of this knowledge will be useful in considering postsecondary testing. Consensus seems to be growing that postsecondary testing should focus on assessing how well colleges are teaching, rather than individual student achievement. (16) The University of Texas System is taking the initiative for a pilot assessment project that will include testing of general academic knowledge and skills in writing, math, reading, and critical thinking. Ultimately, we would do this assessment at all nine of our academic campuses.

4. Policy Questions

Is there a need for a new approach to higher education accountability? Yes, it is time for our nation to focus on the critical outcomes we expect for our students, and to hold our institutions accountable for these results.

Do we know enough now to design a national higher education accountability framework that is highly likely to have a greater impact than the systems in place now? Not yet, although many of the building blocks already exist.

Can a consistent, but flexible, vision be developed of what students should get out of college? Yes, but only if national leadership and institutional will are aligned.

Should a national accountability model be developed for higher education? The long-term goal would be to gather specific, longitudinal, and comparative data that would show which, among like institutions, are most successful in the areas under study, determine the factors of that success, and then apply the knowledge for further improvement. This model should:

• Build on existing systems and sources of data.
• Be aligned with state efforts.
• Create a conceptual framework.
• Utilize multiple kinds of measurements.
• Select meaningful indicators.
• Develop a method to benchmark higher education institutions with similar missions.
• Foster engagement of higher education community.
• Communicate widely about process and results.
• Foster use of results by rewarding success.

Notes:

1) AAHEBulletin.com, "President's Letter," March 2003, aahebulletin.com. return

2) Joseph C. Burke and Henrik Minassians, "Performance Reporting: The Preferred 'No Cost' Accountability Program, the Sixth Annual Report," The Nelson A. Rockefeller Institute of Government, 2002. rockinst.org/quick_tour/higher?ed/current_projects.html. return

3)Joseph C. Burke, personal email communication with Geri Malandra, Feb. 7, 2003. return

4) Institute for Higher Education Policy, Reauthorizing the Higher Education Act: Issues and Options, March 2003, p. 144. return

5) Paul E. Lingenfelter, "Educational Accountability: Setting Standards, Improving Performance," Change, March/April 2003, pp. 19-23. return

6) Jane V. Wellman, "Statewide Higher Education Accountability: Issues and Strategies for Success," Higher Expectations (National Governors Association, 2002), pp. 7-16. return

7) Joseph C. Burke and Henrik P. Minassians, Reporting Higher Education Results: Missing Links in the Performance Chain , New Directions in Institutional Research, No. 116 (Jossey-Bass, Winter 2002), pp. 110-111. return

8) Stephen Burd, "Will Congress Require Colleges to Grade Themselves?" The Chronicle of Higher Education, April 4, 2003. return

9) The American Association of Higher Education is leading a major national initiative on learning assessment; extensive resources are available at http://www.aahe.org/assessment/. return

10) James W. Pelligrino, Naomi Chudowksy, and Robert Glaser, eds., Knowing What Students Know: The Science and Design of Educational Assessment (National Academy Press, 2001), p. 310. return

11) Ibid., p. 299. return

12) Beyond Dead Reckoning: Research Priorities for Redirecting American Higher Education, October 2002. return

13) Roger Benjamin and Richard H. Hersh, "Measuring the Difference College Makes: the RAND/CAE Value Added Assessment Initiative," Peer Review, Vol. 4, No. 2/3, Winter/Spring 2002, pp. 11-15. return

14) Richard J. Shavelson and Leta Huang, "Responding Responsibly to the Frenzy to Assess Learning in Higher Education, Change, January/February 2003, pp. 11-19. return

15) See Jonathan Crane, "The Promise of Value-Added Testing," Policy Report, November 2002 (Progressive Policy Institute, http://www.ppionline.org. return

16) Kate Zernike, "Tests Are Not Just for Kids," New York Times, August 5, 2002. return

Watch the archived webcast at http://edworkforce.house.gov/hearings/hrgarchive.htm (click the link under May 13, 2003)

Read the related news advisory