Conceptual Framework

During the Fall 2000 semester, the Academic Affairs Committee of the Board of Regents requested that the University of Texas System implement a plan to assess student knowledge and skills developed in general education programs and other academic programs across the System. The Chancellor and the Executive Vice Chancellor for Academic Affairs have emphasized the importance of including the assessment of student learning within the overall UT System accountability framework. Therefore, this model will be proposed as a System-wide activity for the academic institutions of the System.

This document will be shared widely to gather input and share plans with all stakeholders. This document includes:

I. The Purpose and Assumptions

At the System level, the purpose of learning assessment is to promote quality, comparability, and information that support policy development. Also, embedded in this purpose is the fulfilling a public duty to report the effectiveness of our programs to critical stakeholders. For example, the Texas Higher Education Coordinating Board (THECB) requires that general education programs be evaluated. The Southern Association of Colleges and Schools (SACS) requires assessment of institutional effectiveness, in which student outcomes must be assessed. And the Council on Higher Education Accreditation (CHEA), a collaborative organization of the regional accrediting agencies, has initiated projects that begin with an assumption that student outcomes should be related to resources and infrastructures throughout universities, not academic programs alone.

At the institution level, the purpose of assessment is to give faculty and administrators information that they can use to improve student learning. Faculty have always assessed individual students in their courses, but until recently few institutions attempted to assess what students learned as a result of their academic programs. Academic assessment asks the question, "How do we know whether students have learned what we attempted to teach them after they have taken our courses?" Effective academic assessments can determine whether academic programs are accomplishing what they intend to accomplish; and, if not, the assessments help make appropriate curricular or pedagogical adjustments so that students' academic success can be increased.

Thus, faculty must specify the learning outcomes for their programs. These learning outcomes are defined as the specific knowledge, skills, abilities and attitudes that students should have acquired after having taken the curriculum that has been designed for them.

For learning assessment to work well, we are proposing a set of principles for institutions to follow. These principles have been adopted from a list provided by the American Association of Higher Education (AAHE).

The assessment of student learning begins with educational values. Assessment is not an end in itself but a vehicle for educational improvement. It practice begins with and enacts a vision of the kind of learning we most value for students. Thus educational values should drive not only what we chose to assess but also how we do so.
Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time. Learning is a complex process. It entails not only what students know but also what they can do with what they know. It involves knowledge, abilities, attitudes, and habits of mind that affect academic success and performance beyond the classroom. Thus, assessment should reflect these understandings by employing a diverse array of methods, including those that call for actual performance, using them overtime so as to reveal change, growth, and increasing degrees of integration.
Assessment works best when the programs it seeks to improve have clear, explicitly stated purposes. Assessment is a goal-oriented process. It entails comparing educational performance with educational purposes and expectations-those derived from the institution's mission, from faculty intentions in program and course design, and from knowledge of students' own goals. Thus, assessment is a process that pushes a campus toward clarity about where to aim and what standards to apply. Clear, shared, implementable goals are the cornerstone for assessment that is focused and useful.
Assessment requires attention to outcomes but also to the experiences that lead to those outcomes. Information about outcomes is of high importance. But to improve outcomes, we need to know about student experience along the way-about curricula, teaching, and the kind of student effort that led to particular outcomes. Assessment can help us understand which students learn best under what conditions; such knowledge help us improve the whole of their learning.
Assessment works best when it is ongoing not episodic. Assessment is a process whose power is cumulative. Improvement is best fostered when it entails a linked series of activities over time. This means tracking the process of individual students or cohorts of students; it may mean collecting samples of student work or using the same instrument year after year.
Assessment fosters wider improvement when representatives from across the educational community are involved. Student learning is a campus wide responsibility, and assessment is a way of enacting that responsibility. Faculty play an especially important role, but assessment's questions cannot be fully addressed without participation by librarians, administrators, and students. Assessment may also involve individuals beyond the campus (alumi/ae, trustees, employers) whose experiences can enrich the sense of appropriate aims and standards of learning.
Assessment makes a difference when it begins with issues of use and illuminates questions that people really care about. Assessment must produce evidence relevant parties find credible, suggestive, and applicable to decisions they need to make. It means thinking in advance about how the information will be used, and by whom. The point of assessment is not to gather data and return "results"; it is a process that starts with questions of decision makers, that involves them in gathering and interpreting of data, and that informs and helps guide continuous improvement.
Through assessment, educators meet responsibilities to students and to the public. There is a compelling public stake in education. We have a responsibility to the publics that support or depend on us to provide information about the ways in which our students meet goals and expectations. But that responsibility goes beyond the reporting of such information; our deeper obligation-to ourselves, our students, and society-is to improve. Those to whom educators are accountable have a corresponding obligation to support such attempts at improvement.

II. Definition of Performance-based Assessment

With the advent of information technology, access to learning opportunities is greater now than ever. And postsecondary organizations are not the only ones providing such learning opportunities. In fact, other organizations have made significant inroads by providing performance-based learning opportunities. It is now possible for sophisticated consumers to obtain skills through different modes of instruction and different times for delivery. Therefore, university leaders have begun to develop programs that can articulate the knowledge, skills, and abilities students are expected to learn and the competencies required for the application of learned curriculum.

Performance-based initiatives are important to communicate to students which competencies are important for them to attain and the extent to which their learning experiences are meeting those expectations. These initiatives are also important to communicate employers or the general public what students know and are able to do. In the next section, we define some of the critical concepts related to performance-based assessment. These concepts have been defined elsewhere by other task forces working on performance-based initiatives.

Key Concepts in Performance-based Assessment:

The following definitions of key concepts have been taken from "Report of the National Postsecondary Education Cooperative Working Group."
Traits and characteristics are the foundation for learning, the innate make-up of individuals on which further experiences can be built.
Skills, abilities, and knowledge are developed through learning experiences, broadly defined to include formally organized postsecondary education learning processes.
Competencies are the result of integrative learning experiences in which skills, abilities, and knowledge interact to form bundles that have currency in relation to the task for which they are assembled.
Demonstrations are the results of applying competencies. It is at this level that performance can be assessed.

In higher education, we typically talk about knowledge, skills, abilities, and competencies as being one and the same. For example, we speak of competent mathematicians and knowledgeable mathematicians. Yet, skills and knowledge are acquired through learning experiences; the different combinations of skills and knowledge one has acquired in a given program define the competencies an individual possesses. These competencies are acquired through integrative learning experiences provided by academic programs. Finally, different competencies are combined to perform or carry out a task. To put it simply, competencies are complementary phenomena that combine skills, abilities, and knowledge.

Performance-based assessment insures that students attain specific knowledge, skills, and abilities important in whatever field they are studying. Using competencies requires the understanding of three institutions:

A description of the competency;
A means of assessing the competency; and
A standard by which someone is judged to be competent.

Typically, curriculum panels of faculty define competencies. The assessment of competencies is accomplished through different methods, including standardized tests, evaluations of student work or portfolios; the standards for judging competence is often set by a master panel of faculty. This process leads to standardizing student outcomes. This process also leads to clarifying the specific knowledge, skills, and abilities students are expected to achieve; the process also helps develop concrete curricular changes, competencies, and performance measures for students.

III. Definition of the General Academic Program

The initial charge from the Board of Regents indicated that we should begin with improving the quality of our undergraduate experience. If this is the initial purpose, then we should focus on the so called "general program or the core curriculum." Thus, institutional representatives should define the competencies to be accomplished in this core curriculum. The Texas Higher Education Coordinating Board has defined the core curriculum for all state colleges and universities (1998, Chapter 5). Thus, we will use such a framework to begin defining the general academic program. This core curriculum includes five areas:

Communication (composition, speech, modern language)
The objective of a communication component of a core curriculum is to enable the student to communicate effectively in clear and correct prose in a style appropriate to the subject, occasion, and audience.

Mathematics
The objective of the mathematics component of the core curriculum is to develop a quantitatively literate college graduate. Every college graduate should be able to apply basic mathematical tools in the solution of real-world problems.

Natural Sciences
The objective of the study of a natural sciences component of a core curriculum is to enable the student to understand, construct, and evaluate relationships in the natural sciences, and to enable the student to understand the bases for building and testing theories.

Humanities and Visual and Performing Arts
The objective of the humanities and visual and performing arts in a core curriculum is to expand students' knowledge of the human condition and human cultures, especially in relation to behaviors, ideas, and values expressed in works of human imagination and thought.

Social and Behavioral Sciences
The objective of a social and behavioral science component of a core curriculum is to increase students' knowledge of how social and behavioral scientists discover, describe, and explain the behaviors and interactions among individuals, groups, institutions, events and ideas. Such knowledge will better equip students to understand themselves and the roles they play in addressing the issues facing humanity.

IV. Design and Method

In order for the UT System to promote quality and inform policy development, the design most appropriate is a longitudinal design with multiple observations. The unit of analysis should be set at the individual level. Thus, institutions must define the academic program, its goals, students who comprise that program, and the outcomes or competencies to be assessed. In our case, the general program is the first two years of student experiences with the core curriculum.

The data collected should be similar across institutions and collected every year. That is, the method, definitions, and metrics should be similar. This will allow for analysis across institutions. Moreover, the institution should collect data from every member of the student population or a random sample of students in the academic program. If a random sample is used, the institution should collect data on some stratified basis to allow for representation of subgroups in the population.

A data set should be maintained and updated every year both at the institutional level. This will allow institutional representatives to provide instant analysis for accrediting organizations, for System accountability purposes, and for program improvement.

Analytical Approach
Value-added assessment is a tool for gauging how much students gain in academic achievement in a given program, i.e., how much "value" has been added to the students by their general program. By aggregating student gains to the institution level, value-added assessment can be used to evaluate academic programs regardless of differences among entering students. The major assumption in this approach is the comparison of students' current achievement to their own past performance and aggregating learning gains at the institution level. For instance, one can use the students' entering ACT or SAT scores on writing, mathematics, and critical thinking skills as the first data point and a test of college academic skills administered at the end of the sophomore year as the second data point. Once we have two data points on the same student, a learning gain can be computed for such a student. The statistical tool is known as Henderson's mixed model, which is an advanced form of analysis of variance.

V. Consultation and Communication

Given the new focus of the assessment program, we need to create a new group (or reappoint the current members) of institutional leaders overseeing the assessment of student learning in each campus. This group should be given a new charge to initiate faculty discussion on developing a set of competencies for students to master in the general program. Representation shall include faculty, staff, and students from component institutions.

Collect and share information about UT System student learning assessment work on a web page as we move forward in this endeavor. Link this page to other sources that will serve as benchmarks.

Develop a process to communicate with policy makers and other stakeholders to gather input and broader support within the community.