The following material arises from the University of Oregon's productivity planning process and is made available here in the hope of engaging the whole university community in that process. This material contains the conclusions of one of the nine productivity working groups; it does NOT represent official university policy.

We solicit your responses and suggestions. Please direct them to the chair of the working group or to Charles Wright, Department of Mathematics. 

(This is part 1 of the report. Part 2 contains the appendices.)

November 19, 1993

MEMORANDUM
TO: Myles Brand
FR: Assessment and Productivies Measurement Group*
RE: Progress Report

Membership: David Conley, DEPM (winter) Pam Daener , Academic Affairs Jane DeGidio, Dean of Students David Hubin, ALS Joe Stone, Arts and Sciences Steadman Upham, Graduate School Norman Wessells, Provost Charles Wright, Mathematics Patricia Gwartney-Gibbs, Sociology, Chair

The Assessment and Productivity Measurement Group's (APMG's) task is "to examine ways that assessment measures or productivity measures can be used at UO," taking into account plans for increased productivity of faculty, students and the UO generally, OSSHE assessment measures, existing UO assessment measures, various levels of measurement and units of analysis (from individual students and faculty to aggregated departments, schools/colleges, and the whole institution), the recommendations of the other Productivity Groups, incentive mechanisms for increasing productivity, and several other suggestions and parameters. It is a huge task.

This memo represents a midstream report on our progress. To begin, we lay out some of the background which frames our thinking. APMG believes that:
 

1) UO, like other major research universities, is a multifaceted institution, serving many audiences (students, parents, alumni, business and industry, state and local governments, accreditation bodies, etc.) in many ways (education, cultural enrichment, inventions, expert consultation and planning, etc.). To boil down the multiple goals and processes of UO into a handful of summary statistics would be a disservice to our audiences and to ourselves.
2) Assessment and productivity studies at UO must be consistent with the University's mission (Appendix A), autonomy, and local culture.
3) Assessment and productivity measurement are separate, although related, concepts.**
4) The immediate goals of the current effort are mainly external and political (e.g., to justify differential fees), but UO can use the effort for internal management, decision- making and evaluation of long-term goals as well (e.g., increasing the quality of undergraduate education, increasing the number of undergraduates, positioning UO nationally);
5) The measures APMG suggests may be used to set goals, to assess how we are doing and how we could do better, and to oblige our multiple audiences.
6) Each measure proposed should be clearly related to a decision or a goal.
7) Significant assessment and productivity measurement activities already exist at UO, but perhaps could be better coordinated and more systematically arrayed.
8) Data collection and reporting should be unobtrusive, inexpensive, and not defeat our purposes by creating new bureaucracy. New and planned technology on campus may assist these goals.
9) Much of the value of undergraduate education goes beyond disciplinary learning, but requires long-term measurement of difficult-to-define concepts (e.g., studies of alumni values, ethics, and civic participation at 10-, 20-, and 30- year intervals).
10) The industrial, input-output metaphor of higher education often implied by critics and evaluators is inappropriate to UO's mission and history (i.e., mere production is not productivity). At the same time, several of UO's multiple audiences are impatient with the grand, precious, ivory tower special pleading that academics frequently use to deflect calls for accountability. While APMG is not settled on an alternative metaphor (UO as a city?), we agree on a way of re-presenting UO.

The best periodic assessment and productivity report, APMG believes, would take the form of a portfolio. The "UO Portfolio" would be issued annually and would contain multiple measures of UO achievements, including tables and graphs, vignettes of faculty members in teaching and research, photo essays of individual students and living groups, video- and audio-tapes of dance, drama and musical performances, sketches of labs, institutes and departments, samples of UO consultation in business, industry, and government, etc. The portfolio should be appropriate to UO's multiple audiences, containing 20-30 items in varying levels of complexity. Some items could be considered "social indicators," which are charted every year (e.g., number of students, number of graduates, pie charts of funding sources), while other items would be reported on a semiannual basis and still others would be unique each year.

The UO Portfolio must reflect the objectives of higher education generally and UO's accomplishments of those objectives specifically. Higher education's objectives might be condensed as the creation of new knowledge and the application of knowledge in teaching and in service to the society. APMG felt that some indicators of all three objectives must be included. The first may be described by measures of CREATIVE PRODUCTIVITY AND SCHOLARLY REPUTATION. The second involves assessing UNDERGRADUATE and GRADUATE EDUCATION. The third goal would be evaluated by measures of CENTRALITY to the university and OSSHE and by SERVICE to institution and the community. To provide a context for the research, teaching and service indicators, measures can be developed for ADMINISTRATIVE AND SUPPORT SERVICES, and FUNDING AND RESOURCES. These seven sub-groups are discussed separately in the paragraphs below. Appendix B contains examples of possible measures that might fall into each of these categories, as well as caveats for their use.

CREATIVE PRODUCTIVITY AND SCHOLARLY REPUTATION

Creative and scholarly research is an integral part of the mission of the University of Oregon. This mission enables the University to attract faculty in a wide range of disciplines who wish to dedicate their careers to the integration of teaching with scholarly and creative pursuits. In a very real sense, therefore, the University is defined by the quality and dedication of the faculty.

How then can the success of the University's faculty in their scholarly and creative endeavors be assessed? The best methods of assessment are those integrated into the structure and normal operations of the University, consistent with the principles of continuous quality improvement.

At the individual level, faculty are evaluated both formally and informally in a variety of ways as part of our faculty retention, promotion, and merit-pay systems. For example, each instructional unit develops specific criteria to be used in merit evaluations of individual faculty in the areas of teaching, research, and service. In addition to internal evaluations, faculty also are often recognized externally as recipients of prestigious awards or by appointment to national and international academies, boards, or societies.

At the program or college level, faculty are collectively evaluated during periodic program reviews involving external evaluators. These reviews deal explicitly with research productivity and the reputation of the faculty within each program. At this time, detailed manifestations of faculty research productivity (i.e., books, refereed papers, performances, commissioned works, scholarly research grants, national and international visibility, etc.) are assess within the context appropriate to the particular discipline. Significantly, program reviews also examine the integration of faculty research with the graduate training of students, the extent and quality of graduate student research, and the pattern of success for graduates of the program. In some instances, program reviews are conducted by accreditation societies or boards. Beyond these periodic program evaluations, many disciplines present occasional evaluations of the productivity and reputation of research departments in the discipline. These offer independent, although often narrow, evaluations of research productivity.

At the University level, detailed evaluation criteria are more difficult to apply due to the diverse nature of our various programs. Thus, evaluations of University research productivity must rely, in the main, on appropriate evidence from its component programs. In some instances, however, aggregations of particular assessment measures (e.g., counts of scholarly research grants and recipients of prestigious awards and appointments) may be meaningful. University membership in the prestigious Association of American Universities, as one of only 56?? public universities, clearly marks our research distinction, as does our attraction to many of the best students from around the world.

UNDERGRADUATE EDUCATION

Providing high quality undergraduate education is a central pillar of the University of Oregon's mission. Since the mid-1980s efforts have been made nationwide to measure the effectiveness and the efficiency of undergraduate education. In its March 1992 report, the OSSHE task force on assessment stressed several key principles to keep in mind about such measurement. Most importantly, the report stressed that assessment must match institutional mission. Further, assessment must make use of multiple indices of outcomes. Institutions must resist simplistic quantification of outcomes such as comparisons of performance on standardized tests, even if those tests are achievement based; moreover, it is completely inappropriate to make statements of outcomes based on testing instruments designed to measure aptitude.

To measure the productivity of undergraduate teaching, APMG recommends using a wide variety of instruments and evaluation approaches and indices.

One set of indicators concerns enrollment and retention, addressing such questions as: Does the institution enroll and retain students who are capable of benefiting from the resources on the campus? What are the entry indicators used for success? Do these indicators and the recruitment efforts produce a cohort of students well suited to the educational program of the institution? What level of attrition is appropriate to the institution? Concrete measures might include average freshman GPA, SAT and placement exam (math, writing, foreign language) scores.

A second set of indicators concerns students' progress toward degrees. Are requirements well articulated and coordinated with the resources marshaled to allow students to meet those requirements? Are there indicators of intermediate achievement that provide calibration of student needs with necessary resources? Are learning and teaching resources marshaled in an efficient and timely manner to ensure that students are learning actively and are progressing toward graduation? What barriers are there to students' progress and how are they overcome? What is the breadth and depth of undergraduate learning and achievement? Concrete intermediate progress indicators might include: student satisfaction and experience surveys, and proportions of students studying abroad, in internships, in honors programs.

A third set of undergraduate education indicators address graduation. Not only is a count graduates important, but assessment of the quality of their learning and of the learning experience, consistent with the UO's goals and mission. Indicators might include: admissions to graduate and professional programs, performance on examinations that measure knowledge and subject area achievement (e.g. GRE Subject Area Tests), initial professional and career placement, and student research and publication

A final set of undergraduate education indicators could come from retrospective (or statistical cohort) studies of alumni, assessing such graduates' professional achievements and long-term satisfaction.

GRADUATE EDUCATION

University instruction of graduate students involves teaching, research, and creative activity at the highest level. Graduate teaching and graduate programs are an extension of the University's research mission and an essential component of outreach to the community, State, and nation. Graduate teaching and research inform the undergraduate programs in the disciplines by focusing faculty and student effort on the most current scholarship, technology, and creative endeavors. The University's graduate programs enable faculty and graduate students to pursue the creation of knowledge while training the next generation of professionals in business, industry, government, and education. Graduate education is a fundamental component of the University's tripartite mission of teaching, research, and service.

Assessment and productivity measures for graduate education might cover enrollment (e.g., counts of students by graduate program, level of study, and demographic/ethnic categories); entry indicators (e.g., GPAs and GRE scores, admissions rates); special recruitment (e.g. text description of departmental or college-wide recruitment plans targeted to special populations of students, and first- and third- person reports of student success); graduation achievement (number of degrees granted by department and University-wide per academic year, cohort completion rates, awards and honors (text listing of internships, fellowships, traineeships, scholarship, teaching, research, citizenship, post-doctoral fellowships); employment and placement profiles and summaries; extramural funding (source and amounts) in support of graduate student research or teaching; student satisfaction and experiences (first person accounts and survey results); teaching innovations and evaluations; student research and publication; faculty teaching load (including reading and conference, recitation, practica, internship, independent study, supervised college teaching, and field study courses); faculty/student collaboration and related professional socialization; expenditures of state dollars (050) on graduate instruction per year divided by the total number of graduate students; and total state dollars (050) spent on GTFs per year divided by the total number of graduate students.

CENTRALITY

Centrality refers to how essential various departments, schools, colleges, and programs are to the University's mission and charter, and how essential various units in the University are to meeting the needs of the State and society in general. Measures of centrality directly reflect the University's statement of purpose and the values of the measures are likely to change little, if at all, over time, but they are important factors in University planning.

SERVICE

Service to the university and service and activities in the public interest are obligations of every faculty member. Such service is normally comprised of two elements: (1) participation in university and departmental governance, and (2) contact with those outside the university through such activities as public lectures, speaking to community groups, performance, commentary, etc.

According to the University Charter, the president and the professors constitute the university, and the faculty should have the immediate government of the institution. Because of this long-standing University of Oregon tradition of faculty governance, the university relies upon effective regular participation of faculty in many aspects of university life beyond teaching, academic advising, and research. Academic and administrative service internal to the university include membership on faculty and governance bodies and serving on departmental, college, school, university or state system committees or task forces. Such commitments can be substantial, including: service on the University Senate; curriculum development; preparation for university or departmental accreditation; evaluation of candidate files for admission; review and evaluation of faculty promotion and tenure files; serving on departmental or university search committees; strategic planning; budgeting; intercollegiate athletics; etc.

Service and activities on behalf of the larger community and the public are also encouraged and recognized as important to the mission and reputation of the university and its departments. Such community and public service reflect professional expertise of the faculty member and often are related to professional growth, scholarship and teaching. Examples of service vary widely by academic discipline but include membership on local, state, national or international committees, conducting workshops, giving public presentations or performances, serving on editorial boards, etc. Faculty may also contribute to the formulation of goals and policies for national, professional and community bodies.

The description and amount of faculty service vary widely from school to school, department to department, and from individual to individual. Quality and effectiveness of service, therefore, are difficult to assess. Departments have different ways of providing service, and different expectations about providing service. However, because of the tradition of faculty governance and participation in the life of the university, and because community and public service often relate closely to professional growth, scholarship and teaching, academic and administrative service and activities on behalf of the larger community are best evaluated at the departmental level. Currently, this is done at tenure, promotion, merit, and post-tenure evaluations with service being one of the criteria. Again, the nature and relative importance of service varies by department and discipline.

Measurement of faculty service would involve counts and narratives of selected items mentioned above. To accomplish this effectively, faculty members would need to complete a self-report of service activities on an annual basis. The completed report will be reviewed at the department level and then forwarded to the appropriate academic dean. Department chairs should seek the counsel of their departmental colleagues in establishing expectations for appropriate types and levels of service activities, both internal to the university and to the public.

ADMINISTRATION AND SUPPORT

[N.B.: This section has received little detailed attention thus far.]

The role of administrators within the University is to act as mediators of the communitarian interests of the faculty, staff, and students. Central leaders (president, provost, deans), on the other hand, represent the fiduciary and legal aspects of the institution and are spokespersons to external audiences. They are directive and engage in policy setting in these arenas. And the division between the communitarian and legal aspects of the University is not always clear. For these purposes, administrators must be principled, ethical, and informed.

For measurement, APMG recommends aggregate summaries (e.g. Administrative FTE per faculty, Administrative FTE per student, etc.), as well as self reviews and FTE/budget summaries for state-funded administrative service and department offices.

FUNDING AND RESOURCES

Funding issues are central to a discussion of productivity. In the case of the University of Oregon, the charge from the Chancellor to develop a productivity plan began from the belief that "we must challenge ourselves to teach the same number of students that we did in 1991-93 with fewer resources available."

Productivity measures are defined as ratios: units produced per unit time or per dollar invested. The measurement in which the State of Oregon arguably has the most interest is the productivity per state general fund dollar invested. Thus one set of measurements could compare UO outputs per state dollar invested with other institutions in the state and elsewhere. Measures of the relative productivity of UO might include student credit hour per state dollar, state dollars per student or per graduate, and state dollars per non-state funded grant or contract dollar; these would all be comparatively easy to determine from data currently available and could claim some validity as measures of productivity.

As a public institution, UO has the obligation to provide access to as many qualified students as is feasible, and that access is affected by tuition policy. Measures which concern public policy makers might involve the ratio of tuition to state dollars and the ratio of state dollars plus tuition to outputs. It is important that UO monitor these ratios to ensure that tuition is not raised more than state funds are reduced without improving the quality of students' education. These ratios sometimes also take the form of costs per student or per external dollar raised or per program or per SCH, since costs must be funded from a combination of revenue sources, by far the largest of which are state appropriations and tuition.

Ultimately, improvements in productivity or efficiency should result in dollar savings for the same level of output, a higher level of output for the same number of dollars, or a higher quality output for the same number of dollars. Since the number of dollars available will be constrained by a combination of legislation and policy, careful monitoring of the impact of the elements of the productivity plan on these funding/cost ratios will be an essential part of making the plan produce the results that are necessary: more output per dollar available.

(This is the end of Part 1. Part 2 contains the appendices.) [Submitted by: Charley Wright
Wed, 15 Dec 93 11:43:04 PST] [Copyright 1993, University of Oregon]