UO Assessment Methods
In measuring the success of Workshop Biology, we were concerned not only that we were able to achieve our goals, but that the workshop course was more effective at achieving those goals than the traditional method of teaching the course. To investigate this, we have continued to offer a traditional version of General Biology as well as the workshop. For the period of the formal study (1991-1994) both were listed in the course catalog as BI 101-102-103, so that they met the same University requirements and attracted the same student population. We have been able to compare outcomes on a set of instruments administered to both classes, and through qualitative methods such as classroom observation. We administered the tests and instruments as pretests and posttests, so that we could determine the actual gains (or losses) made by students over the course of the year. We also compare the outcomes of the workshop course for male and female students and for students of different class levels.
In comparing the workshop and traditional courses, we use the following data collection methods:
These methods are described in more detail below.
We administer instruments to assess students' learning and development in two main areas: concepts and scientific reasoning, and attitudes and values related to science and science classes.
1. Concepts and scientific reasoning. Our assessments of students' conceptual learning includes their ability to apply concepts to novel problems; their scientific reasoning ability, including such skills as making sense of data in both graph and tabular form, identifying deficiencies in experimental designs, and drawing conclusions from experimental results; and their views of scientific knowledge. To assess students' conceptual learning, we use a series of "concepts tests," one for each term. To assess their scientific reasoning abilities, we began with a separate instrument, created using items from Analytical Problems in Biology (Donovan & Allen, 1983), and administered with the other instruments at the beginning and end of the year. In the third year, we incorporated these items into the three concepts tests, partly to get a more longitudinal sampling of student abilities, and partly to spread out the time spent in class taking instruments. The instrument we have developed to assess students' views of scientific knowledge is given at the beginning and end of the year, and is still being developed.
Scoring and analysis: The concepts tests use a combination of essay-type items and multiple-choice items that require written explanation. The tests are identified only with students' social security number, and we shuffle all tests together when scoring them, so we do not know whether one is a pretest or a posttest, or whether it comes from the traditional or the workshop group. The project coordinator and research assistant each score every test independently, check for interrater reliability, and then reconcile any differences to create one dataset.
In scoring the concepts tests, we are normally quite unforgiving with students' answers, only giving full credit when the student has made it clear that he or she fully understands the concept. We wish to distinguish between students with truly complete and accurate understanding of concepts and those with a less clear understanding. Total scores are thus often much lower than one might expect to see on a normal exam.
All of our tests and instruments are analyzed using change scores. A change score is the difference between a student's pretest and his or her posttest, and represents the actual growth taking place during the course. We then compare the means of these change scores in the two classes, effectively controlling for initial differences at the same time we look for significant differences between the two groups.
2. Attitudes and values. The impact of students' values extends throughout the workshop course's activities, from their interest in the subject matter, to their willingness to participate in different kinds of learning activities, to their attitudes toward science classes in general, to their view of the importance of science in society and in their personal lives. We use a variety of both quantitative and qualitative methods to assess changes in students' values. Our instruments include an attitude survey we are developing, which is primarily aimed at assessing students' attitudes toward science classes in general, and is administered at the beginning and end of the year; a set of questions we include on final course evaluations, asking students to evaluate the effectiveness of their course in achieving its goals, and their feelings about the importance of those goals; and the Science Literacy Survey (Champagne, 1989), which asks respondents to rank different aspects of science literacy. For example, is it more important that a scientifically literate person be able to "use scientific information in personal decision-making," to "find scientific information when needed," or to "be able to define basic scientific terms?" While not as useful as a comparison tool, this instrument gives us a good picture of the agreement between our students' values pertaining to science literacy and our own. We usually give this instrument at the beginning and end of the year, but sometimes more frequently.
Scoring and analysis: These instruments are all Likert-format (i.e. 1-5 rating, strongly agree-strongly disagree), but do include space for students to write comments. We score these instruments simply by tabulating the responses, but do not use machine-scored forms (students hate these!).
We administer our own midterm and end-of-term course evaluations in each class every term, in addition to the University's standard final course surveys. Our course evaluations are not identical between the two classes, but normally have several items in common. They usually have a mix of rating-type and open-ended items, the latter analyzed using content analysis. We read through all of the comments once, identify common themes, and then tally the number of comments which fall into each of these categories.
We also ask students to do other reflective writing exercises throughout the term, which helps them understand the value and meaning of what they are learning in addition to giving us important feedback about their experiences in the course. For example, at the end of the year, we often ask students to write a paragraph or two, as a homework assignment, on the following question: "How has this course changed your conception of biology and biologists?" Students' responses showed us that, not only was the course achieving all of its stated goals, but that students were aware of these goals and found them valuable.
The project coordinator and the research assistant attend class sessions as participant-observers. For the first two years of the project, the project coordinator attended every class session; in the third year, this became difficult as more workshop sections were added, and the coordinator and research assistant each attended a sample of class sessions.
Since students do such different kinds of work in the two classes, we generally look at quizzes, exams, papers, and in-class activities for formative assessment of the workshop class itself, and to test specific hypotheses about students' learning (for example, the effects of introducing a new activity or modifying an activity). Occasionally, it is possible to compare students' work in the two classes, as in Fall term of year 3 when both workshop and traditional students completed issues projects, and we were able to compare the kinds of issues students chose, and the evidence they used in support of their arguments.
Each term, we monitor individual students' enrollment patterns to determine how many students remain in the same class the following term, how many students drop altogether, and how many students switch to the other class. We also obtain demographic information from the Registrar's office, including age, gender, class level, major, and college entrance test scores (the UO requires the SAT for admission of new freshmen). We also include a sheet with the instruments that are administered at the beginning of the year which asks students for their educational and career goals, their parents' occupation and educational level, and their reasons for taking their biology course.
Interpreting the results of our assessment efforts can be complex and confusing. The answers do not automatically emerge from the data, any more than answers automatically emerge from any other research. Collaboration is an integral and necessary part of our efforts to construct meaningful interpretations of our observations, interpretations that are shared by the group and thus less prone to personal bias. Also, by gathering both quantitative and qualitative data, we can be more sure of conclusions that are supported by both, while we would be less likely to trust either on its own. Finally, the conclusions we draw feed back into our development process, in which we continually reflect on our goals and our chosen methods of achieving those goals, assessing the fit between our expectations and our observations.
Go to Assessment
Results
