Assessment Results

UO Assessment Results, 1991-1994

The major outcomes of our curriculum evaluation can be summarized as follows:

workshop students appear to display more improvement in conceptual learning and understanding of scientific reasoning than do our traditional general biology students.
workshop students appear to display a greater understanding of and willingness to address complex issues. Students consistently cite the course's relevance as its most important aspect.
workshop students appear to become more involved in their learning experiences and value them more highly, but at the same time develop higher expectations of their learning experiences and are more critical of them.
women and non-traditional learners appear to display particularly high levels of learning and motivation. Also, non-native English speakers do not seem to be at a disadvantage.

While we feel we have sufficient evidence to support these kinds of generalizations at this point, we must add that we have been unable to eliminate all ambiguities or alternative explanations for these results. Additionally, we have not been able to consistently gather useful data throughout the period of the project. A brief chronological summary of our results, including a discussion of difficulties we have encountered in interpreting our data, may help to clarify the limitations of our conclusions to date.

Year 1 (1991-1992)

In Year 1, while we developed and piloted our assessment instruments, we relied primarily on class participant-observation, analysis of student work, and course evaluations to assess the course's effectiveness. Comparisons with the traditional course during this first year would have been difficult anyway, since enrollment in the traditional course was around 200, as usual, while we limited enrollment in the workshop to 60 students, or two lab sections. We can summarize our assessment of the workshop course as follows:

Students tended to have two reactions to the workshop course. Many found it to be an extremely positive learning experience, particularly in terms of the hands-on activities, the issues project's relevance, and the opportunity to work closely with instructors and other students. Others, however, were extremely unhappy with the course's workload, the intellectual demands of many of the assignments, and the ambiguity of expectations. In addition, the course was fairly disorganized this first year and many activities did not work as planned. Perhaps because of this, attrition in the first two terms was high, with 50% of workshop students leaving the course after fall term (normal attrition between terms is around 20-25%).
Attendance and student involvement in course activities was very high all year, but the quality of students' work was not what we had hoped for; it became clear that most students needed more explicit practice in critical thinking, writing, and library research. This was when we came face to face with the problem of misconceptions, also, as we found students unable to simply assimilate many fundamental concepts, and with the problems of students' level of intellectual development. We also needed to rethink our approach to group assignments, grading, and course organization.

Year 2 (1992-1993)

In Year 2, we offered four sections of the workshop course, enrolling up to 120 students. This provides for a better comparison with the traditional course, which enrolled between 140 and 180 students. We have useful data from concepts tests all three terms, a scientific reasoning instrument, a survey of attitudes toward science and science classes, and a survey of value placed on different aspects of science literacy (the "Science Literacy Survey," or SLS; see assessment methods (link) for additional info on instruments). We also continued our qualitative assessment methods (classroom observation, etc.) We can summarize the results as follows:

In the fall, the results of the concepts test were somewhat mixed, the workshop students showing greater improvement in some areas and the traditional students showing greater improvement in others. Workshop students showed significantly greater improvement on both the winter term and spring term concepts tests. In fact, in some areas traditional students' scores actually declined, and their answers demonstrated increased confusion over some concepts. Differences between genders and class levels in the spring was striking, with all class levels improving in the workshop, but freshmen in the traditional class showing a decline in scores. Males improved more than females in the traditional class, but females improved more than males in the workshop.
The workshop students' scores on the scientific reasoning instrument and on the SLS showed significantly greater improvement than did those of the traditional class. Scores on the attitude survey showed no change, except on the scale dealing with one's expectations for success in science classes; traditional students' scores on this scale declined while workshop students' scores showed no change.
We began using activities to improve students' abilities to use the library and critically evaluate evidence they found for their issues projects in the fall of this year. We compared students' bibliographies from this term and from the winter previous (Year 1), and found that using the new activities appeared to be quite effective. Students in the year before were more likely to use books and popular, nonscientific magazines (such as People or Newsweek), while students in the current term, who had completed the library and critical evaluation activities, were more likely to use scientific magazines (Science News, Discover) and scientific journals.
Students' course evaluations in the workshop continued to be more favorable overall than in the traditional class, but we began to see a new phenomenon as well. Workshop students' written comments were, as a rule, lengthy and well thought out, while traditional students' written comments were brief and superficial, as has been the norm in the past. Workshop students' comments were also more critical, often giving specific suggestions for improving an activity and explaining in detail how their suggestions would be more effective. Here are a few examples, beginning with some representative comments from students in the traditional course at the end of Spring term 93:
1. All material was valuable. Stick to the text more.
2. The labs were helpful, and the review sessions. I liked the video. I don't know [what could be improved].
3. The time in lab [was the best]. The lectures [need to be changed].
4. BI 101 workshop version and 102 lecture version were good classes. I enjoy them both. But I can't say the same for the lecture in BI 103. The lecture needs to be improved or changed. The lab is fine.
Here are some representative comments from students in the workshop course at the same time:
1. The format of the tests is excellent. It allows us to explain our reasoning and understanding of the concepts. We also learn more when we get our tests back because we have to be critical of our reasoning in order to understand why we lost or gained points. We can't simply go, "oops, it was 'B'." The soil fauna project had some problems: 1) having an experiment already set up does not help us to learn/experience how to construct a plausible hypothesis/prediction and then learn how to test it. 2) We had to make up a hypothesis to fit the experiment. 3) This also made it difficult to do an accurate write-up. Since we didn't construct the experiment it was easy to leave out crucial "methods" details. For example, how were the soil and litter samples separated?
2. Issues projects and presentations [were the best thing about the class]. With this assignment, I am using what I learn in class to discover how it is relevant to present-day society. The biological aspect of the class comes out in researching my topic and discovering how pertinent material in class is to controversial biological issues. Suggestion: Change the format a little from term to term. Maybe replace the presentations with a field trip + paper one term or make paper revolve around in-class experiments one term.
3. The lab sections [were the best thing], they provided a hands-on approach to what concepts we were learning. The labs showed us how concepts in biology related to everyday life. Only the grading system [should be changed]. I liked first and second term grading better because it was very easy for us as students to figure out where we stood. But I understand this was difficult for you to do because of too many individual assignments. Most of the things that I felt were very useful involved a discussion or a type of group working together. I think this is important because it lets us bounce ideas off each other, and get a better idea of what we were doing.
We saw this as evidence that students were applying their critical thinking abilities to their learning experiences as well!
Enrollment shifts stabilized this second year, with about equal numbers of students dropping or switching between the classes.

Year 3 (1993-1994)

In Year 3, we offered six sections of the workshop course, enrolling up to 180 students; we limited the traditional course to 80. During the first term of Year 3, we conducted a more controlled experiment in which the same instructor taught both courses, using the same goals and as much as possible the same kinds of activities. We were interested in the importance of the workshop format itself, aside from specific activities, in achieving our goals, and in the possibility of achieving workshop goals in a more traditional course format. Some outcomes of this study included:

Workshop students still improved more on the concepts test than did traditional students, but the differences were not as great as they were in previous terms, primarily because the traditional students did better than they had in the past. We refined the concept test given this term by increasing the number of questions and, in scoring, grouped them first by concept area and then by cognitive level. Workshop students' scores improved more in the areas of cell structure, genes and molecular genetics, and cell division; the two groups improved about equally in the areas of Mendelian genetics and photosynthesis and respiration. By grouping questions by cognitive level, we found that workshop students' scores improved more on questions dealing with factual knowledge and understanding, and the application of concepts to novel problems. The two groups improved about equally, however, on questions requring analysis and scientific reasoning. This was also the area in which students improved the least-we hypothesize that these skills take longer than a single term to develop. Females also continued to improve more than males in the workshop, while they showed about equal improvement in the traditional course, and the differences were not quite as pronounced as in the previous term. This term, students of all class levels improved in both courses.
We began using a new set of questions on all final course evaluations asking students how effectively the course had achieved its goals (improving students' understanding of biological concepts, the process of science, the relevance of science to their lives, and their critical thinking abilities), and how important each of those goals were in an introductory biology course. Students in both groups felt that exploring the relevance of science to real life was the most important goal of an introductory biology course, with learning concepts second. Both groups also indicated that this had been the goal most successfully achieved by their course. The two groups' mean ratings on this goal were about the same, while the workshop group's ratings of their course's effectiveness on the other three goals were significantly higher than those of the traditional group.
For the first time, students in both the traditional and the workshop course completed issues projects. Though the preparatory activities in both classes were similar, students in the two classes showed a difference in their choice of topics for a final research assignment. Workshop students more often chose controversial issue-related topics, such as whether gene therapy should be used to treat genetic diseases, while traditional students chose more non-issue, informative topics, such as the details of a human genetic disease. Workshop students also used more appropriate references in their research, and slightly increased their use of scientific journals over the students from fall term of Year 2.
At the end of the term, we asked students about their plans for the following term. 80% of the students in the workshop, as opposed to 59% of traditional students, said they would remain in the same version. Four students (9%) in the workshop said they would switch to the traditional, three of whom cited scheduling difficulties as their reason. 12 (22%) traditional students said they planned to switch to the workshop, 10 of whom stated they felt the format would be more beneficial-only two mentioned their schedule as the main reason.
Our classroom observation of students supported our beliefs that the longer lab time in the workshop, providing greater one-on-one contact with instructors and their peers, led to greater involvement of students in learning activities and enhanced their learning. While the atmosphere in both labs was very positive, it often seemed that workshop students were more engaged in their lab activities and took them more seriously. However, both courses appeared to have done an excellent job in motivating students through the use of relevant issues and increasing their interest in science. Students' comments on course evaluations were very positive in both courses.

Winter and spring terms were conducted as in the past, with different instructors in the two courses. Unfortunately, we experienced some difficulties with instrument administration in both terms. In the winter, only a few students in the traditional course chose to complete the concepts posttest (which, as part of a research project, is necessarily voluntary). In the spring, attendance was so low in the traditional course that less than half were present to take the posttest, and, again, not all of those that were there chose to take the test. This was also the time we administered the year-end posttests of the other instruments, so our sample size for these is also too small to be useful.

In the winter term, we noticed some interesting effects in observing the two classes and looking at students' course evaluations, at least partially due to the interaction of the two instructors. The same instructor that taught during the fall was joined by a faculty member who was new to the project; it was the first time that a tenured, research-oriented faculty member who had not collaborated on developing the original proposal had taught in the workshop. The two instructors each taught half of the lab sections, and split up the assemblies throughout the term. Students had a more difficult time understanding and appreciating the assemblies given by the instructor who was not their lab instructor. Additionally, the two instructors had very different expectations concerning exams; the experienced workshop instructor gave fairly difficult exams and graded fairly strictly. The new workshop instructor, on the other hand, gave extremely difficult exams but graded much more easily. This created difficulties for the instructors the following term, whose exams and grading were much more similar to the first, experienced workshop instructor's. Students felt that the fact that they had "done better" (gotten better grades) on the previous term's more difficult exams meant that the instructors were not doing a good job teaching, testing, or grading them.

Though it appeared from the fall term results that the workshop format, apart from the specific activities, was beneficial to student learning and involvement, we found in spring term that the workshop format alone is not enough. In implementing many aspects of the course this term, we failed to learn from past experience. For example, though keeping portfolios had helped students stay organized in the past, we did not use portfolios again this term. Though we had found the first year that trying too many new things at once only increases confusion, we tried too many new things at once, and failed to carry through on many of them. All of this was complicated by widely varying student expectations formed the term before. We believe that part of this was due, ironically enough, to one of the instructors being away a good deal of the time giving presentations on this project. It was a classic example of the distractions that can occur at the end of a funded project, as one attempts to carry through on the project's activities while looking to future activities, such as dissemination.

There is also the possibility of an effect due to the increased class size of the workshop. Lecture attendance in both classes declined over winter and spring terms; it may be that the larger assemblies this year (again, we had 180 in the workshop and 80 in the traditional) encouraged the kind of disengagement and withdrawal that characterize so many large courses. Our emphasis thoughout this project has been on lab size. A key element of the workshop course has been extended times in small labs, and we have given less attention to the effects of overall class size. We plan to continue exploring this issue as we again offer a large workshop class and a smaller traditional class this year.

Limitations

There are other significant limitations in interpreting many of our results, due to differences in instruments administration and in the ways in which the culture of the two groups affected response patterns. In the workshop, we often attempted to increase student motivation for completing the concepts tests by giving it as an ungraded "practice final," but we have, for the most part, been unable to convince the traditional instructors to do likewise. Thus workshop students may have put more effort into completing the test than did traditional students. In the controlled study conducted in fall of Year 3, both groups did take the concepts test as a practice final, however.

In more general terms, the workshop course's methods have helped to create a culture in which students know their feedback is valued, and in which they get many opportunities to practice giving feedback. Though we consistently give midterm and end-of-term course evaluations in both classes, workshop students get additional opportunties to evaluate their learning experiences, and they are more likely to see evidence that instructors are taking their comments seriously. Though this is clearly evident in differences between responses on the two groups' course evaluations, it is likely that this has an effect on students' responses on other instruments as well, including motivating workshop students to complete the instruments and write longer, more considered responses. Workshop students know they are part of an "experiment" to improve their learning, and though they may not always be happy with the results, most become committed to the idea of working toward improvement.

Additionally, though our goal is to survey all students, because not all students are in attendance on administration days, and not all of those choose to complete the instruments, we inevitably end up with a nonrandom sample. This problem was most serious in the last two terms, as noted previously, but response rate has typically been lowest in the traditional class, and those students in both classes who do not take the instruments are likely to be systematically different in terms of motivation or ability from those who do. Finally, of course, throughout most of the project the two course's goals have been extremely different-we have in effect been comparing apples to oranges. Our assessment methods are weighted heavily toward workshop-type goals. However, this was not true during the Fall 93 experiment, which we are repeating again this fall (1994). We are continuing most assessment efforts as we enter Year 4 of the workshop course. We will continue to present our results at conferences as the project progresses, and are preparing a series of publications which will discuss our findings in more detail.

Conclusions

Despite the limitations discussed above, we believe that it is possible to assess the quality of students' learning and development, and to use that information to make effective decisions about the design of courses. Continually striving to improve our assessment strategies will be a major factor in improving our abilities to make these decisions. Our collaborative efforts to make sense of our findings has been confusing and frustrating at times, but at the same time highly rewarding. We are not in search of "right answers" in the form of the perfect general biology course, since one of the things we've learned is that teaching improvement is a process that continually feeds back in on itself, and is subject to the changing needs of students and the values of individual faculty. Not only do we know more now, we are learning to ask the right questions.

Whether the benefits achieved by the Workshop Biology approach are worth the extra time and effort is a value judgement that each instructor or department must make. What we have come to realize is that the workshop approach offers benefits for instructors as well as for students, in that we come to know our students better and feel a greater sense of accomplishment when the outcomes of our efforts are so clearly evident. We end with two comments from workshop students, who write about what their experience in the course has meant to them.

"When I began taking biology in the fall, I expected it to be a typical science class like those I had taken junior high and high school. That is, one where the teacher wore a white lab coat, we watched movies about deer and elk at least once a week, and did experiments like seeing how fast salt would boil out of water. Basically, nothing we ever did pertained to anything of substantial value in our society, nor did it interest us in the least little bit. Although after taking three terms of biology here at the University of Oregon, I have realized science is much more than white lab coats and beakers of bubbling blue liquid...Science is all around us, and it pertains to almost any aspect of life. When I look in the Register-Guard, for instance, I can find at least one article that is related to science each day. I have learned that in some ways, science can be overwhelming and extremely complicated, yet in others with a little guidance it's just common sense. Through my taking of this course I have come to realize science doesn't have to be a horrible thing. It can only be what I make it, and that can easily be something that is interesting and involves answers to what I find questionable."

"My conception of biology has been drastically broadened after completing this three course cluster. I used to view biology simply as cells, kingdoms, and photosynthesis, failing to realize that biological concepts were around me every day. When I heard from high school science teachers that science is life, I thought that this was simply an attempt to get the students interested in learning the material. Another misconception that I had truthfully answered is that there is more to biology than memorizing data to pass objective tests. Such tests prevent the relating of concepts to social concerns, which I was introduced to in this course. This idea was brought forth in in-class exams in 1st and 2nd quarter where we would have to interpret a hypothesis or a current article and relate it to what was just covered in lecture or lab. This process was modified to the best by having take-home exams where I was able to be thorough in my thinking without the anguish of sitting through a test-taking situation. But the biggest thing I learned in this class is that I should not be scared of science, that I can handle it. This is a direct turn-around to the tragic experiences I was faced with in high school chemistry. I am no longer fearful of biology, treating it as if it were any other course. In fact I might take the freshwater or marine life biology course in the future!"