Section 5.4 Designing and initiating evaluation procedures.
¶Identifying and involving an external evaluator was part of the process of applying for NSF funding. External evaluators conduct assessments to assure the funding agency that adequate progress is being made and to provide feedback to the grantees. The extent to which an evaluator is external can vary. Often evaluators from outside agencies serve this function by visiting periodically to conduct and analyze various assessments but otherwise have little contact. Their expertise is in evaluation procedures; they may or may not have expertise in the focus of a project. The external evaluator for this project, however, was the chair of the OSU department of science and mathematics education. She was assisted by a doctoral student in physics education. They were highly knowledgeable about science education reform issues, the doctoral student could observe class sessions from the perspective of an experienced physics teacher, and both were able to provide on-going feedback through frequent participation in the paradigm faculty's meetings.
Preparing the NSF proposal included devising a 3-year plan to identify and document effective instructional strategies, curricular materials, and factors that encourage sustained interest and enrollment in physics. The evaluation also would compare both affective and cognitive student outcomes for students participating in the reformed curriculum with those in the traditional program as well as develop procedures to evaluate the replication, dissemination, and expansion of the reformed curriculum. In addition, the evaluation would include external review of the new curriculum by physics faculty at other institutions. The evaluator also took responsibility for application for Institutional Review Board approval for obtaining consent from participants to collect data.
After NSF funding was obtained, the evaluator conducted baseline studies to document student progress in the context of the traditional curriculum, with quantitative data such as Graduate Record Examination (GRE) scores, grades in physics courses, numbers of students from various subgroups including gender, minority, and economic factors, as well as attrition rates for different subgroups of students. She also gathered qualitative data such as interviews with students and instructors, class observations, and electronic mail journals. This baseline documentation of the junior year experiences occurred during the first year of the NSF project, 1996-1997, while students were still enrolled in the traditional courses and the faculty were undertaking design of the new courses. A similar baseline study of the senior year experiences occurred during the second year of the project, 1997-1998, while the new junior courses were launched but the seniors were finishing in courses taught in the traditional format. Formal evaluation activities for the new courses occurred in the junior paradigm courses (years 2 and 3) and senior capstone courses (year 3) as they were launched.