SWEPT Evaluation

A comprehensive evaluation process is vital to the long-term success of each SWEPT. It gives evidence of effectiveness, often needed by funders, and strengthens the program. This chapter defines the two major types of evaluation, explains useful evaluation techniques, presents a schedule for the evaluation, and discusses evaluation and use of the results.

Within this web page, you will learn more about Evaluation Types, Methods of Evaluation, Analyzing Data, and the Evaluation Schedule.

Evaluation Types

The two evaluation types most useful for the program are formative and summative. Formative evaluation assesses current, ongoing program activities, provides an internal process that compares the planned program with the actual program, and measures the progress made toward meeting the program goals. This evaluation type helps identify problems threatening the program's viability, enabling the program manager and planning group to make mid-course corrections. The mid-internship meetings, weekly sessions, and informal discussions among teachers, mentors, host-site supervisors, and the program manager are major components of the formative evaluation.

A summative evaluation measures the success of the completed program. It is used to make decisions about the future of the program. Use the results of this evaluation to recruit new host sites, funding sources, and participants, and to publicize the program. The summative evaluation often turns up unanticipated outcomes, identifying aspects of the program that would be otherwise overlooked. The User-friendly Handbook for Project Evaluation: Science, Mathematics, Engineering and Technology Education includes some of these questions as appropriate for a summative evaluation.

• Was the project successful? What were the strengths and weaknesses?

• To what extent did the project or program meet the overall goal(s)?

• Did participants benefit from the project? In what ways?

• What components were most effective?

• Were the results worth the costs?

• How will you share what you have learned?

A close examination of the formative and summative evaluation results is necessary to understand the successes and failures of the program. Remember each participant and partner has a different goal and will interpret the program results differently. Understanding both the overall program goals and objectives and those of the individual partners and participants leads to a better analysis of the evaluation results.

Methods of Evaluation

Base the evaluation on the program objectives and goals. Plan the evaluation before implementing the program. Issues to consider when forming the evaluation include:

• Who will analyze the results?

• When will the results be analyzed?

• From what perspective will the objectives be evaluated?

• What will be done with evaluation results?

Every evaluation requires developing the evaluation questions, choosing the appropriate techniques, and collecting and analyzing the information gathered.

Develop Evaluation Questions

Developing good questions requires a clear understanding of the goals and objectives of both the program and the evaluation. Involve the entire planning group to ensure that no important questions are missed. Determine the resources available for the evaluation and narrow the list of questions to be asked. Remember formative and summative evaluations have different purposes and their differences should be reflected in the questions and groups to whom the results are presented.

Pick the Proper Technique

There are a variety of techniques to use in conducting the evaluation. To pick the most appropriate techniques decide on the information needed to answer the question. Since no one method completely examines each aspect of the program, use more than one to get a full understanding. Four common techniques are discussions, surveys, observations, and reports.

Discussions

Informal discussions and formal interviews with teachers, students, and mentors give an understanding of program aspects needing improvement. For example, teachers may request an increased amount of time together to develop teaching practice changes, laboratory experiments, class lectures, and informal discussion time to share experiences.

Surveys

Administering surveys before and after the internship provides comparative information. Although they are time-consuming to administer and analyze, surveys given to both students and teachers provide information on changes in teaching methods. Compare strengths and weaknesses in laboratories, lectures, discussions, and problem solving exercises.

Observation

Use an independent observer, attending classes before and after the internship, to identify changes in teaching techniques. The best observation is made under similar conditions, including the time of day, classroom situation, number of students, facility, lecture or laboratory setting, and subject matter and level.

Reports

Have participants prepare reports during the internship. Request general information on the reports to generate a wide variety of feedback. Have teachers address their experiences, make suggestions for program improvement, and anticipate effects of the internship on teaching methods. Ask the mentors to supply analogous reports that provide a different perspective.

Analyzing Data

Analyze the data according to the evaluation plan. Stick with the plan to ensure the important questions are addressed. Reexamine the data to answer additional questions only after completing the initial analysis.

Evaluation Schedule

If resources permit, administer the evaluation before, during, and after the internship program, and again in the following years. Use the evaluation prior to the internship to establish the baseline for measuring changes. At mid-program, determine if program objectives and goals are being met, and identify the perceived benefits and any new objectives for the rest of the internship. After the internship, survey the participants to measure changes in perceptions, attitudes, and teaching methods caused by the program. Use this immediate feedback to get ideas on expanding or improving the program.

During subsequent school years, survey students in classroom situations similar to those encountered during previous years. This input helps determine if long-term program objectives are met.