Analyze the following evaluation situation by identifying

the problem and suggesting alternative procedures.

A supervisor of instruction wanted to conduct an evaluation in order to compare a new independent study

approach with the regular instructional approach in high

school mathematics. A written formal agreement with the

district evaluation staff stated the following arrangements:

a. The evaluation was to help the high school mathematics department chairpersons decide whether to adopt

the independent study approach districtwide.

b. The procedures were to conduct a districtwide comparison of the two approaches, involving 20 percent

of the high school’s mathematics teachers and all of

their students.

c. Mathematics achievement, students’ attitude, and

teacher enthusiasm would be assessed.

d. Teachers would be randomly selected and assigned to

the two different approaches.

The supervisor later decided that the evaluation should

provide feedback to improve the new approach rather than to

decide on adoption. She changed the procedure for assigning

teachers and students to the project, which resulted in their not

being assigned randomly. The evaluation staff—assuming that

the evaluation focus and design, once agreed on, would remain

the same—collected and analyzed data as originally planned.

The evaluators found that students’ attitudes toward

both approaches were similar but that student achievement

and teacher enthusiasm were significantly greater for the

independent study approach. The report judged this

approach as superior and recommended it for adoption.

The supervisor was disappointed that the report did not

help improve the independent study approach. The department chairpersons complained that the findings were not

dependable for two reasons: (1) Many of the teachers

assigned to the independent study approach were biased in

favor of it before the study began, and (2) the students in the

independent study classes were generally high achievers prior

to entering the program.