College Manuals
IMasTs
The Authors
Take a Lookook
6th Gradee
7th Grade
8th Grade
Testimonials
Test Results
Bumblebee
Student Success
Select Science

Test Results

Summary

During the development of the sixth grade modules student learning outcomes were assessed with the mathematics and science sub-tests of the TerraNova Multiple Assessments. Pre and post-tests were administered during the second field test to IMaST students and a similar comparison group. The results showed that IMaST slightly higher in mathematics, but it was not statistically significant. Results from the science sub-test, on the other hand, indicated that the IMaST students improved their performance more than did the comparison group.

For the 7th and 8th grades a version of the TIMSS test was used to compare the performance of IMaST students with a comparison group from the same schools. The external evaluator's analysis concluded that the IMaST students tend to score significantly higher in both math and science than the Non-IMaST students. The difference is especially apparent where items are measuring math problem solving and science processes. Consistent differences were found for both seventh and eight grade students. The science differences appeared to be proportionally larger in scientific processes and reasoning for the IMaST group than for the Non-IMaST group. Additionally, the 8th grade IMaST students scored at or above the expected US average, while the Non-IMaST comparison groups scored below the expected USA mean. IMaST seventh grade students scored well above the USA mean while Non-IMaST seventh graders scored around the USA mean.

Feedback obtained by the external evaluator from teachers and, especially, students indicate that this integrated math, science, and technology curriculum is effective. Students liked the way that everything fit together. The fact that they could not really tell when they were doing technology, math, or science was for them a positive. They saw reasons for the mathematics that was being carried out. In the words of one of the students, “we get numbers by collecting data, we don’t just practice anymore”. The teachers rated integration as one of the most positive attributes of the curriculum. Some seemed pleasantly surprised at the impact it had on student behavior and learning.

The following excerpts are from reports submitted by the external evaluator employed by the National Science Foundation to determine if the IMaST Project was meeting the goals set forth in the original proposal.

6th Grade

The impact on student behavior and attitudes. The most direct evidence pertaining to student behavior and attitudes came from on-site visits and interviews with student groups conducted by this evaluator. Two classrooms where the IMaST curriculum was being used were observed in each of the past two years. After observation, a group of four to five students met to respond to a series of structured questions with probes.

From the student’s point of view, the IMaST curriculum is, apparently, truly integrated. They seemed not to notice the distinctions among the three topical areas. They worked at constructing the equipment (technology) necessary to provide data to study the patterns (mathematics) that they discovered in their investigations of scientific phenomena. According to the various responses, this was one of the facets that they really appreciated.

Classroom observations illustrated the high level of student motivation with the curriculum. In the case of all observed classrooms, the teacher or one of the team organized the activities for the day. The students apparently knew the drill. They read the instructions, responded to probing questions, and asked for directions. Then, as though they were “let loose” they converged on their particular group’s (usually four students) table and began work on their various tasks. In one class they were designing watersheds and wells, in another the students began work on measuring the dimensions and volume of various containers.

The noise level in the classrooms was moderate and a function of in-group discussions regarding their various tasks. Some students were constructing or measuring while others were recording data. There seemed to be a very high rate of on task behavior. The teachers’ role was to circulate and guide by asking questions rather than providing answers. The students seemed to be reluctant to stop their work at the end of the class period.

Student interviews confirmed the apparent high level of motivation. The students seemed to have a sense of the unique nature of IMaST. In response to questions regarding how IMaST is different and how they had to study differently, they were able to verbalize the integrated nature of the curriculum (“It’s all mixed together, it’s great”) and what they called the many “hands-on” experiments. One stated “we get numbers by collecting data, we don’t just have to practice anymore”. Two other student comments spoke to the constructivist nature of the tasks: “You put your mind to it rather than just putting it on a page”, and “You need to figure out the answer, you have to be logical”.

Other questions regarding what they liked about IMaST and whether they would recommend it to their peers prompted very positive assessments about their experiences. They enjoyed the experiments and the de-emphasis of drill and homework. They enjoyed the practical hands-on experiences and “collecting data and figuring things out like we did when we calculated speed”. Most felt no hesitation in highly recommending the curriculum to their younger friends and siblings.

The students did have some concerns. A major item was the lack of clear directions and the reticence on the part of the teacher to explain things (which is a major part of the design of discovery curriculum). Some students felt that being the recorder of the data was boring and writing reports was a major chore.

The student’s behavior in the classrooms and their responses during the interview indicated their overwhelming enthusiasm for IMaST Plus. Obviously, the students enjoy conducting the experiments and the “hands-on” nature of the activities. Although there was some reticence for preparing data bases and writing the reports, one got the sense that they had to find something of a distasteful nature to report to this evaluator. Most of them, apparently, do see the value of what one teacher termed “re-thinking” as they summarized their experiences in writing.

These sixth grade students have a rather good sense of the impact of IMaST on their own learning. Observations during the class period found them digging into the data and discussing issues in order to get closure to the task. During the interviews they told this evaluator, in their own words, how creating their own knowledge leads to better retention and renders the knowledge more useful in the future. Furthermore, they seemed to recognize that their high level of motivation makes learning more efficient.

The Morgan-Jinks Efficacy Scale contains three sub-scales including Talent, Context, and Effort. The sample data across all schools showed a slight and not statistically significant loss for both groups on all scales. In addition, there was no apparent difference between the two groups in their change from pre to post assessments on any of the three sub-scales. In light of the more positive data from the direct observations and student interviews, the absence of observed impact from the Efficacy Scale may be due more to internal validity issues than the lack of experimental results. There may be a general decrease in measured efficacy for all students from the beginning of the school year to the end simply due to response bias on the part of the students. No direct indication of reliability of the sub-scales was obtained. However, the intercorrelations between the pre and post-tests were .59 for Talent, .32 for Context, and .40 for Effort. Even with the time differential of eight months, those values seem quite low and indicate a good deal of measurement error in the data. Those low relationship indices would render positive impacts difficult to detect.

The impact on student learning: teacher’s perception, and TerraNova test scores. This section will discuss two sources of evidence for examining the impact on student learning. First, data were obtained from items imbedded into the structured phone interview conducted during this past spring and, secondly, Terra Nova test scores were obtained.

Teacher perceptions. Five items in the phone interview addressed each of (1) student motivation, (2) student initiative, (3) skill development, (4) academic preparation, and (5) cognitive integration. The teachers were asked to respond on a five point likert scale where “1” is a rating of “high” and “5” indicates a rating of “low”. The responses to those items follow:

The item read: “Compared to other curricula with which you are familiar, what is the impact of the IMaST curriculum on:”

Comments included: “The kids really get into it, in some cases they would go beyond the assignments.” “IMaST engages students & develops a sense of control.” “Students are active the entire class.” “It motivates disinterested kids.” “Some LC's and modules were more interesting than others.” “ Kids are highly motivated-they come in excited.” “The students now see a reason for mathematics.” “The kids just love coming to class.” “Some students say they don’t like it, but they certainly don’t act like it.” “Motivation depends on the kids-the hands on type jump into it while the ‘academic’ types are a little more hesitant.”

Comments included: “The program teaches kids to gain initiative.” “They take initiative but do need encouragement once in a while.” “Students get such a sense of ownership.” “This is a lower performing group, so they tend to hold back.” “Classroom management is a challenge.”

Comments included: “It had a major impact on student learning.” “The students understand what they are doing-that makes their learning so much more effective.” “As the students went through the year they just got better and better.” “I expect them to be very well prepared because of the problem solving skills.” “Gets them to think differently.” “They may be a little less prepared for traditional classes-there is less algorithm but better understanding.” “They learned how to learn.” “On a museum field trip the IMaST students showed a much higher understanding and asked much better questions.”

Comments included: “They'll be better prepared for high school than with any other curriculum.” “I expect IMaST will help the students to be able to compete academically at any level.” “I am confident that they are academically prepared.” “Transfer from math activities to worksheets is difficult.” “I expect their future academic work to be high.” “With supplements, academic preparation is a ‘1’.” “I’m a little worried about their test performance.” “They will be well prepared for science as well as math.” “This kind of instruction has to positively impact retention.” “I’m waiting for test scores to see about skills.” “Kids seem ready to go to the next level.”

Comments include: “Integration is a definite strength.” “Integration is great idea.” “Writing helps language skills.” “I love that aspect, the way it fits together and with reality.”

In comparison to other curricular packages with which the teachers are familiar, they gave IMaST relatively high ratings. Eighteen of the nineteen teachers rated student motivation on the high end of the continuum. Fourteen rated student skill development, academic preparation, and cognitive integration as “1” or “2”, while thirteen of them gave those high marks for student initiative. The comments verified those ratings. Many mentioned the eagerness with which the students approach the various tasks. The low ratings (below “3”) occurred in a couple teacher’s estimate of “skill development” and “academic preparation”. Those ratings came primarily from one school where other comments from the teachers indicated a hesitancy to put constructivist teaching methods, in general, into practice.

TerraNova Test reslts. Each of the field test schools were asked to supply a comparison group of students who would not experience the IMaST curriculum. The groups were to be as similar as possible in terms of socioeconomic and ability factors to the IMaST group. The mathematics and science sub-tests of the TerraNova Multiple Assessment package were administered to the IMaST and the appropriate comparison group as a pre test in the fall (October) and as a post test in the spring (late April or May).

Data collection problems occurred at some of the sites. Parental approval for participation in the data collection was not obtained at one site. No record of membership to the IMaST group vs. the comparison group was kept at another site. The comparison group at one of the sites performed much lower on the pre-test than did the IMaST group, and, consequently, regression toward the mean would bias the comparison between the groups. Finally, implementation of IMaST was quite limited at one of the sites, so that the academic impact could be quite limited.

Those schools were not included in the data set from which comparisons were made. The final set included seven of the sites where, in all but one case, one comparison class was available for each IMaST group. The exception, site 6 in Table I., included two IMaST classes and one comparison class. The seven sites included in the analysis, along with the number of students tested at each one, appears in Table I.

Table I.
Frequency of IMaST and Control Students
Within the Final Test Schools

An analysis of Covariance was conducted between the IMaST and the Comparison group on each of the Mathematics and Science sub-tests of the TerraNova. Normal Curve Equivalency (NCE) scores were used in the analysis. NCE’s have the advantage of being immediately comparable to expected performance according to national norms, since NCE’s are anchored to a mean of 50 in both the spring and fall testing periods. Zero gain from fall to spring, for any student as well as the group, indicates normal or expected gain. NCE’s allow direct observation of performance level (by comparing scores or means to 50), as well as gain from fall to spring by comparing the amount of gain to zero which is the expected or normal gain.

The ANCOVA model examined the difference between groups on post-test performance while covarying out (holding constant) performance on the pre test. The analysis examines the difference between post-test means that are adjusted for group difference on the pre-test. Authorities in research (i.e. Glass and Hopkins, 1984) have argued that analysis of covariance provides a more valid estimate of change than does the direct observation of gain scores.

The results of the TerraNova Mathematics sub-test appear in Table II. The F-value of .08 is not statistically significant. Furthermore, the adjusted mean for the IMaST group of 62.33 is only slightly higher than that of 61.01 for the comparison group. The results provide no evidence to indicate that the performance of the two groups were different in Mathematics. Compared to the national NCE mean of 50, both groups seemed to maintain a relatively high level of performance around a mean level of 60. Furthermore, both groups showed improvement in terms of NCE’s from fall to spring.

Table II.
Analysis of Covariance of TerraNova
Mathematics NCE post scores using pre
Scores as the Covariate

Dependent Variable: Mathematics NCE scores

Sum of

Means of TerraNova Mathematics NCE scores by Group

* The number of students varies from that in Table I. due to missing data on either the pre or post test for any given student.

The results for the TerraNova Science sub-test appear in Table III. The F-value of 13.22, p = .0003, indicates a statistically significant difference between the means of the comparison and IMaST groups on Science post-test performance. The adjusted post-test mean of 64.85 for the IMaST group is higher than is that of 61.30 for the comparison group. The results indicate that the IMaST students performed better on the post-test than did those students who did not experience IMaST. Both groups began the fall with relatively strong performance compared to the national mean of 50; each was around the 60 value. Both groups gained as well. However the IMaST students showed more of an increase than did the comparison students.

Table III.
Analysis of Covariance of TerraNova
Science NCE post scores using pre
Scores as the Covariate

Dependent Variable: Science NCE scores

Means of TerraNova Science NCE scores by Group

In general the results of the analysis are mixed. Although no statistically significant difference was found between the two groups on the mathematics sub-test, differences beyond that which could be due to chance were found on the science sub-test. The results in math are not so much negative as they are inconclusive. The results yielded no evidence of the impact of IMaST on student mathematics performance either positively or negatively. Retaining hypotheses of no difference does not support the assumption that no impact occurred, rather that the results suggest nothing definitive.

7th and 8th Grade

More than 1000 students participated in both the pre and post testing at the eighth grade level, and an additional 400 at the 7th grade level. The distribution between IMaST and comparison group numbers was nearly equal in each of the two grade levels. An Analysis of Covariance was conducted on each of the sub-scales as well as the math and science total score. The tables below provide information on the mathematics and science posttests means adjusted for pretest performance by analysis of covariance, and are grouped by participation in IMaST or Non-IMaST. While school scores vary, overall the IMaST students did very well in comparison to the Non-IMaST comparison groups at each grade level.

In addition to comparing IMaST students to Non-IMaST students within schools, student performance was compared with the expected average score for USA students based on the TIMSS results. The expected score on each item used in these sub-tests was calculated and a total expected score for the math and science sub-scales and total score was tabulated. The TIMSS study administered tests near the end of the school year, so these expected scores should provide a benchmark for comparison of posttest scores. For more information on the TIMSS tests themselves, and complete analysis of their results, refer to the TIMSS web site at http://timss.bc.edu/timss2003.html

The following data tables show the composite results of all IMaST pilot students that completed both pretests and posttests for the 1998-1999 school year. Schools without comparison groups or incomplete testing results were eliminated to remove potential bias. Table I. provides the results for seventh grade students, while Table II presents the eighth grade results. The tables contain the TIMSS expected score, the post-test mean adjusted for pre-tests differences for the IMaST and Non-IMaST group for each sub-scale and total of math and science.

Table I.
Seventh Grade Results

* Statistically significant difference between the IMaST and Non-IMaST means.

Table II.
Eighth Grade Results

* Statistically significant difference between the IMaST and Non-IMaST means.

The external evaluator's analysis concluded that the IMaST students tend to score significantly higher in both math and science than the Non-IMaST students. The difference is especially apparent where items are measuring math problems solving and science processes. Consistent differences were found for both seventh and eight grade students. The science differences appeared to be proportionally larger in scientific processes and reasoning for the IMaST group than for the Non-IMaST group. Additionally, the 8th grade IMaST students scored at or above the expected US average, while the Non-IMaST comparison groups scored below the expected USA mean. IMaST seventh grade students scored well above the USA mean while Non-IMaST seventh graders scored around the USA mean.

We are pleased with the overall performance of the IMaST students. While individual school results varied, and many other variables could have also impacted the students' performance across these sites, the data provides clear evidence that IMaST students do outperform the comparison students across sites. Additionally, the IMaST students scored well above the expected USA score on this set of TIMSS items, providing hope that we can improve our position internationally in mathematics and science.

The external evaluator noted that the most unique component of the IMaST curriculum is the inclusion of technology as a primary thrust. He suggested that the hands on nature of technology may well form a solid cognitive foundation for the new information being learned in mathematics and science. However, he indicated that other factors such as the economic status of the community could have been a contributing factor. The total number of schools was not high enough to factor out all of these contingencies. So it can only be concluded that technology activities may increase understanding in mathematics and science. This is an issue in need of further research.