 |
Asia-Pacific Forum on Science Learning and Teaching, Volume
Ling L. Liang, Sufen Chen, Xian Chen, Osman Nafiz Kaya, April Dean Adams, Monica Macklin and Jazlin Ebenezer Assessing preservice elementary teachers views on the nature of scientific knowledge: A dual-response instrument |
Both validity and reliability are important indicators of the quality of any quantitative instrument. However, due to the empirical components involved in the development of SUSSI, the conventional concepts of validity and reliability may not apply well (Aikenhead & Ryan, 1992; Rubba, Schoneweg Bradford, & Harkness, 1996) because an empirically based instrument is developed from a qualitative perspective which focuses more on credibility, trustworthiness and authenticity of data (Erlandson, Harris, Skipper, & Allen, 1993) than on consistency across constructs and measurements. Therefore, both quantitative and qualitative internal validity and reliability issues were examined in this study.
Validity. The face validity and content validity were evaluated by a panel of nine experts (seven science educators and two scientists) who were teaching NOS and/or who were knowledgeable about NOS-related research. The agreement level of each Likert scale item was between 78% and 100%. Credibility, trustworthiness, and authenticity of SUSSI were achieved by modifying existing items drawn from empirical studies and literature, and by analyzing the data from multiple sources, i.e., the participants' selected responses to the Likert items, the participants' constructed responses to the open-ended questions, and follow-up interviews. In the current study, we will focus our report on the data analysis results based on the participants' selected and constructed responses.
By reviewing the completed survey forms, we found that the participants did not provide their constructed responses to all open-ended questions. To code the data, we first randomly assigned each of the American respondents' surveys a numerical number between 1 and 209. Then we scored the first 60 constructed responses by each theme, using the scoring guide described in the Methods section (Table 1). The selection of 60 constructed responses for each theme appeared to be sufficient in our case, as no completely new explanations or examples emerged after about 50 constructed responses were analyzed. For the Likert scale items, the respondents' views were classified as Naïve Views if none of the four responses received a score > 3 within each theme; the respondents' views were classified as Informed Views if all four responses received a score >3 within each theme. Finally, the percentage of the "naïve/informed" views based on the analysis of the participants' responses to the Likert items was compared with those based on the analysis of the open-ended questions, by theme.
It was found that three items (3D, 6A, and 6D) in SUSSI yielded results not supported by the constructed responses. First, 3D states that "Scientific theories explain scientific laws". About 47% of the respondents chose to "agree/strongly agree" with the statement. However, none of the participants mentioned or explained such a relationship in their constructed response section. This lack of understanding was confirmed during the follow-up interviews. When asked about why they thought that "scientific theories explain scientific laws," the interviewees often replied "I do not know." In addition, 33% of the participants responded "uncertain" to this statement. This level of uncertainty was the highest among all of the 24 Likert scale items. This suggested that this item should either be modified or removed. Secondly, for 6A and 6D, about 91% of respondents agreed that scientists use a variety of methods to produce fruitful results (6A), and 83% of participants agreed that experiments are not the only means used in the development of scientific knowledge (6D). However, a closer examination of the participants' constructed responses in the open-ended sections revealed that a number of the respondents equated the term "different methods" with different steps within the scientific method, or with different experiments. Moreover, very few respondents were able to provide valid examples of different types of scientific methods. For instance, one preservice teacher agreed with both Likert statements 6A and 6D. However, when asked to explain whether scientists follow a single, universal scientific method, the same student responded that "for most experiments I do think that all scientists use the scientific method because it is the way you are supposed to conduct experiments." No evidence indicated that this student was aware of any alternative types of methods in addition to experimentation, or the scientific method. We therefore proposed to remove the item 6D and modify the item 6A to "scientists use different types of methods to conduct scientific investigations" (see Appendix A). Table 2 presents the comparison of the participants' responses to the Likert items and open-ended questions by theme, excluding the three items discussed above.
Table 2: Comparison of the American Preservice Teachers Responses to the Likert Items and Open-Ended Questions by Theme
Target Aspect
Naïve Views (%) Informed Views (%)
LR* CR** LR CR Observations and Inferences [1A-D] 2 3 35 35 Tentativeness [2A-D] 0 3 40 5 Scientific theories and laws [3A-C] 90 98 0 0 Social and cultural embeddedness [4A-D] 5 8 21 7 Creativity and Imagination [5A-D] 48 42 15 10 Scientific methods [6B-C] 30 33 13 14 Note: *LR = Responses to the Likert items; **CR=Constructed responses to the open-ended questions. The percentage was calculated based on 60 responses per theme.
For the Likert items, the student views were classified as Naïve Views if none of the four responses received a score > 3 within each theme; the student views were classified as Informed Views if all four responses received a score >3 within each theme.
The constructed responses to the open-ended questions were classified according to the rubric described in the methodology section and Table 1.
By reviewing the data presented in Table 2, for those classified as "naïve views," high levels of agreement between the responses to the two different types of items by each theme were consistently demonstrated. For those classified as "informed views," it was found that the percentage of informed views of responses in the constructed section was lower than that in the Likert responses section. This was partially due to the fact that most constructed responses were too brief to fully answer the open-ended questions. In addition, identified slight mismatches between the Likert items and the open-ended question in certain themes might also have contributed to the observed discrepancies. For instance, within the "tentativeness" theme, while the Likert items addressed both the evolutionary and revolutionary aspects of the tentative nature of scientific theories, in the open-ended section participants were asked to explain why they thought scientific theories change or do not change. Therefore many respondents answered whether or not they thought theories would change, without mentioning the nature of the change (i.e., cumulative, on-going modifications and/or replacement of old theories with new ones). According to our scoring rubric, for a constructed response to be classified as an "informed view," the student was expected to explain both whether and how theories may be changed, i.e., evolutionary and revolutionary/reinterpretation aspects. We therefore proposed to modify the open-ended question by asking respondents why they think scientific theories do not change, or how (in what ways) scientific theories may be changed (see Appendix A).
Reliability. When we selected the six target aspects of NOS in our study, we anticipated that these aspects were not independent, but were in fact inter-related. In the Delphi study conducted by Osborne et al. (2003), nine themes emerged from the participating experts' comments. The authors also confirmed that many participating experts felt that some of the NOS ideas were intertwined and not resolvable into separate propositions. In Tsai and Liu's (2005) study, five subscales were identified in an exploratory factor analysis. Meanwhile, the correlation analysis also revealed that several factors are significantly correlated to each other.
Our study further confirmed the interdependency between certain NOS aspects. An examination of Table 3 revealed that the preservice teachers' understanding of the "observation and inference" (theory-laden NOS) aspect was correlated with their views of the "tentative," the "social and cultural embedded" nature of science, and methodology of science. This suggests that the respondents who possess informed views of the theory-laden NOS were more likely to recognize the tentativeness, methodology of scientific investigations, and social/cultural influences on the development of scientific knowledge. This seems plausible because if someone understands that the development of scientific knowledge depends on both observations and inferences, and that perspectives of current science and the scientist guide both observations and inferences, then it would become more natural to recognize that multiple perspectives may contribute to valid multiple interpretations of observations, and that scientists may use different methods to conduct scientific investigations. Moreover, scientific theories/laws may therefore change in light of new observations/interpretations and be influenced by social and cultural factors.
In our study, we calculated the overall Cronbach alpha for the entire instrument (α = 0.69), the Cronbach alpha values for each subscale, and the correlation coefficients among the subscales (Table 3). According to Hatcher and Stepanski (1994), for social studies, a Cronbach alpha as low as 0.55 can still be recognized and accepted for statistical consideration. Considering there is only a small number of items in each subscale, the results indicate that SUSSI achieved a satisfactory level of internal consistency. According to Table 2, the percentage of respondents who demonstrated informed views on all Likert statements within the theme of "scientific theories and laws" was zero. Likewise, no constructed responses were classified as "informed" views according to the scoring guide. Among the NOS themes under study, the participants showed the highest level of misunderstanding and confusion about the nature of scientific theories and laws. This result is not very surprising and is consistent with what was reported in the literature (McComas, 1998). In Table 3, it was also found that the subscale alpha values for the themes of "theories and laws" and "scientific methods" appeared lower than those for the other themes, and the participants' responses to these two themes were significantly correlated (r=0.61, p<0.01). However, we decided to keep those items because of the demonstrated high consistency between the Likert and the open-ended responses as presented in Table 2, and because both aspects represented the two most widely held misconceptions of NOS (McComas, 1998). For instance, understanding the nature of theories is critical to the current controversy over the teaching of evolution in schools. People who reject evolution as "just a theory" are demonstrating the common misunderstanding of the nature of scientific theories. We believe that this aspect of NOS should be emphasized in science textbooks as well as in assessment.
Table 3: Cronbach Alpha Values for Each Subscale and the Correlations among the Subscales of SUSSI (n=209, overall α = 0.69)
SUSSI Aspects Observations and Inferences Tentativeness Scientific theories and laws Social and cultural embeddedness Creativity and Imagination
Scientific methods
(1A, 1B, 1C, 1D) (2A, 2B, 2C, 2D)
(3A, 3B, 3C) (4A, 4B, 4C, 4D) (5A, 5B, 5C, 5D)
(6B, 6C) (α = 0.61) (α = 0.56) (α = 0.48) (α = 0.64) (α = 0.89) (α = 0.44) Observations and Inferences 1
.16*
0.037
0.26**
0.083
0.14*
Tentativeness 1
0.021
0.24**
0.029
0.028
Scientific theories and laws 1
-0.12
-0.02
0.61**
Social and cultural embeddedness 1
0.14
0.01
Creativity and Imagination 1
0.1
Scientific methods 1
Notes: * p<0.05; ** p<0.01
Copyright (C) 2008 HKIEd APFSLT. Volume 9, Issue 1, Article 1 (Jun., 2008). All Rights Reserved.
