What teachers assess and its consequences

To understand what each teacher's assessment valued, teachers were invited to talk during the interviews about the nature of the questions they asked on their exams and the qualities they wanted to see in their students. The examination papers they provided also supplied tangible evidence of what they assessed.

Analysis of the data suggested that the teachers based their assessments on two types of criteria: cognitive and non-cognitive.

1. Cognitive criteria

Cognitive achievement is related to the acquisition of knowledge and learning skills. The kind of knowledge and skills assessed in Turkish schools is defined in the official assessment policy, which states that:

…in addition to assessing the acquisition of knowledge, qualities of comprehension, application, analysis, synthesis and evaluation should also be assessed (MONE, 2001b: Article 5k).

The official policy emphasizes the importance of assessing students' higher-order thinking skills. However, the application of this approach in the classroom is dependent upon whether teachers' assessments consider these different levels of thinking.

In this study, the cognitive achievement that teachers assessed was apparent both in their definitions of their assessments during the interviews and in the examination papers they provided. Two of the teachers (T11 and T17) defined biology as a subject "based on text, so memorization is inevitable." All of the other teachers emphasized that they wanted students to go beyond memorization. One of the teachers (T7) said:

Sometimes they say, "let's not do the lesson. You just give us a list of questions, and on the exam ask from this list." They will memorize the answers. What I try to do is to prevent memorization. How can I do this? How can I teach them think critically? How can I see their abilities and understanding differently? I do not think I do this but I try. While I try to do this students tend to do exactly the opposite. I do not believe that memorization is any good for students. They just memorize but do not know what it means. They need to know how to use their brains.

Although most teachers indicated that they did not want to encourage memorization, analysis of the interview data and the examination papers together suggested otherwise. The data showed that students were expected to absorb knowledge transmitted by the teacher or found in the textbook in a ready-to-use format and then reproduce that information on the exams. The main concern of the teachers, thus, was not to ascertain what the students knew but to determine whether specific facts were known. One teacher's (T10) comments exemplified this:

I present them the important parts of the topic and I ask them on the exam. During the lessons, I tell them that "I ask this…I ask that on the exam, be careful.

This explanation defines assessment as a process of cueing rehearsed answers in the lessons ( Wiggins, 1990).

The ability to remember important information was frequently mentioned by the teachers:

I ask short-answer, easily-remembered important details. I already dictate notes during the lessons and on the exams I try to ask them… (T23).

This highlights the overemphasis of the assessment of low-level skills (i.e., recall versus recognition).

The practice of providing students with an answer that they can simply memorize in preparation for the exam reduces the cognitive demands associated with the exam (Strage, Tyler, Thomas and Rohver, 1987; Bol and Strage, 1993). Through this approach, the teacher sends a message to students about what is important. Therefore, it is not surprising that the students only learn the answers to these questions (Wiliam, 2003). This means that students are engaging in less productive types of studying. Furthermore, the goals of science learning that call for critical thinking and more sophisticated study strategies may not be realized (Bol and Strage, 1996).

It is interesting to note that two of the teachers admitted that they did not prefer asking questions requiring interpretation or critical thinking in their exams, especially for the low-achieving groups of students. These teachers expressed concern about the inclusion of critical thinking questions in their exams, pointing out that their students were not able to cope with such questions. One of the teachers said:

I do not ask questions requiring explanation or interpretation… They cannot do that… It should be asked though…. Yet students cannot even write what is written in their textbook or a sentence they wrote in their notebook during the lessons, which I had dictated to them or they had copied from the blackboard, let alone adding something on to that…. in some high achieving classes sometimes I do that… but especially Year 9 students… for example, they are very bad at interpreting graphs… (T16).

This comment highlights the teacher’s stance regarding the weaknesses of the students. Although s/he thinks that questions requiring explanation and interpretation by the students should be asked, s/he avoids such questions, and students are not challenged because s/he see students as unable to answer such questions.

Hence, students are sometimes not asked interpretation or thinking questions because of the concern that they cannot cope. Furthermore, assignments requiring research are avoided because the students do not know how to do research. Additionally, because students are not seen as knowledgeable enough or able to make valuable contribution to the lessons, their input is not asked for in the lessons. One of the teachers said, "when I call on students they only repeat what is written in the textbook... they do not say anything different at all" (T16). One teacher indicated that previous years of students were not taught in an environment where they were invited to contribute or discuss their ideas, which suggests that the students in general are not used to these teaching and learning processes and that they do not have thinking abilities. In addition, the social and family background of the students and the students' lack of ability were also indicated as the reasons for their low expectations of teachers. In any case, underachievement was not thought of as the responsibility of the teacher, and the teachers repeatedly indicated that they could not do much to remedy accumulated weaknesses in the students.

The analysis of the questions in the exam papers also supported the teachers' emphasis on assessing basic facts and asking what is given in the textbook and by the teacher in the lessons. In other words, the exams supported the practice of cueing rehearsed answers in the lessons. Table II provides the distribution of the questions in the exam papers according to Bloom’s taxonomy of cognitive levels (Bloom, 1956) and provides the criteria on which the questions were grouped.

As can be seen in Table II, 249 out of 370 questions (67%) were knowledge questions. These questions mainly asked for the knowledge of basic facts, details, terms and definitions, which essentially required students to write pre-defined answers. The words identified in these questions were: "what," "where," "name," "state," "define" and "list." According to the analysis results, the questions in this group fall into three groups: definition questions, list questions and questions asking general facts. Examples of the questions include the following: "What is muscle tonus?" (definition); "Name the tropism movements" (list); and "In which canal is the corti organ found?" (general facts). These types of questions require students to recall the definition given by the textbook or the teacher.

Table II. Distribution of the examination questions according to Bloom’s taxonomy (n=370)

		Questions		Total %
		n	%	Total %
Knowledge	Definitions/descriptions	29	7.8	67.3
	List	37	10.0
	General facts (Where is…?; How many…?)	183	49.5
Comprehension	Comparison (What is the difference between…?)	12	3.2	16.7
Comprehension	Explanation (why is this so?)	50	13.5	16.7
Application	Application of standard algorithms and making simple calculations	42	11.4	11.4
Analysis	Read graphs, make inferences/deduce/conclude	10	2.7	4.6
Analysis	Relationships (How are these concepts related?)	7	1.9	4.6

(Source: Examination papers)

Comprehension questions test students’ understanding of scientific knowledge, and their knowledge of facts, theories and procedures are assumed. The questions in this group require students to make comparisons, order steps in a process, recognize meaning or explain why. An example of a comprehension question is: “Why does glycolysis occur in the cytoplasm in aerobic respiration?” (explanation). The answer to this question is: “Because its enzymes are in the cytoplasm.”

It is noteworthy that the answers to 35 of the 62 comprehension questions on the exam papers were drawn from the examples given in the lessons. Examples below illustrate this:

Question: What is the role of the air sacs in the birds’ skeleton? (Explanation)

Student’s answer: To be able to fly easier and faster using the air.

This question seems to be a good example of a question that requires students to think. However, at the same time it is the exact example that is written in the students’ notebooks: "Birds’ skeletons are adjusted to enable them to fly. In their skeletons, there are air sacs to reduce the body weight of the bird." Hence, the question requires nothing more than what is written in the student’s notebook.

Another example is:

Question: What is the difference between regeneration in human epidermis and that in a sea star? Explain.

(Comparison)

Student’s answer: Different. Sea star can regenerate a new sea star from an arm that had broken off but, since humans have various tissues (muscle, skeleton), the wound just heals.

One might argue that the level of this question is "analysis" if the student has not previously been told what the difference is (or read it from the textbook). However, this was also an example directly from the students’ notebooks.Hence, the previously given answer is sought.

Application of knowledge questions were considered to be the questions that asked students to use known algorithms to conduct simple calculations, as illustrated below:

Question: There are 1500 nucleotides in a DNA molecule. If the ratio of A+T/G+C equals to 2/3, what are the numbers of A and G?

(apply known equations and conduct calculations)

The students are expected to calculate the values of A and G. To answer this question, students should know that A=T and C=G. The rest requires application of this simple mathematical calculation.

Analysis questions required reasoning and the identification of relationships between concepts; in other words, students were tested on their ability to explain how one thing is related to another thing. Only 17 questions fell into this category. Three main sub-groups could be identified from the analysis: considering the relationships between different concepts and with daily life, reading graphs and making inferences, and identifying the relationships between concepts.

The question below tests students’ ability to identify the relationship between two concepts:

Question: Which of the following illustrates the relationship between contract-relax stretching of a muscle and the heat production? (reading graphs and identifying relationships)

( ---------- muscle stretching, ------------ heat production)

amount

stimulus time

A B C D E

The question below tests students’ ability to identify the relationship between aging and mitochondrial DNA:

Question: What is the relationship between aging and mitochondrial DNA? (relationships)

Overall, the analysis of the exam papers supported the interview data results that the knowledge of basic or "important" facts is seen as an important cognitive criterion and that the teachers mainly checked the extent of students' ability to recall (short answer) and recognize (multiple choice or true/false) these facts.

This emphasis on low-level knowledge or thinking skills is also suggested by other studies (Crooks, 1988; Linn, 1990; Gallagher, 1991; White and Gunstone, 1992; Bol and Strage, 1996; Dindar, 1996; Turan, 1998). Bol and Strage (1996) found that teachers excluded items that required higher-order thinking skills from their exams based on a perception that their students would be unsuccessful. Additionally, White and Gunstone (1992) reported that the majority of assessments used by many teachers measured only the recall of closed, low-level science information and algorithmic skills. They indicated that oral questions and written tests were often restricted to short answer questions, which do not measure understanding.

As indicated previously, the nature of learning is determined by the type of assessment that is anticipated (Crooks 1988; Ramsden, 1992; Biggs and Moore, 1993; Bol and Strage, 1996; Scouller, 1996; Cizek, 1997) because students inevitably use the strategy that they think will best achieve the highest grade. Hence, assessment has a crucial effect on what and how students learn. Given the evidence from previous studies, a likely consequence of teachers’ assessments targeting low-level skills could be that they may encourage lower-level learning in students. They may also hinder students' ability to think and confine them to a very limited range of knowledge and activities embodied in the textbooks (Bol and Strage, 1996). The present study found that teachers' assessment predominantly targeted low-level skills. Therefore, this is important to consider in an attempt to improve student learning in schools.

2. Non-cognitive criteria

The term "non-cognitive criteria" is used to indicate the qualities that are not directly associated with students' academic progress. Official assessment policy states that:

….in addition to formal exam results while assigning end-of-term marks, teachers should consider students' qualities, such as participation in class activities, science attitudes, ability to observe, conduct research, take responsibility, work in groups and share ideas (MONE, 2001b; Article 5i).

Although judgment of cognitive achievement dominated their assessment practices, teachers in this study considered other additional factors. However, the factors that teachers considered in their assessments did not comprise all of the items in the policy. For example, qualities like the ability to observe, conduct research and work in groups were not an option for most of the teachers in the study because they did not conduct laboratory work, assign individual or group projects or use group work in their teaching. The qualities that the study’s teachers considered were effort, general in-class behavior and personal characteristics.

2.1. Effort

The teachers in this study shared the view that effort was an important component of end-of-term marks, and 23 out of 24 teachers mentioned taking effort into account while assigning end-of-term marks. The teachers’ main reason for considering effort in end-of-term marks was a concern for making the assessment fair by assigning extra marks to those who studied during the term. One of the teachers defined assessment as a means to show students that their efforts were appreciated by the teacher:

In the end, we need to distinguish between students who study and who do not so that students will see that their effort is appreciated and rewarded. Assessment is the only way to do that (T17).

One of the teachers discussed how she distinguished between the students who studied and those who did not:

….. in the exams, I ask one or two questions they can answer easily and questions that can be counted as of medium difficulty, and then there are one or two difficult questions. I want to see how many of them can answer these questions. These last ones show who really studied (T8).

By asking questions of different difficulty levels on the exams, the teacher (T8) made judgments about his/her students regarding the effort they exerted. This teacher went on to say that "these students are rewarded at the end-of-the term" with extra marks.

It is noteworthy that six of the teachers indicated that students who tried hard or put effort in during the term would be given a passing grade, even if the formal exam results were failure. One of the teachers said:

I have some students, I know that they study, they really put an effort but they cannot be successful on the written exams. I cannot fail them (T24).

Taking effort into account in this sense was a means to compensate for the effort expended. The interviewees mentioned that written exams might not always reflect the students correctly. They argued that students might not show their real performance in the written exams for different reasons, such as illness or lack of preparation, or that students might know the answer but could not remember it during the exam. More importantly, the teachers also indicated that students might not be the kind of students who were good at memorizing and that is why they received low marks on the written exams. Therefore, they believed that it was necessary to consider effort when assigning end-of-term marks.

Such a practice also seems to operate as a reward system based on the relationship between marks and student motivation. The underlying assumption is that if students know that a particular behavior will earn them a mark, they will behave accordingly. Indeed, students may be motivated by the prospect of getting high marks and expend extra effort. However, Stiggins and Conklin (1992) warned that such a relationship may hold true for good students, but for students whose experienced such a system and failed in the past, marks will be nothing more than a reminder of failure, and they will stop trying. Such practices may also lead to the students only trying for marks (Rowntree, 1987; Kohn, 1993) or, as one of the interviewees in this study indicated, result in students with a "materialist view of school education."

2.2. General conduct and personal characteristics

General conduct in the classroom (i.e. paying attention to the teacher, keeping quiet and obeying classroom rules) and personal characteristics (i.e. politeness and showing respect to others) were also mentioned by all of the teachers as factors that they considered when assigning end-of term marks. One teacher said:

At the end of the term while assigning marks, I consider their participation, effort and certainly their personal characteristics, too (T23).

Another teacher indicated:

If a student's general behavior has been good during the term, we raise his/her end-of-term mark (T10).

The teachers' emphasis on students' general conduct and personal characteristics was mainly a reflection of societal values. All of the teachers interviewed indicated that they saw themselves mainly responsible for educating students and that the mission of teaching came later. In other words, "making good citizens" was the primary responsibility of teachers.

Furthermore, the teachers’ practice of considering general conduct and behavior during the term when assigning end-of-term marks also served managerial purposes. Again, linking a reward in the form of extra marks to the end-of-term marks for good behavior would make it easier to manage student behavior in the classroom.

Overall, it seemed that marks were treated as currency in the teachers' hands and that good marks were paid to students for their good traits. However, marks were generally used in the positive way, as one teacher said: "The oral exam marks I assign are not lower than the average of the written exam marks, even higher than that" (T3). Thus, in general, the teachers explicitly admitted being generous with marks to students who showed some evidence of having worked hard and/or behaved well. The teachers frequently indicated increasing students' end-of-term marks based on the consideration of students' effort and behaviors. Similar findings were also reported by previous studies (Brookhart, 1991; Cross and Frary, 1996; Cimer, 2004).

Rowntree (1987) reported that students with personalities favored by their teachers were assigned better grades than their objective performances seemed to justify:

The teachers' knowledge may be used with prejudice - positively or negatively. If he receives generally "good vibrations" from a student he may over rate the student on many constructs that contribute in no way to the "vibrations" conversely, the student from whom he has conceived a distaste or lack of affection may be unfairly "marked down" on constructs that were not influential in shaping the early poor impression (p. 115).

Brookhart (1991) showed that grading often consisted of a "hodgepodge" of attitude, effort and achievement. Similar findings concerning the mixed nature of grades were also reported by Cross and Frary (1996). They reported that teachers raised the grades of low-ability students and those students who showed high effort. Thus, they took student conduct and attitude into consideration when assigning grades. More importantly, the students largely confirmed and supported the use of conduct and attitudes for determining grades and indicated that the students' ability level should be considered to be fair to all students.