What does a test “test”?

All over the world, at all levels of education, mathematics teachers set tests for their students, and give grades on the basis of those tests.

Tests are usually thought to be reliable indicators (if not measures) of how well students  engage with the material being taught to them, and is often believed to be related in some correlational way with their mathematical ability.

Where a student ranks – their class standing – on a test may well be an indicator of both those things.

I want to argue, however, that what a test really tests is the teacher. A test, in my view and experience, tests a teacher’s ability to set a test at the “right” level.

Typically, students will be awarded a percentage score on a test, indicating the fraction of available points a student obtained as a result of the correct answers they gave.

Standards for grading answers vary widely. Some teachers give partial credit, some do not. Some teachers are lenient in relation to answers that require interpretation, others are not.

A certain number of percentage points is taken to be a pass. Below that critical percentage a student is deemed to have “failed”, above it the student has “passed”.

In many cultures, particularly that of North America, a mere “pass” is regarded by students and their parents as a shameful outcome. In my experience the overwhelming majority of North American students see themselves as A or B students, and certainly not as D students.

Grading standards vary widely in the United States.

One variant, not uncommon in my experience is to convert percentages into letter grades as follow:

A = 90% or higher

B = 80%–89%

C = 70%–79%

D = 60%–69%

F = 0%–59%

The interpretation of these letter grades varies quite a bit.

For example, college students often may not count a grade below C as a “pass” for the purposes of their major, but may count a “D as a “pass” if it is a subject outside their major. For example, many engineering students can carry a D in calculus as a passing grade.

The typical college level passing percentage in Australia is 50%, and in the United Kingdom it is 40%.

Curiously, academic standards in these countries, overall, seem to be in inverse proportion to the passing percentage. The UK has generally higher academic standards than Australia , which, in turn, has generally higher academic standards than does the Unites States.  What I mean by “higher academic standards” is that an any given level of academic attainment one will find UK students to be generally more advanced in knowledge skills and achievement than Australian students, who in turn are generally more able than students in the United States, at the same grade level.

How does a teacher construct a test?

A colleague changed the order of the questions on a test for a course he taught each year, and obtained disastrous results. Typically his students would study tests from previous years in preparation for that year’s test. The mere fact of changing the order of questions was enough to send a number of students into a tail spin.

Teachers can easily set a test for which all but the best and brightest students will fail to pass. This does not require setting material that students have never seen: just pushing the standards a little on reasoning and rigor will generally do it. Or just change the order of the questions from previous years’ tests.

Teachers who do this regularly will be seen by their colleagues as being too tough, unrealistic, and out of touch.

On the other hand it is easy to set a test for which all but the least diligent students will pass, and for which most students will do very well: simply focus on routine calculational questions, all of which have been practiced often beforehand.

Teachers who do this will be seen by their colleagues as too easy, too soft, and not fulfilling their obligations to properly assess students.

So a conscientious teacher wants to steer a course between being overly demanding, and being too soft.

Using various skills and prior experience, therefore, a teacher will generally set a test that has a likely outcome of a reasonable number of students passing, not too many top marks, and a few, but not too many failures.

In other words, what is being tested when a teacher constructs a test is, in fact, the teacher’s ability to set an “appropriate”, or satisfactory, test.

The myth of objectivity

Many people, especially administrators, like test scores because they are apparently “objective”.

This imagined objectivity is illusory for several reasons.

We have already addressed how a teacher needs to carefully construct a test so that not everyone fails, not everyone does exceptionally well, and is such that their colleagues are satisfied they are doing an adequate and appropriate job of assessment.

There is no objective standard for this: it is simply a matter of practice embedded in a particular culture.

Even within a given test culture there is the problem, in testing mathematics, of giving partial credit, of deciding what to do if a student makes a simple calculational error but then proceeds more or less correctly in their application and reasoning following that.

A final grade- percentage or letter – is an average of points obtained as a result of many such decisions.

There is very little that is objective about this process.

To see why, one needs only give the same test papers to a colleague, with the same grading instructions, and see how the awarded grades vary. Even more telling is for a teacher to re-grade test papers a week or so later and see how consistent are the grades.

The pain of believing in the objectivity of test scores

Students who fail to pass mathematics tests usually believe they are failures.

They are not.

They may not have studied all the material at a depth that would make a pass or higher grade likely, they may have been ill for some or all a semester, they may have had family problems, they have have seriously misunderstood a teacher’s intentions and instructions. They may have attention deficit disorder, be highly anxious, or be on medications that affect their ability to focus.

Al of these factors are real and should be understood as real mitigating factors.

Believing in the objectivity of a test and the consequent view of oneself as a failure is highly damaging. It lowers self-esteem and inhibits future effort.

Conversely, students who do study conscientiously  who are relatively free of stress,  and who score highly on tests are very likely to see their success as an intrinsic ability: they begin to see themselves as “A” students. Such students may well have high or even exceptional ability, but a class test is not providing evidence for that. What the test provides evidence for is that some factor, or combination of factors, lead a student to a good outcome. Those factors can include high ability, but almost certainly will include time spent studying, effective study habits, belonging to a study group of peers, asking questions of the teacher, taking notes, listening carefully, a low stress environment, and a dash of luck. Believing in the objectivity of tests leads a student to attribute their success on tests to only one of the factors: ability.

Should we use tests?

I don’t know.

I don’t any more because I see tests as testing my ability to set them appropriately, and not testing the intrinsic ability of students.

I can get an estimate of student work habits from tests, but frankly I’m not much interested in a test to tell me this when I see them regularly in class working on projects. I can see how much work they are putting in: I do not need a summative test to tell me that.

Franky I feel we put too much emphasis on tests and examinations. My own preference is for more collaborative project work in which students can exercise their ability to think, to reason, to plan, and to work toward a goal, utilizing their skills and talents in conjunction with others.

Education, for me, should be a win-win activity, not something we carry out to determine – by phony means in my view – who is a success and who a failure.