*Shanghai and Singapore have been greyed out for 2003 and 2006 because neither took part in those two rounds. Since it is hard to assess a trend when the population base changes, I think that*

**Note:****either**Shanghai and Singapore should be included for all rounds,

**or**both should be excluded for all rounds. I have chosen to include them, and, based on their results in 2009 and 2012, I have assumed that they would have finished atop the PISA table had they competed in the earlier rounds.

With Shanghai and Singapore included, the perspective on the Canadian trend changes somewhat. However, some slippage is definitely happening, and for many people the trend is alarming.

But I wonder if we are

But I wonder if we are

*concentrating too closely on the rankings*. Here is the same table when grades are included:**OK. This doesn't look like the PISA scores that you have seen. The results shown above are percentage grades, and the PISA consortium does not report the scores this way. Instead, they use a numerical scale that ranges from about 300 to over 600.**

*Note:*I have a fondness for percentage grades. They provide me with a more serviceable scale than grades out of 600. For example, to me, a class average of 67% indicates an ordinary but acceptable performance, while I am not sure what an average of 527 on the PISA scale conveys.

The conversion from the PISA scale to percentage grades in the table above was done linearly:

Percent_grade = 60 + (Pisa_Score − 482) × (10 / 62.3).

As is evident, Canada has remained remarkably stable: 68%, 67%, 67%, 66%.

###
**Proficiency Levels**

The PISA results are intended to be a measure of mathematical proficiency, and PISA describes six different levels. The grade ranges for each level are as follows:

**Highly proficient**or

**Basic proficiency.**

#### Some features of the PISA proficiency levels

- For each round, the PISA consortium adjusts the grades so that the proficiency levels retain the same meaning over the years. A percentage score of 65% in 2003 has exactly the same meaning as a percentage score of 65% in 2012. In other words, PISA takes care to avoid grade inflation.

- On the percentage scale, each Level band is 10 points wide. On the PISA scale, the bands are 62.3 points wide.

- The PISA scales are designed so that the OECD mean is 500 (or very close to 500). On the percentage scale, the OECD mean is 63%.

- Over the four rounds from 2003 to 2012, Level 6 has
**never**been achieved by any of the participating regions. Level 5 has been achieved by**one**participant. Level 4 has been achieved by**five**participants. In the four rounds from 2003 to 2012, Canada has resided in the Level 3 band.

### Is the slippage significant?

The difference between Canada’s score in 2003 and 2012 is

**statistically significant**. The words mean that, with high probability, the difference in the scores is real and is not a result of the chance that arises by testing only a sample of the population. Statistical significance tells us that the statistics are

**reliable**, but it doesn't tell us anything about whether or not the statistics are

**important**.

(The word "significant" has considerable emotional impact. I remember a Statistics professor telling us that the word should be outlawed when we are discussing statistical results.)

*dependence on intensive memorization and rote practice of the basic algorithms is a poor strategy for learning anything but the most trivial mathematics.*

In any event, when I look at the percentages for Canada from 2003 to 2012, I see changes, but I don't think that they are earth-shattering. Would you be concerned if one section of your Calculus course averaged 68% and another section averaged 66%? I’ve run into that situation teaching back-to-back sections of the same course, and I was not convinced that the section with the higher score was the superior one.

* * *

In a few weeks, the OECD will release the results of the 2015 PISA round. I am very interested in seeing if Canada's results are significantly different in a practical sense. My expectation (

*based on no evidence whatsoever*) is that the results will be much the same as before, and will fall within the 65% to 67% range. ( Hey! A Halifax weatherman had a very successful forecasting run by predicting tomorrow's weather to be the same as today's.) I'll do a Trumpian gloat if my prediction is correct, and a Clintonian sob if it's wrong.

### References

[2] 2006 PISA report (Math scores are on page 53)

[3] 2009 PISA report (Math scores are on page 134)

[4] 2012 PISA report (Math scores are on page 5)

[5] PISA 2012 Technical report (Proficiency levels are described on pages 297 - 301)

[6]

*Ten Questions for Mathematics Teachers… and How PISA Can Help Answer Them*, OECD Publishing, Paris (2016).

## No comments:

## Post a Comment