[ Previous ] [ Contents ] [ Next ]

A measure of academic effectiveness for secondary schools

The foundation of the Report Card is an overall rating of each school's academic performance. Building on student-results data provided by the Ministry of Education, we rate each school on a scale from zero to 10.

We base our overall rating of each school's academic performance on the students' results in four core academic courses. They are Secondary V level courses in the language of instruction and the second language and Secondary IV level courses in the History of Quebec and Canada and the Physical Sciences. The results used as indicators are:

• average uniform examination mark;
• percentage of uniform examinations failed;
• school level grade inflation;
• difference between the examination results of male and female students in Secondary V level language of instruction courses, and
• difference between the examination results of male and female students in Secondary IV level physical sciences.
• a measure of the likelihood that students enrolled at the school will complete their selected program of studies in a timely manner.

The first five indicators demonstrate the school's success in equipping all its students with the knowledge and skills embodied in the curricula. The last indicator demonstrates the extent to which the school is successful in keeping its students on task and on time as they complete the last two years of their secondary school program.

We have selected this set of indicators because they provide systematic insight into a school's performance. ⁸ Because they are based on annually generated data, we can assess not only each school's performance in a year but also its improvement or deterioration over time.

For each school, under the heading Résultats aux épreuves, the table lists the average raw uniform examination mark achieved by its students in each of the four core courses at the June examination sitting in each school year. For the purposes of determining the trend, if any, over time and the Overall rating out of 10, the average marks for all four courses are combined to produce an overall average mark. Detailed explanations of the method used to calculate trends and the overall ratings are included in the relevant sections below.

Examinations are designed to achieve a distribution of results reflecting the inevitable differences in students' mastery of the course work. Differences among students in interests, abilities, motivation, and work-habits will, of course, have some impact upon the final results. However, there are recognizable differences from school to school within a district in the average results on the provincial uniform examinations. There is also variation within schools in the results obtained in different subject areas. Such differences in outcomes cannot be explained solely by the personal and family characteristics of the student body. It seems reasonable, therefore, to include these average uniform examination marks for each school as one indicator of effective teaching.

During the secondary school years, students must make a number of decisions of considerable significance about their education. They will choose the priority that they will assign to their studies. They will chose among optional courses. They will plan their post-secondary educational or career paths.

Will these young people make good decisions? It is unrealistic to presume that they can do so without advice, encouragement, and support. What practical, well-informed counselling can they call upon? While parents, in the main, are willing to help, many lack the information they need to be able to provide good advice. It falls, therefore, to the schools to shoulder some responsibility for advising students and their parents about these and other educational choices. Of course, wise students will seek guidance not only from the counsellors designated by the schools but also from teachers and administrators, parents and other relatives. Where students have strong support from family and community, the school's responsibility for counselling may be lighter; where students do not have such strong support, the school's role may be more challenging.

One of the most important decisions that students must make is to stay in school and complete their chosen programs of study in a timely manner. This year we have introduced a new indicator--the Promotion rate (noted in the tables as Taux de promotion)--which measures the proportion of students in each school who have decided to continue their studies. While there are factors not related to education--absence or emigration from the school or province, sickness, death, and the like--that can affect the data, there is no reason to expect these factors to influence particular schools systematically. Accordingly, we take variations in the Promotion rate to be an indicator of the extent to which students are being well coached in their educational choices. It is a composite result calculated from two measures.

The first component of the Promotion rate indicator gives credit to schools for the extent to which their students remain in school. While some students may require more time to complete the general program than is normally the case and other students may transfer from the general program into a less rigorous program of study, we believe that, as the minimum, schools should encourage and assist students to finish a program of secondary-school study. This component was determined as follows. First, we calculated the proportion of the school's Secondary IV students who receive their general program diploma at the end of the school year or return in the following year to enroll at level Secondary IV or Secondary V in either of the general or professional programs for school-aged students. Then, we multiplied the result by the proportion of the school's Secondary V students who either receive their diploma at the end of the school year or return in the following year to enroll at level Secondary V in either of the general or professional programs for school-aged students.

The second component of the Promotion rate indicator provides a more rigorous test of the school's ability to ensure that its students stay on task. It was calculated by multiplying the proportion of the general program students at the school who enrolled in Secondary IV at the beginning of the year and either received their general program diploma or were promoted to the Secondary V level at the end of the year by the proportion of the school's students who registered in Secondary V at the beginning of school year and who obtained their diploma in the same school year.

Note that neither of the two components used in the calculation of the Promotion rate indicator is a measure of the results of a single cohort of students. Instead, we calculate the results for an "instant cohort" comprising the Secondary IV and Secondary V students enrolled at the school in the same year. ⁹ Using a real student cohort, for example that of students who began Secondary IV in September of 1998 and were scheduled to receive their diplomas in June of 2000, would not measure the effectiveness of the individual school but that of the school system because the available data reports student achievement within the entire education system as a whole. Thus, students at one school in Secondary IV could receive their diploma at another school in the following years. Which school should get credit for these students' timeliness? A further advantage of the "instant-cohort" method of calculation is that it reflects more accurately the effectiveness of the school in a single school year by taking into account the results for students in both Secondary IV and Secondary V. Thus, the Promotion rate indicator is compatible with the other indicators used in the Report Card. The use of the "instant cohort" follows methodology developed by France's national ministry of education. ¹⁰

Finally, we averaged these two components to calculate the composite Promotion rate.

For each school, this indicator (noted in the tables as Surestimation par l'école (%)) measures the extent to which the average "school" mark--the accumulation of all the results from tests, essays, quizzes and so on given in class--exceeds the average uniform examination mark obtained in the four core courses. Where a school's average examination mark is higher than the average school mark, the school is assigned a zero on this indicator.

Effective teaching includes regular testing of students' knowledge so that they may be aware of their progress. As a systematic policy, inflation of school-awarded grades will be counterproductive. Students who believe they are already successful when they are not will be less likely to invest the extra effort needed to master the course material. In the end, they will be poorer for not having achieved the level of understanding that they could have achieved through additional study.

The effectiveness of school-based assessments can be determined by a comparison to external assessments of the students. The same authority--the Ministry of Education--that designed the courses administers the uniform final examinations. These examinations will test the students' knowledge of the material contained in the courses. If the marks assigned by the school reflect a level of achievement that the student subsequently achieves or exceeds on the uniform examination, then the school has not deceived the student into believing that learning has occurred when it has not. It seems reasonable, therefore, to use this indicator as a second measure of effective teaching.

Effective schools will ensure that all their students are assisted and encouraged to reach their potential regardless of any real or perceived disadvantages resulting from personal or family characteristics. At such schools, teachers will take into account the characteristics of their students when they develop and execute their lesson plans. In doing so, they will reduce the probability that systematic differences in achievement are experienced by sub-populations within the student body.

For each school, this indicator (noted in the tables as Échec (%)) provides the combined rate of failure (as a percentage) on the uniform examinations that form part of the four core courses. It was derived by dividing the sum, for each school, of the uniform examinations written by the students where a failing grade was awarded by the total number of such uniform examinations written by the students of that school.

In part, effective teaching can be measured by the ability of all the students to pass any uniform examination that is a requirement for successful completion of a course. Schools have the responsibility of preparing their students to pass these final examinations.

There is good reason to have confidence in this indicator as a measure of equitable teaching. First, these courses are very important to students regardless of their post-secondary plans. In order to obtain a general program diploma, students must successfully complete two of these courses (language of instruction at the Secondary V level and History of Canada and Quebec at the Secondary IV level). Anglophone students must also successfully complete French as a second language at the Secondary V level. The Secondary IV level Physical Science course is a prerequisite for a variety of CEGEP courses. Second, since each of these courses has prerequisite courses, their successful completion also reflects how well students have been prepared in the lower grades. Since successful completion of the courses is critical for all students and requires demonstrated success in previous courses, it seems reasonable to use the percentage of uniform examinations failed as an indicator of the effectiveness of the school in meeting the needs of all its students. ¹¹

In a study of gender differences in the academic results of British Columbia students, it was found that "there appears to be no compelling evidence that girls and boys should, given effective teaching and counselling, experience differential rates of success." ¹²

However, the data from Quebec's Ministry of Education upon which this study is based provides evidence that there are systematic differences in the results of these groups on the Ministry's uniform final examinations. For example, the 1999/2000 results show that on average female students score about 4 percentage points higher on the language of instruction examinations than male students do, and about 1 percentage point higher on the Secondary IV physical science examinations.

The indicators--Écarte sexes (%): langue maternelle and Écarte sexes (%): sciences physiques--are calculated by determining the difference between the two sexes in the average uniform examination results on each of the courses. ¹³

Where both English and French language of instruction examinations were written at the school, the differences between the two sexes were weight-averaged according to the enrolment in each of the languages.

Why are female students seemingly at an advantage over male students in their relative exam results? Schools with a low gender gap are more successful than are others in helping students of both genders to reach their potential.

While each of the indicators is important, it is almost always the case that any school does better on some indicators than on others. So, just as a teacher must make a decision about a student's overall performance, we need an overall indicator of school performance. Just as teachers combine test scores, homework, and class participation to rate a student, we have combined all the indicators to produce an overall school rating, the Overall rating out of 10--in the tables, Cote globale (sur 10).

To derive this rating, the results for each of the indicators, for each year, were first standardized. Standardization is a statistical procedure whereby sets of raw data with different characteristics are converted into sets of values with "standard" statistical properties. Standardized values can be combined and compared. In the case of the Average examination mark (%) indicator, the standardized scores for all four of the course average marks were first combined and then re-standardized to produce a standardized overall average uniform final examination mark. For all the other indicators, the raw indicator values were standardized directly.

The standardized scores were then weighted and combined to produce an overall standardized score. Note that for 1999/2000, Promotion rate became the sixth indicator to contribute to the calculation of the Overall rating out of 10: in the previous years, only the other five indicators were used. Finally, this overall standardized score was converted into an overall rating. (Explanatory notes on the calculation of the Overall rating out of 10 are contained in Appendix 1.)

The Overall rating out of 10--Cote globale (sur 10)--answers the question, "In general, how is the school doing, academically?" It is from this Overall rating out of 10 that the school's provincial rank and its rank within the administrative region are determined.

[ Previous ] [ Contents ] [ Next ]

info@fraserinstitute.ca You can contact us at the above email address for any comments or information requests. Please report any dead links or technical problems.

If you know someone who would be interested in this web page, please enter their email address below, and we will forward this URL to them: Email Address:

Last Modified: October 20, 2000.