The
Economic Freedom
Network

The Fraser Institute

Report Card on Alberta's High Schools : 2001 Edition

[Previous] [Contents] [Next]

A measure of academic effectiveness for schools

The foundation of the Report Card is an overall rating of each school's academic performance. Building on data about student results provided by Alberta Learning, we rate each school on a scale from zero to 10. We base our overall rating of each school's academic performance on seven indicators:

average diploma examination mark
percentage of diploma examinations failed
difference between the school mark and examination mark in diploma courses
difference between male and female students in the value of indicator (3) for English 30 only
difference between male and female students in the value of indicator (3) for Mathematics 30 only
diploma courses taken per student
diploma completion rate.

We have selected this set of indicators because they provide systematic insight into a school's performance. Because they are based on annually generated data, we can assess not only each school's performance in a year but also its improvement or deterioration over time.

Three indicators of effective teaching

1 Average diploma examination mark

This indicator (in the tables Average exam mark) is the average percentage achieved by a school's day students on the uniform final examinations in all of the diploma courses.¹⁴ For each school, the indicator is the average of the mean scores achieved by the school's students in each of the diploma examinations at all sittings during the year, weighted by the relative number of students who completed the course.

Examinations are designed to achieve a distribution of results reflecting the differences in students' mastery of the course work. Differences among students in interests, abilities, motivation, and work-habits will inevitably have some impact upon the final results. There are, however, recognizable differences from school to school within a district in the average results on the provincial examinations. There is also variation within schools in the results obtained in different subject areas. Such differences in outcomes cannot be wholly explained by the individual and family characteristics of the school's students. It seems reasonable, therefore, to include the average examination mark for each school as one indicator of effective teaching.

2 Percentage of diploma examinations failed

For each school, this indicator (in the tables Percentage of exams failed) provides the rate of failure (as a percentage) in the diploma examinations. It was derived by dividing the sum, for each school, of all diploma examinations written where a failing grade was awarded by the total number of such examinations written by the students of that school.

In part, effective teaching can be measured by the ability of the students to pass any uniform examination that is a requirement for successful completion of a course. Schools have the responsibility of preparing their students to pass these final examinations.

There is good reason to have confidence in this indicator as a measure of effective teaching. A student need only successfully complete two diploma courses in order to graduate. Such a student's course of study may not include the prerequisites for all post-secondary educational options but it will be sufficient for graduation from high school. Thus, students enroll in the diploma courses, in large measure, because they want to take them. Further, their success in grade 12 reflects to a certain extent how well students have been prepared in the lower grades. All of the diploma courses have prerequisite courses. Indeed, depending on the school, admission to some of the grade 12 courses may require that the student have received a prescribed minimum grade in the prerequisite lower-level course. Since the decision to take diploma courses is, for the most part, voluntary and requires demonstrated success in previous courses, it seems reasonable to use the percentage of examinations failed in these courses as an additional indicator of the effectiveness of the teaching in secondary schools.

3 Difference between school mark and examination mark

For each school, this indicator (in the tables School vs exam mark difference) gives the average of the absolute value of the difference between the average mark obtained on the diploma examinations and the average "school" mark--the accumulation of all the results from tests, essays, quizzes, and so on given in class--for all the diploma courses.¹⁵

Effective teaching includes regular testing so that students may be aware of their progress. For such assessment to be useful, it must reflect the student's understanding of the course accurately. As a systematic policy, inflation of school-awarded grades will be counterproductive. Students who believe they are already successful when they are not will be less likely to invest the extra effort needed to master the course material. In the end, they will be poorer for not having achieved the level of understanding that they could have gained through additional study. On the other hand, the systematic deflation of grades can work to the detriment of students in those situations where post-secondary admissions and scholarship awards are, in part, based on school assessments. Students may also lose interest in a subject when their actual understanding of the material is disparaged by inadequate recognition.

The effectiveness of school-based assessments can be determined by a comparison to external assessments of the students. For each diploma course, Alberta Learning, the same authority that designed the course, administers its uniform examination. This examination will test the students' knowledge of the material contained in the course. If the mark assigned by the school is a reasonably accurate reflection of students' understanding, it should be roughly the same as the mark gained on the diploma examination. Thus, if a school has accurately assessed a student as consistently working at a C+ level, the student's examination result will be at a similar level. If, however, a school is consistently granting marks substantially different from those achieved by its students on the final examinations, then the school is not providing an accurate indicator of the extent to which knowledge of the course material is being acquired.

An indication of consistency in teaching and assessment

The Gender gap indicators

Research¹⁶ has shown that, in British Columbia's secondary schools, there are systematic differences between the academic results achieved by boys and those achieved by girls. These differences are particularly apparent where the local school makes the assessments. These findings are supported by data from Alberta Learning. However, the same research found that "there appears to be no compelling evidence that girls and boys should, given effective teaching and counselling, experience differential rates of success."¹⁷ Further, "[t]he differences described by each indicator vary from school to school over a considerable range of values."¹⁸

The Gender gap indicators measure the difference, if any, in the average Mathematics 30 and English 30 school marks for boys and girls when their respective average examination marks in the same courses are taken into account. For each course, the indicator value is determined according to the formula:

(Female school mark - Female exam mark) - (Male school mark - Male exam mark)

The indicator reports the size of the difference and the more successful sex.

The Gender gap indicators are affected by at least two factors. If the components of the curriculum tested at the school level are different from those tested on the diploma examination, a high gender gap indicates that the favoured sex is, on average, more successful in acquiring the skills and knowledge embodied in those aspects of the curriculum tested at the school level. If the components of the curriculum tested at the school level are the same as those tested on the diploma examination, then a high gender gap indicates that the school-based assessment may be biased in favour of one sex or may include factors in the assessment other than understanding of the curriculum. In either case, schools experiencing high gender gaps should investigate classroom practice to determine why one sex receives better grades than the other.

Two indicators of practical, well-informed counselling

While they are attending secondary school, students must make a number of decisions of considerable significance about their education. They will, for instance, annually decide whether to begin or continue learning a second language. In grade 10, they are required to choose between different streams in several core subject areas. In grade 12, they may face the choice of completing high school or abandoning it in favour of full-time work.

Will these young people make good decisions? It is unrealistic to presume that they can do so without advice. What practical, well-informed counselling can they call upon? While parents, in the main, are willing to help, many lack the information they need to be able to provide good advice. It falls, therefore, to the schools to shoulder some responsibility for advising students and their parents about educational choices.

The final two indicators used in the calculation of the Overall rating out of 10 assess the counsel given by the schools by measuring the quality of the decisions taken by the students about their education. Of course, wise students will seek guidance not only from the counsellors designated by the schools but also from teachers and administrators, parents, and other relatives. Where students have strong support from family and community, the school's responsibility for counselling may be lighter; where students do not have such strong support, the school's role may be more challenging. These indicators measure the school's success in using the tools at its disposal to help students make good decisions about their education.

There are two very important decisions that senior students must make. First, they must decide whether or not to take a number of academically challenging diploma courses. Second, having made it through school to the end of September in grade 12, they must decide whether to stick it out, do the work, and graduate with their class. Effective counselling will encourage students to make appropriate choices.

1 Diploma courses taken per student

This indicator (in the tables Courses taken per student) measures the average number of diploma courses completed by those students at the school who completed their third year of high school during the reported year. It is derived by summing this same statistic for all the diploma courses.

In their senior years, students have freedom to choose from a considerable variety of courses. Their choices will have an impact upon their literacy, numeracy, and analytical skills upon graduation. Their choices also affect the post-secondary options open to them.

Diploma courses offer study at the senior level in a variety of core disciplines: English language arts (or French for francophone students), Mathematics, the sciences, and the humanities. Alberta Learning has developed courses in each discipline that reflect the post-secondary ambitions of different groups of students and, far from being courses only for a university-bound elite, these courses teach skills and knowledge that will benefit students no matter what they plan to do after graduation. Further, it is the marks obtained in these courses that are commonly used by post-secondary institutions--institutes of technology and community colleges as well as universities--to assess the applicant's readiness for further study and for admission to programs with limited enrollment. Thus, for most students a decision to take advantage of these courses is a good one and a school that is successful in encouraging students to take these courses shows that it offers practical, well-informed counselling.

2 Diploma completion rate

This indicator reports the percentage of first-time grade 12 students who received a diploma in the reported school year. It is derived from data provided by Alberta Learning.

Graduation from secondary school retains considerable value since it increases options for post-secondary education. Further, graduates from secondary school who decide to enter the work force immediately will, on average, find more job opportunities than those who have not graduated.

By completing the 11 years of schooling in preparation for the final high-school year, students have already demonstrated a reasonable ability to handle the basic courses offered by the school. Moreover, for the majority of students, the minimum requirements for graduation are not onerous. The chance that students will not graduate solely because they are unable to meet the intellectual demands of the curriculum is, therefore, relatively small.

Nevertheless, the graduation rate varies quite widely from school to school throughout the province. While there are factors not related to education--emigration from the province, sickness, death, and the like--that can affect the data, there is no reason to expect these factors to influence particular schools systematically. Accordingly, we take variations in the graduation rate to be an indicator of the extent to which students are being well coached in their educational choices.

In general, how is the school doing academically?
The Overall rating out of 10

While each of the indicators is important, it is almost always the case that any school does better on some indicators than on others. So, just as a teacher must make a decision about a student's overall performance, we need an overall indicator of school performance (in the tables Overall rating out of 10). Just as teachers combine test scores, homework, and class participation to rate a student, we have combined all the indicators to produce an overall school rating. The overall rating of school performance answers the question, "In general, how is the school doing, academically?"

To derive this rating, the results for each of the indicators, for each of the five years were first standardized. Standardization is a statistical procedure whereby sets of raw data with different characteristics are converted into sets of values with "standard" statistical properties. Standardized values can readily be combined and compared.

The standardized data were then combined as required to produce seven standardized scores--one for each indicator--for each school, for each year. The seven standardized scores were weighted and combined to produce an overall standardized score. Finally, this score was converted into an overall rating out of 10. It is from this Overall rating out of 10 that the school's provincial rank is determined.

For schools where only boys or girls were enrolled, there are, of course, no results for the Gender gap indicators. In these cases the Overall rating is derived using the remaining five indicators. Because no diploma completion data were available for the year, 1995/1996, the overall rating was calculated using the remaining six indicators. (See Appendix 1 for an explanation of the calculation of the Overall rating out of 10.)