The
Economic Freedom
Network

The Fraser Institute

Report Card on Alberta's High Schools : 2001 Edition:

[Contents] [Next]

Introduction

Parents want better schools. Students want better schools. Teachers, counsellors, principals, superintendents, members of local school boards, and officials from Alberta Learning want better schools. Taxpayers and employers want better schools. While there is agreement about the need for better schools, there is no agreement about how to improve our schools. One thing, however, is certain. Any effective plan to make our schools better will require that we regularly determine whether or not the school is meeting its objectives. We must measure each school's performance.

The Report Card on Alberta's High Schools: 2001 Edition (hereafter, Report Card) collects a variety of relevant, objective indicators of school performance into one easily accessible public document so that all interested parties can analyze and compare the performance of individual schools. In this way, the Report Card encourages and assists those seeking to improve their schools.

The Report Card facilitates school improvement

How will the Report Card lead to better schools? Simply measuring results is certainly no guarantee of improvement. Regular measurement of performance is, however, a necessary component of any plan for improvement. Fifty years ago, Dr. Joseph Juran¹ and others pointed out the role of measurement in building more effective organizations. Juran recommended the adoption of a "quality-spiral" approach to improvement. This approach is simple. The starting point in the spiral is the definition of the school's objectives--what is it supposed to do? Then, historical performance against these objectives is documented so that the school has a benchmark against which to compare improvements. Once the benchmark is established, the school then adopts a short-term goal for improvement; develops a plan designed to achieve that goal; executes the plan; measures the results; revises the goal or plan as required; executes the plan and, thereafter, continues the spiral of action, measurement, and planning toward improvement into the future. It is to the continuous improvement of all Alberta schools that the Report Card is dedicated.

The use of the measurement of results as the basis for improvement is widespread. In many jurisdictions, data relevant to education have become routinely available. For example, the United Kingdom's Department for Education and Employment annually publishes detailed measurements of the performance of primary schools, secondary schools, and colleges, which it distributes widely.² Education authorities in Alberta and other provinces annually release data related to K-12 school performance.³ However, the mere availability of raw data to the public is not sufficient. Our experience in British Columbia, Alberta, and Quebec suggests that action toward improvement is encouraged when clear conclusions are drawn from the data, disseminated broadly, and debated. Education authorities in California and Oregon subscribe to this notion: both have moved beyond simply collecting and publishing performance data. California, under the authority of the Public Schools Accountability Act of 1999, uses an Academic Performance Index⁴ (a single statistic much like the Report Card's Overall rating out of 10) to rate its elementary, middle, and secondary schools. In Oregon, the Department of Education rates each of its public schools from Exceptional to Unacceptable in student performance, student behaviour, and school characteristics. It then uses these three ratings as the basis for an overall rating of school performance.⁵

In Canada, The Fraser Institute introduced the first report card on secondary schools in British Columbia in 1998;⁶ in 1999, it was followed by the Report Card on Alberta's High Schools.⁷ In October 2000, the Fraser Institute and the Montreal Economic Institute published the inaugural edition of the Report Card on Quebec's Secondary Schools⁸ and, last month, the Fraser Institute introduced the inaugural edition of the Report Card on Ontario's Secondary Schools.⁹

First, we must talk

The Report Card will only serve its purpose when its findings are openly discussed among all those with an interest in a school. But, frank, useful discussion can be difficult to initiate. Teachers and school officials sometimes adopt a defensive attitude and see the Report Card as an attack on their ability as professionals. It is not. Teachers, counsellors, and school administrators should be committed to continual professional development and, as every educator knows, feedback is a necessary component of learning. The Report Card provides a variety of relevant, objective feedback.

Educators would perhaps prefer that school performance data not be made public. They may worry that parents do not have the time or the expertise to analyze and interpret such information correctly. Naturally, there are aspects of the Report Card that require interpretation but a broader understanding of school results will undoubtedly follow from discussion and debate among all those concerned with the effectiveness of our schools.

Teachers and principals may fear that parents and taxpayers will understand the results perfectly well and that, as a result, if the school's performance is poor, they will demand change. Dissatisfaction among parents can be a powerful motivator of improvement. Here, in the words of its principal, is what happened at one school in rural British Columbia when it found itself at the bottom of the Report Card ranking:

the fallout or publicity it brought [my school] has allowed me great license in instituting change. For that I thank you (although my thanks is somewhat like a patient thanking a dentist after a painful root canal!!!)

Surely, when teachers, parents, students, administrators, and taxpayers all have easy access to data on school performance and they share the will to discuss it frankly and in detail, Alberta schools and, therefore, the province's students will be the better for it.

Some schools do better than others

The Alberta Report Card, like all the other editions, demonstrates that some schools do better than others. Even when we take into account factors such as the students' family background, which are commonly thought to dictate the degree of success among students, individual school results differ. This finding confirms research results from other countries.¹⁰ Indeed, it will come as no great surprise to experienced parents and educators that the data consistently suggest that what goes on in the schools makes a difference to student success and that some schools make more of a difference than others.

Unfortunately, while educators are eager to trumpet the positive aspects of their school, they are less willing to discuss its shortcomings publicly. The Report Card provides objective results--good and bad--and offers educators an opportunity to accept poor results for what they are--a starting point from which to improve.

Comparisons are at the heart of the improvement process

Many school authorities in Alberta use student report cards that include both the student's result and the median mark for each subject in which the student is enrolled. They also show any previous marks awarded to the student earlier in the year. Historical data like this will show if the student's marks are improving or deteriorating. All such comparative statistics enable students and parents to understand the results shown on the report card better.

Likewise, comparison of results among schools provides a better understanding of the effectiveness of each school. By comparing a school's latest results with those of earlier years, we can see if the school is improving or not. By comparing a school's results with those of neighbouring schools or of schools with similar school and student characteristics, we can identify more successful schools and learn from them. Reference to overall provincial results establishes an individual school's level of achievement in a broader context.

While the Report Card is not about which schools won and which schools lost, there is great benefit in identifying schools that are particularly successful. By studying the proven techniques used in schools where students are successful, less effective schools may find ways to improve. This advantage is not lost on the United Kingdom's Department of Education and Employment. Its Beacon Schools¹¹ program identifies schools across the country that have demonstrated expertise in a wide variety of challenging aspects of the management of schools and the teaching and counselling of their students.

Comparisons are at the heart of improvement and making comparisons between schools is made simpler and more meaningful by the Report Card's indicators, ratings, and rankings.

What should we measure?

While schools in Alberta may serve different student bodies or offer specialized courses or curricula, there are certain basic tasks common to all. The school's teachers should design and execute lesson plans that take into account the differences among students inevitably present in every school. They should ensure that their students master the skills and acquire the knowledge presented in each course. They should develop and use evaluation methods that provide accurate, timely feedback to students and parents about the student's progress. Effective schools will encourage their students to complete their secondary-school studies in a timely manner. They should help their students prepare for a variety of post-secondary opportunities by encouraging them to enroll in relevant, challenging courses. The Report Card presents objective evidence of the extent to which each of the province's schools meet these basic goals.

Our choice of school-performance indicators is dependent on the availability of relevant data. We use only data generated annually and maintained by Alberta Learning so that we can make comparisons from school to school and from year to year.

From these data, for each school, for the five school years 1995/1996 through 1999/2000, we calculated seven indicators of school performance.

1 Average diploma examination mark

This indicator (in the tables Average exam mark) is the average percentage achieved on the uniform final examinations in all of the diploma courses.

2 Percentage of diploma examinations failed

This indicator (in the tables Percentage of exams failed) provides the rate of failure (as a percentage) on the diploma examinations.

3 School versus exam mark difference in diploma courses

For each school, this indicator (in the tables School vs exam mark difference) gives the average of the absolute value of the difference between the average mark obtained on the diploma examinations and the average "school" mark--the accumulation of all the results from tests, essays, quizzes, and so on given in class--for all the diploma courses.

4 English 30 gender gap

5 Mathematics 30 gender gap

The Gender gap indicators measure the difference, if any, in the average English 30 and Mathematics 30 school marks for boys and girls when their respective average examination marks in the same courses are taken into account.

6 Diploma courses taken per student

This indicator (in the tables Courses taken per student) measures the average number of diploma courses completed by the students at a school who completed their third year of high school during the year.

7 Diploma completion rate

This indicator measures the number of first-time grade 12 students who received a diploma in the year.

From these seven indicators, each school's annual Overall rating out of 10 is determined. The overall ratings are intended to answer the important question: "Generally, how is your school doing academically?"

While the seven indicators chosen for the Report Card provide a useful measure of the effectiveness of the academic program at each school, it is likely that the inclusion of additional measures of school effectiveness would make the Report Card even more useful. We plan to add more indicators as relevant data become available and we encourage all interested parties to suggest new measures of school effectiveness that they believe will improve the Report Card.

The Report Card can help parents choose

Where parents can choose among several schools for their children, the Report Card provides a valuable tool for making a decision. Because it makes comparisons easy, the Report Card alerts parents to those nearby schools that appear to have more effective academic programs. Parents can also determine whether or not schools of interest are improving over time. By first studying the Report Card, parents will be better prepared to ask relevant questions when they interview the principal and teachers at the schools under consideration. Of course, the choice of a school should not be made solely on the basis of any one source of information. Nevertheless, the Report Card provides a detailed picture of each school that is not easily available elsewhere.

Taxpayers have a big stake in our schools

Finally, the vast majority of Alberta's students attend schools that are wholly or largely financed by taxpayers. For the school year 2000/2001, Alberta's taxpayers will spend more than three billion dollars to operate and maintain the province's elementary and secondary schools. A public expenditure of such magnitude necessitates continued, independent measurement of the system's results. The measurements should be easily available to any interested taxpayer.

Does the Report Card have an impact?

While it is too early to gauge the effect of the Report Card on the performance of Alberta's schools precisely, there is strong anecdotal evidence that where it has been introduced, its impact has been positive. Response to its publication is evidence of its effect.

A growing number of school administrators and teachers routinely study the document in search of insights into their school's performance. We regularly receive correspondence demonstrating their interest:

Regardless of the changes we have made, the results of your report have made us seriously reflect on the academic issues of this school . . . We are not doing a very good job with our academics. That is fact. The fact your report makes it public has helped us to speed up the process of [improvement] . . . THANKS!

A teacher in the Greater Vancouver area

While many Ministries of Education annually produce data that may be helpful to schools, educators can see the value of the ratings and comparisons included in the Report Card. A district administrator whose schools were not included in the Second Annual Report Card on Alberta's High Schools (2000) asked:

Is there any reason why our school was not included? Is it possible for you to do an analysis of our school and to rank it in comparison to [other] schools and programs? I am getting requests from the principal, parents, and trustees about this and any information from you would be most helpful.

The Report Card encourages educators to use all relevant data about school performance when they make their improvement plans.

Parents are equally interested in the Report Card. Inquiries from parents tend to be of two kinds. First, they seek objective information about the school at which their child is currently enrolled. They use this information as the basis for discussion with school administrators. Often the Report Card is the only source of information with which they are familiar. Second, in choosing a school for their children, they ask for detailed interpretation of the individual school results. Typical of the responses from parents is the following:

I recently read "Grading Alberta Schools" in the Calgary Herald. I appreciated the work that went into this. I feel it's about time we have more tools to evaluate our education system, make improvements and make our administration accountable for the tax dollars they consume.

A parent in Alberta

Overall, the Report Cards are seen as a useful aid to the continuing effort to improve our schools. In October of 2000, we published the first annual Report Card on Quebec's Secondary Schools. Shortly thereafter, a poll of 525 parents and non-parents in Quebec asked for opinions on the merits of a variety of methods for improving the province's schools.¹² More than 75% of respondents considered the Report Card an important innovation.

What is new in this edition of the Report Card?

Several aspects of the Report Card have been improved for this edition. In the past, the ratings of some schools have been affected by the results of students attending continuing education classes and other alternative programs. This year, Alberta Learning provided data that include only the results for day students--these are the students who attend regular daytime classes at the school. As a result, schools can now be more fairly compared. Other changes to the specifications of our data request allowed us to include substantially more schools in this year's Report Card. In this edition, we report results for 276 schools, up from just 222 schools last year.

In addition, we have made a number of improvements to the indicators and to the methods of calculation that make the Report Card more useful.

The Trends indicator tracks changes in school performance over time.

The Report Card provides more than just a snapshot of each school's performance. We now report five years of historical data. Because it is sometimes hard to see if change is occurring simply by scanning the historical data, we have analyzed it to determine whether the school has experienced statistically significant improvement or deterioration on each of the indicators and the Overall rating out of 10. The results of the analysis are reported in the Trends column in the detailed tables.

Improved Gender gap indicators contribute to the Overall rating out of 10

As we discussed in detail in the second edition of the Report Card,¹³ there is widespread concern that, in some schools, boys and girls are not equally successful in academics. Last year we introduced a measure of this "gender gap." This year, we improved the indicator's design and included it in the calculation of the Overall rating out of 10. For the first time, each school's rating will be affected by the extent to which the school ensures that both boys and girls are able to succeed.

A new method of calculating the Overall rating out of 10
and the Trends indicator

This year, we have adopted a different method of calculating the Overall rating out of 10. The raw data is first transformed into "standardized" or "Z" scores. This transformation is a well-accepted statistical method used to make differing sets of data more comparable. For example, by first standardizing the examination marks for all courses taken by students at a school, we can compare a school's most recent average examination mark with its historical results more accurately. This is particularly important as the number of years of data that we report increases.

Since the several changes that we have made to the data and methods produce somewhat different results and to ensure that historical data remain comparable, we have recalculated the ratings for all five years reported. This recalculation also allowed us to reflect the Gender gap in the historical results.

Focus on the results in the most popular diploma courses

In this edition, we report the six diploma courses most frequently taken at each school and the average examination results in these courses. This feature provides readers with course-by-course detail that is not available elsewhere in the Report Card.