Alberta Secondary Schools Report Card, 2000: A Measure of Academic Effectiveness for High Schools
The foundation of the Second Annual Report Card on Alberta's High Schools is an overall rating of each school's academic performance. Building on student results data provided by Alberta Learning, we rate each school on a scale from zero to 10.
How does the school perform on key academic indicators?
We base our overall rating of each school's academic performance on five indicators:
- Average diploma examination mark
- Percentage of diploma examinations failed
- Difference between the school mark and examination mark in diploma courses
- Diploma courses taken per student
- Diploma completion rate.
We have selected this set of indicators because they provide systematic insight into a school's performance. 7 Because they are based on annually generated data, we can assess not only each school's performance in a year but also its improvement or deterioration over time.
To make the indicators as transparent as possible we have kept manipulation of the data from Alberta Learning to the very minimum required. The process by which the five indicators are developed involves no significant editing of the raw data. Thus, parents, administrators, teachers, or other interested parties can replicate our measures with a minimum of effort.
Three indicators of effective teaching
The indicators of effective teaching are based on a different set of data than that used previously. Whereas the 1999 edition analyzed diploma course results only for students who were enrolled in grade 12 at the time they took the course, the new set of data includes diploma-course results for all students taking the course regardless of the grade level in which they were enrolled. This improvement provides a more complete picture of student achievement in the diploma courses.
In order that the historical course results remain comparable, all the affected indicators have been recalculated using the new data-set. For this reason, there may be slight variations between the historic indicator values in this edition and the corresponding values published last year.
1 Average diploma examination mark
This indicator (noted in the tables as Average exam mark) is the average percentage achieved by a school's students on the uniform final examinations in all of the diploma courses. For each school, the indicator is the average of the mean scores achieved by the school's students in each of the diploma examinations at all sittings during the year, weighted by the relative number of students who completed the course.
Examinations are designed to achieve a distribution of results reflecting the inevitable differences in students' mastery of the course work. Differences among students in interests, abilities, motivation, and work-habits will, of course, have some impact upon the final results. However, there are recognizable differences from school to school within a district in the average results on the diploma examinations. There is also variation within schools in the results obtained in different subject areas (see page 13). Such differences in outcomes cannot be explained solely by the personal and family characteristics of the student body. It seems reasonable, therefore, to include the average examination mark for each school as one indicator of effective teaching.
2 Percentage of diploma examinations failed
For each school, this indicator (noted in the tables as Percentage of exams failed) provides the rate of failure (as a percentage) in the diploma examinations. It was derived by dividing the sum, for each school, of all diploma examinations written where a failing grade was awarded by the total number of course completions by the students of that school.
In part, effective teaching can be measured by the ability of the students to pass any uniform examination that is a requirement for successful completion of a course. Schools have the responsibility of preparing their students to pass these final examinations.
There is good reason to have confidence in this indicator as a measure of effective teaching. In most cases, a student need only successfully complete two diploma courses in order to obtain a diploma. Such a student's course of study may not include the prerequisites for all post-secondary educational options but it will be sufficient to obtain a diploma. Thus, students enrol in the diploma courses, in large measure, because they want to take them. Further, the success in diploma courses reflects to a certain extent how well students have been prepared in the lower grades. All of the diploma courses have prerequisites. Indeed, depending on the school, admission to the diploma course may require that the student have received a prescribed minimum grade in the prerequisite lower-level course. Since the decision to take diploma courses is, for the most part, voluntary and requires demonstrated success in previous courses, it seems reasonable to use the percentage of diploma examinations failed as an additional indicator of the effectiveness of the teaching in high schools.
3 Difference between examination mark and school mark
For each school, this indicator (noted in the tables as School vs exam mark difference) gives the average of the absolute value of the difference between the average mark obtained on the diploma examinations and the average "school" mark--the accumulation of all the results from tests, essays, quizzes, and so on given in class--for all the diploma courses.8
Effective teaching includes regular testing of students' knowledge so that they may be aware of their progress. For such assessment to be useful, it must accurately reflect the student's understanding of the course material. As a systematic policy, inflation of school-awarded grades will be counterproductive. Students who believe they are already successful when they are not will be less likely to invest the extra effort needed to master the course material. In the end, they will be poorer for not having achieved the level of understanding that they could have achieved through additional study. On the other hand, the systematic deflation of grades can work to the detriment of students in those situations where post-secondary admissions and scholarship awards are, in part, based on school assessments.
The effectiveness of school-based assessments can be determined by a comparison to external assessments of the students. Alberta Learning, the same authority that designed the course, administers the uniform diploma examination. This examination tests the students' knowledge of the material contained in the course. If the marks assigned by the school are a reasonably accurate reflection of students' understanding, they should be roughly the same as the mark gained on the diploma examination. Thus, if a school has accurately assessed a student as consistently working at a C+ level, the student's examination result will be at a similar level. If, however, on average a school is consistently granting marks substantially different than those achieved by its students on the final examinations, then the school is not providing an accurate indicator of the extent to which the knowledge of the course material is being acquired.
Two indicators of practical, well-informed counselling
During the high school years, students must make a number of decisions of considerable significance about their education. They will, for instance, annually decide whether to start or continue learning a second language. In grade 10, they are required to choose between different streams in Mathematics, Social Studies, English, and Science. A year or two later, they may face the choice of completing high school or abandoning it in favour of full-time work.
Will these young people make good decisions? It is unrealistic to presume that they can do so without advice. What practical, well-informed counselling can they call upon? While parents, in the main, are willing to help, many lack the information they need to be able to provide good advice. It falls, therefore, to the schools to shoulder some responsibility for advising students and their parents about educational choices.
The final two indicators used in the calculation of the Overall rating out of ten assess the counsel given by the schools by measuring the quality of the decisions taken by the students about their education. Of course, wise students will seek guidance not only from the counsellors designated by the schools but also from teachers and administrators, parents and other relatives. Where students have strong support from family and community, the school's responsibility for counselling may be lighter; where students do not have such strong support, the school's role may be more challenging. These indicators measure the school's success in using the tools at its disposal to help students make good decisions about their education.
There are two very important decisions that senior students must make. First, they must decide whether or not to take a number of academically challenging diploma courses. Second, having made it through school and enrolled in grade 12, they must decide whether to stick it out, do the work, and graduate with their class.
A decision in the negative would be comfortable for a student, especially one who lacks the kind of support that we are trying to measure. Students can quite easily rationalize taking less rigorous courses in grade 12 on the basis that these courses more closely parallel their present interests. Likewise, there are all sorts of reasons that can be advanced for deferring graduation: "The few courses I need can be picked up later." "I'm going to fail anyway, so why try?" "There's a job that pays $15.82 an hour available right now, so I can't afford to stay in school." The list is conveniently long. The decisions to be measured have been chosen because students without well-informed counsel may well select the more comfortable yet perhaps less productive options.
1 Diploma courses taken per student
The method used to calculate this indicator has changed substantially from that used in the last edition. Noted in the tables as Courses taken per student, it measures the average total number of diploma courses completed by those students at the school who completed their third year of high school during the reported year. It is derived by summing the course participation rates provided by Alberta Learning for all the diploma courses taken by students enrolled at that school.
In their senior years, students have freedom to choose among a considerable variety of courses. Their choices will have an impact upon their literacy, numeracy, and analytical skills upon graduation. Their choices also affect the variety of post-secondary options open to them.
Diploma courses offer study at the senior level in a variety of core disciplines: English, French, the sciences, mathematics, and the humanities. Course offerings include alternatives that reflect the post-secondary ambitions of different groups of students. So, far from being courses only for a university-bound elite, these courses teach skills and knowledge that will benefit students no matter what they plan to do after graduation. Further, it is the marks obtained in these courses that are commonly used by post-secondary institutions--institutes of technology and community colleges as well as universities--to assess the applicant's readiness for further study and for admission to programs with limited enrolment. Thus, for most students a decision to take advantage of these courses is a good one and a school that is successful in encouraging students to take these courses shows that it offers practical, well-informed counselling.
2 Diploma completion rate
For each school, this indicator reports the percentage of first-time grade 12 students who received a diploma in the reported school year.
A high-school diploma retains considerable value since it increases options for post-secondary education. Further, graduates from high school who decide to enter the work-force immediately will on average find more job opportunities than those who have not graduated.
By completing the 11 years of schooling in preparation for the final high school year, students have already demonstrated a reasonable ability to handle the basic courses offered by the school. Moreover, for the majority of students, the minimum requirements for receipt of a diploma are not particularly onerous. The chance that students will not earn a diploma solely because they are unable to meet the intellectual demands of the curriculum is, therefore, relatively small.
Nevertheless, the diploma-completion rate varies quite widely from school to school throughout the province. While there are factors not related to education--absence or emigration from the province, sickness, death, and the like--that can affect the data, there is no reason to expect these factors to influence particular schools systematically. Accordingly, we take variations in the graduation rate to be an indicator of the extent to which students are being well coached in their educational choices.
In general, how is the school doing academically? The overall rating
While each of the indicators is important, it is almost always the case that any school does better on some indicators than on others. So, just as a teacher must make a decision about a student's overall performance, we need an overall indicator of school performance. Just as teachers combine test scores, homework, and class participation to rate a student, we have combined all the indicators to produce an overall school rating. Noted in the tables as Overall rating out of ten, it answers the question, "In general, how is the school doing academically?"
To derive this rating, the results for all the years were converted into a score out of 10. This was accomplished using the following procedure. For each indicator, the results from the base-year (1995/96) were sorted from highest to lowest. They were then divided into 10 ranges and each range (decile) was assigned a score between 10 and 1. The range that included the top 10 percent of results was given a 10; the next range, a 9; and so on. The results from each subsequent year were then assigned the number score corresponding to the range of values established in the base-year into which each fell.
The number scores for the five indicators were then averaged to produce the annual overall rating for each school. The decile range tables for each of the indicators are provided in Appendix 2.
info@fraserinstitute.ca
4th Floor, 1770 Burrard Street, Vancouver, BC, Canada, V6J 3G7
Tel: (604) 688-0221 Fax: (604) 688-8539 Book Orders: 1-800-665-3558 ext. 580
You can contact us at the above email address for any comments or information requests. Please report any dead links or technical problems.