[Previous] [Contents] [Next]

Appendix 2: Why do schools differ in their overall ratings?

The effectiveness of a school likely depends on a great many factors such as: the quality of the school's principal, teachers, and counsellors; the capacity and motivation of its students; the physical resources and technology at the disposal of the school; the regulatory environment in which the school operates; curriculum; the enthusiasm for, and participation in, school life displayed by the parents. Each of those factors could explain a part of the observed differences in each school's academic results. Verifying and quantifying the relationship between each of these variables and school performance is beyond the scope of this Report Card. However, since it is often suggested that school rankings are merely the reflection of the wealth accumulated by the families of each school's students, a start must be made to systematically quantify the association between socio-economic variables and school performance. By combining 1996 Census data with enrolment data sorted by postal code and by-school, we have developed a reasonable socio-economic profile of the student families for each of the schools in the Report Card. A start has been made.

First, we studied the relationship between socio-economic factors, certain factors of school organization, and the overall rating out of 10. Next, the same relationships were explored with several of the individual indicators that make up the overall rating. Finally, the effect of school size (total student enrolment) was considered.

Association of socio-economic and institutional characteristics of schools with the overall rating out of 10

A standard multiple regression was performed between the overall rating out of 10 as the dependent variable and seven independent variables--five relevant, student family socio-economic characteristics derived from 1996 Census data and two school organizational variables.

Analysis was performed using Version 10.0.0 of SPSS (www.spss.com). Following preliminary analysis of the results, logarithmic transformation of three variables--APEINC, APGTP and ENROLMENT--was calculated to reduce skewness and improve normality, linearity, and homo­scedasticity of residuals.

Twenty-nine univariate and multivariate out­liers were dropped from the analysis, leaving a sample of N = 449 for subsequent analysis. APEINC was strongly correlated to the average number of years of education of the most educated parent. The following analyses use only APEINC as an independent variable, but either can be used in place of the other with similar effect. Table 1 displays the correlations among the variables, the unstandardized regression coefficients (B), and intercept, the standardized regression coefficients (), the partial correlations (sr²_i), R2, and adjusted R2.

The R statistic for the regression was significantly different from zero. The model explained 39% of the variance and five independent variables demonstrated statistically significant associations with the overall rating.

Organizational variables, AUTHTYPE and LNENROLMENT, are the two most powerful explanatory variables. They account for 26.4% of the total variance in the ratings, a minimum of 68% of the explained variance of the model ((0.206 + .058) / 0.39) and 91 % of the unique variability of the model ((0.264) / 0.29), measured by the sum of their sr² coefficients divided by the total unique variability. In comparison, the three most explanatory socio-economic variables, %PPNKOL, AVGAGEPP and LNAPEINC, the only significant socio-economic variables, cumulated 2.4% of the total variance in the ratings, 6.2% of the explained variance of the model (0.024 / 0.39) and 8.3% of the unique variability of the model ((0.007 + 0.008 + 0.009) / 0.29).

Specifically, under the same socio-economic conditions, private management of schools adds 2.1 to the overall rating out of 10 compared to public management. It should be noted that the concept of management includes a variety of conditions such as Ministry regulation and subsidies (which are different for public and private sectors); the educational policies and teaching practices of the school; and its policy regarding student admission. In particular, because the selection policies of Quebec's secondary schools are not yet documented, we cannot assume that the contribution of the school management factor is solely the result of good management or teaching practices. At present, we must accept that selective admissions policies in any school may have a considerable affect on the overall rating. Nevertheless, school organization and management characteristics must be considered to be independent variables worthy of further study.

The finding that organizational variables apparently contribute more than socio-economic variables to the explanation of overall rating is important. It suggests that variables controlled by the Ministry and by administrators of schools, such as the type of management employed and the size of enrolment, may have a greater influence on the overall rating out of 10 than socio-economic family characteristics such as the family's structure, income, parent age, or language used or understood. It strongly suggests that underachievement by students or schools is not a problem without a solution. Changes in policy and management may improve outcomes. In short, schools matter.

Even family income (measured here by LNAPEINC), the variable that is most often cited as a leading contributor, is far less powerful in unique variability than organizational variables. It should be noted, however, that the Pearson correlation coefficient (.424) between LNAPEINC and AUTHTYPE shows a possibility of an indirect effect of income that a special multiple regression analyzing the parents' choice of school confirms. The probability of choosing a private school is shown to be positively associated with income. This is an indirect effect of income on the overall rating that is not included in the unique variability of the income variable. Income's indirect effect would be confirmed if it was excluded from the model and the result was an increase in the unique variability of AUTHTYPE.

Thus, income may play a greater role in the shared variability than is suggested by this preliminary analysis. However, the unique contribution of AUTHTYPE to the overall rating cannot be overlooked. Even if higher income contributes to more frequent choice of private schools, it is still the characteristics of those schools that provide the best explanation for their performance in the overall rating.

Finally, the age of the principal parent (the mother in two-parent families) and the principal parent's knowledge of official language appear to play a significant role.

Explanation for examination marks, failure rate, grade inflation by schools and graduation rate

The overall rating out of 10 is a composite index of four indicators. Are examination marks (EM), the failure rate (FR), and grade inflation by schools (SVEMD) explained by the same factors as the overall rating? In-depth analysis of these indicators is required for a better understanding of the overall rating. Table 2 shows the condensed results of a standard regression considering three of these variables as well as a graduation rate statistic that we plan to include in the calculation of the overall rating next year. More in-depth analysis will also include the gender-gap indicators.

As the overall rating out of 10 draws heavily (50%) on the examination marks, it is not surprising that one finds similar results in a regression of examination marks as one finds in a regression of the overall rating. A regression of examination results (EM) has almost the same R2 and sr² coefficients. Private management adds more than one standard deviation to the overall standardized examination scores. In this model, only the language variable, %PPNKOL, does not display a statistically significant association with the dependent variable.

A regression of the failure rate produces much the same results.

Probably due to the truncated nature of the distribution of SVEMD (less than half of schools inflate marks), this model is less effective. But the three strongest variables, LNENROLMENT, AUTHTYPE, and LNAPEINC still show significant associations with the dependent variable.

The graduation rate will be an important measure of school performance because it reflects the success of a cohort of students over their secondary-school career. It measures on-going school effectiveness over five years. However, the model shows relatively weaker explanatory power. One explanation for this, of course, is that since no data are as yet available for private schools, we cannot take the AUTHTYPE variable into account. The significance of the exclusion of AUTHTYPE is clear. If it were dropped from the EM model, we would have to subtract .211 (sr² value of AUTHTYPE in the EM model) from the R2 coefficient of .41, not so far from what the GR model obtained.

In conclusion, we can say that three of the four indicators used in the overall rating act in a roughly similar fashion. They are pushed or pulled in the same direction by variations in the same independent variables.

The special issue of school size

Of some importance in the GR model is the negative sign of the significant effect of school size, measured by LNENROLMENT. This variable acts in the opposite direction to that observed in the EM model. Bigger schools in the other models were associated with greater success. That does not seem to be the case when graduation rate data is considered. More study is certainly called for.

Table 3 shows the results of two standard regressions on examination marks for public and private schools.

Because school size plays a significant role in the examination-marks model for all schools, we expected that a similar effect would be observed when subgroups of schools were considered. Table 3 shows that the size of the school's enrolment has an effect in the two subgroups considered here. LNENROLMENT explained 2.6% of the total variance in examination marks in public schools. But the same variable explained 23.4% of the total variance in private schools. It would appear that the strong effect of the school size in the examination-marks regression results reported in table 2 (4.7% of the total variance explained) was dramatically influenced by the much stronger effect of the school size in the private sector.

In the case of public schools, the influence of school size is ambiguous. First, note that the average public school has an enrolment of 931 students versus just 518 students in the average private school (see figure 1). This factor in itself may account for the diminished influence of school size in the public schools. Certainly, much more work can be done to determine the effect of school size on performance.

Why do private schools seem to improve with enrolment size? As we have noted before, much more work can be done with these data.

Key to names of independent variables

%LPAR: % of target families in which there is only one parent that resides in the home

%PPNKOL: % of target families in which the principal parent claims no knowledge of either official language

AVGAGEPP: average age of the principal parent in the target families

APEINC: average parental employment income

LNAPEINC: Logarithm of APEINC

APGTP: Average Parental Governmental Transfer Payment Income

LNAPGTP: Logarithm of APGTP

AUTHTYPE: Public or private school management

ENROLMENT: Total enrolment in the school

LNENROLMENT: the logarithm of ENROLMENT

Table 1 Standard multiple regression of socio-economic and organizational variables on the overall rating out of 10

Table 2 Standard multiple regression of socio-economic and organizational variables on examination marks, failure rate, grade inflation by schools, and graduation rate

Table 3 Standard multiple regression on examination marks for public
and private schools

Figure 1: Distributions of enrolments at public and private schools

[Previous] [Contents] [Next]

info@fraserinstitute.ca You can contact us at the above email address for any comments or information requests. Please report any dead links or technical problems.

If you know someone who would be interested in this web page, please enter their email address below, and we will forward this URL to them: Email Address:

Last Modified: October 20, 2000.