[Previous] [Contents] [Next]

Appendix 2: Why do schools differ in their overall ratings?

The effectiveness of a school probably depends on a variety of factors including the leadership and management skills of the administrators; the expertise and enthusiasm of the teachers and counselors; the physical, financial, and technological resources at the school's disposal, the regulations under which the school operates, and the quality of its curriculum. But since the characteristics of the student body--both individual and family--are not the same from school to school, other factors will likely affect each school's Overall rating out of 10. The abilities, aptitudes, and motivations of its students, the enthusiasm of the parents for education, and the degree to which they participate in their children's school life will also have a role to play. While the currently available data does not allow us to quantify the contribution of all such variables, with each new edition of the Report Card we will continue to improve our estimate of the contribution--or value added--that each of Quebec's secondary schools makes to their students' academic success.

In order to determine the school's contribution, we first compiled a variety of statistics for all of the schools. We determined average student family characteristics using by-postal-code enrollment data and 1996 Census statistics; the proportion of special-needs and late-entry students at the school; and, two school characteristics--sector membership (private or public) and the size of the student enrollment at the school. We analyzed the relationship between these factors and the Overall rating out of 10. We then looked more closely at the relationship of these factors and the indicators that make up the Overall rating out of 10. Finally, we re-examined the apparent affect of school enrollment size on the indicators and the overall rating.

A standard multiple regression was carried out with the Overall rating out of 10 as the dependent variable and four independent variables--average parental employment income; the proportion of late-entry students in the Secondary IV class; school-enrollment size; and school-sector membership. We noted that average parental employment income is strongly correlated with average number of years of education of the most educated parent. While the following analysis uses the former, a similar explanatory model can be built with the latter. The four independent variables are referred to hereafter as INCOME, LATE, NUMBER OF STUDENTS, and SECTOR.

The analysis was carried out using SPSS, version 10.0.0--a statistical software package. After preliminary work, we ran the regression using the natural logs of INCOME and NUMBER Of STUDENTS to reduce dissymmetry and to improve normality, linearity, and the homoscedasticity of the residual variances. The analysis was based on a sample of 462 schools.

Table 1 shows the correlation between the variables, the unstandardized coefficients of regression (B), the standardized coefficients of regression (), the partial correlations (sr2), R2, and adjusted R2.

At first glance (see Column 2 of the table), LATE exhibits the highest correlation (r = 0.693) with the Overall rating out of 10, followed by SECTOR (r = 0.541), INCOME (r = 0.416) and NUMBER OF STUDENTS (r = 0.146). However, when multiple regression is used to more carefully analyze the relationship between the variables, the relative importance of the four variables is found to be somewhat different.

First, note that the regression results (column 6) indicate a statistically significant association between each of the four independent variables and the Overall rating out of 10. The standardized regression coefficients () in column 7 indicate the relative influence of each of the independent variables on the Overall rating. Note that their order of importance now appears to be LATE, SECTOR, NUMBER Of STUDENTS, and INCOME. Lastly, sr2 indicates the unique contribution to R2 of each variable when it is considered as a part of the multivariate model. The order of importance reflected by the sr2 correlations is the same as that indicated by the standardized coefficients .

These four independent variables explain 60% of the variation in the Overall rating among schools. This is a substantial improvement over last year's model (R2 equals 0.39) especially considering that the number of independent variables has been reduced from seven to just four.

The use of the LATE variable is the principal cause of the improvement of the model. By adding a component that seems to address a variety of student characteristics, the model gains significantly in explanatory value. A comparison of this year's regression results with those of last year, suggests that LATE absorbs a good deal of the explanatory power of the income variable. Indeed, the LATE variable may be viewed as the effect of the accumulated history of the students in that it is likely to be influenced by the innate characteristics of the pupils, the socio-economic characteristics of their families, the effect of the schools attended by the students prior to enrollment in Secondary IV, and the students general intellectual, social, and personal development from birth through to enrollment in Secondary IV.

Of course, the strength of LATE makes it difficult to distinguish between socio-economic effects and the effects of individual student characteristics. However, the INCOME variable remains statistically significant even when LATE is controlled for. This suggests that the socio-economic characteristics of student families have a continuing effect during the last two years of secondary school.

The statistical significance of the NUMBER OF STUDENTS variable indicates that the size of the school plays a role in the explanation of variances in Overall rating (see details below.

Lastly, differences in SECTOR--public or private ownership--explain 8.5% of the variation in the Overall rating among schools when school enrollment size and non-school variables are taken into account. This estimate is certainly more precise than that of last year. In addition, as regards school years previous to Secondary IV, the introduction of the LATE variable will, to some degree, control for the effect of student selection by private schools and the non-selective nature of most public school admissions as well as for the effect of self-selection of schools by students and parents.

Can the results on individual indicators also be explained by these variables?

The Overall rating out of 10 is a composite index calculated using values for six indicators. Do the same factors explaining the Overall rating also explain the results for average examination marks, fail rate, school-level grade inflation, and promotion rate? A thorough analysis of these relationships is essential to a better understanding of the Overall rating.

Table 2 summarizes the results of a standard regression on four of these indicators.

As the average examination marks contribute heavily (40% weighting) to the Overall rating out of 10, it is not surprising that similar results--R2 = 0.55 and sr2 = 0.26--are achieved when the independent variables are regressed on average examination marks rather than the Overall rating.

The same is true when the fail rate is substituted as the dependent variable. Income, on the other hand, does not demonstrate a significant association with Fail rate.

Only LATE and SECTOR are significantly associated with Promotion rate (R2 = 0.68).

Because of the truncated nature of the distribution of School level grade inflation (less than half of the schools show evidence of grade inflation), this regression model proves ineffective in explaining the dependent variable.

Thus, we can conclude that three of the four indicators composing the Overall rating are affected by the independent variables in the model in more or less similar ways.

The results in table 2 suggest that the size of the school's enrollment is positively associated with academic performance: that is, the larger the school, the better its performance on the Report Card's indicators. The association is true for average examination marks, fail rate, and school-level grade inflation. It is not the case for the promotion rate. Why would larger schools systematically produce better results? Why is the promotion rate different in this regard?

Table 3 reports the results of a regression on the average exam marks and promotion rates for both public and private schools of the previously used independent variables with the exception, of course, of SECTOR.

Since school size is apparently associated with average exam marks when the regression sample includes all schools, a similar association might be expected when the regression is run separately on private and public schools. Table 3 shows that this is, in fact, the case. However, for private schools, the regression coefficient B is twice the size of the corresponding public-school statistic. Further, the promotion rate at public schools seems not to be associated with school size, while in private schools there is a significant relationship.

While school size is seemingly more closely related to school performance in the private sector than in the public sector, further study is required to confirm this result.

[Previous] [Contents] [Next]

info@fraserinstitute.ca You can contact us at the above email address for any comments or information requests. Please report any dead links or technical problems.

If you know someone who would be interested in this web page, please enter their email address below, and we will forward this URL to them: Email Address:

Last Modified: October 20, 2000.