Fraser Institute Logo

[Search]
[Media Releases]
[Events]
[Online Publications]
[Order Publications]
[Student]
[Radio]
[National Media Archive]
[Membership]
[Other Resources]
[About Us]


The
Economic Freedom
Network

 

Promoting I/M and the lack of critical assessment

Although I/M programs fall under provincial jurisdiction, Environment Canada has been sponsoring ``clinics'' for testing vehicle emissions for 12 years and actively promoting I/M since at least 1989 (Environment Canada 1997b, 1989). In 1992, the Canadian Council of Ministers of the Environment (CCME) gave Environment Canada the mandate to co-ordinate the development of a national I/M Code of Practice.

Click Here to View Figure 9

It was clearly stated that, for the development of the Code of Practice, ``the three major sources of information for I/M programs at this time [were] the AirCare program, EPA's Enhanced I/M program and California's Smog Check'' (Ballantyne [undated]). There was apparently no concern that the objectivity of the process would be severely limited by relying upon three government agencies who were avid promoters of their own I/M programs. Nor was there any apparent concern that Sierra Research, the same consultant instrumental in designing AirCare, had a short time before produced a major report (1993) that had been highly criticized and rejected by the California I/M Review Committee (1993) for which it had been written (Klein and Saraceni 1994).

Before fully investigating to what extent excess vehicle emissions had a negative effect upon health or the environment in Canada or whether a I/M program would be an effective strategy for reducing emissions, Environment Canada formed the CCME I/M Working Group to develop the ``best'' or ``most effective'' I/M program. Most of the members of the Working Group had a clear vested interest in establishing I/M programs in Canada (see figure 9). Other members of the Working Group had no opportunity to acquire any independent knowledge of I/M programs and therefore had to rely almost entirely on CCME background material for answers to technical and scientific questions. Environment Canada apparently found no need for input from independent scientists who could read technical reports, understand the statistics, and help guide the others.

Formal meetings of the Working Group were held in 1994 to write the original Code of Practice and again in 1997 to make revisions. Environment Canada commissioned two background documents, one for the original Code and the second for the revised code. Both were written by the same consulting firm. Responding to an APA report claiming that the Environment Canada background document for the original code was biased and promoted I/M uncritically, Douglas Cope, the consultant, wrote: ``Of course the draft Background Report, produced by D. Cope Enterprises, favored I/M, this was what my firm was contracted to do!''33

In his letter, Mr Cope wrote that Environment Canada and the CCME had decided in favour of I/M programs before the contract to develop the Code of Practice was let to his firm. The background report confirms that one purpose of developing a Code of Practice was clearly to promote I/M programs: ``The existence of a CCME Code of Practice will also assist provincial agencies in convincing both management and politicians that I/M should be implemented'' (CCME 1993: 2-8).

The manner in which the views and interests of both the non-governmental I/M lobby and various bureaucrats were promoted is indicated by table 5, which categorizes the list of references found in the Code, the revised Code, and the two background documents34 that were made available to working group members by Environment Canada.

Click Here to View Table 5

The US EPA, the CCME, AirCare, and ProtectAir, all recommended or were actively promoting expensive centralized, dynamometer I/M programs; Sierra Research was involved in the design of British Columbia's AirCare Program; suppliers and contractors have a clear (and perfectly legal and reasonable) interest in promoting any I/M program where they can sell their equipment and services. References to American states or other Canadian departments generally related to minor technical details or incidental background material. Almost none of this material has gone through the scientific validation process of peer review.

However, among the 9 ``other'' documents (which include a few newspaper clippings) there are two very interesting reports. The first is a report from the United States General Accounting Office (GAO) that describes a very serious problem:

[T]here are drawbacks to the [EPA's recommended] IM240 test procedure for both the identification and repair of out-of-compliance vehicles that could hamper the test procedure's effectiveness. Our review of EPA data found that over 25 percent of the vehicles that EPA tested using the IM240 test procedure failed an initial emissions test but passed a second emissions test, even though no repairs were made to the vehicles. These results raise questions about whether the IM240 test procedure is reliable in identifying out-of-compliance vehicles and whether inaccurate identification of emission problems could result in unnecessary repairs. (US GAO 1992: 1-2)

This report should have set alarm bells ringing when it was cited in the first background report, not only because it questions the validity of scheduled testing but also because the first CCME Code of Practice strongly recommended using the IM240 test procedure. Instead, this GAO report was not quoted but merely described as having ``suggested there were repeatability problems'' which were possibly ``not the fault of IM240;'' there followed a couple of sentences of exonerating speculation (CCME 1993: 5-17).

The second ``other'' CCME reference is to a peer-reviewed article published in 1987 by the Society of Automotive Engineers (SAE). Harold Haskew, the principal author of this article, is a well-respected expert in vehicle emissions. The article was listed among the references to the first background report but it does not appear to be cited anywhere in the text. It states:

Based on the in-use emission performance of today's [1987] closed loop systems and the cost of inspecting vehicles, it is difficult to support the need for an I/M program. If there are other justifications for an I/M program, a more meaningful approach would be to use a combination of a parameter [i.e., visual] inspection and the on-board diagnostic system. (Haskew et al. 1987)

If the interrogation of a modern vehicle's on-board diagnostic (OBD) system were made the primary feature of a modern emissions reduction program, it could cost less, be more convenient, and provide a better means of identifying vehicles with emissions problems than the dynamometer testing recommend by the CCME. However, this option was never seriously explored in the CCME material.

Environment Canada also overlooked a large number of scientific works that raise doubts about the effectiveness of its recommended I/M program and about I/M in general. For instance,

We compare roadside survey results at I/M and non-I/M locations in California, and observe identical tampering and overall failure rates at the two types of locations. We also show that motorists are taking steps to ``pass the test'' and that the high-emitting vehicles' idle-emissions performance and tampering rates in the roadside surveys are unaffected by the Smog Check test ... EPA's national tampering surveys also show little difference in tampering rates among areas with decentralized, centralized and no I/M programs in the United States. (Lawson et al. 1993: 1567)

Although the CCME strongly recommends the IM240 procedure, it does not accurately reflect reality:

Ironically, the IM240 [recommended by the CCME] is derived from Federal Urban Dynamometer Driving Schedule (FTP), which both the EPA and CARB [California Air Resources Board] have stated does not represent real-world driving and emission patterns.

... there is no solid scientific base to presume success of the enhanced I/M programs in reducing on-road emissions, the goal which they were originally supposed to meet. (Zhang et al. 1996: 1449)

Another study showed that Colorado's IM240 test produced a 12 percent average false high emitter identification rate (14 percent for 1991-1996 vehicles) and a 23 percent average false pass rate (Gallagher et al. 1997: 3-4).

I/M programs evaluated with the US EPA computer model often show impressive emission reductions, but when ambient air measurements are made, or other real-world means of measuring effectiveness such as remote sensing or roadside pull-overs are used, those reductions all but disappear, indicating a flawed computer model.

Unfortunately, our investigation and synthesis leaves us with little confidence in the ability of existing regulatory agency methodologies (including computer based models, and supporting data and analyses) to realistically and reliably assess and forecast performance, compare alternatives and provide a rational guide to policy. (Aroesty 1994: ix)

Ambient air measurements show that Minnesota's centralized I/M program provides no significant reduction of the target pollutant (carbon monoxide) (Scherrer and Kittelson 1994).

Although roadside remote sensing (RRS) has been summarily dismissed in the CCME background document, it is a technology that can cut identification costs by over 90 percent and has many other advantages over conventional I/M techniques.

Use of roadside remote sensing resulted in very low cost identification of high emitting vehicles (HEV). The HEV identification costs for this study were about $100 per vehicle; they would have been as low a $9 per vehicle if had we been able to pull over every passing HEV for repairs35 (Lawson et al. 1996b).

  • On-road remote sensing of CO and HC (VOCs) emissions in California find that only a small percentage (10 to 20 percent) of gross polluters dominate CO and HC hot exhaust emissions.

  • The remote sensor is a highly effective tool for identifying high emitting vehicles on the road, and in also effective at targeting tampered and defective vehicles.

  • A significant finding is that high-emitting vehicles exhibit greater variability in their emissions than clean vehicles, regardless of the test method used. The vehicles most likely to exhibit variable emissions are late-model computer-controlled vehicles that are not deliberately tampered but have broken emission control components. (Stedman et al. 1994; 130,000 measurements)

    Finally, it is clear that the CCME is recommending an antiquated technology:

    Improving the inspection part of I/M, including separating inspection from maintenance facilities, does not hold much promise for widespread effective repairs. Moreover, EPA's inspection programs, either existing or enhanced, will probably not be justified for modern vehicles, in competition with the new identification technologies. (Ross et al. 1995: 39)

    Recent research continues to confirm the sorry record of I/M. A peer-reviewed study of the Colorado ``high-tech'' I/M program showed a CO reduction that was between 4 percent (\xb1 2 percent) and 7 percent (\xb1 2 percent), much less than the 23 percent claimed by the program's administration; reductions in VOCs and NOx were not detectable. The study also provides strong evidence of the ``summer cottage effect'' whereby motorists with vehicles emitting excessive emissions register them outside the I/M area to avoid the test, and suggests that the program might even be increasing NOx emissions (Stedman et al. 1997, 1998).

    I/M programs and the US EPA

    Even in Canada, the US EPA's Office of Mobile Sources is the prime authority on I/M programs. Federal and provincial bureaucrats in Canada rely on the US EPA for I/M program design, computer models, and claims of effectiveness; they have also adopted the EPA's strategy of ignoring all the science that has shown I/M programs to have little or no impact on air quality.

    Before the 1990s, there was very little information about I/M programs except for a number of technical reports--most were no more than drafts--issued by the US EPA that invariably concluded I/M programs to be effective. A review of nearly 30 of the US EPA's technical reports found that every one suffered from one, and often many, very serious flaws (Coninx 1996a):

  • no independent peer (scientific) review

  • no proper citations and references

  • no attention to uncertainty and the degree of confidence that can be placed on results

  • no controls

  • no attention to clear statistical bias

  • very small sample sizes

  • questionable selection criteria or criteria not fully disclosed, which may lead to bias

  • naive, self-serving, or incorrect assumptions and conclusions

  • no attention to potential conflict of interest in evaluation or self-evaluation by contractors

  • inappropriate rejection of results that do not support the concept of I/M programs

  • incomplete and biased summaries and abstracts of the reports.

    Virtually all the US EPA research on I/M programs found in the technical reviews is of such poor quality that it is unlikely that a reputable scientific journal would consider any of them for publication. It is difficult to imagine how any government official who has taken the trouble to read the technical reviews--rather than skimming over the abstracts--could be convinced of the value of I/M programs as an effective strategy for the reduction of emissions.

    In the early 1990s, research on I/M programs from sources other than the US EPA began to be published. These articles showed that existing I/M programs provided little or no environmental benefit (see, for example, Lawson et al.1990, 1993). In 1991, the US EPA's entire strategy of ozone reduction was questioned in a 500-page book published by the US National Research Council (US NRC 1991) and the US General Accounting Office (GAO) began a series of reports that were highly critical of the quality of the science done by the US EPA and of the computer models that the US EPA used to predict the effectiveness of its I/M programs (e.g., US GAO 1992 1994, 1997).

    Instead of encouraging independent research into I/M programs, the US EPA has either ignored it or attacked it. See, for example, the letter that the EPA sent to Congressman Robert K. Dornan of California following a University of Minnesota study that found the Minnesota I/M program to have produced no significant air quality benefit (Scherrer and Kittleson 1994; full response, including the substance of the letter, reproduced in Coninx 1996a). The US EPA attacked the Minnesota study again in its Briefing Book, which is used to promote I/M programs (US EPA 1995).

    In fact, there is little official will to find the truth about the effectiveness of I/M programs. The US EPA, which in 1993 had 706 active contracts worth an estimated total of $13.7 billion (US) (US GAO 1994: 17), claims that it does not have the resources to submit the computer model that it uses to predict and evaluate I/M effectiveness to the standard scientific validation method of peer review even though the GAO lists 14 ``major limitations'' in the EPA computer model (US GAO 1997). Some of the most compelling proof of the computer model's tendency to greatly over-estimate the benefits of I/M programs comes from analyzing the US EPA's own field data, which consisted of data from 44,000 vehicles collected between 1985 and 1992 (Lawson et al. 1996a). The US EPA will not suffer such embarrassment again because they stopped collecting the data after 1992.

    Environment Canada and the ministries of the environment in British Columbia and Ontario have long used the US EPA computer models or slightly modified models to sell I/M programs. The contractor hired to design AirCare II is required to use a MOBILE 5C (the US EPA MOBILE model 5 modified for Canadian vehicles, which were subject to less stringent emission standards before 1988) or similar computer model to determine the impact of AirCare on regional emissions (AirCare 1997: para 1.3.2). A recently published scientific study using two Canadian computer models based on the US EPA's models found that one under-predicted emissions by up to 36 percent, while the other over-predicted emissions by up to 24 percent (Gertler et al. 1997). RAND, the American think tank, expressed little confidence in such computer models and found that an independent inquiry into I/M models and evaluation methods was urgently needed (Aroesty 1994).

    The almost pathetic weakness of the US EPA's claims about the effectiveness of I/M programs is illustrated by the 1995 Congressional testimony of Mary D. Nichols, EPA Assistant Administrator for Air and Radiation and the US EPA's spokeswoman for I/M programs (Nichols 1995). On page 196 of the testimony, Ms. Nichols claims that the EPA has ``not only one study but many studies'' that show that emission reductions as a result of the I/M program in Minnesota are 30 percent for VOCs, 30 percent for CO, and 15 percent for NOx. On the next page, she says that these studies are based on ``actual emissions of automobiles'' and not on a computer model. Ms. Nichols then refers to a study by ``someone named Tiao.'' This was George Tiao who had, in fact, conducted studies on the I/M programs in Oregon and Arizona (Tiao et al. 1982, 1989). The latest was published two years before the start of the Minnesota I/M program and a close review of both studies reveals that neither showed any significant air quality improvement as a result of an I/M program without considerable non-scientific manipulations. Finally, near the bottom of the next page of her testimony, Ms. Nichols admits that she does not have a study that contradicts a University of Minnesota study (Nicols 1995) that showed that the Minnesota I/M program produced no significant improvement in air quality.

    This slippery type of response from I/M stakeholders both inside and outside of government is typical. If I/M programs really work so well, where is the science that confirms this? Instead, I/M stakeholders often quote from reports that were commissioned, and paid for, by the same government agencies that are charged with running and promoting the I/M program. Consultants who write such reports have revealed an uncanny tendency to find the results that please their clients.





     info@fraserinstitute.ca

    You can contact us at the above email address for any comments or information requests. Please report any dead links or technical problems.

     
    If you know someone who would be interested in this web page, please enter their email address below, and we will forward this URL to them:
    Email Address:
    Last Modified: Wednesday, October 20, 1999.