NUSAP analysis of VOC emissions from paint

Introduction

Emissions of VOC (Volatile Organic Compounds) from paint in the Netherlands are monitored in the framework of VOC emission reduction policies. The annual emission figure is calculated from a number of inputs: national sales statistics of paint for five different sectors, drafted by an umbrella organization of paint producers; paint import statistics from Statistics Netherlands (lump sum for all imported paint, not differentiated to different paint types); an assumption on the average VOC percentage in imported paint; an assumption on how imported paint is distributed over the five sectors; and expert guesses for paint-related thinner use during application of the paint.
 

Method

A NUSAP-based protocol for the assessment of uncertainty and strength in emission data was developed and used (Risbey et al., 2001), which builds inter alia on the Stanford Protocol (Spetzler and von Holstein, 1975) for expert elicitation of probability density functions to represent quantifiable uncertainty and extends it with a procedure to review and elicit parameter strength, using a pedigree matrix. The expert elicitation systematically makes explicit and utilizes unwritten insights in the heads of experts on the uncertainty in emission data, focusing on limitations, strengths and weaknesses of the available knowledge base.
 
Pedigree conveys an evaluative account of the production process of information, and indicates different aspects of the underpinning of the numbers and scientific status of the knowledge used. Pedigree is expressed by means of a set of pedigree criteria to assess these different aspects. The pedigree criteria used in this case are: proxy, empirical basis, methodological rigor, and validation. Assessment of pedigree involves qualitative expert judgment. To minimize arbitrariness and subjectivity in measuring strength, a pedigree matrix is used to code qualitative expert judgments for each criterion into a discrete numeral scale from 0 (weak) to 4 (strong) with linguistic descriptions (modes) of each level on the scale. The table below presents the pedigree matrix we used in this case study.
 
 
Code
Proxy
Empirical
Method
Validation
4
Exact measure
Large sample direct measurements
Best available practice
Compared with indep. mmts of same variable
3
Good fit or measure
Small sample direct measurements
Reliable method commonly accepted
Compared with indep. mmts of closely related variable
2
Well correlated
Modeled/
derived data
Acceptable method limited consensus on reliability
Compared with mmts not independent
1
Weak correlation
Educated guesses / rule of thumb estimate
Preliminary methods unknown reliability
Weak / indirect validation
0
Not clearly related
Crude speculation
No discernible rigor
No validation
Pedigree matrix for emission monitoring. Note that the columns are independent.
 
The expert elicitation interviews start with an introduction of the task of encoding uncertainty and a discussion of pitfalls and biases associated with expert elicitation (such as motivational bias overconfidence, representativeness, anchoring, bounded rationality, lamp-posting, and implicit assumptions).
 

Results

Pedigree scores for input parameters. The strength-column, averages and normalizes the scores on a scale from 0 to 1.Note: NS=National Sales, Th%=Thinner use during application of paint, (SHI, B&S, DIY, CAR and IND refer to each of the 5 sectors).

 
Next, the expert is asked to indicate strengths and weaknesses in the knowledge base available for each parameter. This starts with an open discussion and then moves to the pedigree criteria that are discussed one by one for each parameter, ending with a score for each criterion (see table).
The protocol is designed to stimulate creative thinking on conceivable sources of error and bias. We identified 5 disputable basic assumptions in the monitoring calculation, and 15 sources of error and 4 conceivable sources of motivational bias in the data production.
In a next step in the interview, the expert is asked to quantify the uncertainty in each parameter as a PDF using a simplified version of the Stanford protocol (see Risbey et al., 2001 for details). We used the PDFs elicited as input for a Monte Carlo analysis to assess propagation of parameter uncertainty and the relative contribution of uncertainty in each parameter to the overall uncertainty in VOC emission from paint. We found that a range of ±15% around the average for total 1998 VOC emission from paint (52 ktonne) captures 95% of the calculated distribution.
 
We further analyzed the uncertainty using a NUSAP Diagnostic Diagram (Fig. 1) to combine results from the sensitivity analysis (relative contribution to variance, Y-axis) and pedigree (strength, X-axis). Note that the strength axis is inverted, left-hand corresponds to a strong and right-hand side to a weak knowledge base.
 
Diagnostic diagram for VOC from paint
 
The Diagnostic Diagram identified uncertainty regarding the assumed VOC percentage of imported paint as the most problematic. Other input quantities in the VOC monitoring calculations whose uncertainty was diagnosed to be 'important' are: assumed percentage of additional thinner use for paint applied in industry, the overlap between the paint import statistics and the national paint sales statistics, and import in volumes below the import statistics reporting threshold. 
 

Conclusions

The diagnostic diagram identified the assumption for VOC percentage of imported paint as the weakest spot in the monitoring of VOC emissions. The results are highly sensitive to this assumption whereas its pedigree strength is low. Other input quantities in the VOC monitoring calculations whose uncertainty was diagnosed to be ‘important’ are: the assumed percentage of additional thinner use for paint applied in industry, the overlap between the CBS paint import number and the VVVF national paint sales numbers, and the import below the reporting threshold for firms to report their imports to the CBS.
Overall, the study shows that NUSAP provides a strong diagnostic tool that yields a richer insight in sources and nature of uncertainty than Monte Carlo analysis alone. The NUSAP elicitation procedure stimulates scrutinization of method and underlying assumptions and effectively facilitates structured creative thinking on sources of error. The combination of quantitative and qualitative aspects of uncertainty in data and parameters is helpful in setting priority for uncertainty management and quality improvement.

Documentation of this case study

J.P. van der Sluijs James Risbey and Jerry Ravetz (2005), Uncertainty Assessment of VOC emissions from Paint in the Netherlands, Environmental Monitoring and Assessment, 105, p. 229-259.

J.P. van der Sluijs, James Risbey and J. Ravtz, Uncertainty Assessment of VOC emissions from Paint in the Netherlands, Department of Science Technology and Society, Utrecht University, 2002, 90 pp.

James Risbey, J.P. van der Sluijs, and J. Ravetz, Protocol for the assessment of uncertainty and strength in emission monitoring, NW&S report