Table of Contents |  Print This Page

Audit Validation | Synopsis From: The Internal Responsibility System In Ontario Mines

  • Revised: September 2009
  • Content last reviewed: June 2009
  • Final Report: The Trial Audit & Recommendations
  • Revised: Steering Committee 23 October 2000 (September 2000; 31 July 2000)
  • Ian M. Plummer, Peter W. Strahlendorf, Michael G. Holliday

To validate the audit tool, we compared responses to groups of questions with the safety record of the mine. The safety statistic we primarily used of this comparison was total medical injury (TMI) frequency. [ 1 ]. In order to make the comparison meaningful, we needed to calculate frequencies over a reasonably long period. Because the safety statistics of one of the six mines only started in 1997, we used the period 1997 to April 2000.

Complete analysis of the vast amount of data collected from the six mines visited in the trial audit would have taken many months more than the time and resources available to this project. Accordingly, in the validation, we chose to concentrate on two facets of the data contained in the questionnaires. One involved a question that probed what respondents believed the IRS should be, and the other looked at groups of questions that have a bearing on various aspects of actual IRS performance.

Individual Perception of the IRS

The first question (Question 1a) on every questionnaire was a multiple-choice question and consisted of 13 separate statements. It asked respondents about what they thought the IRS should be, and was administered to all levels in the responsibility system. Because the same question was asked of everybody, the responses give a very good overview of the current understanding of the IRS at the mines--see Table C.

Table C: Summary of Answers to Question 1a as Percentages
Note: "n=" represents the number of respondents for each position
Multiple choice answers a) to k) Wkr
n=
151
FLS
n=
34
MM
n=
19
Mgr
n=
8
Exec
n=
10
Dir
n=
4
JHSC
n=
26
HSC
n=
7
Insp
n=
8
Avg.
n=
267
a) everyone (from the rockface to the Boardroom) looks after health and safety as part of doing their job; 88 91 100 100 100 100 100 100 100 92
b) the workers are the only ones to look after their own and co-worker safety; 11 6 0 0 0 0 0 0 13 7
c) the supervisors and managers co-operate with the workers to head off and solve problems; 71 94 95 100 80 100 96 100 100 81
d)we all work together to make things better by reducing or eliminating the health and safety risks; 81 97 95 100 100 100 96 100 88 88
e) the Health and Safety Committee is directly responsible to make the workplace safe; 24 15 11 13 20 50 19 0 25 21
f) the Company President has some responsibility for the worker’s health and safety; 67 79 79 100 100 100 81 86 78 75
g)the supervisor only makes sure the workers follow the rulses and procedures 9 3 0 0 0 0 4 0 0 6
h) if the worker cannot fix the problem, the supervisor will help find the answer; 77 91 79 100 80 100 88 86 75 82
i) if the worker cannot fix the problem, the only way out is to refuse to work; 11 3 0 0 0 0 0 0 0 7
j) the Health and Safety Coordinator finds the problems and tells the workers to fix them; 11 6 0 0 0 0 4 0 0 7
k) the Health and Safety Committee are resources; 66 79 79 100 100 100 92 100 88 75
l) the Health and Safety Committee help resolve issues when other ways fail; 68 76 89 50 40 25 77 57 88 70
m) we go beyond fixing problems and try to improve the health and safety in the work processes." 66 79 84 100 90 100 88 100 88 75
"SCORE" 60 80 85 91 82 81 85 91 82 70%

The overall understanding of the IRS at the mines involved in the trial audit, using the scoring scheme in the footnote below, [ 2 ] stood at 70 percent. In general, understanding improves as one moves up the line of direct responsibility within the mine site. Mine workers, as a group, show the lowest level of understanding and mine managers show the highest; understanding decreased somewhat at the corporate level. The health and safety component (JHSC members and health and safety co-ordinators) also showed high levels of understanding. Similar distributions in understanding were also seen when the mines were looked at on an individual basis.

The success of this multiple-choice question in giving a "snapshot" of the level of understanding of the IRS at a mine site became apparent to us early in the trial audit visits. And, because of this, we took the opportunity to combine the multiple-choice question with a supplementary question asking the respondents for an assessment of how the IRS was actually working at their workplace. In this question, respondents were asked to say whether they "strongly disagreed", somewhat disagreed", somewhat agreed", or "strongly agreed" with the following statement: "Given my answer to the Question 1a, I believe the IRS is working effectively in my workplace". [ 3 ] This supplementary question (Question 1b) was used at four of the six mines.

Working on the assumption that individual respondents' assessments of the IRS in their workplaces (i.e., Question 1b) are of greater validity the higher they scored on the understanding part of the question (i.e., Question 1a), we multiplied each individual's score for 1b by their score for 1a. The resulting product is a measure of the effectiveness of the mine's IRS according to that individual. We looked at the effectiveness of the mine's IRS , as judged by mine-workers' responses to Questions 1a and 1b (mine workers were the only group that really contained enough responses to make a comparison valid), and compared them with that mine's safety record. The result of this comparison is shown in Figure A. There is a remarkably good relationship between the effectiveness of a mine's IRS (as measured by worker responses to Questions 1a and 1b) and its safety performance-the higher the score to the Q1a*Q1b combination, the lower the TMI frequency for the mine is likely to be. [ 4 ]

Figure A: The Relationship Between a Mine's Safety Performance and the Effectiveness of Its IRS (as measured by workers' responses to Questions 1a and 1b)

A good relationship between the effectiveness of a mine's IRS (as measured by worker responses to Questions 1a and 1b) and its safety performance is represented in the graph-the higher the score to the Q1a*Q1b combination, the lower the TMI frequency for the mine is likely to be.

Figure B: The Relationship Between a Mine's Safety Performance and Its Average IRS Performance Score

A good correlation between the scores 
for the nine indicators, taken as a whole, and the safety records of the six mines is represented in the graph-the 
higher the score, the lower the TMI frequency.

IRS Performance Indicators

Many of the questions were designed to probe the flow of information and responsibility up and down, and within, the responsibility chain. To investigate relationships with safety statistics, we chose the clusters of questions dealing with indicators of IRS performance. These clusters dealt with the nine indicators of IRS performance shown below and involved all levels of the responsibility system.

  • Beliefs [34] [ 5 ]
  • Responsibility [4]
  • Accountability [8]
  • H&S concerns & responses [16]
  • Initiatives & responses [26]
  • IRS issues & responses [45]
  • Demonstrating leadership [39]
  • Work planning [8]
  • Information flow [6]

To make the comparison with safety records, we arbitrarily took the average of the "scores" of all nine indicators (individual questions were scored in a similar way to those described in the Section above). The result, for the six mines, is shown in Figure B. There is good correlation between the scores for the nine indicators, taken as a whole, and the safety records of the six mines-the higher the score, the lower the TMI frequency.

[ 1: We also looked at lost-time injury (LTI) frequency. But, because LTIs are of considerably smaller magnitude than TMIs, correlations with questionnaire responses were not as obvious. ]

[ 2: SCORE: Each individual's understanding of the IRS was scored by assigning 12.5% to each of a, c, d, f, h, k, l, and m and -20% to each of b, e, g, i, and j, and adding the result. Thus respondents would score 100% if they ticked a, c, d, f, h, k, l, and m, but no others, and 0% if they ticked all 13 alternatives or if they ticked none at all (-100% would be achieved if b, e, g, i, and j, but no others were ticked). In cases where there were negative scores, we assigned 0%. ]

[ 3: SCORE: We assigned 0% to "strongly disagree". 33% to "somewhat disagree", 67% to "somewhat agree", and 100% to "strongly agree". ]

[ 4: The least-squares linear-regression line shown in Figure A, is significant (as measured by the Student's t test) at the 1% level (p = <0.01)-in other words, such a line would be likely to occur by chance only one time out of a hundred. ]

[ 5: The figures in brackets indicate the number of questions in each suite. ]

Previous | Next