Role of the Radiation Safety Officer: Final Evaluation Report

PDF, 110 pages, 2,6 Mb

Top of Page

Acknowledgements

The Canadian Nuclear Safety Commission (CNSC) would like to acknowledge and thank the following contributors:

The Ottawa Hospital for the valuable assistance that was provided in the planning of this evaluation:

We extend our appreciation to Michèle Légaré, Director and Corporate Radiation and Laser Safety Officer, for her participation as a radiation protection program expert on the Evaluation Advisory Committee. Her contribution was instrumental in helping the project evaluate potential factors that may lead to successful implementation of a radiation protection program for CNSC licensees.

Ms. Légaré organized a day of job shadowing to help CNSC evaluators better understand the role of radiation safety officers. She also hosted evaluators during a radiation safety committee meeting at The Ottawa Hospital. Participation in both of these activities provided valuable insight into the role of radiation safety officers in a health care facility.

Thank you to Ali Soushtarian, Nadia Zaid, Jon Aro, Alex Wolf and Praveen Ratra for sharing information on a variety of topics related to radiation safety such as internal permits, record-keeping practices, policies and overall program management.

Members of the radiation safety officer working group who provided expert advice and guidance to evaluators:

We extend our appreciation to the following individuals who provided valuable feedback on the evaluation planning documents, including the terms of reference, the theory of change, multiple interview guides and surveys:

  • Kate Scheel, Program Manager, Ionizing Radiation, Simon Fraser University
  • Stéphane Jean-François, Certified Health Physicist, Radioprotection Inc.
  • Jeff Dovyak, Radiation Safety Coordinator, Winnipeg Regional Health Authority

External members who provided input and feedback on the evaluation methodology, scope, recommendation and report, and participated in the Evaluation Advisory Committee:

  • John Mayne, Advisor on Public Sector Performance
  • Steve Montague, Partner, Performance Management Network Inc.

Top of Page

Executive summary

Context

The CNSC recognizes the critical importance of radiation safety officers (RSOs) who work for nuclear substances and radiation devices licensees. It also recognizes the significance of recent industry trends. For these reasons, the CNSC undertook this evaluation to analyze the contributing factors that lead to success for RSOs in the medical and academic/research sectors. While the CNSC’s REGDOC‑1.6.1, Licence Application Guide: Nuclear Substances and Radiation Devices, provides some guidance to RSOs, the evaluation sought to understand how infrastructure, institutional, interpersonal and individual Footnote 1 factors can also have an impact.

Results of this evaluation will support the development of a new regulatory document that will provide additional guidance for licensees on the design and implementation of an effective radiation protection program.

This evaluation focuses on nuclear substances and radiation devices RSOs. Class II RSOs are out of scope of this evaluation.

Nuclear substances and radiation devices RSOs

With their expertise and knowledge of radiation protection, RSOs are critical to the implementation of the licensees’ radiation protection programs. RSOs’ contributions help to ensure that licensees are compliant with regulatory requirements, that doses to workers and the public are controlled and that licensees have an enhanced safety and security culture.

Licensees are responsible for putting in place safety and control measures through the implementation of a radiation protection program. The CNSC reviews these measures during its assessment of licence applications and verifies the measures through compliance activities. Through these oversight activities, the CNSC closely monitors trends in the licensees’ operational structures and performance.

During the evaluation, due consideration was given, through 11 case studies, to the different contexts in which RSOs operate. In the medical sector, operational contexts can be vastly different. This sector can include a wide variety of licensees – from large amalgamated hospitals, with complex operation and multiple sites, to small community hospitals with only one site. While RSOs working in the academic/research sector may also have complex operations, they do not face the same pressures as RSOs in the medical sector. In general, significant budget constraints and the important focus on patient care add an additional layer of complexity to the role of RSOs in the medical sector.

Methodology

The evaluation was conducted in 2017–18 and used standard evaluation and research methodology, Treasury Board Secretariat (TBS) standards and policies, and the good practices of the Canadian federal government.

To answer the evaluation questions, a mixed-methods approach was used, blending various data collection strategies. The rationale in using this approach is that the assessment of RSOs’ success factors and their influence on the radiation protection programs is mostly qualitative and relies, for the most part, on the judgment and experience of the stakeholders.

Evaluation results are based on information collected through:

  • document and literature review
  • key informant interviews
  • online surveys
  • case studies to examine the role of RSOs who work for specific licensees operating in different contexts and with different compliance records
  • comparative analysis to explore the effectiveness of the role of the RSO when compared to the role of nuclear substances and radiation devices RSOs internationally (United States and United Kingdom) and other similar roles nationally such as:
    • biosafety officers
    • flight safety officers
    • occupational health and safety officers

The evaluation also relied on data analysis spanning a 10-year time frame (2007–17). CNSC staff provided over 40,000 rows of data to the evaluation team in support of this analysis, extracting the data from three sources:

  • the Licence Operating Users Integrated System (LOUIS)
  • administrative monetary penalty (AMP) notices of violation
  • the Event Information Tracking System (EITS)

Recommendations

The following recommendations are proposed:

  • 1.The CNSC’s Directorate of Nuclear Substance Regulation (DNSR) should provide regulatory guidance to RSOs with respect to:
    • the characteristics of a successful internal audit/inspection program (frequency, format, timing, templates)
    • adequate RSO resourcing levels based on the nature and magnitude of the licensed activity
    • the content of RSO work descriptions based on the nature and magnitude of the licensed activity
    • the appropriate composition and purpose of radiation safety committees given the different contexts in which RSOs operate
  • 2. DNSR should enhance its existing compliance promotion strategy to support RSOs with their continuous improvement activities.

Observations

DNSR should consider, in collaboration with external partners:

  • further promoting the nuclear substances and radiation devices mentorship opportunities for RSOs that are currently described on the CNSC’s external website
  • exploring the possibility of supporting the development of an RSO training program

Top of Page

Introduction

This report presents the findings, conclusions and recommendations from the evaluation of the role of nuclear substances and radiation devices radiation safety officers (RSOs) from the medical and academic/research sectors. Undertaken by the Canadian Nuclear Safety Commission (CNSC) between July 2017 and July 2018, it looks primarily at the contribution of the RSO to the effectiveness of the radiation protection program.

The work was conducted in accordance with the Treasury Board of Canada Secretariat’s Policy on Results and Standards on Evaluation and employed a mixed-methods approach to collect both qualitative and quantitative evidence.

Report structure

This evaluation report has been divided into chapters to allow the reader to select the level of information that meets their needs. While the report can be read as a comprehensive piece on the evaluation, each chapter offers different levels of information.

  • Chapter 1 can be read as a standalone report. It provides an essential summary of the evaluation and offers the reader an overview of the evaluation purpose, consultation strategy and key findings. Where generalizations are made on effectiveness, these are based on triangulation of evidence (presented in chapter 2).
  • Chapter 2 is written from an evaluator’s perspective. It presents the data behind the logic and offers the reader the supporting evidence from which the findings, conclusions and recommendations have been drawn. Chapter 2 summarizes the data from six technical reports that form the basis of the evidence collected.
  • Chapter 3 outlines the detailed recommendations and observations.

Purpose of the evaluation

The need for this evaluation was identified by the Director General of the Directorate of Nuclear Substance Regulation (DNSR) in response to performance trends in the industry, and in DNSR’s presentation of the Annual Regulatory Oversight Report on the Use of Nuclear Substances in Canada: 2015 to the Commission.

The Commission asked CNSC staff to evaluate and recommend a regulatory approach for nuclear substances and radiation devices RSOs in the medical and academic/research sectors, because regulatory oversight identified weaknesses in implementation of radiation protection programs with some licensees undergoing transitions.

The evaluation results are intended to support the development of regulatory guidance for licensees on the design and implementation of effective radiation protection programs.

Program description

Radiation protection programs are required for every licensee to ensure that contamination levels and radiation doses received by workers are monitored, controlled and maintained below regulatory dose limits, and kept as low as reasonably achievable (ALARA). In general, these programs ensure that licensees are in compliance with regulatory requirements. All licensees are required to designate a minimum of one person who is responsible for implementing the radiation protection program. This person is named the radiation safety officer (RSO). The purpose of having a person dedicated to radiation protection within the licensee’s organization is to ensure effective implementation of the radiation protection program and the overall safety performance of the licensee.

REGDOC-1.6.1,Licence Application Guide: Nuclear Substances and Radiation Devices,provides guidance on qualifications and duties of RSOs. Some of these responsibilities, as listed in appendix C of REGDOC-1.6.1, indicate that RSOs may need to:

  • assess the qualifications and competence of workers who will use nuclear substances and radiation devices to determine whether they can do so safely and in compliance with regulations and the licence
  • ensure that workers who are required to use nuclear substances and radiation devices are adequately trained in radiation safety and radiation protection procedures
  • authorize qualified workers to use nuclear substances or operate radiation devices
  • ensure that workers whose duties may occasionally expose them to nuclear substances and radiation devices, such as cleaners, administration or other support staff, receive appropriate training in radiation safety
  • communicate with all workers and management
  • monitor, advise and consult on issues related to the handling of nuclear substances and radiation devices in accordance with regulations and licence conditions
  • review requests for authorization to purchase or use nuclear substances and radiation devices in order to ensure that the radioactive material and the proposed handling and location of storage are acceptable and comply with the regulations and licence requirements

Designating RSOs

The designation of RSOs for nuclear substances and radiation devices licences is the responsibility of the person accountable for the management and control of the licensed activity. Known as the applicant authority, this person is generally a member of senior management within the organization with sufficient delegated authority to direct human and financial resources to address any issue of non-compliance identified by the CNSC.

Certification

Nuclear substances and radiation devices RSOs are not required to be certified by an independent body or by the regulator. They are required to maintain sufficient knowledge related to the proposed licensed activities and the applicable regulatory requirements to enable the licensee to effectively manage activities with nuclear substances and radiation devices in accordance with the terms of the CNSC licence. The RSOs’ qualifications are included in the licence applications. A CNSC licensing officer will assess, as part of the regulator’s review of the application, if the RSO listed in it has sufficient knowledge and expertise with regard to the applicant’s proposed activities. This assessment will form part of the CNSC’s recommendation on the application.

Corporate and site RSOs

Depending on the complexity of the licensee’s radiation protection program, a licensee may designate a corporate RSO to oversee the management of the program at the corporate level. Site RSOs report to the corporate RSO and are responsible for overseeing the program at the site to ensure that the corporate program is effectively implemented at the local level.

Depending on the structure and size of the organization, it may be necessary to designate several site RSOs for the same location. The authority, role, responsibility and qualifications of each site RSO must be clearly defined and communicated throughout the organization.

Top of Page

Chapter 1 – Results of the evaluation

Setting the stage

Through collaboration with DNSR and RSOs across Canada, the CNSC’s program evaluation team launched the evaluation of the role of RSOs in 2017.

The goal of this project was to gather evidence for DNSR and the Commission on the key factors that impact RSOs’ success in their roles. The work took into consideration the variety of licensee operations such as level of complexity of operations, differences in organization size and the geographic disparity of sites. The findings will support the development of a new regulatory document that will provide additional guidance for licensees on the design and implementation of an effective radiation protection program.

Sectors

Five sectors of nuclear substances and radiation devices

Nuclear substances and radiation devices are used in a broad range of applications in Canada. As a result, the CNSC has grouped them by five sectors, according to their primary uses, which are: medical, academic/research, industrial, waste nuclear substance and commercial.

In the past decade, a number of licensees in these sectors have implemented significant changes in the way they organize and conduct business. In the public sector, several provinces, including Alberta, Quebec and Nova Scotia, have transitioned to provincial or regional governance structures for health facilities. The objective of these transitions is to improve coordination of the delivery of health services, reduce administrative overhead and leverage cost efficiencies.

This project focused on two sectors – medical and academic/research. DNSR selected these sectors because they have more complex radiation protection programs and share similarities in RSO functioning. They are also representative of a changing landscape, with a number of amalgamations having taken place in recent years. Approximately 380 RSOs work in these two sectors. Through early engagement, several RSOs have demonstrated an interest in supporting the CNSC in better understanding the impact of the RSOs’ role on licensees’ radiation protection program performance.

The project included two licensees from the commercial sector. This provided some idea of how the results of the project could be applied to this sector.

It is important to mention that CNSC licensing and compliance activities have not identified a systemic problem that would indicate an issue with the qualifications or suitability of RSOs as a whole. However, given the critical importance of RSOs to radiation safety, this evaluation is an opportunity to review the CNSC’s oversight strategy, in order to identify opportunities to more proactively influence licensees’ radiation safety performance.

Engagement strategy

From January to March 2018, the evaluation team reached out to a wide range of people directly involved in radiation protection programs in both hospitals and universities. The evaluators gathered qualitative data through interviews and online surveys. They conducted over 90 in-person interviews and visited 11 licensees. They supplemented these sources of information with licensee compliance data, a document and literature review, and comparative analysis research. They analyzed data by triangulating information gathered from these different methods.

While the primary stakeholders were RSOs, the team also consulted other stakeholders, including:

  • nuclear medicine technologists
  • chiefs of nuclear medicine
  • cardiologist technologists
  • graduate students
  • housekeeping and security personnel
  • physicians
  • professors and associate professors
  • veterinarians and veterinary technologists
  • senior managers

The ultimate goal of the engagement phase was to collect as much information as possible on the factors that help RSOs succeed in their role, given the different contexts in which they operate. Throughout this phase, the CNSC evaluation team was impressed by the high level of stakeholder participation and contribution to the project.

RSOs’ four areas of activity

RSOs have a wide range of responsibilities. Working with the RSO community, the evaluation team developed a straightforward approach that grouped their work into four main areas. These four areas reflect the core activities that form the daily work of RSOs. This structure allowed for the identification of factors that impact the success of RSOs for each activity area, which are described below.

Area 1 – Training activities

RSOs deliver training to ensure that workers adopt safe work practices. Included in the training activities are the design and delivery of the course content, as well as the evaluation, update and maintenance of the training content to ensure that it is up to date and relevant. RSOs offer training in many different ways – classroom-style, one-on-one or through an online platform.

Area 2 – Communication and advice activities

A large part of the RSOs’ work involves sharing performance information with management and providing timely information and feedback to workers. RSOs also act as the main point of contact with the CNSC and have exchanges on regulatory issues with their licensing officers.

Area 3 – Monitoring, controlling and reporting activities

These activities consist of ensuring that nuclear substances and radiation devices are handled in accordance with regulations and licence requirements. They can encompass designating nuclear energy workers (NEWs), developing a security plan, designing monitoring programs, monitoring occupational radiation doses received by workers, conducting inspection activities, and much more.

Area 4 – Continuous improvement activities

RSOs undertake these activities to ensure that they are aware of industry good practices that can be applied to their own radiation protection program. This may include reviewing changes in regulatory requirements, monitoring reportable events in Canada and internationally, or attending conferences and training.

Other influencing elements

Limiting the scope of this project to these four areas of activity would not provide a comprehensive picture of the RSOs’ contribution to radiation protection programs. For a more complete appreciation of the other elements that directly impact the RSO, the team explored other themes. These included the following.

  • Governance – Are the right governance models in place?
  • Certification – Are there advantages to considering certification for nuclear substances and radiation devices RSOs?
  • Alignment across regulatory bodies – Is there an impact to alignment or lack of alignment across regulatory bodies?
  • Relationship with the CNSC – Does the relationship with the regulator allow for open communication and effective resolution of non-compliance?

The RSOs’ influence on stakeholder behaviour

This evaluation focused on understanding how the RSOs’ efforts in each of the four areas of activity contribute to the effectiveness of radiation protection programs. That meant making a distinction between the RSOs’ core activities and radiation protection programs as a whole.

To help with this, the evaluation team adopted the COM-B model Footnote 2, a recognized tool in evaluation, to guide the team’s understanding of behaviour. This model allowed the team to look more closely at the conditions that should exist for people to change their behaviour. In simple terms, the COM-B model looks at the following three elements necessary for behaviour change to occur.

  • Capabilities – Do people have the right knowledge and skills?
  • Opportunity – What outside factors help or hinder the desired behaviour changes to occur?
  • Motivation – How engaged, motivated and committed are people?

For example, it’s possible to think that worker participation in a radiation protection training session will lead to a greater understanding of the topic. But for the training to be understood, a number of different factors or conditions need to be present, such as timeliness of its delivery and suitability of the content for the specific audience.

In addition to considering COM-B elements that contribute to behaviour change, the team also considered conditions and factors that can either help or hinder realization of the desired change. Based on the work of Pawson and Tilley (2004), these factors are grouped under four categories called the 4Is: infrastructure, institutional, interpersonal and individual (appendix D). The evaluation looked at the presence or absence of these factors and how they impact the RSOs’ ability to positively influence the performance of the radiation protection programs.

What the evaluation team found

Overall, RSOs in both sectors had the capabilities, opportunity and motivation to deliver effective radiation protection training, disseminate essential information and conduct thorough monitoring and oversight activities. When it came to continuous improvement activities, the degree of access RSOs have to radiation protection good practices and lessons learned is varied. This is due to barriers such as lack of time to attend learning events, lack of funding to attend learning events, lack of management support and lack of adequate access to online resources.

Keeping in mind the wide variety of licensee operations and differences in organizational size, structure and complexity, not all RSOs are equally successful in all four core areas of activity. When the team compared the medical and academic/research sectors, RSOs in the medical sector appeared to face more obstacles than their counterparts in the academic/research sector. This is primarily due to factors such as budget constraints, complexity of operations, and the nature of the RSOs’ work in the medical sector – that is, the need to manage the additional variable of patient care.

RSOs in large institutions in the medical sector face unique challenges. These can be compounded if they lack sufficient time to fulfill their RSO-related duties. They may lack time because they are part-time RSOs managing complex radiation protection programs, the operations are spread across different sites, or direct patient care takes precedence over their RSO duties.

The evaluation team heard from both new and experienced RSOs, in both small and large organizations, that additional regulatory guidance from the CNSC would be beneficial. Several RSOs are looking for additional guidance to support their internal inspection activities. Specifically, they feel that it would be valuable to have templates to help them conduct effective internal inspections, as well as guidance on the frequency and format.

Twitter and facebook logos

Many RSOs put forward ideas such as how to leverage social media to provide just-in-time regulatory information. Others suggested ways the CNSC could enhance its online resources to provide more detailed and practical regulatory information and guidance.

The team noticed that in large hospitals, budget constraints and travel freezes can limit the RSOs’ ability to participate in learning activities. Many RSOs shared their thoughts on how the CNSC could increase its use of technology to facilitate virtual participation in some learning activities.

The need for further regulatory guidance stood out as a recurring theme throughout the engagement with RSOs; therefore, the evaluation recommendations focus on this area.

Training activities

In most cases, the training activities delivered by RSOs are effective in helping workers adopt safe practices. There was no evidence of a wide-ranging problem in the RSOs’ ability to deliver radiation protection training to those who need it.

However, case study evidence identified situations where some RSOs face challenges. For instance, it can be more difficult for part-time RSOs in large hospitals with multiple locations to have enough time to handle all the administrative duties of the training program (identifying the workers who need to be trained, maintaining the content, tracking attendance and delivering the training in a timely manner). A challenge that RSOs working in the commercial sector can face is finding the right time to train shift workers. This is because the workers face operational production pressures and are not able to take time during their shift to attend training. As a result, workers may need to either come in early before a shift or stay late after a shift to attend training.

The evaluation team found that training offers participants a solid understanding of effective radiation protection practices and procedures, particularly if it is timely and tailored to the audience. RSOs value online learning management systems to help alleviate the time required to deliver in-class or refresher training. Those currently without an online learning platform are seeking to adopt one, as they recognize it would be a valuable time saver. Workers were of the opinion that more opportunities to learn through hands-on exercises, such as practising to clean up a mock spill, were needed.

83% of the RSOs feel that nuclear energy workers demonstrate strong motivation and engagement in the learning. They tend to ask questions and show active engagement during the training. Likewise, auxiliary staff also demonstrate good engagement levels in learning, even though their level of involvement in the radiation protection program is indirect.

Most workers feel a sense of responsibility, are motivated and adopt safe radiation protection practices, but it is not always as a direct result of the training. It can result from the organization’s safety culture or from the sense of the possibility of regulatory consequences.

The overall health of an organization’s safety culture also plays a key role in ensuring that safe practices are adopted. For instance, the team often heard that workers keep each other accountable by speaking up when they witness a potentially unsafe work practice.

In terms of the impact of regulatory consequences, RSOs often told the evaluation team that the threat of a potential negative impact to a licence is the primary factor that motivates workers to adopt safe work practices.

Capability Trainees understand practices and procedures
Opportunity They have time to attend training
Motivation They are engaged in learning
COM-B findings

From the COM-B perspective, workers understand radiation protection practices as a result of the training they receive. This understanding is generally attributed to the fact that the majority of RSOs tailor the radiation protection training to the audience. For example, RSOs might offer a more simplified level of training to auxiliary workers using pictograms, particularly if there is a language barrier because English is the workers’ second or third language. More in-depth training accompanied by a comprehensive training manual would be offered to nuclear energy workers and non-nuclear energy workers.

94% of the workers are provided with sufficient time away from their work to attend training. The evaluation team heard that it can be challenging to find the right time to provide training to shift workers in the commercial sector.

The team found that workers were engaged in learning about radiation protection and that the RSOs’ views mostly align with this opinion. Even though auxiliary workers may only be peripherally impacted, they understand the training and are engaged in it.

Key factors

The factors that impact the success of RSO training activities and should be considered in the development of regulatory guidance are outlined below.

Factors that support success Factors that limit success
  • timely training delivery (individual/interpersonal)
  • hands-on, practical training components (individual/interpersonal)
  • audience-specific training content (individual/interpersonal)
  • strong safety culture, specifically pressure from peers to work safely (interpersonal)
  • the potential of negative regulatory consequences (infrastructure)
  • lack of time allocated to the RSO function to ensure that training content is up to date and approach is meaningful to learners (institutional)
  • competing operational priorities such as number of patients to see in a day (institutional)

Communication and advice activities

RSOs in the medical and academic/research sectors are effective in their communication and advice activities. The majority of RSOs in both sectors appear to have the necessary skills, time and ability to communicate and engage with program stakeholders, and these efforts contribute to workers adopting safe radiation protection practices.

The evaluation team identified some situations in the medical sector where workers did not value the RSOs’ communications. In each case, the reason was because the workers did not view the RSO as a credible authority on radiation protection, due to the perception that the RSO lacked experience either in the licensed activity or as an RSO.

The evaluation team found no evidence of systemic problems in the RSOs’ ability to transmit program information to workers, their management or the CNSC. In a few cases, RSOs working in large organizations with multiple sites told the team that they would like to have more time to proactively disseminate information. As it stands, given their workload, their communications tend to be reactive, rather than proactive.

One of the keys to successful communication is the physical proximity of the RSO to the workers. More frequent interactions naturally take place when RSOs are located in the same department, on the same floor or in the same unit as the workers. As a result, workers are more inclined to ask the RSO questions because they see each other on a daily basis. The team observed that corporate RSOs in large organizations in the medical sector rely on the site RSOs at the other locations to act as the main point of contact for workers.

Communication between RSOs and the applicant authority is generally effective, even though it may be very infrequent. 89% of the applicant authorities report receiving advice from the RSO on their organization’s radiation protection program. In small hospitals, RSOs appear to interact with management more frequently than do RSOs in larger organizations. In large or amalgamated organizations, RSOs may speak to the applicant authority monthly, once a year or less often.

The evaluation team found no evidence that either frequent or infrequent communication had a direct impact on the overall success of a radiation protection program. The team heard from the vast majority of RSOs that, as long as no compliance issues arise, the applicant authority is generally not concerned with the details of the radiation protection program. 67% of the applicant authorities read the radiation safety committee meeting minutes they receive from the RSO, and this appears to be sufficient to meet their information needs.

Capability Stakeholders understand the broad program objectives
Opportunity They have access to the RSO
Motivation They acknowledge and value RSO communication
COM-B findings

From the COM-B perspective, the evaluation team found that the information the RSOs disseminate helps workers and management better understand the radiation protection program objectives. The team found no evidence of information gaps or any overall lack of knowledge either from workers or management on the radiation protection program.

In the vast majority of cases, workers are able to receive information and advice from RSOs and feel comfortable reaching out in person, via email or by phone.

Workers and management value information disseminated by RSOs. There are exceptions in the medical sector, where some RSOs are not seen as credible authorities on radiation protection. As a result, workers rely on colleagues, rather than the RSO, for advice. The team also found that management can become more heavily involved in the day-to-day management of the program if they lack confidence in the credibility or authority of the RSO.

Key factors

The factors that impact the success of RSO communication activities and should be considered in the development of regulatory guidance are outlined below.

Factors that support success Factors that limit success
  • the ease of access of the RSO to workers to provide just-in-time advice and information (individual and interpersonal)
  • lack of time allocated to the RSO function to proactively share information with stakeholders (institutional)
  • competing operational priorities such as number of patients to see in a day (institutional)
  • RSO not perceived as a credible authority of radiation protection (individual/interpersonal)

Monitoring, controlling and reporting activities

RSOs in both sectors are generally able to conduct effective monitoring, controlling and reporting activities. As a result, workers make adjustments to their work when needed and licensee management obtains useful performance information.

There are some differences within the medical sector. Full-time RSOs have sufficient time for these activities because they do not juggle as many other tasks such as patient care. They have sufficient time to conduct internal audits, disseminate the audit findings, prepare reports for the CNSC and maintain internal records. Part-time RSOs working in large and small hospitals mostly have enough time for these activities, but it is at the expense of other work. In other words, it’s always a balancing act for them to fulfill their administrative duties while also tending to patient care activities.

One of the keys to a successful internal audit program is timely follow-ups on audit findings. These follow-ups are essential because they ensure that changes are implemented. Part-time RSOs in large hospitals with multiple sites have time to conduct internal audit activities, but they lack sufficient time to verify that the audit findings and recommended changes are implemented. As a result, they may see issues recur over time.

90% of the workers in both sectors, regardless of organization size, receive clear feedback from RSOs on their work performance. This feedback is usually shared in person, via email or through the employee’s line manager.

When RSOs identify areas of non-compliance through internal audit activities, they feel that they are successful in enforcing radiation protection practices. RSOs who are less successful feel that barriers such as worker complacency and the perceived lack of RSO credibility prevent them from achieving success.

RSOs resolve worker non-compliance (such as lack of contamination control) through open dialogue with the worker. If required, the issue is escalated to the employee’s manager. RSOs consistently viewed that this informal enforcement strategy was effective.

Capability Stakeholders increases their knowledge of program performance and gain awareness of their work performance
Opportunity They have time to review the information
Motivation They want to improve the program performance
COM-B findings

From the COM-B perspective, the evaluation team found that the monitoring, controlling and reporting activities provide stakeholders with increased knowledge of program performance. 89% of the applicant authorities receive the information they need to assess the effectiveness of their radiation protection program. The CNSC receives the required reports such as the annual compliance reports on time, and data analysis indicates that the level of compliance with this requirement has been trending upward and reached a level above 80% in 2017.

Workers receive information that helps them improve their work performance and lower their doses. Not all RSOs feel that they have enough time to complete these activities to the extent they deserve, particularly part-time RSOs in large hospitals.

The evaluation team did not identify any issues with stakeholders lacking sufficient time to review the information they receive from RSOs. Management has time to review the radiation safety committee meeting minutes they receive. Workers told the team that they receive and incorporate the feedback they receive following internal inspections.

From a motivation standpoint, in most cases workers demonstrate openness to adopting effective practices that lead to overall program improvement. Applicant authorities, while not generally involved in the daily oversight of the program, believe that they have sufficient information on the program and its performance.

Key factors

The factors that impact the success of RSO monitoring, controlling and reporting activities and should be considered in the development of regulatory guidance are outlined below.

Factors that support success Factors that limit success
  • the RSOs’ ability to communicate effectively with various stakeholders (individual)
  • timely follow-ups on internal audit findings (individual and institutional)
  • lack of time allocated to the RSO function to conduct adequate follow-ups after internal inspections (institutional)
  • RSO not perceived as a credible authority of radiation protection (individual/interpersonal)
  • worker complacency (individual)

Continuous improvement activities

Most RSOs in the academic/research sector have greater access than their counterparts in the medical sector to radiation protection good practices and lessons learned. In general, the RSOs in the academic/research sector told the evaluation team that they participate in continuous learning activities and/or are members of groups, such as the Canadian Radiation Protection Association (CRPA) or the Canadian University RSO listserv (CURSO-l), which share information.

In the medical sector, a number of RSOs have access to continuous improvement activities. There are, however, a number who still struggle to keep abreast of industry best practices and lessons learned.

Some part-time RSOs in large hospitals have little time to dedicate to continuous improvement activities. Others told the evaluation team that there is no funding to attend learning events such as conferences or CNSC outreach events. And others said that they lack management support to participate in these activities.

Overall, information sharing within the RSO community can be limited, mostly in the medical sector. Some RSOs told team members that they had never heard of national organizations such as the CRPA. And even though the CNSC provides a list of CRPA members offering to share knowledge and experience in their specialty areas to licensees, the team met only two RSOs during the evaluation who knew about these potential mentorship opportunities. When it came to finding information online, both CNSC staff and RSOs told the team that it can be difficult to navigate the CNSC website rapidly to find relevant information.

The majority of RSOs who come across radiation protection good practices integrate them into their programs, share them with workers and other site or delegated RSOs. Sharing of good practices with management, including the applicant authority, happens infrequently, unless the applicant authority is a member of the radiation safety committee. Many members of radiation safety committee told the team that the committee was a forum where good practices were shared.

Capability RSOs and management are aware of good practices
Opportunity They have sufficient time to review information received
Motivation They are engaged in program improvement
COM-B findings

From a COM-B perspective, the evaluation team noticed that RSOs in both sectors are aware of good practices in radiation protection if they are members of professional associations and are able to attend continuous learning events.

Management is not necessarily aware of good practices. The day-to-day radiation protection program activities are managed by RSOs, and management trusts that good practices and lessons learned are integrated into the radiation protection program as applicable.

Key factors

The factors that impact the success of RSO continuous improvement activities and should be considered in the development of regulatory guidance are outlined below.

Factors that support success Factors that limit success
  • having support from management to undertake continuous learning activities (institutional)
  • lack of funding to attend learning events (e.g., travel freeze in the medical sector in some provinces) ( institutional)
  • lack of adequate CNSC online resources (e.g., difficulty to navigate the CNSC website rapidly to find relevant information) (infrastructure)
  • lack of opportunities to connect and exchange with other RSOs (infrastructure and interpersonal)

Success factors at a glance

Area of activity Factors that support success Factors that limit success
Training
  • timely training delivery (individual/interpersonal)
  • hands-on, practical training components (individual/interpersonal)
  • audience-specific training content (individual/interpersonal)
  • strong safety culture, specifically pressure from peers to work safely (interpersonal)
  • the potential of negative regulatory consequences (infrastructure)
  • lack of time allocated to the RSO function to ensure that training content is up to date and approach is meaningful to learners (institutional)
  • competing operational priorities such as number of patients to see in a day (institutional)
Communication and advice
  • the ease of access of the RSO to workers to provide just-in-time advice and information (individual/interpersonal)
  • lack of time allocated to the RSO function to proactively share information with stakeholders (institutional)
  • competing operational priorities such as number of patients to see in a day (institutional)
  • RSO not perceived as a credible authority of radiation protection (individual/interpersonal)
Monitoring, controlling and reporting
  • the RSOs’ ability to communicate effectively with stakeholders at all levels (individual)
  • timely follow-ups on internal audit findings (individual/institutional)
  • lack of time allocated to the RSO function to conduct adequate follow-ups after internal inspections (institutional)
  • RSO not perceived as a credible authority of radiation protection (individual/interpersonal)
  • worker complacency (individual)
Continuous improvement
  • having support from management for continuous learning activities (institutional)
  • lack of funding to attend learning events (e.g., travel freeze in the medical sector in certain provinces) (institutional)
  • lack of adequate CNSC online resources (infrastructure)
  • lack of opportunities to connect and exchange with other RSOs (infrastructure/interpersonal)

Other influencing elements

1.Are the right governance models in place?

A few RSOs working in the medical sector in organizations that have undergone amalgamation told the evaluation team that the governance structure of their organization can create challenges to their effectiveness. They feel far removed from the applicant authority and perceive that their role is not valued within the organization. Other RSOs feel that they are still able to manage their radiation protection program effectively even though there are several layers of management.

For the most part, RSOs in both sectors told the evaluation team that they feel well supported by their applicant authority and generally communicate with them on an as-needed basis. The team heard from RSOs who rarely or never communicate directly with their applicant authority, and from others who communicate with their applicant authority weekly.

2. Does the CNSC provide sufficient guidance?

Many RSOs feel that the current communication channels used by the CNSC, such as the website and email distribution list, can be improved. For example, a number of RSOs feel that the website is difficult to navigate, and that information is moved and reorganized frequently.

As for the emails that are distributed through the CNSC mailing list, including the DNSR Newsletter, RSOs told the evaluation team that they appreciate the information but that the newsletters are published too infrequently and are not audience specific. RSOs want to receive information from the CNSC that is timely and applicable to their area of work.

In terms of the nature of the guidance that is available, many RSOs would like to have additional regulatory guidance to support them in the day-to-day management of the radiation protection programs. The areas of guidance most frequently mentioned are:

  • how to determine adequate RSO resourcing levels based on the nature of the work and the magnitude of licensee operations
  • what to include in an RSO work description to ensure that the role of the RSO is accurately captured and defined, not tacked on to a job description as an afterthought
  • support for internal audits including guidance on the audit frequency, format and templates
  • guidance on radiation safety committee composition and purpose
3. Would there be advantages to certification?

Results are mixed among RSOs regarding the advantages of requiring certification. 49% of the RSOs surveyed told the evaluation team that certification could be beneficial as it would:

  • provide an opportunity for RSOs to take part in continuous learning
  • ensure a standardized base level of technical and regulatory knowledge
  • help to ensure that the RSOs are competent and qualified
  • increase the RSOs’ credibility or weight/influence within their organization

Almost half of the RSOs interviewed who were in favour of certification do not see it as a way to directly improve radiation protection program performance, but as a means to support access to continuous learning opportunities. In other words, if there is a regulatory requirement to maintain a certification, it would likely necessitate regular training or continuous learning. As such, this would be a way for RSOs with limited learning and development opportunities to access continuous learning.

The following quote illustrates the majority view in support of certification: “I think that there should be RSO certification. Credentials to maintain a certain level of education should be required and mandatory attendance for continuing education and refresher courses. I think the consequences of not having these things [lead to] a very poor understanding of radiation safety, which then gets passed through the organization. This perpetuates the lack of safety in these areas, lack of understanding turns into complacency […].

Almost half of the RSOs in the medical sector feel that the CNSC could work with the CRPA or another professional body to develop a standardized and recognized RSO training program.

4. Is there an issue with the alignment across regulatory bodies?

The majority of RSOs see no issues in terms of the need for greater alignment across regulatory bodies. Those who did told the evaluation team that greater alignment across regulatory bodies could help to avoid overlap and duplication of work. For example, in the medical sector, some RSOs feel that the CNSC should work with the provincial ministries of labour to align the timing of their inspection efforts so as to avoid back-to-back inspections within a short period of time.

Another example came from an RSO working in the commercial sector. The RSO cited a potential misalignment between the Health Canada requirements concerning the addition of saline solution to a vial after the radioactive substance has already been added as opposed to adding the saline solution first. In the RSO’s view, the first method can contribute to extremity dose, but if the saline could be introduced first in the vial, there would be no exposure. 

5. Does the relationship with the regulator allow for open communication and effective resolution of non-compliance?

91% of the RSOs in both sectors feel that their relationship with the CNSC allows for open communication and resolution of non-compliance. The evaluation team heard from some RSOs who feel that communication with the CNSC is ineffective because they were asked by their licensing officer to limit their communications to email. This might result in numerous exchanges to resolve an issue that could have been resolved efficiently over the phone.

4% of the RSOs felt that their relationship with the CNSC allowed for very little open communication. Barriers that hinder the relationship with the CNSC include:

  • lack of clarity on who to contact within the CNSC
  • inability to reach someone when calling the CNSC
  • lack of clarity in CNSC expectations (i.e., RSOs feel that the CNSC seems hesitant to offer a clear interpretation of its regulation or be forthcoming with its expectations; this is viewed as counterproductive)
  • delays in obtaining inspection reports
  • nature of the relationship between regulator and licensee (at times seen as adversarial); a small minority of RSOs feel that the CNSC penalizes based on the “tiniest of infractions”, based on interpretation, not the regulations

Most applicant authorities believe that they have a collaborative relationship with the CNSC that allows for efficient resolution of non-compliance. Only one applicant authority stressed that the CNSC was not collaborative and that the regulator has adopted a policing approach which is seen as counterproductive in the pursuit of safety.

Stakeholder views

An alternate approach to presenting the information collected during the evaluation is to summarize the viewpoints of each group. This section presents the views of the different stakeholder groups with the aim of showcasing the different perspectives.

RSOs

  • Most believe that the biggest obstacle they face is the lack of time to fulfill their RSO duties.
  • RSOs feel that the CNSC could offer more guidance on RSO work descriptions, RSO resourcing levels based on magnitude of operations and internal audit format and frequency.
  • Most offer training to workers different ways: some leverage e-learning solutions, others rely on a third party because they do not have time, and others use a combination of online, in-class and training manual/materials.
  • RSOs recognize the importance of internal audit follow-ups, but not all RSOs have time to complete this task.
  • They recognize that workers are generally receptive to their advice, but sometimes feel that workers can be too overwhelmed with their daily work to incorporate the feedback.
  • Some feel that they have effective radiation safety committees, while others feel that their committee is ineffective and identified challenges in determining appropriate committee composition and purpose.
  • Some feel that they do not have access to continuous learning opportunities.
  • RSOs believe that taking a collaborative approach to problem solving supports worker engagement.
  • They believe that there is an opportunity to clarify the roles and responsibilities of the RSO.
  • Some face challenges when dealing with the CNSC licensing officer – ineffective communication, multiple back and forth, unclear expectations, and frequent staff turnover.

Workers (including nuclear energy workers, non-nuclear energy workers, auxiliary staff, physicians, graduate students, professors)

  • Workers believe that the RSO offers comprehensive training that helps them understand how to protect themselves and would like more hands-on practical exercises.
  • They are almost always able to reach the RSO when they have questions.
  • They recognize that having an RSO that is in close proximity to them is helpful and facilitates communication and daily interactions.
  • Most workers consider the RSO to be the expert when it comes to regulatory requirements.
  • Workers receive clear and specific feedback from the RSO after internal inspections and believe that the feedback leads to improvements, such as lower dosimeter readings.

Applicant authorities

  • Applicant authorities feel that existing governance models are effective.
  • They believe that radiation safety committees support the integration of good practices into the radiation protection program.
  • They recognize that open communication between the RSO and the workers helps to promote engagement.
  • Some value more frequent interactions with RSOs; others are satisfied with infrequent interactions (i.e., no news is good news).
  • Some feel that the CNSC should take a more collaborative approach with licensees.

Radiation safety committee members

  • Committee members see the committee as serving multiple purposes: ensuring compliance with regulatory requirements, ensuring that staff follow protocols, acting as a platform for information sharing across disciplines, ensuring that best practices are implemented, and reviewing incidents and recommending solutions.
  • They value the contribution of RSOs to the committee, except where the RSO is not viewed as being a credible authority on radiation protection.
  • They feel that the CNSC needs to be clear on expectations and provide guidance on the time requirement for RSO roles/activities based on the magnitude of operations.
  • They recognize that there is room to improve worker training (more tailoring, more hands-on, releasing workers to attend training).
  • They believe that their organization adjusts their practices as a result of internal audit findings or as a result of implementing best practices.

DNSR staff

  • DNSR staff recognize that not all RSOs have the resources (financial, time or human) they need to effectively run their radiation protection programs.
  • They believe that the lack of time allocated to the RSO function is the most frequent resourcing issue facing radiation protection programs.
  • They feel that, while it is a regulatory expectation that RSOs have authority to stop work or restrict operations, in practice not all RSOs feel comfortable doing so or receive support from management to do so.
  • They hear directly from RSOs that they want clear direction and requirements from the CNSC, particularly on the time requirement for RSO roles/activities based on the magnitude of operations.
  • They believe that to support effective communication between the applicant authority and the RSO there needs to be a clearer understanding of roles and responsibilities.

Case study takeaways

The evaluation team completed 11 case studies during the evaluation. The studies were designed to look at the role of RSOs in different contexts. The team also provided an in-depth understanding of how the role of the RSO and associated “success factors” influences the radiation protection program performance Footnote 3 in these varied situations.

To better understand success factors and barriers in different contexts, in other words, “what works for whom, in which context”, the case studies have been grouped into five categories:

  1. Medical, large complex with geographic disparity
  2. Medical, large complex with location proximity
  3. Medical, small, one site
  4. University/academic research
  5. Commercial sector

Key takeaways

RSOs in the medical sector, working in large hospitals with complex programs and geographic disparity of sites, faced challenges if the following circumstances were present:

  • a lack of management support for radiation protection
  • inadequate time allocated to the RSO function and lack of clear RSO roles and responsibilities
  • competing priorities between expeditious patient care and worker radiation protection

RSOs in the medical sector working in large, complex hospitals without geographic disparity, faced challenges if the following circumstances were present:

  • inadequate time allocated to the RSO function
  • perceptions that the RSO was not a credible and trusted authority on radiation protection

The small hospital had consistently good regulatory compliance. Management demonstrated a high level of involvement in the radiation protection program. There was open communication between the RSO, the workers and management.  

Organizations in the academic/research sectors had consistently good regulatory compliance. Support from management was strong and RSOs had significant access to learning and development opportunities. One barrier to effective communication between the RSO and workers was observed in a situation where the RSO was not physically located with the workers.

Among commercial sector organizations, RSOs faced challenges if the following circumstances were present:

  • inadequate time allocated to the RSO function
  • a lack of time for shift workers to attend radiation protection training and/or refreshers

Top of Page

Chapter 2 – Detailed findings and supporting evidence

This chapter is written from an evaluator’s perspective. It presents the data behind the logic and the supporting evidence from which the findings, conclusions and recommendations have been drawn.

This chapter summarizes the technical reports of the following data sources.

Case study

Eleven case studies were completed to examine the role of RSOs who work for specific licensees operating in different contexts and with different compliance records. The objective of the case studies was to provide an in-depth understanding of how the role of the RSO and associated “success factors” influences the radiation protection program performance Footnote 4 in these varied situations.

In consultation with the CNSC’s regional site coordinators, criteria were established for the selection of case studies. These focus on licensees in the medical and/or academic/research sectors who have experienced situations of change such as:

  • a change in RSO
  • a change in radiation protection program performance over time, either improvement or decline
  • a change in work environment/organization; e.g., complex environments, multiple sites across provinces, amalgamation

The case studies were also selected to examine different regulatory situations such as:

  • licensees with both nuclear substances / radiation devices and Class II licences
  • licensees with a radiation safety committee
  • licensees with significant reported events
Comparative analysis

The comparative analysis explored the effectiveness of the role of the RSO when compared to:

  • the role of nuclear substances and radiation devices RSOs internationally (United States and United Kingdom)
  • other similar roles nationally such as:
    • biosafety officers (Public Health Agency of Canada)
    • flight safety officers (Transport Canada)
    • occupational health and safety officers
    • Class II RSOs (CNSC)

The intent is to provide a comprehensive picture of the RSO role, its regulatory expectations, guidance, success factors and influence in relation to other similar roles. The evaluators conducted comparative analysis by sharing information via email and phone interviews with the United States Nuclear Regulatory Commission (U.S. NRC), the United Kingdom’s Health and Safety Executive (HSE) and other regulator program officials, as well as by reviewing regulatory documents. In seeking to understand what contributes to the effectiveness of the RSO’s role, the following areas were compared:

  • legislation
  • regulatory guidance
  • certification
  • governance models and reporting relationship
  • oversight committee structure
  • reporting requirements
  • level of RSO authority
Interviews

Interviews were central to this evaluation in bringing both a breadth and depth of data. The interviews covered all 22 evaluation questions. 97 interviews were completed with 102 CNSC and licensee stakeholders. Interviewparticipants by stakeholder group consisted of:

CNSC internal interview stakeholders (n=12):

  • DNSR and DERPA Footnote 5 management: n=4
  • DNSR staff (both from compliance and licensing): n=9

External interview participants were selected from two groups (n=89):

  • Group 1 – Representative sample participants (n=3)
    • RSOs: n=3
  • Group 2 – Specific to case study participants (n=86)
    • RSOs (either corporate or site): n=28
    • Radiation safety committee members: n=16
    • Applicant authorities: n=9
    • Workers (nuclear energy workers (NEWs) and non-NEWs, including physicians): n=33
Literature review

The purpose of the literature reviewwas to gather open source information on similar roles that are used by other federal and/or provincial/territorial regulatory organizations, as well as on good practices related to regulation and associated guidance. The review strengthened the theory of change behind the role and interventions of the RSO. The scope of the literature review was adjusted based on evaluation schedule and interview findings.

Survey

The RSO survey was designed as a census of the RSOs in the medical and academic/research sectors across Canada. The evaluators administered the survey to licensees with high/medium- and low-risk use types. This report summarizes the results of the high/medium-risk respondents only. Low-risk respondent results were excluded because the level of complexity of their operations is very different from those in the high/medium-risk group. Combining the results of both groups would have misrepresented the survey data. Licensees with low-risk use types operate in very different environments than licensees with high/medium-risk use types. They are generally from much smaller organizations, typically with only one CNSC licence. Although they possess nuclear substances, it is possible that these substances are accessed only once per year, and they are always handled by the same individual.

RSO survey features:

  • 55 questions (multiple choice and open ended)
  • Total respondents: 255/342 (high/medium risk)
  • Response rate: 75% (high/medium risk)
  • Average completion time: 42 minutes
  • Data collection time frame: February 6, 2018 to March 31, 2018
  • Data gathered on all 22 evaluation questions

The survey of applicant authorities was designed as a randomized sample of the population (approximately 191), which represents a confidence level of 95% and confidence interval (or margin of error) of +/-5.

Applicant authority survey features:

  • 10 questions (multiple choice and open ended)
  • Total respondents: 49/185
  • Response rate: 26%
  • Average completion time: 9 minutes
  • Data collection time frame: February 6, 2018 to March 1, 2018
  • Data gathered on evaluation questions (3, 7, 8, 10, 13, 18, 20, 21 and 22) related to issues of communication, authority, governance and regulatory guidance.

As the CNSC does not maintain contact information on workers in the radiation protection programs, the workers were identified by 20 RSOs who were selected at random. Each RSO was asked to provide the email addresses of two to four workers from their organization who would be willing to complete the survey.

Worker survey features:

  • 10 questions (multiple choice and open ended)
  • Total respondents: 19/29
  • Response rate: 66%
  • Average completion time: 6 minutes
  • Data collection time frame: February 6, 2018 to March 1, 2018
  • Data gathered on evaluation questions (2, 5, 6, 9, 11, 12) related to issues of training, communication and feedback, and barriers/enablers to adoption of safe practices

All surveys were developed internally and administered by a third party. Respondents were able to complete the survey in either official language. During the survey period, two reminders were sent to participants who had not completed the survey. In an effort to further increase survey participation, CNSC staff made follow-up phone calls to RSOs who had not completed the survey.

The survey results are anonymous and the data collected is protected and handled per the TBS Policy on Results and supporting guidance.

Training activities

A key activity as part of the RSO’s role in overseeing radiation protection programs is to ensure adequate training for workers who are required to use or handle nuclear substances and radiation devices Footnote 6. Training activities delivered by the RSO include the design, delivery and evaluation of training (formal such as in-class and informal such as workshops and online training).

The CNSC’s series of regulatory documents provides licensees with detailed guidance on the elements of an effective training system Footnote 7.

Findings

Across both sectors, RSOs are generally successful in undertaking radiation protection training. The vast majority of RSOs believe that through training, they are able to convey radiation protection program objectives to workers. As a result, workers feel that they have a good understanding of the role they play in radiation protection and adopt safe radiation protection practices.

Mostly in the medical sector, competing operational requirements, the lack of time (i.e., pressure to do more with less time) and worker complacency are some issues that can arise and can impede workers from adopting safe work practices.

Important factors that motivate workers across both sectors to adopt effective radiation protection practices include the quality of training (including a hands-on component), timeliness of the training, pressure from peers to be safe and the fear of potential regulatory consequences (i.e., the loss of a licence or the issuance of an administrative monetary penalty).

Supporting evidence

Reach – Is the RSO able to reach the workers who require training?

To be effective in delivering training, RSOs should be able to identify and contact workers. In turn, workers Footnote 8 should be able to attend the required training. For any of this to be possible, the radiation protection program needs to be properly resourced in terms of financial, human or physical resources.

95% of the RSOs surveyed (n=259 Footnote 9) believe that they are successful at identifying and contacting the workers who require radiation protection training, even when their program is not properly resourced.

Conditions that prevent workers from attending radiation protection training are as follows:

  • It is not clear who requires training
  • There are geographic limitations
  • Workers are not released from their duties to attend
  • It is difficult to reach workers

Corporate RSOs rely on site RSOs or other managers (i.e., the line managers of housekeeping or security personnel) to identify who needs to be trained and to reach out to workers who require training.

94% of the RSOs surveyed (243/255) believe that workers who require radiation protection training are able to attend the training. It can occasionally be a challenge to have workers released from their duties to attend training.

77% RSOs surveyed in the medical and academic/research sectors (193/251 Footnote 10) believe that they are equipped with the necessary resources (i.e., financial resources, human resources, training facilities and time) to develop and maintain an effective radiation protection program in their organization.

23% of the RSOs surveyed (58/251) expressed concern over a lack of resources. This view was more prevalent among part-time RSOs in the medical sector managing complex radiation protection programs with multiple locations. This group reports that they lack time because they must balance operational work, such as direct patient care, with their other RSO-related duties.

They feel that it is insufficient to have a part-time RSO manage a complex program spread across multiple sites. The following quote illustrates this view: “Takes years to design training. Not enough to have a part-time RSO who also has significant other duties to fulfill.”

89% of the CNSC staff from the Directorate of Nuclear Substance Regulation (DNSR) (8/9) do not believe that all RSOs have adequate resources to develop and maintain an effective radiation protection program in their organization. They believe that this varies significantly from one organization to the next, depending on the size of the organization and complexity of the operations.

A challenge facing RSOs working in the commercial sector is finding the right time to train shift workers. Offering training to workers after an overnight shift does not lend itself to optimal learning conditions. Some indicated that having an online learning management system would be very beneficial, but that their management does not support an investment in a learning platform. The following quote illustrates this view: “I try to find times that work but with shift work it can be difficult; online training would be useful as people could all do [training] at a convenient time.”

The comparative analysis with other federal regulators reveals that, as part of their compliance promotion activities, some organizations provide licensees with additional resources to support licensees in the conduct of their training activities. The Public Health Agency of Canada (PHAC) has developed an online laboratory biosafety and biosecurity learning portal accessible to all its licensees. This portal includes learning material and e-learning courses on the principles and practices of biosafety and biosecurity to support regulated parties.

Through this portal, the biological safety officers (BSOs) are able to track worker course completion and test results, thus reducing the administrative burden on them to create and maintain such tracking mechanisms. The courses are reported to be popular with BSOs and many of them have decided to make the courses mandatory training for their laboratory personnel.

Knowledge – Do workers understand radiation protection practices and procedures?

As a result of their participation in training activities, workers should have increased their skills and gained an understanding of effective radiation protection practices. By gaining these skills and increasing their knowledge, workers should know what is expected of them in the conduct of their daily activities. The evaluation sought to understand if, through the training activities, RSOs are able to influence the workers’ understanding of radiation protection program objectives. Conditions that promote this increased understanding include the workers’ capabilities and motivation, the tailoring of the training to its audience, and the timeliness of the training.

100% of the RSOs interviewed across both sectors (28/28) indicate that they tailor the radiation protection training content to meet the needs of different audiences. This was also reported to be the case when RSOs rely on a third party to deliver the radiation protection training.

Tailoring a training program generally means offering different levels of information. A simplified level of training may be offered to auxiliary workers using pictograms, particularly if there is a language barrier because English is not the workers’ first language. More in-depth training accompanied by a comprehensive training manual would be offered to nuclear energy workers and non-nuclear energy workers.

A minority of RSOs feel that training could be further tailored to include more specific examples and practical applications, but that lack of time prevents them from working on this.

89% of the RSOs interviewed (16/18 Footnote 11) feel that they offer training in a timely manner. They generally offer training either in a group setting or one-on-one. The down side of offering one-on-one training is that it can be time consuming for RSOs. Some RSOs believe that having a learning management system allows them to deliver timely training to workers.

100% of the workers (29/29) feel that the training is offered to them in a timely manner. Many explained that they received timely training as it is a prerequisite for obtaining their thermoluminescent dosimeter (TLD) Footnote 12 and their site access card.

91% of the RSOs interviewed (21/23) believe that workers recognize the importance of radiation protection. They feel that, for the most part, workers want the radiation protection program to succeed. This is demonstrated when they ask questions and are actively engaged during the training.

16% of the RSOs interviewed (3/19) do not feel that workers are engaged and believe that they attend training simply because “they have to”. Other RSOs believe that the level of worker engagement in learning depends on the trainee’s role in the organization. For example, nuclear energy workers demonstrate high engagement levels because they have completed their education in a field that already provides a good base of knowledge of radiation protection. They also demonstrate high levels of interest in radiation safety due to the nature of their work with patients.

Image indicates that 94% of the RSOs believe that workers understand the radiation protection program objectives and 100% of the workers believe that they have a strong understanding of the radiation protection program objectives.

94% of the RSOs (n=251 Footnote 13) believe that workers understand the objectives of the radiation protection program as a result of the training they receive. Some RSOs use the training test results as an indicator of the workers’ level of understanding. 100% of the workers surveyed (20/20) believe that the training received helps them understand the role they play in radiation protection.

Behaviour – As a result of the training they receive, do workers adopt safe practices?

As a result of having increased their knowledge of effective work practices through training, radiation protection program workers should adopt safe radiation protection practices in their workplace. Conditions that promote this behavior include having: management support; motivated workers who want to positively influence the radiation protection program; and RSOs who are perceived to be credible, trusted and recognized as the authority on radiation protection. The evaluation sought to understand if workers were adopting effective practices, and if not, what barriers or challenges prevented this from happening.

100% of the RSOs interviewed across both sectors (28/28) feel that workers adopt effective radiation protection practices and procedures as a result of their participation in radiation protection training. The following quote illustrated this view: “Without training, workers would not be aware of the regulations (and the workplace policies and procedures that support the regulations).”

RSOs also mentioned that, to their knowledge, workers were not intentionally being non-compliant with radiation protection practices and procedures. The view was that most situations of worker non-compliance were related to lack of knowledge and awareness. This is why, they said, periodic radiation protection refresher training is valuable.

A strong radiation protection program and peer pressure to adopt safe practices are among the factors that support workers in adopting effective radiation protection practices and procedures.

Image indicates that 94% of the workers believe that they adopt safe work practices as a result of training.

94% of the workers (n=46 Footnote 14) feel that they adopt effective radiation protection practices as a result of attending training. The following quote illustrates this view: “The training certainly did help; [it] taught me how to protect myself and how to handle substances to ensure I was comfortable handling the substances.”

4% of the workers interviewed and surveyed (2/49), all of whom were physicians, feel that as a result of their profession, they do not need additional training in order to adopt effective radiation protection practices. They do not see a need to take part in the training because radioactive substances are handled by workers (i.e., technologists) and not by them directly.

When prompted, both RSOs and workers identified a number of challenges that can prevent workers from adopting safe radiation protection practices:

  • competing operational requirements and limited time
  • views that safe practices are an unnecessary burden
  • complacency and/or lack of motivation to adopt safe practices

Compliance data analysis validates that for the most part workers adopt safe radiation protection practices. Between 2007 and 2017, the percentage of compliance in the proper use of equipment, clothing and procedures at the work site has been at 95% or higher (figure 1). This shows that workers do adopt safe radiation protection practices.

Figure 1: High compliance in use of appropriate equipment, clothing and procedures

Twitter and facebook logos
Description

The data from 2007 to 2017 shows a high level of compliance.

  2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
% of compliance 97% 95% 98% 97% 94% 98% 99% 98% 99% 97% 95%
Case study findings relating to training

In the medical sector, particularly in large hospitals with complex radiation protection programs and a geographic disparity of sites, the RSO function and overall radiation protection program may be inadequately resourced.

This impacts the ability of RSOs to deliver training activities. This was apparent in a case study where the RSO from a large complex hospital with multiple sites was part-time. The RSO had identified areas to improve the radiation protection training program, making the training more hands-on, but struggled to find time to deliver the training across all sites and to have workers from all sites removed from their operational duties for training purposes.

Reviewing the theory of change
Training Pathway
Description

A graphic representation of a pathway that is a sequence of steps that are required from the RSO to get from training activities to their impacts. The graphic reads from bottom to top. The principal assumptions and risks related to the training pathway are listed at the bottom. Above on the left, the graphic shows the pathway sequence divided in 4 categories: activities, reach, knowledge and behaviour. On the right, the graphic shows the assumptions and risks related to reach, knowledge and behaviour.

The principal activities/outputs assumptions and risks of the training pathway are:

  • RSO is qualified (training/experience)
  • RSO has the required soft skills (i.e. leadership, communication, persuasion)
  • RSO is knowledgeable of regulatory requirements
  • RSO is supported by management (including AA and when applicable RSC)
  • Adequate resources are available to implement/maintain the RPP ($, time, etc.)
  • RSO is committed (wants to positively influence)

Narrative for training pathway

Training activities

Training activities include the design, development, delivery and oversight of formal and informal training, info sessions, workshops and material

Reach

Reach is the extent to which the RSO, in the conduct of the training activities, is able to influence RPP users. In order for the RSO to be effective in reach, the evaluation assumes that:

  • RSO is able to identify all stakeholders for which RPP training is required
  • RSO is able to contact trainees
  • Trainees attend required training

Knowledge

The immediate outcome of the training pathway is for trainees to understand RPP practices and procedures (capabilities). To do so, they need to have time to attend training (opportunity) and have to be engaged in learning (motivation).

For the trainees to gain this knowledge and understand RPP practices and procedures, it is assumed that:

  • Training is understood by stakeholders (capabilities)
  • Training is audience specific (capabilities)
  • Training includes hands-on practical exercises (capabilities)
  • Training is timely (opportunity)
  • Workers realize the importance of learning RPP skills and good practices (motivation)
  • Workers feel peer pressure to adopt safe practices (motivation)
  • Workers do not want to risk any potential negative regulatory consequences (motivation)

Behaviour

The expected behaviour is that stakeholders adopt safe RP practices. In order for stakeholders to adopt safe RP practices the evaluation assumes that:

  • RPP trainees know how to adopt safe RP practices
  • RPP practices are supported by management and RSC
  • Trainee is motivated to positively influence the RPP
  • RSO is credible, trusted and recognized as RP authority

In addition to the capacity change outcomes related to capabilities, hands-on practical components are viewed by workers as extremely valuable. They help them better understand training program components, than do traditional theory-based training programs. Additional factors that influence the workers’ motivation in adopting safe radiation protection practices include 1) peer pressure, and 2) the fear of potential negative regulatory consequences.

From the RSO’s perspective, the factor that most significantly impacts their ability to design and deliver training is the lack of time. Case study evidence highlights that among large medical, complex organizations with geographic disparity, compliance issues were observed when adequate resourcing of the radiation protection program was lacking. This is primarily due to the lack of time allocated to the RSO function.

For part-time RSOs in the medical sector, it can be difficult to have sufficient time to design training program elements such as hands-on exercises.

Communication and advice activities

Part of the RSOs’ work involves sharing performance information with management and providing timely information and feedback to workers. RSOs also act as the main point of contact with the CNSC and of exchange on regulatory issues with licensing officers. These communication and advisory activities are central to the RSO role Footnote 15 and are considered to be an important factor in improving radiation safety performance.

Certain communication activities are outlined in regulations, such as the requirement to inform nuclear energy workers in writing of their designation and the associated risks.

Findings

Communications and advice produced by RSOs reaches workers and management as intended. Workers and management understand the radiation protection program objectives and important practices and procedures. Workers value the information shared by RSOs and it is important that the RSO be a credible and trusted authority on radiation protection. Having an RSO located in close proximity to workers is favourable as it increases the frequency of communication between RSOs and workers.

RSOs believe that having strong management support and having sufficient time allocated to their role are conditions that increase their personal engagement levels at work. For workers, having access to ongoing training and having practical hands-on training (as opposed to strictly theory-based training) are conditions that increase their personal engagement levels at work. Worker engagement can be impacted when there are operational pressures and the worker feels that they are required to do more with less time.

Supporting evidence

Reach – Is the RSO able to reach workers, management and the CNSC?

To be effective in reaching their audience, RSOs should be able to identify and contact workers, management and radiation safety committee members. RSOs must also have a strong understanding of the licensee’s operations as they relate to the radiation protection program to ensure that they reach out to the right audience, at the right time, on the right topic.

Across both sectors, all RSOs believe that they communicate effectively with workers, management or radiation safety committee members. In most cases, site RSOs communicate and provide advice to front-line workers while corporate RSOs communicate and provide advice to management and radiation safety committee members.

Some RSOs working in large amalgamated hospitals report facing challenges in being able to communicate directly with the applicant authority because of too many management layers.

This is similar to what the evaluation team heard from CNSC staff from DNSR, who expressed mixed views on RSOs’ ability to always effectively reach the applicant authority. Some DNSR staff believe that, while this is the CNSC’s expectation, as described in REGDOC-1.6.1, in reality many RSOs are unable to reach the applicant authority because of multiple layers of management. The following quote illustrates this view: “Large organizations may require the RSO to jump through many hoops to get to the applicant authority.”

Case study evidence indicates that even an inadequate relationship between RSOs and applicant authorities does not automatically negatively impact the performance of the radiation protection program. Some RSOs in large amalgamated hospitals operate successful radiation protection programs, even with very little communication with their applicant authority.

100% of the RSOs interviewed (28/28) feel that they have sufficient knowledge of the licensees’ operations to effectively communicate and offer advice on radiation protection matters. 11% of the RSOs (3/28) indicated that new RSOs can encounter a steep learning curve before feeling comfortable with licensee operations. This is especially true in complex radiation protection programs when there are a number of different operations. Case study evidence also supports this, especially for large complex operations in the medical sector managing a number of different licences.

There are variations when it comes to the frequency of RSO interactions with program stakeholders. Some RSOs communicate more frequently with certain stakeholders, such as front-line workers, than with others, based on the information needs of each group.

For example, RSOs may interact with nuclear medicine technologists on a daily basis, but  interact with housekeeping staff only once a year (figure 2).

Figure 2: Frequency of RSO communication with stakeholder groups

Figure 2
Description
  On a daily basis At least once a week At least once a month 1 to 3 times a year Once a year or less otrher Does not apply
Workers 47% 18% 13% 9% 4% 11% 0%
Auxiliary staff 21% 14% 13% 28% 13% 12% 0%
Management including the AA 9% 26% 17% 25% 11% 4% 8%

The data analysis examined compliance with the regulatory requirement to keep nuclear energy workers informed of risks, rights and obligations. Data shows that for the most part RSOs do reach workers (figure 3). The compliance in this area has significantly increased since 2007, reaching a level of 99% compliance in 2017.

Figure 3: Annual compliance with requirement to keep nuclear energy workers informed of risks, rights and obligations

Figure 3
Description

The data from 2007 to 2017 shows the compliance is trending upward

  2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
Compliant 84% 85% 82% 93% 87% 93% 92% 96% 93% 96% 99%
Non-compliant 16% 15% 18% 7% 13% 7% 8% 4% 7% 4% 1%

Knowledge and awareness – Do workers and management have an increased understanding of the radiation protection program objectives?

As a result of receiving information and advice from RSOs, workers, management and radiation safety committee members should have a strong understanding of their organization’s radiation protection program objectives. The evaluation team sought to understand whether there are any barriers or challenges that would restrict this.

96% of the workers in both sectors (n=50 Footnote 16) feel that the communication flow with RSOs is effective. They do not report any challenges when reaching out to their RSO. Many expressed that the RSO was very approachable, which made them feel comfortable about asking questions. The following quote illustrates this view: “Yes. If I need any guidance or clarification, the RSO is very approachable.” They believe that the physical proximity of the RSO to the operations facilitates open communication.

In both sectors, 6% of the workers interviewed (n=2) face challenges when attempting to communicate with their RSO. In one case, the worker does not ask the RSO any questions because the worker feels that the RSO lacks expertise. In another case, a worker is not able to ask the RSO questions because the RSO is generally unavailable due to operational work pressures.

When it comes to applicant authorities, 91% (n=53 Footnote 17) feel that they receive relevant and timely information and advice from the RSO. 67% (n=9 Footnote 18) of applicant authorities keep themselves informed of issues concerning the radiation protection program by reading the minutes of the radiation safety committee meetings or attending the committee meetings.

Workers and management both report having a solid understanding the objectives of the radiation protection program. 74% of the RSOs surveyed (180/244) also feel that workers have a strong understanding of the objectives of the radiation protection program. But they do not have the same perspective when it came to managements’ understanding of the radiation protection program objectives.

50% of the RSOs surveyed (123/244) feel that management understands the objectives of the radiation protection program. Those who feel otherwise attribute lack of understanding to managements’ overall lack of knowledge of radiation protection or lack of interest in the program. The following quote from an RSO in the academic/research sector illustrates this view: “Senior management doesn’t always understand the objectives of the program; therefore you need to have good communication skills as you need to speak their language. As an example, you have to talk to them about the reputation risk to the university of non-compliance”.

Data analysis examined whether the requirement to post RSO contact information is met. Results indicate overall good compliance with the requirement to post the name and job title of a person who can initiate an emergency procedure 24 hours a day in regard to radiation protection (figure 4). The compliance with this requirement has significantly increased since 2007 from 91% to 98% in 2017.

Figure 4: Significant increase in compliance with posting of radiation protection emergency contact

Figure 4
Description

The data from 2007 to 2017 shows the compliance is above 90% and trending upward

  2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
% of compliance 91% 92% 91% 92% 94% 97% 99% 97% 98% 99% 98%

Behaviour – As a result of the communication and advice produced by the RSO, are workers and management engaged in radiation protection programs?

As stakeholders gain a better understanding of the objectives of the radiation protection program, they should demonstrate their engagement in radiation protection and act as the eyes and ears of the RSO. Behaviour relating to engagement can manifest itself in various ways. Workers who routinely adopt a questioning attitude and value safety are engaged. This engagement will lead workers to adopt safe practices and increase compliance with regulatory requirements. The evaluation sought to understand what conditions promote robust stakeholder engagement.

95% of the RSOs surveyed (230/241) believe that they are engaged in the radiation protection program of their organization. 100% of the RSOs interviewed (28/28) report having high levels of personal engagement.

When asked what contributes to their high levels of engagement, RSOs identified the following three factors most frequently:

  • the RSO’s focus on the safety of workers
  • the RSO’s strong sense of personal commitment and desire for excellence
  • strong management support

83% of the RSOs surveyed (199/241) feel that nuclear energy workers and non-nuclear energy workers are engaged in the radiation protection program. Elements that contribute to high engagement levels are:

  • the RSO’s availability, frequency of contact with workers and visibility
  • strong management support
  • a healthy organizational safety culture (including peer pressure to adopt safe practices)
  • a comprehensive training program, including refreshers
  • trust and collaboration among colleagues

The view shared by many was that RSOs and workers have to operate in a “no blame, no shame” environment that fosters openness and transparency.

Some RSOs recognize that issues with worker engagement can arise when there are too many competing priorities. For example, these RSOs pointed out the difficulty a nuclear technologist faces in meeting the need to treat a certain number of patients per day while still having to follow standard operating procedures/protocols for radiation safety.

The importance of prioritizing safety over productivity is well documented in international literature from the International Atomic Energy Agency, other sources, and within CNSC regulatory and guidance documents to licensees. CNSC REGDOC-2.1.2, Management System: Safety Culture,provides guidance to licensees and indicates that licensees “should make safety the utmost priority – overriding the demands of production and project schedules”, acknowledging that production demands can negatively impact worker engagement levels and the licensee safety culture.

The RSOs’ impression of worker engagement levels is aligned with what workers and radiation safety committee members report. The vast majority of workers believe that they are engaged in the radiation protection program of their organization.

Factors that can limit the engagement of workers towards radiation protection are:

  • poor safety culture within the organization
  • lack of accountability (lack of clear roles and responsibilities)
  • lack of management support for the RSO and the radiation protection program
  • focus and priority on patient care, rather than safety of workers

100% of the applicant authorities surveyed (46/46) believe that they are engaged in the radiation protection program. When prompted to share what contributes to their high level of engagement, they highlighted the following factors present in their work environment:

  • the applicant authority’s close working relationship with the RSO
  • ensuring of compliance with regulatory requirements and understanding of the consequences of non-compliance
  • an organizational culture that values radiation protection and is focused on employee safety
  • a radiation safety committee that functions effectively

26% of the RSOs surveyed (62/240) do not feel that applicant authorities are engaged and noted that factors such as lack of time and interest in radiation protection limited their engagement. The following quote from an applicant authority illustrates this view: “No news is good news on this file.”

Case study findings relating to communication and advice

In the medical sector, RSOs working in large hospitals with complex radiation protection programs and a geographic disparity of sites can find it difficult to have face-to-face communication with all stakeholders. This makes the role of the site RSO crucial to the success of radiation protection programs.

In the medical sector, regardless of organization size, when the RSOs lack solid understanding of the licensee’s various operations, this impacts the value of the communication and advice. Workers in this context do not value or trust the information they receive from their RSO and are less inclined to see the RSO as a key partner to their operations.

Reviewing the theory of change
Communication pathway
Description

A graphic representation of a pathway that is a sequence of steps that are required from the RSO to get from communication and advice activities to their impacts. The graphic reads from bottom to top. The principal assumptions and risks related to the communication and advice pathway are listed at the bottom. Above on the left, the graphic shows the pathway sequence divided in 4 categories: activities, reach, knowledge and behaviour. On the right, the graphic shows the assumptions and risks related to reach, knowledge and behaviour.

The principal activities/outputs assumptions and risks of the communication and advice pathway are:

  • RSO is qualified (training/experience)
  • RSO has the required soft skills (i.e. leadership, communication, persuasion)
  • RSO is knowledgeable of regulatory requirements
  • RSO is supported by management (including AA and when applicable RSC)
  • Adequate resources are available to implement/maintain the RPP ($, time, etc.)
  • RSO is committed (wants to positively influence)

Narrative for Communication and advice pathway

Communication and advice activities

Communication and advisory activities include the development and dissemination of info sessions, messages, newsletters and emails.

Reach

Reach is the extent to which the RSO, in the conduct of the communication and advice activities, is able to influence RPP users (including NEWs and non-NEWs) and management.

In order for the RSO to be effective in reach, the evaluation assumes that:

  • RSO is able to identify all RPP users and management (including RSC members)
  • RSO understands licensee operations as they relate to RPP
  • RSO is able to contact RPP users and management

Knowledge

Immediate outcome of the communication and advice pathway is for RPP users and management to understand broad licensee RPP objectives (capabilities). To do so, they need to have access to the RSO (opportunity) and acknowledge and value RPP communication (motivation).

For the RPP users and management to gain this knowledge and understand broad licensee RPP objectives, it is assumed that:

  • RPP communications (including performance information and feedback) are understood and relevant (capabilities)
  • RSO engages with RPP users (NEWs and non-NEWs) and management (opportunities)
  • RSO is located in close proximity to workers and able to provide just-in-time advice and guidance (opportunities)
  • Workers and management are engaged in communication (motivation)
  • Workers and management recognize the importance of RPP (motivation)

Behaviour

The intermediate outcome of this pathway is for the RPP users and management to become engaged in RPP. To achieve this, it is assumed that:

  • Workers and management:
    • Understand their role in relation to RP practices
    • Recognize their influence on RPP performance
    • Are committed (want to positively influence)
    • Value RPP in support of their business/operations (alignment among objective)
  • RSO is credible, trusted and recognized as RP authority

One additional assumption was identified for the communication and advice pathway. In addition to the original capacity change assumptions related to opportunity, it is valuable to have the RSO located in close proximity to the workers. Case study evidence highlights that when the RSO was not located in proximity to workers, they were not able to provide valuable just-in-time advice and feedback on practices to workers because they could not observe the workers in action. Having the RSO’s work location close to the operations enables daily walk-throughs and quick exchanges for seeking advice and providing clarifications. These exchanges became more difficult and more formalized when attempted over the phone or via email instead of in person.

From the RSOs’ perspective, the factor that most significantly impacted their ability to communicate and provide advice to workers and management was the lack of time. They reported that if they had more time, their communications would be more proactive.

Monitoring, controlling and reporting activities

RSOs are responsible for the monitoring, controlling and reporting activities of the radiation protection program, to ensure that the handling of nuclear substances and radiation devices is done in accordance with regulations and licence requirements.

This involves completing a number of tasks such as designating nuclear energy workers, developing a security plan, designing and developing monitoring programs, monitoring occupational radiation doses received by workers, undertaking auditing activities, investigating and reporting on incidents of overexposure, preparing annual reports and responding to CNSC inquiries.

The information gathered through monitoring, controlling and reporting activities serves to generate a broad picture of the radiation protection program’s effectiveness and to provide different groups of stakeholders with program information.

Findings

RSOs have access to information that allows them to adequately assess the effectiveness of their radiation protection programs. Information and reports generated by RSOs reach their target audience and, for the most part, those stakeholders find them useful.

This information leads licensees to improve their practices and procedures and helps the CNSC to prepare for its planned inspections.

In most cases, RSOs use the information to prioritize where to allocate their internal inspection effort, making the allocation of internal inspection effort risk informed. Having adequate time to complete follow-ups after internal audit activities is key to a successful audit program.

Supporting evidence

Reach Is the information produced by the RSO reaching management, workers, radiation safety committee members and the CNSC as intended?

To effectively share performance information, RSOs should know the information needs of program stakeholders. This is fairly easy for defined groups such as management, radiation safety committee members and the CNSC, but for workers this can become complicated as the group can be large and diverse.

Secondary data analysis indicates that for the most part RSOs know who forms part of the population of workers, particularly when it comes to nuclear energy workers, as they keep records of their names and job categories (figure 5). Annual compliance with this regulatory requirement has been consistently high, at 99% or 100% over the past few years.

Figure 5: RSOs keep records of nuclear energy workers names and job categories

Figure 5
Description

The data from 2007 to 2017 shows a high level of compliance

  2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
Compliant 96% 96% 89% 97% 98% 99% 99% 100% 99% 100% 99%
Non-compliant 4% 4% 11% 3% 2% 1% 1% 0% 1% 0% 1%

83% of the RSOs (n=212 Footnote 19) believe that information and reports they produce reach the target audiences such as workers, management, radiation safety committee members or the CNSC. Many indicated that they disseminate information to workers through their management. A minority of part-time RSOs in the medical sector in large hospitals with multiple locations felt that they were not always able to reach their target audience. They feel that, due to the lack of adequate resourcing of the RSO function, their ability to monitor and report adequately is limited.

The majority of stakeholders interviewed and surveyed believe that they receive the information on the radiation protection program that they need. 90% of the workers interviewed (26/29) feel that they receive clear feedback on their work performance. A minority of workers indicate that they do not receive it from the RSO, but that they receive it directly from their line manager.

92% of the applicant authorities surveyed (45/49) and 94% of the radiation safety committee members surveyed (15/16) believe that they receive relevant and timely program performance information from RSOs. A minority either do not receive information or are unsure if they do.

The evaluators further explored time, as it could be a barrier or challenge to the RSOs’ ability to undertake their monitoring, controlling and reporting activities. 73% of the RSOs interviewed (16/22) feel that they have sufficient time to prepare reports.

Some part-time RSOs feel that they lack time to produce reports. And some full-time RSO recognize that being full-time in their role allows them to have enough time to be effective in undertaking monitoring, controlling and reporting activities.

RSO survey responses demonstrate that when pressed for time, RSOs prioritize responding to inquiries from the CNSC and workers and focus less on internal investigations and reports.

Similarly, CNSC staff from the Directorate of Nuclear Substance Regulation (DNSR) have mixed views on whether RSOs have sufficient time to prepare reports. Many feel that full-time RSOs usually have sufficient time to prepare reports. Part-time RSOs managing complex radiation protection programs in the medical sector usually lack time to prepare reports. This can be because the size of the organization and competing organizational priorities such as patient care impact the amount of time that is attributed to the RSO function.

Knowledge – Do workers, licensee management and the CNSC have increased knowledge of the radiation protection program performance as a result of the information they receive?

As a result of the performance information provided by RSOs, licensee management and the CNSC should have increased knowledge of the radiation protection program performance in the licensee’s organization. For this to occur, workers and management should have time to review the information and want to improve the program performance.

RSOs should be able to analyze and produce program performance information and have sufficient authority to take corrective action as required.

98% of the RSOs (n=259 Footnote 20) believe that they are able to adequately assess the information of the radiation protection program in order to take corrective action as required.

Case study evidence highlights that hospitals having undergone an amalgamation may be using a number of different information technology platforms. As these different systems are not always connected or compatible with each other, corporate RSOs rely on front-line workers and site RSOs to obtain the information needed to assess the effectiveness of the radiation protection program.

83% of the RSOs interviewed (20/24) feel that they have sufficient authority to successfully implement and maintain their organization’s radiation protection program. They also feel that they have sufficient authority to take corrective action and stop work as required.

Some RSOs feel they have insufficient authority to take corrective action. They report that management does not support them in decisions to stop work. The following quote from an RSO illustrates this view: “I can make recommendations, but the manager has the final say. I’ve made recommendations that were rejected.” Other RSOs report that they have insufficient authority to take corrective action because of a lack of clear roles and responsibilities in their work description.

95% of the applicant authorities surveyed (44/46) believe that they have delegated sufficient authority to the RSO. The minority view was that they did not know whether or not sufficient authority had been delegated.

Across both sectors, 64% of the RSOs surveyed (152/236) report having enforced radiation protection practices or procedures in their organization. Of this group, 96% (146/152) were successful in doing so. Challenges to enforcing radiation practices or procedures are:

  • worker complacency
  • worker failure to understand their obligations
  • unclear CNSC regulations
  • lack of management support of RSO authority in the organization
Image indicates that 96% of the RSOs believe that they are successful at enforcing radiation protection practices.

89% of the RSOs surveyed (211/236) feel that workers are motivated to improve the radiation protection program performance. In instances of worker non-compliance, RSOs report using strategies such as speaking with workers and escalating issues to management through a progressive disciplinary approach. These strategies were reported to be effective.

Radiation protection program stakeholders increase their knowledge of the radiation protection program performance based on the information they receive from their RSOs. 92% of the applicant authorities surveyed (45/49) believe that they increase their knowledge of program performance by receiving and reviewing relevant and timely information such as performance data and investigation reports from the RSOs.

8% of the applicant authorities surveyed (4/49) feel that they do not receive any information from their RSOs or do not know if they receive information from their RSOs.

81% of the radiation safety committee members interviewed (13/16) feel that they receive sufficient information from the RSOs to assess the effectiveness of the radiation protection program. 19% (3/16) believe that performance information is not shared either because committee meetings are not well managed or because the RSO function is not properly resourced in their organization.

90% of the workers (n=44 Footnote 21) feel that they receive clear feedback from RSOs on their work performance. A minority of workers report not receiving feedback on their work performance because they receive it directly from their manager and not the RSO. Aligned with what workers report, 88% of the RSOs interviewed (15/17) are confident that program-related performance information such as dose information is easily accessible to workers, either directly or via their manager.

In accordance with their licence, every DNSR licensee must submit an annual compliance report (ACR) and each licence has a licence condition specifying the due date for the report submission. Licensees also must report any events related to their licensed activities. This allows the CNSC to verify that appropriate action has taken place. The evaluators attempted to determine whether the information that is provided in the ACRs is useful in supporting the CNSC’s assessment of licensee compliance.

Data analysis shows that for the most part licensees are submitting their ACR on time to the CNSC (figure 6). The timeliness of submission of the ACR to the CNSC has significantly improved over recent years, with over 80% of the licensees submitting on time in 2017.

Figure 6: Percentage of ACRs submitted on time

Figure 6
Description

The data from 2007 to 2017 shows the compliance is satisfactory and trending upward

  2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
% of ACR submitted on time 62% 65% 66% 64% 70% 68% 76% 75% 66% 77% 83%

53% of the RSOs interviewed (8/15) are either unsure or do not think that the information they provide to the CNSC is useful in supporting the CNSC’s assessment of their compliance. 29% of the RSOs interviewed (8/28) do not see the value or usefulness of the ACR. Some RSOs believe that the purpose of the ACR is unclear and that the CNSC should clarify the purpose and communicate it to the licensees. Opposite to the view from the RSOs, 70% of the DNSR staff interviewed (7/9) believe that the information they receive from the RSOs is useful in supporting the CNSC’s assessment of licensees’ compliance. The minority view from DNSR staff was that the ACRs are not very useful but rather a “check-the-box exercise” and that information (such as event reports) and discussions with RSOs are more useful.

Behaviour – As a result of the information and reports produced by the RSO…

  • Is the RSO inspection work risk informed?
  • Do licensees adjust their practices or procedures?
  • Does the CNSC consider the licensee input in its assessment of compliance?

1. Is the RSO inspection effort risk informed?

In the context of their duties, RSOs are required to develop and implement programs to inspect and review licensed activities and implement remedial actions to correct any deficiencies Footnote 22. These are often referred to as internal audits or internal inspections. The RSOs’ inspection efforts should be risk informed and aligned with CNSC compliance expectations. In other words, RSOs may focus their attention on inspecting areas where radiation protection performance is lower or considered to be “at risk”.

90% of the RSOs surveyed (211/234) consider the past performance of their organization’s radiation protection program as part of their internal audit and monitoring activities. A minority did not believe that they consider past performance because they used a predetermined inspection frequency that does not change regardless of how they assess performance.

RSOs with successful audit programs believe that sufficient time needs to be allocated to the RSO for this task to be well performed. Appropriate resourcing of the function is essential to ensure that internal audits are performed frequently and that follow-ups and improvement measures are undertaken. The following quote illustrates this view: “There’s no point in doing internal audits if you can’t rapidly follow up on actions.” Other internal audit success factors highlighted by RSOs are:

  1. quality of communication with workers
  2. frequency of the internal audit activities
  3. strong record-keeping practices

Some RSOs find it useful to adopt the CNSC Type II inspection worksheet to develop their own internal audit checklist. This also helps them prepare for CNSC inspections. It was suggested that the CNSC should provide the inspection worksheet online in an editable format (MS Word versus PDF).

2. Do licensees adjust their practices or procedures?

The performance results including internal and external inspection findings that are gathered and shared by RSOs, should lead licensees to make adjustments to their procedures and practices.

93% of the RSOs surveyed (236/255) believe that their organization adjusts its practices and procedures based on the assessment of the radiation protection program performance. Low dosimetry readings of workers are often cited as an example of a result that demonstrates improvements over the years.

Those who do not see their organization improving its practices or procedures believe that it is because the radiation protection program is not sufficiently resourced. As a result, their organization is still very reactive; it relies on the regulator’s inspections and enforcement actions to drive change.

In agreement with the RSOs’ views, 91% of the applicant authorities (n=53 Footnote 23) and radiation safety committee members interviewed (94%, 15/16) believe that they receive program information from RSOs that leads to improvements in practices or procedures. The minority view was that they are unsure and unable to determine if the program is making improvements based on performance information.

At the individual level, 100% of the workers interviewed (20/20) feel that they receive clear feedback from the RSO on their work performance that leads to adjustments to their work practices.

Figure 7: Licensees believe that they adjust their practices based on the assessment of their performance

Figure 7
Description

93% of RSOs, 91% of AAs and 90% of workers believe they adjust their practices based on the assessment of their performance

3. Does the CNSC consider the licensee input in its assessment of compliance?

Overall the view is that the CNSC relies mostly on its own inspection findings, not on the ACR or the event reports, to assess licensee performance. The ACR is seen as a tool to help inspectors prepare for an inspection, but the ACR does not influence the level of compliance verification effort applied by the CNSC.

The event reports can to some degree influence the allocation of inspection effort and this was seen through case study evidence. The 11 case study organizations reported a total of 243 events between the period of 2007 and 2017. Most events were the result of loss of containment. Based on the information captured in the Event Information Tracking System (EITS), only one event had safety significance. A 2017 event was rated as a level 1 on the International Nuclear and Radiological Event Scale (INES), which means it was an anomaly.

This specific event was related to skin contamination exceeding the regulatory extremity dose limit for a nuclear energy worker and led the CNSC to increase its compliance verification effort for that licensee. Also in reviewing the Type I inspection reports for these 11 organizations, a number of reports highlighted that reported events formed part of the basis for the CNSC to undertake a Type I inspection.  

Case study findings relating to monitoring, controlling and reporting

In the medical sector, RSOs working in large hospitals with complex programs and a geographic disparity of numerous sites, feel that there is a lack of CNSC guidance.

Although the relationship between RSOs and licencing officers was good, the lack of CNSC guidance led to many exchanges to address compliance issues, making these exchanges inefficient at times.

In a few cases, the RSO was also faced with a lack of authority to enforce radiation protection policies and procedures, which limited the RSO’s influence on workers to improve their practices based on their performance.

RSOs working in small hospitals are able to monitor the practices of workers on a daily basis. This leads to frequent and useful reporting to workers and management on program performance.

In the university and academic/research sector, RSOs operated in an environment conducive to monitoring, controlling and reporting on the radiation protection program. One RSO experienced challenges in being effective because their office has been moved and they are no longer working with the operations side. This limits the RSO’s ability to provide just-in-time performance feedback to workers.

Reviewing the theory of change
Monitoring, Controlling and Reporting Pathway
Description

A graphic representation of a pathway that is a sequence of steps that are required from the RSO to get from monitoring, controlling and reporting activities to their impacts. The graphic reads from bottom to top. The principal assumptions and risks related to the monitoring, controlling and reporting pathway are listed at the bottom. Above on the left, the graphic shows the pathway sequence divided in 4 categories: activities, reach, knowledge and behaviour. On the right, the graphic shows the assumptions and risks related to reach, knowledge and behaviour.

The principal activities/outputs assumptions and risks of the monitoring, controlling and reporting pathway are:

  • RSO is qualified (training/experience)
  • RSO has the required soft skills (i.e. leadership, communication, persuasion)
  • RSO is knowledgeable of regulatory requirements
  • RSO is supported by management (including AA and when applicable RSC)
  • Adequate resources are available to implement/maintain the RPP ($, time, etc.)
  • RSO is committed (wants to positively influence)

Narrative for Monitoring, controlling and reporting pathway

Monitoring, controlling and reporting activities

These activities consist of performing or overseeing: audit program, purchase authorization, investigation, reports (events/annual); list of NEWs; dosimetry record; licence renewal; response to CNSC requests.

Reach

Reach is the extent to which the RSO, in the conduct of the monitoring, controlling and reporting activities, is able to influence stakeholders involved with RPP (including users and management) and CNSC.

In order for the RSO to be effective in reach, the evaluation assumes that:

  • RSO is able to identify users, management and CNSC information needs (who, what)
  • RSO is able to provide information to users, management and CNSC at the appropriate time
  • RSO is able to contact workers, management and CNSC

Knowledge

The immediate outcomes of the monitoring, controlling and reporting pathway is for licensee management and CNSC have an increased knowledge of RPP performance and directly involved RPP users gain awareness of their own RPP performance (capabilities). To do so, RPP users and management must have time to review RPP performance (opportunity) and want to improve program performance (motivation).

For the RPP users and management to increase their knowledge of RPP performance, it is assumed that:

  • RPP performance information and feedback is understood and relevant (capabilities)
  • Licensee management (including radiation safety committee) and CNSC obtain timely information on RPP performance (opportunity)
  • Workers value receiving feedback on their performance (motivation)

Behaviour

The intermediate outcomes of this pathway are for licensees to adjust their practices or procedures, for the RSO inspection effort to be risk informed and for the CNSC to consider licensee input in its assessment of compliance.

For these behaviours changes to occur, it is assumed that:

  • CNSC and licensee relationship allows for open communication and resolution of non-compliance
  • RSO considers RP performance as part of inspection planning
  • Consequences of non-compliance (e.g. penalties/AMPS) are clearly communicated
  • RSO has authority to enforce RP policies and procedures
  • Workers and management are committed to improving their RP practices
  • RSO is credible, trusted and recognized as RP authority

For this pathway, no capacity change assumptions were identified as missing. What emerged was the importance of the credibility of the RSO as a trusted source of information on radiation protection. In the case studies where the RSO was not seen as a credible or trusted source of information (either because they were perceived as lacking experience working in the licensed activity or as an RSO), the workers’ motivation was impacted as they did not value receiving feedback on their work performance from the RSO. Also, workers did not always find that the information communicated by the RSO to be relevant to their operations/activities.

Some RSOs operate in high knowledge environments such as universities. For this reason, they may not be perceived as credible and trusted experts on radiation by key radiation protection program stakeholders.

From the RSOs’ perspective, RSOs lack time to conduct audit follow-ups. This was also highlighted through the case studies, where the lack of time led to RSO activities/tasks being prioritized in a way that resulted in dropping or delaying some activities such as internal audit follow-ups.

Continuous improvements activities

RSOs undertake continuous improvement activities to keep abreast of industry good practices and lessons learned in order to improve the licensee’s radiation protection program.

RSOs come across good practices and lessons learned by reviewing upcoming changes in regulatory requirements, keeping abreast of reportable events in Canada and internationally, attending conferences and participating in training and CNSC outreach events.

Once aware of the good practices and lessons learned, RSOs should share this information with colleagues and integrate the relevant practices into their radiation protection program.

Findings

The degree of access to radiation protection good practices and lessons learned is varied due to barriers such as lack of time and/or funding to attend learning events, lack of management support or lack of adequate access to online resources.

When RSOs do report having access to good practices and lessons learned, they share this information with workers and integrate it into radiation protection programs. In most organizations, the radiation safety committee is the forum where discussions on good practices take place. Sharing of good practices with management, including the applicant authority, is less common unless the applicant authority is a member of the radiation safety committee. Sharing of information within the broad RSO community is limited, particularly for some RSOs in the medical sector.

Supporting evidence

Reach – Do RSOs have access to radiation protection good practices and lessons learned?

70% of the RSOs (n=179 Footnote 24) believe that they have access to radiation protection good practices and lessons learned. RSOs report receiving this information from a number of sources, including other RSOs, the CNSC, the internet, the Canadian Radiation Protection Association (CRPA), the DNSR Newsletter, the Canadian University RSO listserv (CURSO-l) and the Canadian Organization of Medical Physicists (COMP). About half of the RSOs interviewed attend CRPA training events or the annual conference, and find these activities useful. Other RSOs participate in training through vendors and attend CNSC outreach sessions, and find these useful.

Some RSOs in the medical sector do not feel that they have access to radiation protection good practices and lessons learned, mainly due to barriers including lack of time, lack of budget, lack of available information on good practices and lack of management support for continuous learning.

58% of the RSOs surveyed (135/233) believe that they are able to keep abreast of good practices in radiation protection; 5% of the RSOs surveyed (12/233) can only do so “very little”. They feel unable to do so because of:

  • lack of time
  • lack of budget to attend conferences or other learning events
  • lack of available information on radiation protection good practices

“I do not have funds and time available to attend conferences or events. Moreover, it is not encouraged by supervisors as it is considered trivial to develop skills in radiation safety.” – RSO

Many RSOs in the medical sector feel that they are restricted by a travel budget freeze that limits their ability to participate in learning events. To overcome this barrier, some RSOs have paid out of pocket to attend learning events.

It is not a CNSC requirement that RSOs set aside time to maintain or increase their knowledge of radiation safety best practices. The Public Health Agency of Canada (PHAC) and the Canadian Food Inspection Agency (CFIA) provide clear guidance concerning their expectations on continuous learning for biological safety advisors. The Canadian Biosafety Handbook states that “A successful biosafety program is regularly reviewed at the program management level and continually improved to remain relevant, applicable, and effective.” In addition, biological safety advisors are encouraged to update and improve their knowledge of biosafety-related topics on a regular basis. It is also an expectation that senior management “continuously improve […] the biosafety program” Footnote 25.

CNSC staff in DNSR have mixed views on whether RSOs have access to information on radiation protection good practices and lessons learned. They feel that the CNSC offers good information online but also recognize that the CNSC’s website can be challenging to navigate. During the 2018 CRPA conference workshop, RSOs suggested including a section on frequently asked questions on the CNSC website with information on past incidents and their resolution. It was also suggested that general guidance on how to troubleshoot common problems could be included, and that this would greatly benefit new RSOs.

Many RSOs feel that the DNSR Newsletter distribution strategy could be improved. This topic recurred during discussions on continuous improvement, both during interviews and during the CNSC workshop at the CRPA annual conference in May 2018. RSOs highlighted that the newsletter is a useful tool as it provides good information on the maintenance and improvement of radiation protection programs. Improvements to the newsletter were suggested, such as having more frequent publications to ensure that information is disseminated in a timely manner.

Another suggestion brought forward at the conference was for the CNSC to increase the use of webinars as part of the DNSR outreach strategy. This could provide a forum for RSOs and the regulator to exchange information on and discuss regulatory issues. It could also address the challenge faced by RSOs due to budget restraints and travel freezes in the medical sector.

RSOs also offered their views on the CNSC email distribution list. Many RSOs would prefer to have a personalized email distribution list that would send relevant information to DNSR RSOs. The current distribution approach uses the generic CNSC distribution list and RSOs feel inundated with CNSC communications. As a result of this information overload, recipients may not read all emails and may overlook important information. They also suggest that just-in-time information could be pushed out via social media (e.g., Twitter, Facebook) rather than via the newsletter or in an email.

Knowledge – Are good practices and lessons learned shared with stakeholders?

As a result of having access to radiation protection good practices and lessons learned, the RSOs should be informed of industry good practices and disseminate this information to concerned stakeholders. The evaluation sought to understand if RSOs shared good practices and lessons learned with workers, management and other RSOs.

96% of the RSOs surveyed (223/233) share good practices and lessons learned with site RSOs and workers. They share them through different channels: in person, via email, during team meetings, in bulletin board posts or through training and/or hands-on demonstrations. Sharing of good practices with RSOs in the larger community is less frequent. Survey results reveal that 60% (140/233) of the RSOs believe that they share good practices with other RSOs. RSOs who believe otherwise were prompted to explain why. They responded that it was because:

  • they have no opportunities to connect with other RSOs
  • they do not know who to contact
  • they do not know that an RSO community exists

Sharing good practices with management is not as prevalent as it is with workers. 57% of the RSOs surveyed (133/233) feel that they share good practices that relate to radiation protection with the applicant authority. In contrast, 98% of the applicant authorities (n=55 Footnote 26) feel that they receive information from RSOs on good practices. Many applicant authorities report they obtain this information by participating in radiation safety committee meetings or by reading the committee minutes. Where organizations have a radiation safety committee, all stakeholders agree that the committee is a good forum for sharing good practices.

Behaviour – Under which conditions do RSOs integrate good practices and lessons learned into the radiation protection program?

As a result of the time spent by RSOs keeping informed of industry good practices through training and conferences and by sharing information with other industry RSOs, feedback loops should be in place to ensure that the program is continually updated and improved.

This includes improving the training that is offered to workers, and integrating any new information on good practices into their communications with all stakeholders. The evaluation sought to understand which conditions promoted the integration of good practices into radiation protection programs.

Image indicates that 92% of the RSOs believe that they integrate good practices and lessons learned into their radiation protection program.

92% of the RSOs (n=239 Footnote 27) believe that when they access good practices and lessons learned, they are integrated into their program. This view was also supported by 96% of the applicant authorities surveyed (45/47). They also believe that the RSO integrates radiation protection good practices into the radiation protection program of the organization. Examples of program improvements include: the review of standard operating procedures; the radiation safety committee oversight; and the improvements to training programs. A minority of RSOs believe that they do very little to integrate good practices, due to lack of time or access to information.

CNSC staff in DNSR have mixed views on whether RSOs have integrated good practices into the radiation protection program. Some feel that the integration of good practices increases if RSOs are engaged and if there is a driver for improvement within the organization. This driver could be an event that compels the RSO and the organization to improve the program.

Some CRPA members have offered to share their knowledge and experience in their specialty areas with licensees. Their contact information is listed on the CNSC’s external website Footnote 28. Any interested RSO can contact the mentors directly. Of the 75 RSOs present at the 2018 CRPA annual conference workshop offered by the CNSC on the evaluation of the role of RSOs, only two knew of the existence of the mentorship program for RSOs that is promoted on the CNSC’s website.

RSOs, applicant authorities and radiation safety committee members identified conditions that promote the integration of good practices into their program. These conditions are stakeholder (management and worker) support, RSO visibility and accessibility, RSO collaborative approach and sufficient resourcing of the RSO function.

Case study findings relating to continuous improvement

In the medical sector, for those RSOs working in large hospitals with complex programs and with geographic disparity of numerous sites, the lack of time allocated to the RSO function limits the ability of the RSO to invest in the program’s continuous improvement.

Reviewing the theory of change
Continuous improvement pathway
Description

A graphic representation of a pathway that is a sequence of steps that are required from the RSO to get from continuous improvement activities to their impacts. The graphic reads from bottom to top. The principal assumptions and risks related to the continuous improvement pathway are listed at the bottom. Above on the left, the graphic shows the pathway sequence divided in 4 categories: activities, reach, knowledge and behaviour. On the right, the graphic shows the assumptions and risks related to reach, knowledge and behaviour.

The principal activities/outputs assumptions and risks of the continuous improvement pathway are:

  • RSO is qualified (training/experience)
  • RSO has the required soft skills (i.e. leadership, communication, persuasion)
  • RSO is knowledgeable of regulatory requirements
  • RSO is supported by management (including AA and when applicable RSC)
  • Adequate resources are available to implement/maintain the RPP ($, time, etc.)
  • RSO has access to industry good practices and OPEX
  • RSO is committed (wants to positively influence)

Narrative for continuous improvement pathway

Continuous improvement activities

These activities consist of reviewing and/or contributing to good practices, lessons learned and consultation on draft regulatory documents.

Reach

Reach is the extent to which the RSO, in the conduct of continuous improvement activities, is able to reach other RSOs, site RSOs, RSC, RPP management (including AA).

In order for the RSO to be effective in reach, the evaluation assumes that:

  • RSO is able to identify who would benefit from specific good practices
  • RSO is able to identify which good practices/OPEX applies to their licensee RPP
  • RSO is able to provide timely information to Site RSOs, RSC and RPP management

Knowledge

The immediate outcomes of the continuous improvement pathway are for RSO, site RSO, RSC and RPP management to be aware of RP good practices(capabilities). To do so, they need to have sufficient time to review information received (opportunity) and they need to be engaged in RPP improvement (motivation).

For the RSO, site RSO, RSC and RPP management to integrate RP good practices in RPP, it is assumed that:

  • RP good practices and OPEX shared is understood and relevant (capabilities)
  • Site RSOs, RSC and RPP management see how they can support RPP continuous improvement (opportunities)
  • RSO can network or connect with other RSOs in the broader community (opportunities)
  • Technology allows for virtual participation in learning events (opportunities)
  • Site RSOs, RSC and RPP management value receiving RP good practices (motivation)

Behaviour

The intermediate outcomes of the continuous improvement are for RSO to integrate RP good practices in RPP.

For these behaviours changes to occur, it is assumed that:

  • RSO is able to incorporate good practices in RP and modifies training, communication and advice, monitoring/controlling and reporting as required
  • RSO is supported to bring about change
  • RSC and RPP management (including applicant authority) are motivated to positively influence RPP
  • RSO is credible, trusted and recognized as RP authority

Two capacity change assumptions were identified as missing from the original theory of change under the continuous improvement pathway. The first was the assumption that RSOs have an opportunity to connect with other RSOS. It is important for RSOs to be able to network and exchange with other RSOs to share best practices. Case studies highlight that RSOs overseeing radiation protection programs that have had performance issues hesitate to reach out to the CNSC when they have questions. They can also be isolated from the broader community (geographically and/or because of a persistent lack of management support or funding to attend learning events). As a result, they are unsure of whom to ask for advice and guidance on particular issues.

The other capacity change assumption that was missing was the opportunity for RSOs to participate in learning events virtually. Given the wide range of technology options, it would be reasonable to assume that provisions for virtual participation can be made for many learning events. Those RSOs who are unable to travel due to budget restrictions would then be able to participate in events without having to incur travel costs.

In terms of the overarching activity assumptions, what was missing from the original theory of change was the assumption that RSOs have access to industry good practices and OPEX (operational experience). From the RSO’s perspective, many who were unable to attend learning events felt that management did not understand the value of participating in continuous learning activities.

Other influencing elements

The evaluation team was particularly interested in understanding if there were consequences that emerged as a result of the way the CNSC regulates nuclear substances and radiation devices licensees.

In addition to the four key areas of RSO activity (training; communication and advice; monitoring, controlling and reporting; and continuous improvement), other elements were explored. These elements include the adequacy of governance models, relationship with the applicant authority, certification, and alignment across regulatory bodies and the relationship with the regulator.

Are the right governance models in place?

56% of the RSOs surveyed (53/95) feel that their governance structure supports them in successfully doing their job. 44% (42/95) believe that their governance structure can impede them from successfully doing their job. The main impediments are:

  • an RSO’s level in the organization’s hierarchy
  • a lack of time to fulfill RSO duties as a result of too many other duties
  • lack of management support for the RSO role
  • little value or importance being placed on the RSO role
  • limited access to management

All CNSC staff from DNSR who participated in the evaluation feel that the governance structure of the RSO’s institution directly impacts their ability to fulfill their role and responsibilities. A barrier to the RSO’s effectiveness can exist in large institutions where a number of management layers lie between the RSO and the applicant authority. The following quote illustrates this view: “Large organizations may require the RSO to jump through many hoops to get to the applicant authority.”

Applicant authorities have the most positive views among all groups on the effectiveness of their governance structure. 96% of the applicant authorities surveyed (44/46) believe that their organization’s governance structure allows the RSO to be effective in their role. Only one applicant authority did not believe their governance structure allowed the RSO to be effective, while another felt that they were not in a position to answer the question.

Relationship with the applicant authority

It is unclear whether the current relationship as described in REGDOC-1.6.1, Licence Application Guide: Nuclear Substances and Radiation Devices, between RSOs and the applicant authority is the right one. The evidence suggests that the governance structure, especially in large, complex organizations, can make it challenging for RSOs in the medical sector to be effective in their role when many management layers separate them from the applicant authority. No other model or approach was suggested. The view expressed was that in large organizations, reducing the management layers between the RSO and the applicant authority would be beneficial and increase RSO effectiveness.

79% of the RSOs surveyed (177/225) feel well supported by the applicant authority. 15% of the RSOs (34/225) do not feel well supported. When prompted to further explain why they do not feel supported, they identified the following themes:

  • the applicant authority is not involved in radiation protection
  • there has never been any contact between the RSO and the applicant authority
  • the applicant authority does not trust the RSO

Case studies compared large hospitals in the medical sector that had undergone amalgamation. The findings reveal that the relationship between the RSO and the applicant authority is not a necessary condition for the success of a radiation protection program. Other factors such as having an adequately resourced program and support for their direct manager are also significant predictors of program success.

In practice, the CNSC encourages licensees to designate an applicant authority at a senior level in the organization. Adopting a different approach, PHAC licensees determine the appropriate governance model and reporting relationship (between the biological safety officer and senior management) for their organization.

In large organizations such as universities, the licence holder can be the vice-president of research; in small organizations the RSO can also be the licence holder. The representative from PHAC indicated that there have not been any issues with this model and that lines of communication seem to be open between the RSO and senior management. PHAC’s perspective is that this approach is effective, as it allows each licensee to adopt a model that works for the nature and magnitude of their operations.

Some CNSC staff from DNSR believe the reporting relationship model between the RSO and the applicant authority is adequate. Others feel that in large organizations the applicant authority and the RSO are separated by too many layers of management. As a result, the applicant authority is not always aware of any issues with the radiation protection program unless the CNSC becomes involved.

REGDOC-1.6.1 provides guidance and expectations on the radiation protection program governance and management structure. It notes that “the person responsible for radiation safety, […] should have the capacity for direct reporting to the applicant authority”. This regulatory document is well aligned with good practices found in literature, such as the study from Hofmann, Jacobs and Landry (1995), which notes that an organization can improve its safety attitudes and behaviours by appointing safety representatives. That research also shows that simply appointing safety representatives is not enough. Safety representatives need to have legitimate power within the organization and must have access to individuals in decision-making positions.

Does the CNSC provide sufficient guidance?

To facilitate and clarify expectations around the development and implementation of an effective radiation protection program, the CNSC is currently preparing a new regulatory document for nuclear substances and radiation devices licensees (REGDOC-1.6.2). The findings and conclusions from this evaluation could inform the development of this regulatory document.

The evaluation sought to understand whether the guidance currently available to licensees is sufficient (particularly in areas such as RSO roles and responsibilities, and the implementation and maintenance of the radiation protection program).

Currently, the expectations for nuclear substances and radiation devices licensees with respect to an effective radiation protection program can be found in CNSC publications:

  • REGDOC-1.6.1, the licence application guide for nuclear substances and radiation devices
  • G-121, the regulatory guide that provides information to medical and academic/research licensees on how to design and implement radiation protection programs
  • REGDOC-2.2.2, the regulatory guide that provides guidance to licensees for developing radiation safety training programs for workers

REGDOC-1.6.1provides guidance and expectations on the role and responsibilities of RSOs Footnote 29. As part of the licence application process, the CNSC assesses whether RSOs have sufficient knowledge and expertise with regard to the applicant’s proposed activities. This is done by reviewing the job description of the applicant’s RSO, including roles and responsibilities, qualifications and authority. Research from Hofmann, Jacobs and Landry (1995), which highlights the importance of having job descriptions that have clear roles and responsibilities regarding safety, would also support the CNSC in clarifying expectations around RSO job descriptions.

Both RSOs and CNSC staff from DNSR have mixed views on whether or not the CNSC provides sufficient guidance on the RSO role and responsibilities. Some CNSC staff from DNSR feel that the CNSC provides good information in the licence application guide and the regulatory documents.

Others feel that the CNSC is not sufficiently prescriptive and that this leaves licensees uncertain of the regulator’s expectations. The following quote illustrates this view: “The REGs are grey – expectations are unclear. Leads to the interpretation of the intent of the REGs.”

CNSC staff from DNSR, radiation safety committee members and RSOs identified that further guidance in the following areas would be beneficial:

  • specific regulator expectations concerning internal audits (format, frequency, timing)
  • an appropriate RSO job description (roles and responsibilities and level of effort required based on the nature and magnitude of licensee operations)
  • adequate resourcing levels based on the nature and magnitude of the licensed activity
  • role of the radiation safety committee (composition and purpose)

On the topic of level of effort allocated to the RSO function based on the nature and magnitude of the licensee operations, the evaluation looked at what could be learned from others nationally or internationally.

Nationally, a formula is being developed for Class II RSOs and tested by Class II licensing officers to determine the appropriate level of effort, given the licensed activity. No evidence has been found of nuclear substances and radiation devices licensing officers using a similar formula or approach in their assessment. In the United States, the licence applicants are required to identify the number of hours per week allocated to the RSO to perform their duties.

Internationally, the United States Nuclear Regulatory Commission (U.S. NRC) in its review of applications does not use a formula to determine if the RSO function is adequately resourced. Experienced licensing staff perform this assessment case by case, using their professional judgment. They take a number of factors into consideration when making the determination: the number of staff, the magnitude of operations, the geographic disparity of the organization and the proposed number of hours per week that have been allocated to the RSO to perform their duties. In comparison, the CNSC currently does not request information such as number of hours allocated to the RSO function to support this type of assessment by licensing officers.

In the United Kingdom, the Health and Safety Executive (HSE) uses a ratio of 1 radiation protection supervisor per 20 employees as a general rule to ensure a successful program. This ratio is not found in regulations or guidance documents but has been informally communicated to industry.

Many RSOs and applicant authorities would like to see the CNSC take on a larger role in the sharing of best practices across licensees. Some of this sharing already occurs via the DNSR Newsletter, but RSOs feel that it is published too infrequently, stressing that “by the time they receive it, the information is considered to be old news”. Some of these RSOs suggested that the CNSC transmit to them relevant information using social media to increase its timeliness.

The overarching sentiment is that the regulator is aware of industry best practices and should do more to share this information with licensees as part of the CNSC’s compliance promotion strategy.

Would there be advantages to certification?

The question of CNSC certification for nuclear substances and radiation devices RSOs was an area of interest because the CNSC certifies Class II RSOs. Also, nuclear substances and radiation devices RSOs in other countries such as the United States are certified.

All stakeholder groups expressed mixed views on the potential benefit of certification for nuclear substances and radiation devices RSOs. Some even questioned the need for Class II RSO certification Footnote 30 by the CNSC.

RSOs with limited access to continuous learning opportunities consider that certification would be a means to ensure access to continuous learning. They believe that funding for these activities is not available, particularly in times of fiscal restraint, because there is no requirement for continuous learning.

“When an RSO spends the majority of the time enrolling personnel for online training and updating user lists, having certification would not benefit these situations. Having a strong, open CNSC relationship to ask questions and learn in a positive environment would.” – RSO

Close to half of the RSOs do not see a significant value in CNSC-led certification. Rather, this group feels that the CNSC could work with organizations such as the CRPA to develop a standardized and recognized RSO training program.

RSOs who supported having certification identified the following potential benefits:

  • certification could be an opportunity for RSOs to take part in continuous learning
  • certification could ensure a standardized base level of technical and regulatory knowledge
  • certification could ensure that the RSO in the position is competent and qualified
  • certification could increase the RSOs’ credibility or weight/influence within their organization

The following quote illustrates the support for certification: “I think that there should be RSO certification. Credentials to maintain a certain level of education should be required and mandatory attendance for continuing education and refresher courses […].”

A few RSOs wrote comments in the survey expressing that they recognized that professional associations could play a role in certification. The following quote illustrates these views: “I believe recognition of existing certifications (i.e., COMP and CRPA) are more important. I believe the regulator could look at some certifications to see if they meet a minimum standard and leave it to those organizations to certify individuals.”

One applicant authority commented on the question of certification, cautioning that “certification could be dangerous, not the solution for nuclear substances, even if it is appropriate for Class II RSOs; what might be required for nuclear substances RSOs is requirement and support for continuous education.” Another applicant authority feels that the certification process could be useful and that the regulator “…should work with [a] training institute to build RSO capacity and build opportunities to connect within [the] community of RSOs.”

The majority of CNSC staff from DNSR (75%, 6/8) believe that certification is neither necessary, nor the solution to improve RSO effectiveness. Most do not believe that certification would be of value because it would not guarantee that RSOs would more positively influence the radiation protection program performance. The following quote illustrates this view: “Certification is not a guarantee of a good RSO, as it doesn’t assess all of the important attributes of the role. On the other hand, certification may give more weight to the role.” Many indicate that the main challenges facing RSOs is the lack of time allocated to their role and limited access to continuous learning. The view was that certification would not eliminate these challenges.

25% of the CNSC staff from DNSR (2/8) believe that the CNSC should certify nuclear substances and radiation devices RSOs in the medical and academic/research sectors. The following quote illustrates this view: “Because there is no certification, there is a risk of a lack of consistency of the role of RSOs.”

The evaluation reviewed different approaches to certification of “safety officers” internationally and nationally. On the international front, both the United States and United Kingdom models served as comparators. Domestically, the evaluation examined the approach used to certify Class II licensees. This revealed different certification models that fall under two categories: 1) certification led by the regulator, and 2) certification led by recognized professional associations. No evidence was found on the relative effectiveness of these models, although the United Kingdom through the Health and Safety Executive is transitioning from regulator-led certification to board-led certification. The United Kingdom recognizes that professional bodies are well positioned to assess the qualification and competence of radiological protection advisors. In Canada, the CRPA has expressed interest to take on the role of qualifying RSOs through certification.

In the United States, RSOs must be certified by a specialty board through a certification process recognized by the U.S. NRC, as stated in the Code of Federal Regulators. The U.S. NRC’s website lists the names of the certification boards it recognizes. To have its certification process recognized, a specialty board requires that candidate RSOs meet criteria such as education, professional experience and training.

In the United Kingdom, the employers must ensure that the radiological protection advisors meet the criteria of competence Footnote 31. The UK Health and Safety Executive is currently revising the certification process to recognize certification offered through professional bodies. The view is that the professional bodies are better positioned than the regulator to certify the advisors.

In Canada, all licensees who operate or service Class II facilities or prescribed equipment must have a certified RSO or a qualified temporary replacement. At a minimum, Class II licensees must have one certified RSO. However, depending on the organizational structure, several positions may be eligible for undergoing certification. RSO candidates must possess certain qualifications before they can be considered for certification. The certification process is initiated by the CNSC upon receipt of a completed application from the licensee. Upon review of the application, if an assessment of the material provided indicates that the candidate appears capable of performing the duties of the position, CNSC staff will contact the candidate to schedule a certification examination. If the candidate is able to clearly demonstrate their knowledge as it relates to the position of RSO within their organization, the CNSC will issue a certificate for the individual, detailing the location, organization and types of facilities for which the person is certified to serve as RSO. Once certification is issued, it remains valid for the period for which the RSO is employed by the licensee (no expiry and no requirement to renew). 

There is no requirement from PHAC for biological safety officers to be certified. It was explored in the past, and the Canadian College of Microbiologists (as the Board) would have offered the certification. However, the College did not pursue certification after PHAC decided not to require certification for biological safety officers.

The Canadian Radiation Protection Association represents approximately 280 RSOs, mainly from the academic/research and medical sectors. It operates a program for radiation safety professionals to demonstrate their knowledge and commitment to radiation protection through the Registered Radiation Safety Professional Program known as CRPA (R).

Within the Canada Labour Code, there is no requirement for occupational health and safety representatives to be certified. The Canada Labour Code states: “Every employer shall, for each work place controlled by the employer at which fewer than twenty employees are normally employed or for which an employer is not required to establish a work […] appoint the person selected […] as the health and safety representative”. Health and safety representatives receive training on health and safety per the Canada Labour Code, but do not have any certification requirements.

The CRPA has a designation process, CRPA (R). The designation of Registered Radiation Safety Professional (RRSP) is determined by academic achievement, experience in the radiation field and successful completion of an exam on identified core competencies, which is administrated by the CRPA. The purpose of the designation is to ensure that a regulator, employer or member of the public can have confidence in the qualifications of an RRSP with the CRPA (R) credential. To maintain the designation, professional or service contributions that relate to radiation safety must be submitted and reviewed by an examination committee annually. In this context, designation aims at confirming that candidates met core competencies identified by the CRPA Footnote 32.

Is there an issue with the alignment across regulatory bodies?

61% of the RSOs surveyed (67/109) believe that there are negative consequences due to the lack of alignment across regulatory bodies. Greater alignment could avoid overlap and duplication of work. The following quote illustrates this view: “Leads to confusion and misunderstanding or misinterpretation of the regulations and how to enact them. All bodies need to be on the same page and have the same knowledge base to be able to advise licensees in radiation safety.”

An example was provided by an RSO working in the commercial sector. The RSO cited a potential misalignment between the HC requirements concerning the addition of saline solution to a vial after the radioactive substance has already been added as opposed to adding the saline solution first. In the RSO’s view, this can contribute to extremity dose, but if the saline could be introduced first in the vial, there would be no exposure.

The European ALARA workshop presentation, entitled “Use of an electronic finger dosimeter in optimisation of finger doses by C. J. Martin, M. Whitby, T. Hilditch and D. Anstee”, is a study that was done on extremity exposure from radiopharmaceutical preparation which showed this step to be a significant contributor to extremity dose.

Regulatory alignment – within the CNSC

Some RSOs identified alignment within the CNSC as an area for improvement. The following quote illustrates this view: “I think lack of alignment within the CNSC (i.e., security, nuclear substances and Class II) is more significant than with other agencies.”

Other RSOs feel that there is a lack of alignment between the CNSC’s Nuclear Substances and Radiation Devices Licensing Division and Nuclear Security Division. Since the implementation of REGDOC-2.12.3, Security of Nuclear Substances: Sealed Sources, in 2013, RSOs feel that there is a lack of clarity with respect to the submission process for a security plan.

Does the relationship with the regulator allow for open communication and effective resolution of non-compliance?

Image indicates that 91% of the RSOs feel confident that their relationship with the CNSC allows for open communication and resolution of non-compliance.

91% of the RSOs in the medical and academic/research sectors (n=234 Footnote 33) are confident that their relationship with the CNSC allows for open communication and resolution of non-compliance.

A minority of RSOs believe that the relationship with the CNSC does not allow for efficient resolution of non-compliance. Some believe that effective communication with the CNSC is limited because they can only communicate with the licensing officer via email. To illustrate, one RSO reported that this resulted in numerous exchanges to resolve what was considered to be simple issue that could have been resolved more efficiently over the phone.

Other factors reported by RSOs that contribute to ineffective communication are:

  • lack of clarity on who to contact within the CNSC
  • inability to reach someone when calling the CNSC
  • lack of clarity in CNSC expectations
  • delays in obtaining inspection reports
  • lack of alignment between licensing and compliance groups

Applicant authorities and RSOs views were in agreement. 86% of the applicant authorities interviewed (6/7) believe that they have a collaborative relationship with the CNSC which allows for efficient resolution of non-compliance. Only one applicant authority stressed that the CNSC was not collaborative and that the CNSC has adopted a policing approach which is seen as counterproductive in the pursuit of safety.

Top of Page

Chapter 3 – Observations and recommendations

The following observations and recommendations stem from the findings and conclusions that were drawn across the different lines of evidence.

Recommendations require a formal management response and are made where strong support and multiple lines of evidence indicate that the program should undertake corrective action.

Observations do not require a formal management response and are made where an important theme was identified but may be supported by only a single line of evidence, where the observation is not directly related to a finding, or in areas that might require further detailed management analysis before a decision is made about the benefit of pursuing a corrective action.

Recommendations

The following recommendations are proposed:

  • 1. The CNSC’s Directorate of Nuclear Substance Regulation (DNSR) should provide regulatory guidance to RSOs with respect to:
    • the characteristics of a successful internal audit/inspection program (frequency, format, timing, templates)
    • adequate RSO resourcing levels based on the nature and magnitude of the licensed activity
    • the content of RSO work descriptions based on the nature and magnitude of the licensed activity
    • the appropriate composition and purpose of radiation safety committees given the different contexts in which RSOs operate
  • 2. DNSR should enhance its existing compliance promotion strategy to support RSOs with their continuous improvement activities.

Observations

DNSR should consider, in collaboration with external partners:

  • further promoting the nuclear substances and radiation devices mentorship opportunities for RSOs that are currently described on the CNSC’s external website
  • exploring the possibility of supporting the development of an RSO training program

Top of Page

Appendix A – Evaluation approach and methodology

Before starting the evaluation work, there was no universal representation of the role of the RSO and its influence on the radiation protection program (RPP). As a result, the evaluation team prepared an initial draft logic chain based on CNSC REGDOC-1.6.1, Licence Application Guide: Nuclear Substances and Radiation Devices, appendix C: “Qualifications and Duties of the Radiation Safety Officer.”

This draft was further refined following presentation at the CNSC’s Operations Management Committee in March 2017 and through consultation with a number of key stakeholders, including:

  • CNSC employees working in licensing and compliance in the Directorate of Nuclear Substances and Radiation Devices (DNSR) and employees working in the Directorate of Environmental and Radiation Protection and Assessment (DERPA)
  • Nuclear substances and radiation devices RSOsactively working in the medical and academic/research sectors:
    • Stéphane Jean-François, Certified Health Physicist, Radioprotection Inc.
    • Kate Scheel, Program Manager, Ionizing Radiation, Simon Fraser University
    • Jeff Dovyak, Radiation Safety Coordinator, Winnipeg Regional Health Authority

Stakeholders

For the purposes of this evaluation, the primary stakeholders are:

Internal:

  • DNSR, particularly the Nuclear Substances and Radiation Devices Licensing Division (NSRDLD) and the Operations Inspection Division (OID)
  • DERPA, particularly the Radiation Protection Division (RPD)
  • Vice-President, Technical Support Branch
  • Executive Vice-President and Chief Regulatory Operations Officer

External:

  • Nuclear substances and radiation devices licensee management and staff
    • radiation safety officers
    • nuclear energy workers and non-nuclear energy workers, such as researchers, physicians, technicians, technologists and other impacted auxiliary workers (administrative, technical, cleaning and security support)
    • radiation safety committee members
    • licensee management (including the applicant authority)
  • the CRPA, whose mission is to “ensure the safe use of radiation by providing scientific knowledge, education, expertise and policy guidance for radiation protection” Footnote 34.

Logic chain and theory of change

The RSO role logic chain presents a holistic view of the RSO’s role and its influence. This provides clarity to better understand how the role of the RSO contributes to the licensee’s RPP. See figure A.1: RSO role logic chain.

A number of key terms and concepts are used throughout the evaluation.

Logic chain:The logic chain (figure A.1) is used to provide the evaluation unit, advisory committee and external members with a visual model of how the role of the RSO is supposed to influence the RPP performance. It supports the development of appropriate data collection tools and as such it was developed early in the evaluation.

Pathways:Pathways refer to the sequence of steps that are required to get from an activity to its impacts. In this evaluation there are four pathways: 1) training, 2) communication and advice, 3) monitoring, controlling and reporting, and 4) continuous improvement. Individual logic chains and their associated theory of change have been developed for each pathway and are shown in chapter 2 of the report.

Assumptions and risks:Assumptions are the events and conditions that are needed for the cause to lead to the effect. For example, for the RSO to deliver the training to the RPP users, the evaluation assumes that the RSO is able to correctly identify and communicate with his target audience (i.e., trainees). Assumptions that are deemed “at risk” are described as those less likely to occur. For example, the evaluation assumes that the RSO has the adequate financial resources to implement the RPP.

External influences:External influences are the other conditions that must exist to help ensure that the assumptions are correct and the change is realized. For example, the evaluation recognizes the need for the existence of funding stability and the regulatory context. External influences are captured on the logic chain but presented outside of the four pathways.

Figure A.1: RSO role logic chain

RSO role logic chain
Description

A graphic representation of the RSO role evaluation logic chain mainly focused on persuasion-type intervention. The graphic reads from bottom to top. The principal assumptions and risks related to the logic chain are listed at the bottom. Above, the logic chain is divided in 4 pathways: 1) training, 2) communication and advice, 3) monitoring, controlling and reporting, and 4) continuous improvement. Each pathways is divided in 4 sections: 1) activities/outputs, 2) reach, 3) capacity changes (immediate outcomes) and 4) behaviour (intermediate outcomes). A box above the 4 pathways is showing the direct benefit of the 4 pathways which is an increased compliance with regulatory requirement. There are two boxes above the direct benefit one that are showing the longer-term benefits of the four pathways. The longer terms benefits are reduction and control of doses to workers and to the public and licensee enhances safety and security culture. The last box at the top of the logic chain shows the ultimate outcome which is that nuclear substances and radiation devices licensees operate safely and securely.

The principal activities/outputs assumptions and risks of the continuous improvement pathway are:

  • RSO is qualified (training/experience)
  • RSO has the required soft skills (i.e. leadership, communication, persuasion)
  • RSO is knowledgeable of regulatory requirements
  • RSO is supported by management (including AA and when applicable RSC)
  • Adequate resources are available to implement/maintain the RPP ($, time, etc.)
  • RSO is committed (wants to positively influence)

There are two boxes on left of the graphic outlining the key stakeholders and the external and organizational influences. The key stakeholders are:

  • Radiation safety officer (RSO)
  • Site radiation safety officer
  • Licensee radiation protection program (RPP) management
  • Applicant authority (AA)
  • CNSC
  • Radiation Safety Committee (RSC)
  • Licensee RPP users including nuclear energy workers (NEWs) and non-NEWS; these range from researchers, physicians, technicians, technologists, to auxiliary workers including administrative, technical, security and cleaning
  • Contractors/Volunteers

The external an organization influences are:

  • Funding availability
  • Regulatory context/expectation
  • Competing agendas
  • State of safety culture
  • Social, economic, geo-political, environment and/or technological factors

Training pathway

The training activities consist of the design, development, delivery or oversight of formal and informal training, info sessions, workshop and material.

Reach is the extent to which the RSO, in the conduct of the training activities, is able to influence RPP users (including NEWs and non-NEWs).

The immediate outcome of the training pathway is for trainees to understand RPP practices and procedures (capacity). To do so, they need to have time to attend training (opportunity) and have to be engaged in learning (motivation)

The intermediate outcome is that stakeholders adopt safe RP practices.

Communication and advice pathway

Communication and advisory activities include the development and dissemination of info sessions, messages, newsletters and emails.

Reach is the extent to which the RSO, in the conduct of the communication and advice activities, is able to influence RPP users (including NEW and non-NEW) and management.

The immediate outcome is that RPP users and management understand broad licensee RPP objectives (capacity). To do so, they need to have access to the RSO (opportunity) and they need to acknowledge the value of RPP communication (motivation).

The intermediate outcome of the communication and advice pathway is for RPP users and management to be engaged in RPP.

Monitoring, controlling and reporting pathway

Monitoring, controlling and reporting activities consist performing or overseeing: audit program, purchase authorization, investigation, reports (events/annual), list of NEWs, dosimetry records, licence renewal, response to CNSC requests.

Reach is the extent to which the RSO, in the conduct of the monitoring, controlling and reporting activities, is able to influence stakeholders involved with RPP (including users and management) and CNSC.

The immediate outcome of the monitoring, controlling and reporting pathway is for licensee management and CNSC have an increased knowledge of RPP performance and directly involved RPP users gain awareness of their own RPP performance (capabilities). To do so, RPP users and management must have time to review RPP performance (opportunity) and want to improve program performance (motivation).

The intermediate outcomes of the monitoring, controlling and reporting pathway are that licensee adjusts their practices and procedures, RSO inspection effort is risk informed and CNSC considers licensee input in its assessment of compliance.

Continuous improvement pathway

These activities consist of reviewing and/or contributing to: good practices, lessons learned and consultation on draft regulatory documents.

Reach is the extent to which the RSO, in the conduct of continuous improvement activities, is able to reach other RSOs, site RSOs, RSC, RPP management (including AA).

The immediate outcomes of the continuous improvement pathway is for RSOs, stite RSOs, RSC and RPP management to be aware of RP good practices (capabilities). To do so, they need to have sufficient time to review information received (opportunity) and they need to be engaged in RPP improvement (motivation).

The intermediate outcomes of the continuous improvement is for RSO to integrate RP good practices in RPP.

Contribution analysis and realist evaluation approaches

The logic chain Footnote 35 for this evaluation is inspired by aspects of approaches known as contribution analysis and realist evaluation.

  • Contribution analysis assesses the contribution a program is making (in this case the RPP through the RSO’s contribution) to observed results Footnote 36.
  • Realist evaluation recognizes that interventions take place in complex systems and seeks to understand “What works for whom in what circumstances and in what respect, and how?” Footnote 37.

Consequently, the logic chain was designed to look at the contribution the RSO makes to a licensee’s RPP and is based on the activities that are central to the role of the RSO.

The narrative that follows serves as a guide to explain the linkages within the logic chain (figure A.1). For ease of comprehension, the logic chain should be read from the bottom to the top to align with this narrative. Nested logic chains representing each of the four core activities undertaken by the RSO are also shown.

Principal assumptions and risks

While there are some specific assumptions that relate to each pathway, a number of principal assumptions underpin the logic chain. In order for the activities in each pathway to take place, the evaluation seeks to understand if the RSO has the necessary skills and level of commitment, and is not constrained in any way that would prevent effective performance of their role. The following assumptions are deemed as likely necessary for the theory of change to unfold as anticipated.

  • the RSO is qualified (has the necessary training and/or experience)
  • the RSO is knowledgeable of regulatory requirements
  • the RSO is supported by management (including applicant authority and, when applicable, the radiation safety committee (RSC)
  • adequate resources are available to the RSO to implement and maintain the RPP (money, time, staff, etc.)
  • the RSO has the required soft skills (i.e., leadership, communication, persuasion)
  • the RSO is committed (wants to positively influence RPP performance)

Activities (outputs)

The RSO operates in a complex system and must influence a number of stakeholders to create the desired impact to the RPP. The RSO undertakes a number of activities Footnote 38 in the context of the RPP and, in undertaking these activities, certain outputs are produced. (While many tasks and outputs might be undertaken, the focus is on the key functions). The activities have been grouped under four main categories (also known as pathways):

  1. Training activities
  2. Communication and advice activities
  3. Monitoring, controlling and reporting activities
  4. Continuous improvement activities
1. Training activities

Training activities form a central part of the RSO role. They include design, delivery and evaluation of training (formal such as in-class and informal such as workshops and online training), to a variety of stakeholders (nuclear energy workers, researchers, physicians, technicians, technologists and auxiliary workers) to ensure safe use of nuclear substances and radiation devices per the RPP procedures and practices. It also encompasses the development and maintenance of related training materials.

2. Communication and advice activities

Communication and advisory activities are also central to the RSO role. They include ensuring effective communication to all levels of stakeholders on a variety of RP (radiation protection) issues and proactively disseminating information (by using different media such as internal communication bulletins, newsletters, email and posters). The communications activities can also involve two-way communications between key RPP stakeholders (nuclear energy workers, researchers, physicians, technicians, technologists and auxiliary workers) and the RSO, where the RSO provides advice. This can be initiated directly by the RSO or in response to RP stakeholder inquiries.

3. Monitoring, controlling and reporting (surveillance) activities

These activities mainly consist of ensuring that the handling of nuclear substances and radiation devices is done in accordance with regulatory requirements, and internal processes and procedures. This involves informing nuclear energy workers, developing a security plan, designing and developing monitoring programs, monitoring occupational radiation doses received by workers, inspecting/auditing licensee activities, investigating and reporting on abnormal events, preparing annual reports (both for internal management and for the CNSC) and responding to CNSC inquiries.

4. Continuous improvement

Continuous improvement activities describe what the RSOs undertake to ensure that they keep abreast of industry good practices and lessons learned in order to improve the licensee’s RPP. These activities can include identifying and addressing weaknesses in work procedures, reviewing upcoming changes in regulatory requirements, keeping abreast of reportable events in Canada and internationally, attending conferences and participating in training.

Reach

Reach is the extent to which the RSOs, in the conduct of their activities and the production of their outputs, engage with others in an effort to influence the targeted population. The logic chain incorporates reach because the RSO must influence the knowledge/awareness and behaviours of other individuals in order to have a successful RPP. Stakeholder groups that the RSO needs to reach typically include nuclear energy workers, researchers, physicians, technicians, technologists, auxiliary workers, licensee management (including the applicant authority), radiation safety committee members and the CNSC.

Outcomes (immediate to ultimate)

The outcomes Footnote 39 are presented in a continuum approach from immediate outcomes that relate to gaining knowledge to long-term/ultimate outcomes that relate to safety and security. The RSOs have control over their outputs (activities) but only influence over the outcomes. Accordingly, the higher up the continuum towards the ultimate outcome, the lower the RSO’s degree of influence.

The immediate outcomes refer to the knowledge gained/generated by and between stakeholders that can lead to a change in attitude Footnote 40. The increase in knowledge gained at the immediate level outcome enables stakeholders to move up the continuum to intermediate outcomes where a change in behaviour can be observed in the stakeholders.

Capacity changes (immediate outcome)

Immediate outcomes are formed out of the awareness, the knowledge (capability Footnote 41), opportunities Footnote 42 and motivations Footnote 43 of the key stakeholders from activities undertaken by the RSOs and their outputs. The RSOs do not have control over the achievement of these outcomes but have direct influence over them. The conduct of the activities/outputs should allow stakeholders to gain knowledge or increase their awareness in four main areas:

  1. understanding of RPP practices and procedures
  2. understanding of the broad RPP objectives
  3. understanding the RPP’s performance
  4. awareness of radiation protection (RP) good practices

1. Trainees understand RPP practices and procedures, have time to attend training and are engaged in learning

Trainees should have time to attend training and, as a result of their participation in training activities, trainees should understand RPP practices/procedures. By gaining this knowledge trainees should be clear on what is expected of them in the conduct of their daily activities which relate to the RPP.

2. Stakeholders understand broad RPP objectives, have access to the RSOs and value RPP communications

As a result of communication and advice activities undertaken by the RSOs, stakeholders should have a clear understanding of the RPP objective and be aware of the RPP environment in which they are involved. Accordingly they need to know who the RP point of contact is and, when required, have access to their RP contact. Should stakeholders have concerns or questions while carrying out their duties, they should now be equipped with sufficient base knowledge to appropriately reach out.

3. Stakeholders obtain information on RPP performance, licensee wants to improve RP performance and has the time to do so

The RSOs gather information through monitoring, controlling and reporting activities to generate a complete picture of the RPP’s overall effectiveness. In turn, the licensee management chain and the CNSC are provided with information on the RPP performance, while those directly involved in the RPP obtain feedback on the effectiveness of their current procedures and practices on radiation safety. These stakeholders need to have time to review the information and be motivated to improve the RP performance.

4. Stakeholders are aware of RP good practices, have the time for reviews and are engaged in RPP improvements

Through continuous improvement activities, the RSO should be well informed on industry good practices. The RSO gathers information that can support improvements to the RPP and associated training/communication and disseminates it to concerned stakeholders (e.g., shares information received via the CNSC distribution list and obtained through training and while attending conferences). Stakeholders need to have the time and motivation to improve the RPP.

Behaviour (intermediate outcome)

Intermediate outcomes are related to a change in behaviour as a result of the knowledge gained. The change in behaviour should directly contribute to an enhanced licensee safety and security culture. Behaviour changes that should be observed in six key areas are as follows:

  1. Stakeholders adopt safe RP practices.
  2. Stakeholders are engaged in RPP.
  3. Stakeholders adjust practices/procedures based on performance results.
  4. RSO inspection effort is risk informed.
  5. Reports contribute to the CNSC’s assessment of licensee compliance.
  6. The RSO integrates RP good practices in RPP training and communications.

1. Stakeholders adopt safe RP practices

As of a result of acquiring an increased understanding of RPP context and their respective roles and responsibilities, stakeholders should be adopting safe RP practices.

2. Stakeholders are engaged in RPP

In addition, as stakeholders now have a better understanding of the objective of the RPP and know who to contact in case of questions or concerns, they should be demonstrating their engagement in RP and act as eyes and ears for the RSO. This would typically be observed by stakeholders reaching out to the RSO and asking questions to improve current practices.

3. Stakeholders adjust practices/procedures based on performance results

As a result of the information that is gathered and shared with respective stakeholders, adjustments to procedures and practices are agreed to and implemented based on inspection results.

4. RSO inspection effort is risk informed

As a result of the RSO having information on RPP performance, the internal RSO inspection effort should be risk informed and aligned with CNSC compliance expectations. In other words, RSOs may focus their attention on inspecting areas where RP performance is lower or “at risk”.

5. Reports contribute to the CNSC’s assessment of licensee compliance

As a result of the licensees’ report submissions (e.g., annual compliance reports, events), the CNSC has relevant information that contributes to its assessment of licensees’ compliance.

6. The RSO integrates RP good practices in RPP training and communication

As a result of time spent keeping abreast of industry good practices, the RSO should be able to continually update and improve the RPP and associated training/communication components.

Direct benefit and safety impact (ultimate outcome)

These changes in behaviours of key stakeholders would result in direct benefits, including increased compliance with regulatory requirements, enhanced licensee safety and security culture and reduction/control of doses to workers and the public. The combination of these direct benefits should in turn result in the achievement of the ultimate outcome, which is for nuclear substances and radiation devices licensees to operate safely and securely.

Data sources

Literature and document review

The evaluation team examined over 70 documents as part of the literature and document review. The team performed a targeted literature review using journal articles, books and grey publications and focused on understanding how organizational design, level of authority, and regulatory basis can influence safety within an organization.

The document review consisted of collecting and reviewing existing documents (e.g., regulations, regulatory documents), in order to better understand the regulatory requirements as they relate to the role of the RSO. This included documents relevant to the role of RSOs for licensing and compliance activities with respect to nuclear substances and radiation devices.

Secondary data analysis

Secondary data analysis was performed to support the evaluation. DNSR staff extracted over 40,000 rows of data and provided them to the evaluation team in support of this analysis, extracting the data from three sources:

  • the Licence Operating Users Integrated System (LOUIS)
  • administrative monetary penalty (AMP) notices of violation
  • the Event Information Tracking System (EITS)

Comparative analysis

The comparative analysis explored the effectiveness of the role of the RSO when compared to:

  • the role of nuclear substances and radiation devices RSOs internationally (United States and United Kingdom)
  • other similar roles nationally such as:
    • biosafety officers
    • flight safety officers
    • occupational health and safety officers
    • Class II RSOs

The intent is to provide a comprehensive picture of the RSO role, including its regulatory expectations, guidance, success factors and influence in relation to other similar roles. The evaluation team conducted comparative analysis through information sharing via email, phone interviews with the U.S. NRC, UK HSE and other regulator program officials, and through a regulatory document review. In seeking to understand what contributes to the effectiveness of the RSO’s role, the following areas were compared:

  • legislation
  • regulatory guidance
  • certification
  • governance models and reporting relationship
  • oversight committee
  • reporting requirements
  • level of RSO authority

Case studies

The evaluation team completed 11 case studies to examine the role of RSOs who work for specific licensees operating in different contexts and with different compliance records. The objective of the case studies was to provide an in-depth understanding of how the role of the RSO and associated “success factors” influences RPP performance in these varied situations.

In consultation with the CNSC’s regional site coordinators, criteria were established for the selection of case studies. These focus on licensees in the medical and/or academic/research sectors who have experienced situations of change such as:

  • change in RSO
  • change in RPP performance over time, either improvement or decline
  • complex environments, multiple sites across province, amalgamation

The case studies were also selected to examine different regulatory situations such as:

  • licensees with both nuclear substances and Class II licences
  • licensees with a radiation safety committee
  • licensees with significant reported events

Interviews

Interviews were conducted for the RSO evaluation with both CNSC staff/management and licensee stakeholders, ranging from RSOs to workers, radiation safety committee members and applicant authorities.

Interviews are central to this evaluation, bringing both a breadth and depth of data. The interview covered all 22 evaluation questions. 97 interviews were completed with over 100 CNSC and licensee stakeholders. Interviewparticipants by stakeholder group consisted of:

  • CNSC internal interview stakeholders (n=13):
    • DNSR and DERPA management: n=4
    • DNSR staff (both from compliance and licensing): n=9
  • External interview participants were selected from two groups (n=89):
    Group 1 – Representative sample participants (n=3)
    • RSO: n=3
  • Group 2 – Specific to case study participants (n=86)
    • RSO (either corporate or site): n=28
    • Radiation safety committee member: n=16
    • Applicant authority: n=9
    • Workers (nuclear energy workers (NEWs) and non-NEWs, including physicians): n=33

Surveys

Three web-based surveys were administered for the evaluation of RSOs. The surveys were designed to complement the data collected through interviews and to increase the representation of RSOs, applicant authorities and RPP workers.

All surveys were developed internally and administered by a third party. Respondents were able to complete the survey in either official language. During the survey period, two reminders were sent to participants who had not completed the survey. In an effort to further increase survey participation, CNSC staff also made follow-up phone calls to RSOs who had not completed the survey.

Each survey has its own potential for bias, which is discussed in the following sections. Therefore, caution should be exercised in generalizing the survey results.

RSO survey

The RSO survey was designed as a census of the RSOs in the medical and academic/research sectors across Canada. The survey was administered to licensees with high/medium-and low-risk use types.

This report summarizes the results of the high/medium-risk respondents only. Low-risk respondent results were excluded because the level of complexity of their operations is very different from those in the high/medium-risk group. They are generally from much smaller organizations, typically with only one CNSC licence. Although they possess nuclear substances and radiation devices, they may be accessed only once per year, and are always handled by the same individual. Combining the results of both the low-risk and medium/high-risk groups would have misrepresented the survey data.

Features:

  • 55 questions (multiple choice and open ended)
  • Total respondents: 255/342 (high/medium risk), 15/33 (low-risk)
  • Response rate: 74% (high/medium risk), 45% (low-risk)
  • Average completion time: 42 minutes
  • Data collection time frame: February 6, 2018 to March 31, 2018
  • Data gathered on all 22 evaluation questions

Considerations:

  • Sufficient representation from RSOs in Quebec was achieved, as 20% of the RSOs completed the survey in French. This number is comparable to the ratio of Quebec’s population in Canada. It is also comparable to the percentage of the Canadian population with French as their first official language, which is 22% (according to 2016 census data).

Applicant authority survey

The survey of applicant authorities was designed as a randomized sample of the population (approximately 191), which represents a confidence level of 95% and confidence interval (or margin of error) of +/-5.

Features:

  • 10 questions (multiple choice and open ended)
  • Total respondents: 49/185
  • Response rate: 26%
  • Average completion time: 9 minutes
  • Data collection time frame: February 6, 2018 to March 1, 2018
  • Data gathered on evaluation questions (3, 7, 8, 10, 13, 18, 20, 21 and 22) related to issues of communication, authority, governance and regulatory guidance

Potential limitation:

  • Even though reminders were sent and the survey was deliberately designed to be short, the response rate of applicant authorities was low at 39%. As a result, there is a potential for non-response bias materializing; this limits the ability to generalize the results.

Worker survey

As the CNSC does not maintain records of RPP worker contacts, the workers were identified by 20 RSOs who were selected at random. Each RSO was asked to provide the email addresses of two to four workers from their organization who would be willing to complete the survey.

Features:

  • 10 questions (multiple choice and open ended)
  • Total respondents:19/29
  • Response rate: 66%
  • Average completion time: 6 minutes
  • Data collection time frame: February 6, 2018 to March 1, 2018
  • Data gathered on evaluation questions (2, 5, 6, 9, 11, 12) related to issues of training, communication and feedback and barriers/enablers to adopting of safe practices

Potential limitation:

  • The survey of workers relied on a non-probability sampling technique to recruit participants. Known as referral sampling or snowball sampling, this method is subject to bias, as RSOs may have recruited (intentionally or not) participants who do not represent the overall population of workers.

Limitations

  • Given the circumstance under which the surveys were administered (in a regulator/licensee relationship), there is the potential for a social desirability bias Footnote 44 to materialize in the surveys.
  • Limited availability of regulatory information on flight safety officers. Information was based on what is publicly available in the Canadian Aviation Regulations.
  • Data from LOUIS is not easily usable and often required manipulation. This increased the potential for error. To mitigate this, the evaluation team adopted peer review, or data and other additional quality control mechanisms.

Top of Page

Appendix B – Evaluation questions

RSO “success factors” – Reach
  1. Are RSOs successful at reaching the stakeholders who require training? If not, what barriers are in play?
  2. Are communications and advice produced by RSOs reaching their targeted audience?
  3. Are information and reports generated by RSOs reaching their targeted audience?
  4. Are RSOs able to access radiation protection good practices and lessons learned?
    1. Is the training offered by third parties to RSOs appropriate and useful?
RSO “success factors” – Awareness and knowledge
  1. Are RSOs able to convey radiation protection objectives to program users (nuclear energy workers and non-nuclear energy workers)? If not what prevents this from happening?
  2. Do trainees understand practices/procedures important to radiation protection? If not, what factors limit their ability to get that understanding?
  3. Can RSOs adequately assess the effectiveness of the radiation protection program in order to take effective corrective action? Do they have access to the information they need to make that assessment?
    1. Does the RSO have sufficient authority to successfully implement and maintain the RPP?
    2. Are there effective strategies to address worker non-compliance?
  4. Does licensee management (including the applicant authority) obtain meaningful and actionable information from the RSO on licensee RPP performance? Similarly, does the CNSC get appropriate information from the RSO on licensee RPP performance?
  5. Are radiation protection program users (nuclear energy workers and non-nuclear energy workers) able to obtain useful feedback from the RSO on their program-related performance?
  6. Are good practices widely shared between RSOs, site or delegated RSOs, radiation safety committee members and the applicant authority? If so, how?
RSO “success factors” – Behaviour
  1. Do RPP users adopt effective radiation protection practices/procedures? If not, what prevents this from happening?
  2. Which conditions promote robust stakeholder engagement in radiation protection programs?
  3. Are licensees able to adjust their practices/procedures based on assessment of their performance against the program’s objectives?
  4. What makes for a successful licensee audit program? Is the RSO’s inspection effort risk informed?
  5. Is the information provided by RSOs to the CNSC useful in supporting the CNSC’s assessment of licensee compliance?
  6. Under which conditions does the RSO integrate RP good practices into their program?
Direct benefit
  1. To what extent do the RSO- related interventions contribute to the following?
    1. increase in regulatory compliance
    2. reduction or improved control in doses
    3. improved safety/security culture of licensees
  2. What are the conditions/factors that maximize the RSO’s contributions?
Program design and delivery
  1. Do any significant unintended consequences (both positive and negative) result from the way RSOs are regulated by the CNSC?
    1. Are there consequences associated with the absence of “credentialing” for nuclear substances and radiation devices RSOs?
    2. Are there consequences of having multiple CNSC licences?
    3. Are there consequences associated with alignment or lack of alignment between regulatory bodies?
  2. Does the governance structure of the RSO’s institution impact the ability of the RSO to discharge their role and responsibilities? If so, how? What adjustments would make the RSO more effective?
  3. Is the current relationship between the applicant authority and the RSO the right approach? Are other models worth considering?
  4. Does the CNSC provide sufficient guidance: 1) on RSO roles and responsibilities and 2) on RPP implementation and maintenance?

Top of Page

Appendix C – Management of the evaluation

Roles and responsibilities

The Regulatory Research and Evaluation Division (RRED) was responsible for the conduct of the evaluation. The senior evaluator was designated to lead the evaluation process with support of other evaluation officers and guidance and oversight of the Director of RRED and Head of Evaluation, Director General of the Strategic Planning Directorate (SPD). The evaluation team was composed of three members:

  • Geneviève Boudrias – Senior Evaluation Officer
  • Heather Crowe – Evaluation Officer
  • Benoît Marcotte – Evaluation Officer

The guidance and oversight function was further supported from a management focus through the Evaluation Advisory Committee (EAC). The EAC was composed of directors general (inclusive of the Head of Evaluation) and external members. Members were responsible for guiding the evaluation by validating deliverables (e.g., terms of reference, preliminary findings, and this final report). Members of this committee played a key role in advising RRED on the evaluation issues and questions that address management needs. The EAC consisted of the following members:

  • Colin Moses, Director General, Directorate of Nuclear Substance Regulation
  • Peter Fundarek, Director, Nuclear Substances and Radiation Devices Licensing Division
  • Michael Rinker, Director General, Directorate of Environmental and Radiation Protection and Assessment
  • Liane Sauer, Director General, Strategic Planning Directorate and Head of Evaluation
  • Michèle Légaré (external member, radiation expert)
  • John Mayne (external member, evaluation expert)

Budget

The evaluation was conducted internally. The largest source of cost was the internal evaluation staff level of effort required. The staff level of effort required, 2 full-time equivalents (FTEs), slightly exceeded the estimate of 1.5 FTEs.

As the number of case studies undertaken increased from what was initially planned, some contingency funds were used to cover travel expenses to conduct onsite interviews at case study facilities.

Table C.1: Evaluation detailed budget

Resource effort FTEs Actual FTEs
Internal staff effort to plan, conduct and report on the evaluation 1.5 2.0
Activity description O&M Actual O&M
Contract with evaluation expert for ongoing support and guidance $17,000 $17,000
Contract for expert participation on the Evaluation Advisory Committee $9,000 $6,000
Contract for third-party survey administration $5,000 $4,388
Travel costs for in-depth case study interviews and data collection (potential of 4 outside NCR, participation of 2 evaluators) $15,000 $16,376
Contingency (for potential travel if some key interviews cannot be completed over the phone) $2,000
Presentation at the Canadian Radiation Protection Association annual conference Unplanned $2,187
Total $47,700 - $49,700  

Timelines

The timelines for planning, conducting and reporting on this evaluation are outlined in table C.2, categorized by phase (planning, conducting and reporting). The planned schedule is identified with an "x". The actual schedule matches the planned schedule, except where identified with an asterisk ‒ the reporting phase – which took more time than initially planned.

Table C.2: Planned versus actual schedule

Year 2017 2018
Phase PLANNING CONDUCTING REPORTING
Activity/Month 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9
Develop logic chain x x                          
Develop terms of reference   x x                        
Year 2017 2018
Phase PLANNING CONDUCTING REPORTING
Activity/Month 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9
Approve terms of reference     x                        
Develop evaluation matrix and data collection tools   x x                        
Present evaluation plan to Commission       x                      
Collect documentation       x                      
Execute document review and data analysis       x x                    
Execute literature review       x x                    
Develop questions and conduct comparative analysis         x x                  
Develop interview questions / guides and surveys       x x x                  
Administer interviews and surveys             x x              
Analyze interviews and surveys                 x            
Analyze case studies collected during interviews                 x            
Code all evidence collected               x              
Draft preliminary findings report and conclusions                 x            
Draft evaluation deck/report                 x x x * * *
Approval                   x x x * * *

Appendix D – The 4 “I”s factors and conditions

Infrastructure (the wider social, economic and cultural setting of a program/intervention including social norms). Examples:

  • Funding stability – adequate resources are available
  • The consequences of non-compliance are clear
  • Union/management environment
  • Regulatory context and expectations

Institutional (the characteristics of the institution involved, such as power structures, authorities and policies). Examples:

  • The RSO is supported by management
  • The RSO has the authority to enforce radiation protection policies and procedures
  • Adequate resources are available to implement/maintain the radiation protection program
  • The RSO is allocated sufficient time for their role
  • Stakeholders value the radiation protection program in support of their business/operations (competing priorities; e.g., safety of procedure to worker versus increasing number of patient procedures per hour)
  • Radiation protection program practices are supported by management and the radiation safety committee

Interpersonal (nature and history of key relationships among stakeholders, such as communication levels, networks, historical relationships and trust among groups). Examples:

  • The RSO is respected and seen as a credible and trusted authority on radiation protection
  • Stakeholders are committed (want to have a positive influence)
  • Stakeholders understand their role in relation to radiation protection practices
  • Stakeholders recognize their influence on radiation protection program performance
  • The relationship between the CNSC and the licensee allows for open communication and resolution of non-compliance

Individual (characteristic and capacities of stakeholders, including commitment factors that affect the inclination to make change). For example, the RSO:

  • is qualified
  • has the required soft skills
  • is knowledgeable of regulatory requirements
  • is committed to the role
  • understands licensee operations as they relate to radiation protection program

Appendix E – References

CNSC reference documents – External

Other external reference material

Footnotes

Footnote 1

R. Pawson and N. Tilley, Realist Evaluation (2004).

Return to footnote 1 referrer

Footnote 2

S. Michie, M. M. v. Stralen and R. West, “The behaviour change wheel: A new method for characterising and designing behaviour change interventions,” Implementation Science 6(42) (2011, 11 pages).

Return to footnote 2 referrer

Footnote 3

As part of the case studies, two distinct approaches were considered: 1) contribution analysis, to help assess the relative contribution the RSO components have made to the RPP performance; and 2) realist questions, to help understand how context and conditions affect the RSO results chain (impact pathway).

Return to footnote 3 referrer

Footnote 4

As part of the case studies, two distinct approaches were considered: 1) contribution analysis, to help assess the relative contribution the RSO components have made to the RPP performance; and 2) realist questions, to help understand how context and conditions impact the RSO results chain (impact pathway).

Return to footnote 4 referrer

Footnote 5

CNSC directorates: Directorate of Nuclear Substance Regulation and Directorate of Environmental and Radiation Protection and Assessment.

Return to footnote 5 referrer

Footnote 6

CNSC, REGDOC-1.6.1, Licence Application Guide: Nuclear Substances and Radiation Devices, Version 2, appendix C: “Qualifications and Duties of the Radiation Safety Officer” (2017).

Return to footnote 6 referrer

Footnote 7

CNSC, REGDOC-2.2.2, Personnel Training, Version 2 (2016).

Return to footnote 7 referrer

Footnote 8

Workers are licensee staff who handle nuclear substances and radiation devices or who may be exposed to them in the context of their duties. This includes nuclear energy workers and non-nuclear energy workers ranging from researchers, physicians, technicians and technologists, to auxiliary workers including administrative, technical, security and housekeeping staff.

Return to footnote 8 referrer

Footnote 9

Includes interview data (18/19) and survey responses (241/255).

Return to footnote 9 referrer

Footnote 10

Survey baseline varies as not all survey respondents answered all of the survey questions.

Return to footnote 10 referrer

Footnote 11

Interview baseline varies as not all 28 interviewees answered all questions.

Return to footnote 11 referrer

Footnote 12

A TLD is a portable device that is used to measure ionizing radiation exposure to workers by measuring the intensity of visible light emitted by a crystal inside the detector when the crystal is heated. The intensity of light emitted is dependent on the radiation exposure.

Return to footnote 12 referrer

Footnote 13

Includes interview data (12/16) and survey responses (239/250).

Return to footnote 13 referrer

Footnote 14

Includes interview data (27/29) and survey responses (19/20).

Return to footnote 14 referrer

Footnote 15

CNSC, REGDOC-1.6.1, Licence Application Guide: Nuclear Substances and Radiation Devices, Version 2, appendix C: “Qualifications and Duties of the Radiation Safety Officer” (2017).

Return to footnote 15 referrer

Footnote 16

Includes interview data (30/32) and survey responses (20/20).

Return to footnote 16 referrer

Footnote 17

Includes interview data (8/9) and survey responses (45/49).

Return to footnote 17 referrer

Footnote 18

Interview data.

Return to footnote 18 referrer

Footnote 19

Includes interview data (18/19) and survey responses (194/237).

Return to footnote 19 referrer

Footnote 20

Includes interview data (28/28) and survey responses (231/237).

Return to footnote 20 referrer

Footnote 21

Includes interview data (26/29) and survey responses (18/20).

Return to footnote 21 referrer

Footnote 22

CNSC, REGDOC-1.6.1, Licence Application Guide: Nuclear Substances and Radiation Devices, 2, appendix C: “Qualifications and Duties of the Radiation Safety Officer” (2017).

Return to footnote 22 referrer

Footnote 23

Includes interview data (9/9) and survey responses (44/49).

Return to footnote 23 referrer

Footnote 24

Includes interview data (22/24) and survey responses (157/233).

Return to footnote 24 referrer

Footnote 25

Public Health Agency of Canada and Canadian Food Inspection Agency, Canadian Biosafety Handbook, Second Edition (2016).

Return to footnote 25 referrer

Footnote 26

Includes interview data (9/9) and survey responses (46/47).

Return to footnote 26 referrer

Footnote 27

Includes interview data (29/29) and survey responses (210/232).

Return to footnote 27 referrer

Footnote 28

Nuclear substances and radiation devices mentorship program.

Return to footnote 28 referrer

Footnote 29

The specific qualifications and duties expected of an RSO are described in appendix C of REGDOC 1.6.1.

Return to footnote 29 referrer

Footnote 30

REGDOC-2.2.3, Personnel Certification: Radiation Safety Officers, explains the RSO certification process and expectations for Class II RSOs. It describes the purpose of certification as one to “ensure that persons seeking certification by the Canadian Nuclear Safety Commission (CNSC) for the position of RSO at a Class II nuclear facility are qualified to carry out the duties of the position in accordance with the Nuclear Safety and Control Act (NSCA) and its regulations”. The certification process has two components: 1) assessment of the candidate’s capabilities to perform the duties of the position, based on the submitted application; 2) assessment of the candidate’s knowledge, based on an examination. It specifies that the examination is normally conducted in the form of an oral examination and that it is tailored to: 1) operational risks of the licensed activity and equipment; 2) the organization’s policies and procedures; and 3) the candidate’s academic background and work experience. The certification has no set expiration or requirement for continuing education; it remains “valid for the period for which the person is employed by the licensee as RSO”, noting that certification can be invalidated or revoked. In this context, certification is undertaken for Class II RSOs to ensure qualification for their role.

Return to footnote 30 referrer

Footnote 31

Criteria of competence include: holding a valid certificate of competence from an organization recognized by the Health and Safety Executive as an Assessing Body; or holding a National or Scottish Vocational Qualification (N/SVQ) level 4 in Radiation Protection Practice issued within the last five years.

Return to footnote 31 referrer

Footnote 32

Canadian Radiation Protection Association.

Return to footnote 32 referrer

Footnote 33

Includes interview data (14/20) and survey responses (220/236).

Return to footnote 33 referrer

Footnote 34

Canadian Radiation Protection Association.

Return to footnote 34 referrer

Footnote 35

This kind of results chain logic is referred to as either a logic chain or a results chain in this paper and, together with the inputs, outputs and influence factors can be considered a logic model on the RSO role as part of the RPP.

Return to footnote 35 referrer

Footnote 36

John Mayne, “Contribution analysis: An approach to exploring cause and effect,” ILAC Brief 16 (2008).

Return to footnote 36 referrer

Footnote 37

R. Pawson and Nick Tilley, Realist Evaluation (2004).

Return to footnote 37 referrer

Footnote 38

The OECD defines activities as actions taken through which inputs and resources are mobilized to produce outputs. Outputs are the products (which may include services) which result from activities.

Return to footnote 38 referrer

Footnote 39

According to the OECD (2010), outcomes are the effects produced by an intervention’s (program’s) activities and outputs.

Return to footnote 39 referrer

Footnote 40

The basic theory is that someone will become aware, gain knowledge and understanding and then take actions or adopt behaviours according to this knowledge. This is known as a theory of reasoned action and/or planned behaviour. All of the following activities can be seen to essentially assume this type of results pathway.

Return to footnote 40 referrer

Footnote 41

Capability is defined as the individual’s psychological and physical capacity to engage in the activity concerned. It includes having the necessary knowledge and skills (Mayne 2017).

Return to footnote 41 referrer

Footnote 42

Opportunity is defined as all the factors that lie outside the individual that make the behaviour possible or prompt it (Mayne 2017).

Return to footnote 42 referrer

Footnote 43

Motivation is defined as all those brain processes that energize and direct behaviour, not just goals and conscious decision making. It includes habitual processes and emotional responding, as well as analytical decision making (Mayne 2017).

Return to footnote 43 referrer

Footnote 44

Social desirability is the tendency to provide answers that put one in a good light with the person who asks the question; it is often motivated by wanting to make a good impression in a social interaction. It can occur when respondents falsely deny engaging in socially undesirable behaviors such as drug use or cheating on one’s spouse, or when they falsely claim to have engaged in desirable behaviors like voting or volunteering. (Dillman, Smyth and Christian, 2014).

Return to footnote 44 referrer

Date modified: