TSB Recommendation A16-14

Transport Canada agrees that it must continually enhance its oversight policies, procedures and training. In fact, independent to this accident or this recommendation, TC committed in its National Oversight Plan for 2016/2017 to conduct an evaluation of its surveillance program.

Further to that commitment, in July 2016, TC launched a Civil Aviation Surveillance Program Evaluation and Update Project. The purpose of this project is to analyze opportunities for improvement identified via inspector feedback, analyze lessons-learned, and leverage business intelligence to update the surveillance program. The goal of the project is to ensure the surveillance program effectively verifies regulatory compliance at appropriate intervals and is effective in carrying out enforcement action, as required. While the project is scheduled to complete in December 2017, the department will not hesitate to implement enhancements sooner, if warranted.

The Board’s recommendation A16-14 will beused as an input to this evaluation work.

In its response, Transport Canada indicated that it launched a Civil Aviation Surveillance Program Evaluation and Update Program that is scheduled to be completed in December 2017. The Board is encouraged that Transport Canada has committed to evaluating its surveillance program, and to considering opportunities for further improvements in order to ensure the effectiveness of its surveillance program.

In a recent briefing to the Board, TC has provided a detailed update on the various program improvement initiatives undertaken since 2015-16. The Board is pleased to note that TC has implemented some concrete actions such as: the establishment of a National Oversight Office, the implementation of an Oversight Advisory Board, the creation of a dedicated team working on surveillance policies and procedures, strengthened oversight planning, risk-based decision-making, timely enforcement actions, and taking temporary measures that will permit an increase in the number of inspections on higher risk areas while the program evaluation and update is being done.

The Board also acknowledges TC’s efforts as it strives to find the right balance between planned and reactive oversight activities, as well as in the use of the various types of oversight tools available. Although TC has implemented numerous improvements it is too early to assess whether or not TC’s actions will adequately address the safety deficiency associated with this recommendation.

Therefore, the response to the recommendation is considered to be Satisfactory Intent.

TC agrees with the recommendation.

In July 2016, TC launched a Civil Aviation Surveillance Program Evaluation and Update Project to analyze improvement opportunities identified via inspector feedback and lessons-learned to update the surveillance program. 

The objectives are to:

• ensure the surveillance program effectively verifies regulatory compliance with safety management systems requirements;  
• confirm that verifications are done at appropriate intervals; and, 
• verify that enforcement action is taken as required. 

Since launching the project, TC has briefed the Chair and members of the TSB on progress.

While the evaluation of the surveillance program continues, TC has already taken action and has:

• Introduced leaner, more efficient tools that effectively determine compliance and risk and provide a better balance between system and process level surveillance (e.g., anunannounced inspection at the ramp of the airports used to determine an air operator’s method for complying with regulatory requirements, such as flightcrew certification, instrument and equipment requirements, weight and balance procedure, dispatch procedures and an overall evaluation of the operations being conducted. Ramp inspections may be conducted at point of origin, en route or destination stops). 
• Introduced Safety Management System verifications as part of inspection tools, other than assessments, which will facilitate more regular checks of enterprises’ SMS to confirm they are working.
• Made extensive use of data interms of examining performance levels and sampling methodologies to determine surveillance intervals.

TC’s surveillance plan for 2018/2019 covers private operators (604), aerial work (702), heliports (305), flight training units and the general aviation community in addition to the higher impact sectors, such as airlines (705), commuters (704) and their associated Approved Maintenance Organizations.

In April 2018, TC also introduced new guidance to inspectors for post-surveillance work, including the documentation of findings and enforcement procedures. The intent of these updates is to strengthen TC’s approach to documenting findings of non-compliance and to better articulate the issue identified through inspection.  This will support companies (that are in receipt of findings) with their development and implementation of more effective corrective action plans. This new guidance will allow unsafe operating practices to be addressed more quickly and effectively.  In cases where certificate holders do not develop Corrective Action Plans, TC will suspend the relevant Canadian Aviation Document.

In its response, Transport Canada indicated that its Civil Aviation Surveillance Program Evaluation and Update Project is currently on-going. While the evaluation of the surveillance program continues, TC has introduced leaner, more efficient tools to determine compliance and risk, has introduced safety management system (SMS) verifications, has made extensive use of data to determine appropriate surveillance intervals, and has introduced new guidance to inspectors for post-surveillance work.

The Board is encouraged by these efforts. However, the Board notes that TC’s response makes no mention of air taxi (703) operations as part of its surveillance plan for 2018/2019, and it does not clearly describe what, if any, changes will be made to enhance existing oversight policies, procedures, and inspector training. Although TC’s actions to date may reduce the risk to the travelling public, the risk will not be substantially reduced or eliminated until its oversight policies, procedures, and training have been enhanced.

Therefore, the response to Recommendation A16-14 is assessed as Satisfactory in Part.

Since 2016, TC has implemented a number of changes to the surveillance program in order to improve oversight of the aviation industry. Among other enhancements, TC has implemented guidance, tools and training to improve the quality of findings, drive decision-making, and improve the risk-based planning methodology to ensure effective oversight.

Developing accurate, comprehensive, and clear findings are an essential function that supports enforcement action where required. Findings are critical communication tools that enable companies to understand and address the identified non-compliances. Further, robust and well supported findings are a necessary foundation upon which all further enforcement action relies, should an operator prove unable or unwilling to rectify identified issues.

SI SUR 029 was developed to provide inspectors with the enhanced guidance and tools to:

• accurately determine the existence of a non-compliance;
• determine what evidence is required to support the finding of non-compliance;
• clearly describe the nature of the non-compliance; and,
• identify whether the finding should be at the system level or process level.

It is critical that all of these elementsare in place in order to allow for an effective enforcement response. In addition to the publication of SI SUR 029, Surveillance Update training was developedand delivered throughout 2018, which included instruction related to SI SUR029.

The Oversight Advisory Board (OAB) was established to support efficient and effective decision-making when handling complex oversight decisions^{Footnote 1}.The process was developed in order to bring decision-makers and subject matter experts together to review concerns and determine appropriate courses of action in relation to complex oversight issues.

The OAB brings several benefits, including:

• circumventing hierarchical processes for a rapid response;
• supporting nationally consistent decision-making; and
• facilitation of information-sharing between operational groups.

Fiscal year 2020-2021 is the third year of transition during which TC is enhancing its surveillance program. The planning methodology relies on the following fundamental concepts: operational latitude, using local intelligence, leveraging inspectors’ professional judgment, and data-driven risk-based decision-making.

SI SUR 028^{Footnote 2} was updated in October 2018 for fiscal year 2019-2020 and is currently being amended in preparation for 2020-2021 planning. The document details enhancements to the risk-based planning process that have been implemented with a view to continuous improvement.

Changes to the surveillance planning methodology for 2020-2021 include^{Footnote 3}:

1. The introduction of a “conceptual National Oversight Plan” which is developed by headquarters, and further refined by the regions based on local intelligence.
2. An enhanced data analysis tool that will assist in the planning of surveillance activities. This tool uses data obtained from the National Aviation Company Information System (NACIS), the Civil Aviation Daily Occurrence Reporting System (CADORS), as well as qualitative data from the regions to group enterprises into peer groups based on risk level.
3. The introduction of aqualitative assessment tool to validate the categorization of enterprises using the quantitative analysis using the primary inspector’s perspectives.
4. Continued use of process inspections as the primary surveillance tool for planned surveillance activities. Systems level surveillance will be conducted on a reactive basis where warranted.

TC’s surveillance plan for 2019-2020 includes surveillance for the following sectors: commercial airlines (705), commuter airlines (704) and their associated Approved Maintenance Organizations, as well as private operators (604), aerial work (702), air taxi (703), heliports (305), flight training units, and the general aviation community.

The Board is encouraged by Transport Canada’s(TC) efforts to improve oversight of the aviation industry. In particular, TC has indicated in its response that it has implemented guidance, tools, and training to improve:

• the quality of findings produced during surveillance activities;
• the decisions made related to the oversight of commercial aviation; and
• the risk-based planning methodology.

The necessary guidance and tools can be found in the following documents:

• Staff Instruction (SI)SUR-027 — Oversight Advisory Board (OAB), published in October 2017;
• SI SUR-029 — Writing Findings for Non-Compliance, published in April 2018;
• SISUR-001 — Surveillance Procedures, updated in January 2019; and
• SI SUR-028 — Surveillance Planning Instructions — Fiscal Year 2019-2020, updated in January 2019.

TC’s emphasis on improving the quality offindings will assist in the conduct of surveillance activities and in post-surveillance decision-making, which may include enforcement action decisions, at the regional and/or national level. TC’s response also highlighted the benefits of the OAB which, when used, can assist with the post-surveillance decision-making process.

TC has provided a general overview of its recent enhancements to its surveillance planning methodology. Specifically, TC refers to operational latitude, local intelligence, and inspectors’ professional judgment, which will be used in conjunction with an enhanced data analysis tool and a qualitative assessment tool. In light of the infancy of these changes, it is not possible to fully assess any potential impact that they will have on TC’s oversight program.

TC now relies on process inspections as the primary surveillance tool for planned surveillance, where as systems level surveillance is done on a reactive basis, when warranted. There are benefits to an increased focus on regulatory compliance, which may be easier to accomplish through process inspections; however, simply verifying regulatory compliance in a cross-section of an organization does not guarantee that commercial aviation operators are capable of effectively managing safety within their organization. TC must also confirm that operators are capable of managing safety risks effectively, whether or not they have a safety management system. Therefore, TC’s increased reliance on process inspections represents a reduced surveillance focus, which will have to be assessed over time.

It is likely thatTC’s planned actions and actions taken to date will reduce the risks associated with the safety deficiency identified in Recommendation A16-14. However, withthe information TC provided, it is not possible to determine at this point if these actions will substantially reduce or eliminate the safety deficiency. That determination will be made once TC’s changes have been fully implemented.

Therefore, the Board considers the response to the recommendation to be Satisfactory in Part.

TC agrees with this recommendation.

Following the original commitments related to TSB recommendation A16-14, TC completed the Surveillance Program Evaluation and Update project in 2018. Since then the civil aviation program has continued through the normal continuous improvement cycle as evidenced by the publication of multiple revisions of staff instructions (SI) defining the surveillance program:

• SI-SUR-001 Issue 8 – Surveillance Procedures^{Footnote 4}
• SI-SUR-029 Issue 2 – Writing Findings of Non-Compliance^{Footnote 5}
• SI-SUR-028 Issue 5 – Surveillance Planning Instructions – Fiscal Year 2020-2021^{Footnote 6}

The training program launched to explain to the inspectorate the changes resulting from the Surveillance Program Evaluation and Update project concluded in the spring of 2020. The vast majority of the Civil Aviation Safety Inspectors, Technical Team Leads and Associate Directors of Operations participated in the two-day scenario based training that was offered in small groups across Canada to facilitate open discussion about the multiple aspect of their surveillance related duties, including: data gathering for risk-based planning, pre-site, on site and post site activities, enforcement, writing and documenting findings of compliance, quality control and continuous improvement and operational latitude concepts. Recurrent surveillance training is to be included to the list of mandatory training at the next revision of CAD-ADM-05: Required Training for Civil Aviation Employees who Develop, Deliver and/or Support Oversight Activities.^{Footnote 7}

The planning exercise for fiscal year 2019-2020 and fiscal year 2020-2021 was done with the support of local intelligence gathered from inspectors directly involved with the enterprises as well as the risk indicators based on data available from central databases. This methodology enabled TC produce a risk-based surveillance plan that reflects the preoccupations of the inspectorate as well as the interpretation of the risk data available to TC as a whole.

The recent COVID-19 pandemic has proven to be an opportunity to leverage the Targeted Inspection (TI) tool that was introduced as a result of the Surveillance Program Evaluation and Update Project. Being a flexible and nimble tool, the TI methodology enabled TC to assess the impact of the pandemic on Canadian certificate holders and to capture emerging safety risks in a changing landscape. More specifically, the purpose of the COVID-19 TI was to:

• Assess certificate holders current levels of risk given the changing landscape of aviation operations;
• Feed data-driven decision-making in order to prioritize surveillance for certificate holders who demonstrate increased levels of risk, and;
• Actively monitor and respond to the changes imposed during the COVID-19 pandemic.

In a matter of 3-4 weeks from start to finish, the inspection was designed, planned, delivered and the results were analyzed.^{Footnote 8} The vast majority of the 2166 operating certificate holders contacted were found to represent very low to low risk. These results are consistent across regions, with inspectors providing only 3 recommendations to increase monitoring and 9 recommendations to initiate a reactive surveillance activity as a result of higher risk.

The COVID 19 pandemic is also driving further improvements to the surveillance program due to the restrictions imposed on travel and physical distancing requirements. Remote surveillance and new surveillance tools aiming directly at regulatory compliance verifications are being considered and piloted to limit on-site presence while maintaining an adequate level of oversight.

It is important to point out that this is not a departure for the system-level approach to surveillance. The vision is to limit on-site presence to conduct necessary compliance verification. The documentation review, documentation sampling, interviews and analysis work required tosupport the system level surveillance will continue in a remote (virtual) environment.

TC continues to rely on the development of its Quality Assurance system to support further improvement of its surveillance system. Following the conclusion of the Surveillance Program Evaluation and Update project and several other Transformation Initiatives, TC is committed to invest the resources necessary to ensure that the surveillance program remains effective at mitigating risks associated with Civil Aviation.

With respect to the enforcement component of the surveillance program, TC has recently moved to a multimodal enforcement model. Civil aviation is working actively with the newly created Centre for Enforcement Expertise to develop procedures and guidance to ensure effective enforcement process.

Finally, TC has launched the Civil Aviation Oversight Application Rationalization project aiming at reducing the number ofoversight related databases and systems while improving workflow, interconnectivity, analysis and reporting capability. This long term project (estimated at 5 years+) is expected to dramatically improve Civil Aviation’s risk-based planning and risk analysis capability once fully implemented.

The Board is pleased that Transport Canada (TC) agrees with this recommendation and is taking steps to improve its surveillance program. In particular, the Board acknowledges that TC has undertaken the following activities:

• TC has made multiple revisions to staff instructions defining the surveillance program, including:
• SI-SUR-001 Issue 8 – Surveillance Procedures
• SI-SUR-029 Issue 2 – Writing Findings of Non-Compliance
• SI-SUR-028 Issue 5 – Surveillance Planning Instructions – Fiscal Year 2020-2021
• TC provided 2-day scenario-based training to its personnel to explain recent changes to TC’s surveillance program.
• TC introduced the Targeted Inspection (TI) tool to allow for a flexible and nimble approach to oversight during the COVID-19 pandemic. Using this tool, TC determined that the vast majority of operators represent a very low to low risk.
• TC is exploring remote surveillance and new surveillance tools intended to verify regulatory compliance.
• TC continues developing its quality assurance system to support further improvements to its surveillance system.
• TC has recently moved to a multimodal enforcement model and is developing procedures and guidance to ensure an effective enforcement process.
• TC has launched the Civil Aviation Oversight Application Rationalization project to reduce oversight-related databases and systems while improving efficiency.

It is likely that TC’s planned actions and actions taken to date will reduce the risks associated with the safety deficiency identified in Recommendation A16-14. However, in light of past instances where non-compliances have gone undetected, or have been allowed to persist, the Board is unable to assess the validity of the risk assessments made using the new TI tool. Furthermore, TC has not provided a clear framework for how it plans to ensure that the frequency and focus of surveillance activities are commensurate with the capability of the operators to effectively manage risk or how moving to a multimodal enforcement model will result in more effective enforcement of commercial aviation.

While the Board recognizes these positive steps, with the information provided, it is not possible to determine at this point if these actions will substantially reduce or eliminate the safety deficiency.

Therefore, the response to Recommendation A16-14 is assessed as Satisfactory in Part.

Transport Canada (TC) agrees with the recommendation.

Since the recommendation was issued in 2016, TC launched in 2016, and completed in 2018, a “Surveillance Program Evaluation and Update” project. TC also established a training program to explain to the inspectorate the changes resulting from this project. The goal of the project was to ensure the surveillance program effectively verifies regulatory compliance at appropriate intervals and is effective in carrying out enforcement action, as required. In 2020, TC has also moved to a multimodal enforcement model. Civil aviation is working actively with the newly created Centre for Enforcement Expertise to develop procedures and guidance to ensure an effective enforcement process.

In our last update in September 2020, the Department committed to:

• Continuously develop its Quality Assurance system to support further improvement of its surveillance system and invest the resources necessary to ensure that the surveillance program remains effective at mitigating risks associated with Civil Aviation; and,
• Launch a Civil Aviation Oversight Application Rationalization (CAOAR) project aimed at reducing the number of oversight related databases and systems while improving workflow, interconnectivity, analysis and reporting capability.

Quality assurance of surveillance findings

Since the previous update, TC has formed the Finding Review Committee in April 2021. Through this initiative, 100% of findings entered by TC inspectors are currently undergoing quality assurance. Findings which do not pass an initial screening are analyzed by the Finding Review Committee which is comprised of representatives from all regions and operational branches. Representatives then provide feedback to their respective operational groups to support continuous improvement in the quality of findings TC is writing. Results from the committee are reported monthly to Civil Aviation senior management.

TC also continues to devote resources to improving the processes and guidance it has in place for inspectors in order to ensure the consistent and effective delivery of the surveillance program. As such, SI-SUR-029 - Writing Findings for Non-Compliance^{Footnote 9} is undergoing a third revision that will provide new tools to aid inspector decision-making. The national consultation process is planned to begin Fall/Winter 2021.

Resources have also been invested in developing a tool for the automation of qualitative risk indicator information though the Surveillance Analytical Risk Algorithm (SARA) program. When fully implemented, it will improve the accuracy and timeliness of data gathering, specifically raising the voice of front line inspectors and leveraging their knowledge of civil aviation enterprises. This project also includes a planning module, which when implemented, will streamline the risk based planning process.

Additionally, TC is reviewing how it balances the oversight workload with the resources available through the surveillance planning methodology.

Civil Aviation Oversight Application Rationalization project

In regards to the Civil Aviation Oversight Application Rationalization (CAOAR) project, TC continues to move it forward even as funding for full implementation is not yet secured past the short-term. The implementation plan is as follows:

• Short term (July –  September 2021): CAOAR is in the process of launching a request for information (RFI) that will complete the last piece of information required to further finetune budgetary and procurement requirements. Meanwhile, the CAOAR team is assisting Civil Aviation Oversight applications’ business owners in mapping and streamlining their processes to help identify master data to be migrated to the future CAOAR solution.
• Medium term (October 2021 – March 2022): The results of the Request for Information (RFI) will help short-list potential vendors and enable Civil Aviation to select a vendor of choice to set up a pilot project for Civil Aviation Medicine. The functionality to be evaluated in the pilot is a comprehensive subset of the functionality available in the overall solution. It is the intent of TC to use the pilot as a Proof of Concept of the overall CAOAR solution. CAOAR will continue to work with Treasury Board Secretariat and with TC’s Digital Services and Transformation Office to secure proper funding to launch a procurement process (Request for Proposal (RFP) OR Advance Contract Award Notice (ACAN)) to secure the CAOAR solution.
• Long term (April 2022 – forward): Once funding is secured, publish CAOAR RFP/ACAN and award contract. With contract in place, the next step is to initiate phased (prioritized) deployment of the CAOAR solution. Depending on approved/allocated project budget, the phased implementation can be accomplished in 2 to 5 years.

This long term project (estimated at 5 years +) is expected to dramatically improve Civil Aviation’s risk-based planning and risk analysis capability once fully implemented.

Adapting oversight to the COVID-19 pandemic

The COVID-19 pandemic has provided a need and opportunity to adapt oversight policies and procedures to a new reality. In order to enable the conduct of remote surveillance activities in response to the challenges posed by the COVID-19 pandemic, TC issued Internal Process Bulletin (IPB) 2020-14 – Remote Surveillance During the COVID-19 Pandemic^{Footnote 10} on December 11, 2020. It provides a decision-making framework for the conduct of remote surveillance in concert with, or in lieu of, on-site activities. The IPB gives guidance on determining whether remote surveillance is appropriate for the activity being performed and the Canadian Aviation Directive (CAD) holder involved as well as the process for initiating remote surveillance with them.

Remote surveillance is still new and as such the full range of its use and effectiveness is not yet known. At present, it has allowed us to conduct oversight while respecting public health orders. Preliminary results suggest that it is well suited to certain activities such as document/record review and interviews. While remote surveillance will never replace on-site activities, as technology improves and the process is refined, it may prove to be a useful tool to in the oversight process by allowing preliminary sampling or interviews to occur, which will maximize on-site activities and refine areas of concern.

The Targeted Inspection (TI) methodology enabled TC to assess the impact of the pandemic on Canadian certificate holders and to capture emerging safety risks in a changing landscape. During the onset of COVID-19, this allowed TC to follow public health orders while maintaining effective oversight by deploying inspectors only where an immediate risk was identified. Through the TI, TC established risk thresholds that would trigger reactive surveillance if any aspect of a certificate’s assessment reached those limits.
As TC prepared to resume regular surveillance, the TI results were considered in conjunction with additional risk indicators to re-prioritize activities for the remainder of the planning year.

In June, TC performed a second round of the COVID-19 TI in order to determine how risks have changed or evolved since last year. The results are still being analyzed.

In its response, Transport Canada (TC) stated that it agrees with this recommendation.

The Board is pleased that TC is continuing to take steps to improve its surveillance program and notes that the following actions have been taken since the last update:

• Quality assurance of surveillance findings
• Created the Finding Review Committee, which carries out a quality assurance function by reviewing all findings entered by TC inspectors. The committee provides feedback to their respective operational groups to support continuous improvement in the quality of findings TC is writing.
• Began a third revision of SI-SUR-029 - Writing Findings for Non-Compliance to provide new tools to help inspector decision-making. The national consultation process was planned for fall/winter 2021; however, TC recently indicated this process has been delayed as work continues on the surveillance program.
• Initiated the development of a tool for the automation of qualitative risk indicator information through the Surveillance Analytical Risk Algorithm (SARA) program. This program is intended to improve the accuracy and timeliness of data gathering and includes a planning module that should streamline the risk-based planning process.
• Civil Aviation Oversight Application Rationalization (CAOAR) project
• Continued moving forward with this project aimed at reducing the number of oversight-related databases and systems, even as funding for full implementation is not yet secured past the short-term. TC has outlined short-, medium-, and long-term goals for this project. According to TC, this long-term project (estimated at 5+ years) is expected to dramatically improve Civil Aviation’s risk-based planning and risk-analysis capability once fully implemented.
• Adapting oversight to the COVID-19 pandemic
• Adapted oversight polices and procedures to conduct surveillance activities remotely in response to the challenges posed by the COVID-19 pandemic.
• Issued Internal Process Bulletin (IPB) 2020-14 – Remote Surveillance During the COVID-19 Pandemic on 11 December 2020, which provides a decision-making framework for conducting remote surveillance in concert with, or in lieu of, on-site activities.

The Board is encouraged by TC’s ongoing efforts to enhance its current oversight policies and training. However, TC’s response does not provide a clear framework for how it plans to ensure that the frequency and focus of surveillance (i.e., systems level vs. process level) and post-surveillance activities, including enforcement, are commensurate with the operator’s ability to effectively manage risk. As a result, it is not possible for the Board to determine, at this point, if these actions will substantially reduce or eliminate the risks associated with the safety deficiencies identified in the recommendation.

Therefore, the response to Recommendation A16-14 is assessed as Satisfactory in Part.