While safety culture is a topic of keen interest in business aviation, measuring the health of a safety culture in the operational setting of a specific organization remains a challenge — one that persists partly because academic researchers in this field typically have focused almost exclusively on large, commercial air transport operators and have ignored business aviation operators. Considering that the U.S. Federal Aviation Administration (FAA) predicts that turbine-powered business aircraft will average 12.5 million flight hours annually between now and 2034, the lack of attention seems to be more than a minor oversight.1 One result is that operators may lack a sound basis for evaluating the quality of the nonpublic, proprietary safety culture measurement tools on the market.
I reached this conclusion after surveying related academic literature and aviation industry sources as part of my planned doctoral dissertation on the effectiveness of the International Standard for Business Aviation Operations (IS-BAO) in the development of healthy safety culture. Essentially, this search for a scientifically designed and validated survey instrument to assess safety culture in business aviation operations produced no results.2 Moreover, the search revealed no peer-reviewed studies on safety culture in business aviation.
This led to a yearlong research project to produce a scientifically tested measurement instrument for the assessment of business aviation safety culture. While the end result was such an instrument, the research also produced some interesting results for students of aviation safety culture in general and of business aviation safety culture in particular: it verified the presence of distinct, measurable constructs that reside within a healthy safety culture. Comprehending these constructs and the questions that define them may help aviation safety managers and other aviation leaders to better assess elements of safety culture in their own organizations.
Safety Culture Defined
The following constructs have been derived from a commonly agreed definition of the term safety culture. In a 2002 study commissioned by the FAA, a group of scholars surveyed and compared safety culture across several industries and produced the following definition:
“Safety culture is the enduring value and priority placed on worker and public safety by everyone in every group at every level of an organization. It refers to the extent to which individuals and groups will commit to personal responsibility for safety; act to preserve, enhance and communicate safety concerns; strive to actively learn, adapt and modify (both individual and organizational) behavior based on lessons learned from mistakes; and be rewarded in a manner consistent with these values.”3
In Implementing Safety Management Systems in Aviation, the authors define the term more succinctly. “Safety culture,” they say, “can be described as the values, beliefs and norms that govern how people act and behave with respect to safety.”4
While both definitions provide an overall context, they do not tell us how people in a healthy safety culture act. More importantly, the definitions do not provide a way to measure the health of the safety culture. Hence, I saw the need to focus on research constructs for better insights into safety subcultures.
Brief Look Back
This effort wasn’t the first to realize that measurement of safety culture required going beyond definitions. One earlier project focused on individual behaviors, or constructs, inside of safety culture to validate a survey of commercial aviation operations in 2006.5 That instrument, the Commercial Aviation Safety Survey (CASS), was designed to validate a safety culture survey for airline operations based on a five-factor model that included organizational commitment, management involvement, pilot empowerment, reporting systems and accountability systems.
After distributing this instrument to the pilots and managers of a large U.S. airline and analyzing the results, the researchers were forced to revise their model to focus on four main factors within safety culture: organizational commitment, operations interactions, formal safety system and informal safety system.
In 2008, a different research team administered the revised four-factor CASS to the flight operations department of a major European airline.6 Their analysis confirmed the existence of a positive, effective safety culture within the organization by focusing on the constructs. The same year, two researchers released an FAA-funded report highlighting their revised instrument, now called the Safety Culture Scale Measurement System (SCISMS).7 Presently, the SCISMS appears, from my literature survey, to be the only industry-validated instrument designed to measure safety culture in aviation organizations.
In a related study, another research team tested a four-factor model using an existing data set from a major international carrier.8 While this model was unorthodox, their results proved that a four-factor construct fit the data better than a single-factor construct model.9,10
Constructs in Subcultures
When I realized there was a need to design an instrument to assess safety culture in business aviation operations, my first step was to turn to the safety subcultures hypothesized by Alan J. Stolzer, Carl D. Halford and John Goglia in their 2008 work, Safety Management Systems in Aviation, summarized in Table 1. While the names chosen for these subcultures, their attributes and key behaviors/constructs had appeared previously in other research, most notably the work of risk theorist James Reason, the Stolzer team was the first to grasp the importance of the subculture context and to group the five subcultures as part of a coherent safety culture description. These subcultures provided only the constructs I needed to describe the performance of people inside the safety and a critical component — the measurable behaviors/constructs — needed for safety culture assessment in business measurable aviation operations.
First Instrument
Entire textbooks have been written about how to construct survey instruments that will accurately collect the data they are intended to collect. Question phraseology, question length, the order of questions and the response mechanism for questions must be taken into account, for example. Once the instrument is designed, multiple pretests and evaluations are required to ensure that the instrument generates consistent data. Knowing that opportunities for testing would be limited, rather than develop questions for the entire survey, the majority of my questions were adapted — with attribution — from an open-source version of the SCISMS and the Safety Culture Checklist designed by Reason.
The result was a 40-question survey instrument. Five questions at the beginning collected data on the respondent and five questions at the end asked for the respondent’s opinion of the survey itself. The remaining 30 questions were written to assess the five safety culture areas identified by the Stolzer team, five questions per subculture, with an additional five questions that assessed the respondent’s opinion of the organization’s leadership. The respondents answered all the safety culture and leadership assessment questions using a Likert scale (1 = strongly disagree, 2 = disagree, 3 = slightly disagree, 4 = neither agree nor disagree, 5 = slightly agree, 6 = agree and 7 = strongly agree).
Test of First Instrument
For the first test of my survey instrument, a sampling frame was derived of the 44 IS-BAO–registered business aircraft operators based in the U.S. states of Illinois, Indiana and Wisconsin. Fourteen operators agreed to participate, but 13 actually completed the survey. The data collection instrument was distributed to 162 respondents affiliated with the 13 participating operators. A total 101 responses were received for a response rate of 62.35 percent.
The first test of the survey instrument was promising in both its validity and its reliability. Nearly all respondents reported no difficulty understanding the questions, and they agreed that the survey was valid in measuring both safety culture and leadership. Measurements of the scale reliability in the subcategory human behaviors/constructs were regarded favorably as well. In every construct area but one, the survey satisfied the research community’s reliability criteria.
But most interestingly, when the survey responses were loaded into a principal component analysis, they did not group precisely according to the five subculture areas described earlier. Instead, they grouped into eight distinct areas, only some of which aligned with the original researchers’ hypothesis. Based on the content of the survey questions, I renamed certain constructs within the model. With these renamed categories, the survey answers (variables) were distributed as depicted in Table 2. Questions devised to assess safety culture leadership factors across three construct areas did not group into one area as expected.
Second Instrument
While the distribution of variables across the new constructs in the first test was instructive, it was not conclusive, largely due to the limited sample size. I was still convinced that the survey variables should group into measurable, understandable constructs. This led to a review of results and a revision of the survey instrument to measure the original research team’s five subculture constructs and one additional area, the Empowered Culture, in which members of the business aviation organization are given authority to recommend revisions to safety guidance and procedures, where appropriate. Additionally, I tightened the focus on leadership and used three questions to gather a limited measurement of the respondents’ views of their leaders. Rather than an even distribution of questions across measurement areas in the revised survey instrument, I revised it to measure the subculture areas as depicted in Table 3.
Second Test
To test the second version of my survey instrument, I targeted 116 operators in California, New Jersey and New York, as well as three operators from the energy industry not located in those states, for a total of 119 operators. Of these operators, 18 agreed to participate in the survey. Survey links were forwarded to 376 respondents, and 232 responses were received, a response rate of 61.7 percent.
The results for the second instrument were even more encouraging in the area of content validity. The respondents typically reported that the survey measured safety culture and leadership, they said they found the questions were easy to understand and not biased, and they agreed the time required for the survey was optimum. Scale reliability for the behavior measurement areas/constructs was higher as well, although one construct area still did not achieve reliable status. Most encouraging was that a confirmatory factor analysis revealed that the hypothesized distribution relationship in Table 3 held true once the appropriate covariances were taken into account. In other words, the seven culture constructs appeared to accurately represent the latent values inherent in the data.
Results and Application
So what is the takeaway from all this? Safety culture is measurable. Not only is business aviation safety culture measurable from an overall perspective, but the subcategories’ individual behaviors/constructs are measurable so that a business aviation organization and its leaders can know the strengths and weaknesses of its safety culture. The descriptions of the final constructs are in Table 4.
Measurement of these constructs requires specifically targeted questions that ask the respondents’ opinions about the aspects of their safety culture. Some of the representative questions for each construct are:
The Informed Culture — My organization places high priority on training;
The Flexible Culture — My organization has a mechanism to approve deviations from existing guidance if something unexpected happens;
The Empowered Culture — Line personnel are actively involved in identifying and resolving safety and/or operational concerns in my organization;
The Reporting Culture — All personnel can report safety discrepancies without fear of negative repercussions;
The Learning Culture — If a safety issue is raised, it will be communicated to everyone in the organization;
The Just Culture — Standards of accountability are consistently applied to all personnel in my organization; and,
The Leadership Culture — Leadership encourages us to actively identify hazards and safety risks and when we do, leaders take prompt action to investigate and mitigate them as practicable.
As noted above, these questions measure respondents’ perceptions/opinions of the area assessed and in this instance, it is perceptions that matter. These constructs are all elements of organizational culture, and according to one of the foremost authorities on the subject, Edgar Schein, organizational culture is largely a function of the perceptions of those involved.
Next Steps
In mid-2014, the International Business Aviation Council (IBAC) was considering incorporating the latest safety culture survey instrument into its safety culture toolkit and distributing this toolkit to IS-BAO–registered operators worldwide for periodic self-assessment. I plan to use the instrument to complete in 2015 research for the doctoral dissertation, as noted, on the effectiveness of IS-BAO implementation and leadership in the development of a healthy safety culture in business aviation operations.
Chris Broyhill is the transportation director at Exelon Corporation and a Ph.D. student. He has held leadership positions in aviation organizations for over 30 years and has been researching safety culture in business aviation for the last three years.
Notes
- FAA. Forecast tables, FAA aerospace forecast fiscal years 2014-2034. Washington, D.C.: FAA, 2014.
- Due to their limited distribution, often at high cost, and their unavailability to academic researchers, such proprietary tools are unlikely to have been subjected to scientific rigor that would help assure their statistical validity and reliability.
- Wiegmann, D.A.; Zhang, H.; von Thaden, T.; Sharma, G.; Mitchell, A. Safety culture: A review. (Technical Report no. ARL-02-3/FAA-02-2). Atlantic City, New Jersey, U.S.: FAA, 2002.
- Stolzer, A.J.; Halford, C.D.; Goglia, J.J. Implementing safety management systems in aviation (Kindle edition). Burlington, Vermont, U.S.: Ashgate, 2011.
- Gibbons, A.M.; von Thaden, T.L.; Wiegmann, D.A. “Development and initial validation of a survey for assessing safety culture within commercial flight operations.” The International Journal of Aviation Psychology, 16(2), 215–238, 2006.
- Von Thaden, T.L.; Kessel, J.; Ruengvisesh, D. “Measuring indicators of safety culture in a major European airline’s flight operations department.” The Proceedings of the 8th International Symposium of the Australian Aviation Psychology Association, Sydney, Australia, April 8–11, 2008.
- Von Thaden, T.L.; Gibbons, A.M. The safety culture indicator scale measurement system (SCISMS). (Document no. HFD-08-03/FAA-08-2). Atlantic City, New Jersey, U.S.: FAA, 2008.
- Kelly, T. K.; Meyer, R. C.; Patankar. M. (2012). Safety culture: Testing the safety culture pyramid with structural equation modeling. (No. Grant # 08-G-014). St Louis, MO: Center for Aviation Safety Research.
- Patankar, M. S.; Sabin, E. J. (2010). The safety culture perspective. In E. Salas and D. Maurino (Eds.), Human Factors in Aviation, second edition. Chennai, India: Elsevier.
- The model tested was a pyramid, first hypothesized by Patankar and Sabin in 2008, and it consisted of four distinct, yet interdependent, layers. “The base of the pyramid included safety values and unquestioned assumptions that serve as the foundation of the model,” their report said. “The second layer is described as safety strategies and consists of leadership strategies, policies, procedures, organizational norms, history, legends and heroes. The third level of the pyramid [represents] the safety climate of the organization and consists of the short- and near-term set of attitudes and opinions surrounding safety. The apex of the pyramid [represents] safety behaviors and consists of individual and group safety-related behaviors within the organization.”