Photo by Tim Olk
Article by Dane Carley
A flight attendant at an employee forum tells the American Airlines CEO Doug Parker, “Our standards suck compared to United and Delta” (Murphy Jr., 2018) in front of a crowd. The CEO’s reply? “We work really hard to match our service to our competitors, and they do the same to us.” (Murphy Jr., 2016). The root cause of the flight attendant’s complaint, according to Murphy Jr. (2018), is that the flight attendant did not feel that the training provided to new employees was adequate, which causes the customers’ experience to suffer. An American Airlines pilot had a similar interaction with CEO Parker seven months earlier when he said, “You keep comparing us to Delta. That’s a mistake. American Airlines never compared themselves to another airline. American Airlines always set the standards and the other airlines copied us” (Murphy Jr., 2018).
These interactions reveal two key items. First, the staff takes a lot of pride in their work and they can sense that the culture that made them proud of their company is slipping away. Second, it reveals that CEO Parker disagrees and that American Airlines is doing well based on the data he regularly reviews. How can two, seemingly, polar opposite opinions exist at the same time? It is easy. Does it happen in your fire department or the fire service in general?
What Are We Measuring?
As we discussed in Tailboard Talk: Is the fire service trading effectiveness for efficiency? (Carley and Nelson, 2018), data will reveal answers to the questions specifically asked but only the questions asked. Data is the perfect witness because it is always objective and it cannot provide information about anything that is not measured. The National Fire Incident Reporting System (NFIRS) is a perfect example with which we are familiar. The data collected can tell a fire chief a ton of information. This information can be sliced and diced even further with programs designed to do so. This information, when compared to other departments, can describe to the chief how his or her department compares to the one in the neighboring city or compares to standards like those provided by the National Fire Protection Association (NFPA), Insurance Services Office (ISO), or for accreditation.
However, with one exception, the data NFIRS collects is limited to measuring inputs and outputs; NFIRS does not measure outcomes or culture. The one exception that is arguably a measure of outcome is dollar loss from fire compared to the value of the buildings in which fire occurs. Ironically, this is the one measure often devalued as being an inaccurate measure. It certainly is not perfect, but it can also be much more telling of a department’s performance than, for example, the number of people who arrive on the scene. Even if 100 people arrive on the scene of a single-family dwelling fire, that does not mean they know how to put out the fire. But, a department that consistently loses 90 percent of the value of a building to fire is an accurate measure of the department’s ability (or inability) to perform at a fire scene.
Based on NFIRS data alone, culture is even harder to measure than fire department performance. In fact, it is impossible to measure culture with NFIRS data. It is possible to measure a department’s current climate through well-written surveys built to produce valid and reliable results, but culture cannot be measured through a one-time survey. This is because a department’s climate describes what is important to the members today; hence, climate can be measured because it is possible to ask questions to measure those aspects. However, culture is something that forms over generations of firefighters telling stories around the kitchen table; holding others accountable for certain actions and expectations; and during training when senior members teach those things that are important to the department.
Therefore, culture can sometimes be measured through surveys if consecutive chiefs of a department intentionally set out to do so by tracking trends in responses to the climate surveys over many years. Otherwise, an organizational psychologist or sociologist can observe a department over an extended time and produce qualitative research that describes a department’s culture. Much of the data for such a project comes from asking the members of the department to describe the department’s culture and combining that to the researcher’s observations.
Data Creates Opposing Points of View
Herein lies the reason why CEO Parker can feel that American Airlines provides a service equal to the competitor’s service while the staff fears for the direction in which American Airlines is headed. CEO Parker likely has tons of data saying that American Airline’s service is as good, or better, than the competitor’s service—things like percentage of flights that arrive on time, the percentage of luggage that is lost, or percentage of seats filled on each flight. It is likely that these numbers are competitive. Since this data measures outcomes to some degree, it is also likely that, unlike NFIRS data, a change in the company’s culture will eventually surface as the employees fail to do what they need to do to maintain the company’s performance. If an airline company’s data only recorded the amount of fuel used, the number of planes flown per day, etc. then it would more closely resemble NFIRS data because it only measures inputs and outputs.
Likewise, many fire departments show the public every day that they are great. Departments pursue and receive accreditation. Departments improve their ISO score. Both occur because of the proper presentation of data—data that is pulled from the NFIRS system. This means that an accredited department or a department with a high ISO rating has proven to one rating organization or the other (or both) that their outputs are acceptable to the rating organization.
However, the flight attendant and the pilot are both an early warning system of sorts that the airline’s culture of service is eroding and that mediocrity has crept into the level of service that is acceptable. And CEO Parker appears to be blind to the future problems because the data he reviews shows the airline is competitive today. The flight attendant specifically pointed out that the training is inadequate and it is affecting customer service. The pilot specifically pointed out that American Airlines used to set the standard, which implies that American Airlines is currently content doing just enough to be the same.
Is your fire department in the same position? Does the line staff seek out ways to be better or to provide a better service only to be told by administrators that the data says what is being done is already great? Does this reply not ring true for the line staff because they see the potential for failure in the future but cannot produce the data to support their position and gain the administration’s attention?
Protecting the Fire Service Culture (When It’s Appropriate)
This happens because members can recognize a culture shift but it cannot be easily measured, particularly with the type of data collected by most fire departments. We described a scenario in Tailboard Talk: Is the fire service trading effectiveness for efficiency? (Carley and Nelson, 2018) where a call taker/dispatcher did exactly what was expected of many call takers across the country—sort through 9-1-1 calls and determine the appropriate action to take. The problem is that the expected action has changed from treating every call as an emergency and responding as such until we know differently to only sending the minimum amount of a response then escalating the call if a bigger problem is found. This is a cultural shift based entirely on data. The data tells us that this works a high percentage of the time; as a general number, but depending on the department, it works at least 90 percent of the time for fire alarms that do not involve a fire, for example.
In such a case, the administration has the data to show how well the system works and how doing this improves other aspects of the department’s response such as engine reliability. Since a full alarm is not responding to every fire alarm, it means engines are not tied up on a fire alarm call in another engine’s area when a second call is received for that engine’s primary response area. These are valid points, which is why accreditation and ISO measure these data points. However, line staff tends to have a visceral reaction to reducing responses. Why? Because members can give specific examples of where having a full response was important the other 10 percent of the time. Furthermore, members can describe how prioritizing calls with some type of emergency medical dispatching system (implying, by extension, that some calls are less important than others are) have affected the department’s culture negatively.
This is why CEO Parker can have such a different opinion from the same airline’s “line staff” and why data is imperfect by itself unless data is collected for all aspects of a department. In the absence of data—specifically data on a department’s culture—it is critical that administrators pay attention to what the line staff is saying. For example, if the line staff is saying that it seems like people do not respond as quickly or seriously to calls as they did 10 years ago, that means the climate is changing; it means that mediocrity is creeping into the department’s operations. It is even more helpful when they have the insight of a timeline and can connect it to certain events in the department. The reason this is important is that if such episodes are not addressed, then the climate will become the department’s culture. In such a culture it appears that mediocrity is acceptable as long as the data shows that its comparable with standards.
However, protecting ideas that have traditionally shaped the fire service’s culture is not always the right thing, either. Data has shown that firefighters have a much higher cancer rate than the general public. Although dirty gear used to be a sign of experience, we now know it is carrying carcinogens. How about hazing the probe? We know that hazing is counterproductive to building effective companies. We have learned these things with data produced by research.
This means that the data used by an administration cannot be ignored because accreditation, ISO, and other measures of performance serve a purpose. In fact, output data collected for accreditation and ISO evaluations is a critical tool for finding specific shortcomings to outcomes. For example, if a department sets an outcome that 90 percent of fires will be contained to the area involved on the first unit’s arrival but cannot meet that outcome, then output data can provide suggestions for why the outcome is not being met (e.g. the effective response force arriving on scene in eight minutes or less is too small). Therefore, it is important to find a solution that incorporates the output data often used by administrations and the observations provided by the line staff.
Measuring Outcomes and Culture
One way to do this is to measure outcomes as discussed our previous piece. Without the proper inputs, outcomes will suffer. Likewise, without the proper outputs, outcomes will also suffer. At some point, without the proper culture, outcomes will also suffer. For example, a department that measures the quality of life of full arrest patients discharged from the hospital is a far superior measure to only measuring the time it took for a firefighter to reach the person in full arrest. A poorly trained firefighter with a bad attitude will result in poor outcome measures for a patient no matter how fast the firefighter arrives at the patient. A department full of poorly trained crews with bad attitudes will result in poor outcome measures for all of the patients, thereby making the quality of life outcome measure an indirect measure of culture.
A second way to measure a department’s performance is to measure the culture itself. This is done by conducting annual surveys built to produce valid and reliable results that measure the department’s climate and then tracking the survey’s trends over several years. The trends reveal the department’s culture if they tracked over several years. This can otherwise be done formally and less frequently by having the department’s culture qualitatively assessed by someone trained to do so. Nonetheless, trends should be tracked to show changes in the culture.
It’s entirely possible to have two opposite opinions of the where a department, or an airline company, stand based on the data the person speaking has reviewed. Both are valid points. CEO Parker is correct in saying that American Airlines is providing equal or better service than competitors. The flight attendant is correct in saying that the service is not equal or better than competitors are. This is possible because each is looking at different data. If these viewpoints are opposed to each other, then a third solution should be considered, because a poor culture will lead to poor outputs and outcomes. Thus, the observation made by the flight attendant and pilot—similar to a department’s line staff making observations about behavior—are ultimately more important because outcomes rely so heavily on an effective culture whereas only measuring outputs is an incomplete measure (Carley, 2016).
Carley, Dane and Nelson, Craig, “Tailboard Talk: Is the fire service trading effectiveness for efficiency?” Fire Engineering, March 2018.
Carley, Dane, Developing outcomes for the Fargo Fire Department,. National Fire Academy, 2016.
Dane Carley has been in the fire service since 1989. He spent 24 enjoyable years pulling hose, throwing ladders, cutting line, and teaching in a variety of capacities and places before being promoted to battalion chief in 2013 for the Fargo (ND) Fire Department. He has had the chance to serve in urban, suburban, rural, and wilderness areas working for city, county, state, and federal agencies over the years. In his spare time, he cowrites for Fire Engineering magazine, coproduces a radio show for Fire Engineering Talk Radio, and coteaches leadership classes that support higher-reliability organizing in the fire service.
MORE DANE CARLEY