BY PETER BRYAN and PAMELA PANE
Whether your agency is completing a strategic plan, participating in the self-assessment accreditation process, justifying current or needed resources, or just performing an analysis, there is a distinct need for accurately measuring effectiveness, analyzing service levels, and developing a “national” fire service database. Law enforcement agencies, for years, have been extremely effective in using statistics to demonstrate needs, secure funding, and implement new programs.
The fire service has been wrestling with the question of how many firefighters a community should have, and we have apparently “accepted” that there is no clear-cut answer. Further, we have failed miserably, in comparison with law enforcement, at establishing quality data and effective systems or analysis of the demands for service (including fire and EMS). Perhaps the fire service has just been too busy trying to keep up with the day-to-day workload and handling crisis issues to find time to effectively, or even adequately, analyze demands for service and resource needs and compile information and data.
Local control affords the fire service the ability to complete planning and analysis by doing any of the following:
- measure the effectiveness of service and organizational development,
- implement strategic planning,
- set a service level policy,
- perform self-assessment/accreditation,
- compare our agency with “like agencies” (can we really compare?), and
- develop “Standards of Cover.”
STANDARDS AS BENCHMARKS
The fire service can use as benchmarks “standards” such as those in Table 1, and it is critical that we develop additional ones.
 |
One of the most critical and unresolved problems in measuring performance is the difficulty of comparing apples to apples. Should it be this difficult for the fire service to measure effectiveness? Of course not. Our law enforcement public safety partners have been doing it for years. So why do we have such difficulty?
AVERAGING VS. PERCENTILE/FRACTILE
For years we used “averaging” to analyze data, especially response times. Many “standards” have referenced using a methodology called “percentile” or “fractile” for approximately the past 10 years. It is time for the fire service to eliminate the term “average” from its response time vocabulary. Averages may or may not still be appropriate for analysis of data that we “count,” such as the number of personnel on a first alarm arriving within a specific time. As an example, averaging the number of personnel over a year might give an agency an appropriate analysis of whether adequate personnel were on-scene.
Analyze time studies (response time, dispatch/call processing, turnout, travel time), however, using the percentile/fractile method. Most standards reference 90 to 95 percent. Using a percentile/fractile goal or standard will generally require a significantly higher level of service when compared to averaging. A good example of such a difference is to take a set of data and illustrate average and 90th/95th percentile for the same response time data. There can be almost an eight-minute difference, based on the method of analyzing the data.
FIGURING PERCENTILE/FRACTILE
The example below, from Inland Counties Emergency Medical Agency (ICEMA), San Bernardino, California, illustrates the correct method for figuring percentile/fractile.
 Tables courtesy of the Inland Counties Emergency Medical Agency (ICEMA), San Bernardino, California. Published with permission. |
Fire information management systems generally use averaging instead of percentile/fractile. However, knowledgeable information technology (IT) departments can compile lists from these systems of time data (information fields) or write routines to analyze data into meaningful listings. The listing of response times is generally done by whole-minute time frames (1-2 minute, 2-3, 3-4, etc.). For each whole-minute range, the numbers of responses within that minute are counted, and the percentage of total response is calculated. Beginning at the lowest whole-minute range, the cumulative percentage is added together until at least 90 percent is achieved (using the 90th percentile/fractile). A cumulative percentage of 89.50 would require using the next range until 90 was found. The whole-minute range now becomes the 90th percentile/fractilethat is, if the 3-4 minute range was at least the 90 cumulative, then the performance was less than 4 minutes response time 90th percentile/fractile. Note: The response times for the tables are for the first piece of qualified apparatus.
 |
null
RANCHO CUCAMONGA ANALYSIS
The following example of a data analysis of the Rancho Cucamonga Fire Protection District apparatus travel time to EMS incidents for the calendar year 2004 indicates that the travel time was less than 4 minutes 55.18 percent of the time. The travel time at the 90th percentile/fractile was less than 7 minutes.
The Rancho Cucamonga Fire Protection District has used the Commission on Fire Accreditation International (CFAI) Self-Assessment Manual as the starting point to analyze data to allow us to move toward self-accreditation. CFAI uses the NFPA standards as discussed in this article and scores based on fractile response time data.
 |
A time study is just one goal or standard for measuring effectiveness. You can analyze each segment of total response and time to manage an incident to final conclusion. Using a “time continuum” (below) can assist with the analysis of each segment, thus allowing incremental improvement for each segment, which can add up to improvement in the overall response time.
When the annual budgets for career fire departments are in the millions, we must ask ourselves why we are not fully analyzing our data and performance and don’t want to be using real numbers.
APPLYING THE PERCENTILE/FRACTILE METHOD
Below are some nationally recognized good practices or standards that use the percentile/fractile methodology.
NFPA 1710
• 4.1.2.1: The fire department shall establish the following time objectives:
- One minute (60 seconds) for turnout time.
- Four minutes (240 seconds) or less for the arrival of the first-arriving engine company at a fire suppression incident and/or 8 minutes (480 seconds) or less for the deployment of a full first-alarm assignment at a fire suppression incident.
- Four minutes (240 seconds) or less for the arrival of a unit with first responder or higher level capability at an emergency medical incident
- Eight minutes (480 seconds) or less for the arrival of an advanced life support unit at an emergency medical incident, where this service is provided by the fire department.
• 4.1.2.2: The fire department shall establish a performance objective of not less than 90 percent for the achievement of each response time objective specified in 4.1.2.1.
NFPA 1221
• 6.4.2: Ninety-five percent of alarms shall be answered within 15 seconds, and 99 percent of alarms shall be answered within 40 seconds.
• 6.4.3: Ninety-five percent of emergency dispatching shall be completed within 60 seconds.
• 6.4.5: Where alarms are transferred from the public safety answering point (PSAP), the transfer procedure shall not exceed 30 seconds for 95 percent of all alarms processed.
Isn’t it about time that we, the national fire service, develop and use appropriate standards and methodology to effectively analyze service delivery? Percentile/fractilenot averageis that methodology. Let’s get unified and consistent.
PETER BRYAN is chief of the Rancho Cucamonga (CA) Fire Protection District, which provides full-service fire, EMS, hazardous materials, technical rescue, and wildland urban interface services. He has served as the chief for Rancho Cucamonga, City of Monrovia (CA), and City of Norco (CA) during his 35 years of fire service.
PAMELA PANE is a management analyst II for the Rancho Cucamonga (CA) Fire Protection District. She has been responsible for the compilation of data and information for the district’s strategic plan.
