By Bill Booth and Dr. Mitch Javidi
Market Overview
In today’s fast-paced environment, traditional classroom learning does not fully meet the training needs of employees in dispersed organizations. Just as the Web is a critical business component, eLearning is now a critical learning component for organizations. Fast, anytime access to engaging content that is tracked and recorded enables organizations to enhance training in employee skills and competencies. Contextual learning in a more informal setting is the wave of the future as younger tech-savvy personnel enter the workforce. With the cost of implementing e-learning tools falling, more governments, businesses, and schools have added online courses and other forms of distance learning to their organizations. As evidence, The Continuing Education Coordinating Board for Emergency Medical Services reported a total of 593,167 course completions in 2009-10 (1).
There has been widespread research by academics, corporations, and scientists on the effectiveness of e-learning. The overarching result of all the recent studies has been that pure online learning is as efficient as pure face-to-face learning and can be better in some cases where the student has no time limitation. It has also been found that classroom learning enhanced by some online learning is the best approach to education. A recent meta-analysis (2) by the U.S. Department of Education proved the above facts. In addition to being effective, it has also been proven in a Forrester Research study (3) that e-learning earns organizations a positive return on investment in less than a year.
The e-learning industry is expected to see a lot more changes in the next five years than what it has seen over the past decade, primarily because of the technological advancements. Mobile instruments (phones, tablets) are expected to become the platforms of choice. Learners will use content when they need it. Games and simulations will become an integral part of workplace learning. We will see the emergence of personalized learning systems. All these will lead to shift in the way we measure performance with an increased emphasis on cost efficiency.
Statement of Situation
Within the public safety sector, in particular to this assessment, the blended learning requirements of theoretical and practical fire services training activities coupled with dramatic budget constraints are creating challenges to agencies nationwide. Although distance education formats have been embraced, the breadth of offerings available (as example, at FDIC 2011 there were 116 education vendors exhibiting over 400 products), the lacking mechanisms to assess quality and value, along with burdensome subscriptions costs and disconnected Learning Management Systems served as impetus for Interact Business Group’s contracted consultation.
Study Overview
The intent for our engagement was to:
- Identify the macro trends of distance education and the associated market impacts to the public safety communities.
- Secure and evaluate common training course modules from leading content development companies.
- Utilizing an established evaluation criterion, define a course-assessment matrix and weighting system.
- Establish an independent evaluation panel from both academia and training directors from notable fire service agencies/organizations.
- Summary of findings and recommendations.
Course Selection
The two main criteria used for selecting the courses included (1) finding common classes utilized by most fire departments and (2) identifying classes currently available from multiple vendors. The courses selected for evaluation were:
- Building Construction
- Firefighter Orientation
- CPR Academic
- Ventilation
- Driver Operator
- Ground Ladders
- EVOC Approach
- Incident Scene Management
- EVOC Assessment
- Personal Protective Equipment
- Fire Control
- Portable Fire Extinguishers
- Fire Hose, Nozzles, and Foam
Course Evaluation Methodology
The evaluation criteria were developed from the following source material:
Quality Matters Project; funding by the Fund for the Improvement of Postsecondary Education (FIPSE); http://www.qmprogram.org/
SOURCE Book (University of Wisconsin); http://www2.uwstout.edu/content/profdev/rubrics/discussionrubric.html
Evaluating Online Learning (U.S.; Department of Education); http://www2.ed.gov/admins/lead/academic/evalonline/evalonline.pdf
Monterey Institute for Technology and Education (MITE); http://www.montereyinstitute.org/ocep/
Taking into consideration the various aspects of online learning courses, 19 different courseware evaluation scoring elements were selected, including:
- Operating Systems and Platforms
- Course Efficiency
- Purchase Price
- Local/State/National Guidelines
- Writing Preciseness
- Course Navigation
- Student Progress Indicator
- Animation-Graphics-Audio
The evaluation methodology employed a scoring scale ranging from 0 to 4 with each point value having a one-word description. Detailed levels of performance scoring guidelines were developed for each scoring element. The following example illustrates the guidelines used for the “student interest” element:
0 points/poor: The course performed poorly.
1 point/deficient: The course content was deficient in some areas. It lacked content necessary to be fully engaging.
2 points/developing: Overall, the course did a reasonable job of presenting the material. Improvements could be made to improve the course.
3 points/proficient: The course fully engages all course-related topics, consistently encourages participation, and references course materials in relation to content.
4 points/exemplary: The course completely fulfilled course expectations, exhibits organization and absolute comprehensiveness of course materials, and relates supportive examples to express knowledge of topics discussed.
Course Auditors
Another critical component of this study was assembling the subject matter expert team that would be reviewing the courses and scoring them in accordance with the described methodology. The seven-person team consisted of experts from all regions of the U.S. with instructional design expertise from academia, fire training academies, and large to mid-sized fire agencies. Brief descriptions of each team member include:
- A training division chief from a large fire district in the Mountain West region
- An assistant director of a fire and rescue institute in the South Atlantic region
- A dean of public safety training and PhD at a community college in the Mid-Atlantic region
- A fire chief from a mid-sized fire department in the West South Central region
- A bureau chief of instruction for a state division of fire standards and training in the New England region
- A retired fire chief and PhD who operates a consulting group in the West North Central region
- A fire chief from a mid-sized volunteer fire department in the South Atlantic region
Findings and Conclusions
The initial impetus for this evaluation study was based on the need for fire departments to review all facets of their training operations and delivery systems. With budget cuts and staffing reductions, departments must find better and more manageable ways of delivering training. While looking for ways to maintain training proficiency and meet budget constraints, departments are increasingly turning toward technology as their favored solution. Hoping to provide the fire service with a better understanding of its distance education options, this study focused on an increasingly important aspect of fire training technology–online learning and courseware. As described earlier in this paper, the proliferation of courseware vendors and their products has been both a blessing and a curse to fire chiefs and training officers. Interact Business Group undertook this effort with the belief that the fire service would benefit from a standardized, independent, and unbiased courseware evaluation system.
This study resulted in many important findings regarding current conditions within the fire service. Those findings include:
- Most departments do not fully understand the quality, scope, and depth of available courseware. They frequently make purchases without having fully-examined all the facts and many training officers are making decisions about hardware and online courseware from an emotional rather than objective perspective. Based at least partly on frustration, there is a pervasive attitude of “we need to do something else,” or “we have to make a change,” or “let’s give it a try.”
- With respect to the actual courseware products, this study found that the level of quality and subscription cost and value from audited content providers (and likely other vendors) varies significantly. Several courses were deemed “outstanding” by the evaluators while some courses that had been sold and used by departments for a number of years was found to have typographical errors, technical reference errors, and video that did not match the narrator’s script.
- Among vendors, there are considerable redundant costs in course development, systems development, and administrative infrastructure.
- Some departments have developed their own courseware internally–in several cases, they were excellent–but have no method to deliver the products to a larger audience. These departments generally expressed enthusiasm about sharing their “home grown” productions with others but had no budget or process to do so.
- In terms of sharing information about courseware, this study found that many courseware purchases and their associated learning management systems are not connected to departmental and/or state and federal tracking systems.
- Amongst fire service agencies, there is no publication or sharing of success or failures with regard to online courseware usage. For instance, one course was judged to be excellent by one evaluator; in this case the evaluator was from a large paid district, while the same course was deemed to be unacceptable because of the lack of graphics and easy to read bullets.
- There is little or no “lessons learned” dialogue. Of particular importance, the study revealed that there is no uniform method of assessing the quality and value of courseware content.
Several key conclusions can be drawn from this study.
- First, a standardized courseware evaluation system should be created and implemented. These evaluations must be performed by independent, unbiased, and trusted professionals.
- Second, the evaluations should be made available to the fire service in a single, easy-to-access on-line location.
- Third, a mechanism needs to be developed to allow end-users of the courseware to offer their own comments about the products and those comments should be available at the same location as the professional evaluations.
By integrating these conclusions, a significant, cost-effective value we believe would be realized in creating a single online location that would function as the e-learning library and purchasing “hub.” In doing so the quality of content would increase, the ability to not only extend training but to quantify learning all while delivering associated economies of scale for lower costs to all parties.
Bill Booth, President of the Interact Business Group, possesses 20 years of experience developing business plans and strategic models for public safety agencies. Throughout the nation, Mr. Booth has assisted and directed public agencies and educational institutions by applying the best traditional business practices in the field of training facility financing, operating, partnering, technology and funding. Under his leadership IBG is considered the leader and pioneer in the field of strategic and business planning for the public safety training industry. Mr. Booth is also the founder of the highly regarded “Seven-Step Business Plan Process,” which provides a step-by-step roadmap that enables planners and decision makers to approach the training facility planning and sustainability method from a strategic/business perspective. Bill may be contacted at bbooth@interactbusinessgroup.com
Mitch Javidi, Ph.D., is the founder of Readiness Network, an internet-based community designed specifically for high-risk industries including public safety, law enforcement, and military. The network creates an ecosystem that acts as the connective tissue for all stakeholders within these industries and facilities the free flow of information, learning, goods and services within a highly secure platform. Dr. Javidi is an accomplished scholar and professional with over 25 years of academic and global business experiences, serving as an adjunct professor at both NC State and Illinois State Universities and founder of several enterprises worldwide. He is also an honorary Member of the US Army Special Operations Command.
REFERENCES
1. http://www.cecbems.org/Default.aspx
2. http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf
3. The ROI Of eLearning by Claire Schooley, April 13, 2009
