Jun 29 2017 City of Baltimore 911 Outage highlights Need For A NG911 Solution
Recently, the city of Baltimore experienced a 911 outage that left the city without 911 services for almost two hours. A formal investigation of the situation still has not provided any explanation on what caused the outage, however, it has been determined that 911 calls were being improperly routed by the carrier (Verizon) to an unstaffed center.
During the outage period, would-be emergency callers received the recorded message:
“Baltimore city emergency center, all operators are busy. Your call will be answered in turn. Please do not hang up.”
At that time, city officials also decided to instruct citizens via traditional media outlets and social media to call 311 instead of 911.
City of Baltimore’s 911 Vulnerability
This situation highlights how fragile the 911 infrastructure is in Baltimore, because it utilizes Centrex technology to receive and route 911 calls that pre-dates the Enhanced 911 (E-911) technology predominantly used today. In the more traditional, E-911 model, 911 calls are routed to a Public Safety Answering Point (PSAP), like the city of Baltimore’s 911 center, by selective routers maintained by Verizon that are able to route emergency calls over specialized E-911 trunks, called CAMA trunks. The Selective Routers are able to route calls over multiple CAMA circuits and can re-route the calls over dedicated analog trunks if no CAMA circuits are available.
For Baltimore city, the traditional E-911 architecture, which has been approved by the National Emergency Numbers Association (NENA), has not been implemented, leaving the city vulnerable to failures that occur by using a single central office for routing of 911 calls. So if the central office switch call routing is compromised in any way, as was the case on Tuesday, June 14, 2016, 911 calls are not able to be delivered to any other location.
Available Option for Disaster Recovery
Baltimore County has been discussing the feasibility of allowing Baltimore to leverage the advanced Next Generation 911 technology available in the Baltimore County backup facility for the past eight years, for use as a disaster recovery site for the city. The Baltimore County Backup facility, located in Hunt Valley, approximately eight miles from Baltimore County’s primary center, currently serves as the geographically diverse, backup PSAP for Baltimore County.
The facility is surrounded by Carroll County, Harford County, Anne Arundel County, Howard County and the city of Baltimore. Because of its close proximity to nearby counties and its use of multitenant technology, Baltimore County has extended invitations to nearby counties to consider using the facility as the geographically diverse backup center for their PSAPs.
To date, no county or city has taken advantage of this generous offer from Baltimore County.
The Baltimore County backup facility is equipped with 20 call taker and supervisor positions that are supported by a fully redundant, geodiverse, Voice Over IP (VoIP) telephone switch that is Next Generation 911 compliant. The infrastructure is able to accept 911 calls over CAMA circuits from any of the neighboring counties and is able to interface with any NENA-compliant Computer Aided Dispatch (CAD) system and Geographic Information System (GIS). More importantly, the telephone switch supporting the center was designed to support a “multitenant” configuration where different PSAPs, using differing technologies, can operate independently in the center at the same time, using the same equipment.
Baltimore County’s Robust NG911 Solution
Below is the system architecture of the Baltimore County 911 infrastructure. It leverages a primary PSAP in Towson, as well as a PSAP in Hunt Valley as a backup center. In the Baltimore County configuration, the backup center can be staffed along with the primary center for normal operation, or they can operate independently of the primary center as a standalone backup center for both emergency and nonemergency call handling.
More importantly, the centers are setup to allow diverse routing of 911 calls over CAMA circuits to either facility to provide call routing redundancy. Emergency or nonemergency calls received at either location are routed to available call takers at either center seamlessly, with no awareness by the call takers or supervisors. This automatic redundant call routing feature is available without any manual activation or setup.
Figure 1. Baltimore County Primary and Backup PSAP Centers
Baltimore County Backup Center for Use by the City of Baltimore
The use of a geographically-separated backup center by Baltimore County exemplifies how the backup facility can also be used by the city of Baltimore as a backup PSAP. Located just 17 miles from Baltimore city’s Primary PSAP, the Hunt Valley facility could be setup in a multitenant configuration to allow city PSAP personnel to staff the center during a major disaster, or simply access the call handling functions remotely in the case of an outage such as the one that occurred in mid-June, 2016.
Since the system uses VoIP technology, call takers and supervisors can be located in any facility that has network connectivity to the VoIP switching infrastructure. The system can even be accessed via laptops and tablets over WiFi if an ad hoc center needs to be established quickly to handle emergency calls. Call takers and supervisors utilize a SoftPhone application such as the application shown below, which provides complete call handling capabilities including voice and video communications and can be run from any computer, laptop, or tablet device running Microsoft Windows.
Figure 2. NG911 SoftPhone Application Phased Implementation Approach
A viable backup PSAP solution for the city of Baltimore that utilizes the Regional Backup Center has been presented by Baltimore County to the city as a phased approach.
In the first phase, Baltimore County has volunteered to use its own personnel to handle 911 calls for the city of Baltimore in case of an outage. The solution, along with a Memorandum of Understanding (MOU) between the city and the county, was presented to the Maryland Emergency Number Systems Board (ENSB) and has been approved for deployment later this year.
As part of the first phase, the existing Baltimore County environment will be expanded to support a queue for Baltimore city emergency calls. CAMA circuits for the city will be provisioned at the Hunt Valley facility and emergency calls will be accepted by the redundant, VoIP telephone switch and routed to Baltimore County personnel at either location (Towson and Hunt Valley) logged in to take calls for the city. Incidents for Baltimore will be entered by county personnel into the Baltimore County CAD system and coordinated with Baltimore police, fire and EMS for dispatching. Call recordings will be logged by the Baltimore County digital logging equipment and tagged as Baltimore city emergency calls. Baltimore County reports will provide separate call and agent statistics for calls handled for the city.
The second phase, which has yet to be presented for funding to the ENSB, allows for Baltimore city personnel to staff the Hunt Valley facility and handle emergency calls independently of the county. In the second phase, the Hunt Valley facility is reconfigured to support a multitenant environment, utilizing the core telephony infrastructure as a shared resource and implementing separate logging equipment and data repository for Baltimore. Such a configuration allows for call handling equipment to be shared and repositories such as the digital recorder and reporting database to be separated.
The multitenant configuration also allows for the Hunt Valley facility to be simultaneously staffed by both city and county personnel in a cooperative manner with each organization handling their own emergency calls using their own preferred infrastructure. Separate CAD and mapping systems for the city and the county can be supported in the solution providing complete independence.
A Cost-Effective Solution
The backup solution proposed for the city of Baltimore offers the most expeditious and cost-effective method for the city to implement a fully operational backup PSAP geographically separated from the current Baltimore city PSAP location. Such a solution would allow the city to handle disaster scenarios such as the riots that occurred late last year that left the 911 center overloaded and unable to assist city residents
The solution also has the advantage of leveraging an existing regional backup center and also leap frogging the city of Baltimore directly to a Next Generation 911 solution. If Baltimore desires, the proposed multitenant configuration can be expanded to support the city of Baltimore’s primary center with little additional cost as compared to implementing a new, NG911 system for “Charm City.”