After four days of ravaging Jamaica, Cuba, and the Bahamas, Hurricane Sandy slammed into the New Jersey shoreline on October 29, 2012, resulting in losses of $30 billion to businesses in the affected area; damaging or destroying 346,000 homes; causing widespread power outages (some of which lasted for several weeks); and killing 37 people. Neither the popular seaside city of Cape May nor Cape Regional Medical Center (CRMC), a 240-bed community hospital in Cape May Courthouse, suffered damage from what turned out to be the deadliest, most destructive hurricane of the 2012 Atlantic hurricane season—and the second-costliest storm of its type in U.S. history.
The extent of Sandy (and the devastation that it wrought) significantly affected the medical center in another way, however. It propelled implementation of a comprehensive backup and disaster recovery solution to the top of the list of priorities, with Off-site Archiving/Disaster Recovery—a service offered by FUJIFILM Medical Systems USA, Inc, under its Synapse® Cloud Services umbrella—going live in August 2013.
CRMC serves a population of 100,000 year-round residents and several million visitors annually. Patients’ imaging needs are handled in-house and at Cape Radiology, a freestanding, hospital-owned imaging center located in nearby Rio Grande, New Jersey. Collectively, these entities perform 90,000 imaging procedures per year.
The hospital’s need for a new approach to disaster recovery had been a topic of discussion among its administrators and other constituents for approximately two or three years, until right before Sandy struck. Secondary copies of the hospital’s database, including images, had long been stored on a server in an off-site facility. The server was aging and had not been properly maintained; a loss of nodes frequently impeded access to necessary data.
Keith Babore, CRMC’s director of imaging services, says, “We knew we were on borrowed time with this solution, based on its age alone. It wasn’t even really a backup and disaster recovery solution, anyway. There had been no question that we were going to have to do something.”
That a new backup and disaster recovery plan was warranted, given the hospital’s coastal location, also did not escape notice. Bob Gates, technology manager, observes, “This area is very vulnerable to hurricanes. We weren’t confident in our ability to retrieve good copies of our images after any floodwaters receded.”
The Wake-up Call
Still, disaster recovery planning remained on the back burner—until Superstorm Sandy “put in a major wake-up call,” Babore states. “The eye of the storm passed right through an area near the hospital. Nearby towns were underwater; had we been in Seaside Heights or Long Beach Island, not that far to the north of Cape May—where the damage was unimaginable—we would have been in big trouble.” Approached shortly after the hurricane about the need for a viable backup and disaster recovery solution, CRMC’s senior leaders displayed what Babore and Gates describe as sticker shock.
To gain decision makers’ support, the hospital’s IT Steering Committee explained to them the consequences of not having a true backup and disaster recovery solution in place—especially in light of the area’s susceptibility to hurricanes, of predictions that changing weather patterns might lead to a higher incidence of storms, and of the unlikelihood of emerging from another storm after having suffered only a few minutes without power (as was the case during Hurricane Sandy). Potential consequences include interruptions in the provision and continuity of patient care, lack of access to prior studies for comparison with new images, and delays in (or the impossibility of) restoring electronic images following a disaster.
All Aboard
Once senior leaders were on board, a cross-functional group comprising radiologists, IT personnel, and imaging staff members convened to conduct roundtable discussions of what was needed in a backup and disaster recovery solution. Many of the same individuals had been involved in a previous effort to select a PACS; this resulted in the implementation of Fujifilm’s Synapse PACS. Radiologists, Gates says, emphasized that they wished the backup to be invisible to them—in other words, no different from the original format. He adds, “From the IT perspective, durability and ease of information access were important.” Other must-have features included rapid image turnaround and a high level of confidence in the quality of the backup images.
Several vendors were invited to present their solutions, with a short list of two or three vendors then created and site visits to the finalists subsequently conducted. Reports from KLAS®, which conducts and publishes research intended to help health-care providers formulate informed decisions, were also consulted during the evaluation process.
The Fujifilm solution received the nod for several key reasons. “We liked the fact that this vendor, unlike several of the others, started out in imaging rather than in radiographic technology,” Babore says. “It made us more confident that the backup images would be of the quality we needed. That the solution is based on Synapse technology was a factor, too, as we knew there would be a single point of accountability. The promised turnaround times were excellent, the pay-per-use business model was very attractive, and the vendor was also willing to be flexible.”
Multiple Storage Points
Now that the solution is in place, images and related data, in addition to being stored on the hospital’s storage network, will be transmitted to Fujifilm’s secure data center in Denver, Colorado, via VPN, for storage in the vendor’s private cloud. A recent upgrade of Internet connections in the Cape May area allows transmission to occur at a speed of up to 100 megabits per second. Like many Synapse users, CRMC’s radiology department and Cape Radiology create both lossless compressed (original) and lossy compressed (clinical) versions of images for storage at the center.
The lossless images are stored on offline media, and the lossy images are stored online for seven years. A duplicate of the hospital’s database is stored at a second off-site Fujifilm facility to protect its entire Synapse configuration, in the event of a disaster. Such features as redundant power protection, redundant fire suppression, and a physical plant constructed to be impervious to environmental conditions such as flooding further safeguard the data.
In mid-July, CRMC was in the process of conducting tests to determine the volume of data that can comfortably be sent to the data center at any single juncture. “We don’t want anything we send to be any more than 24 hours old, and our goal is to get as close to real-time transmission as possible,” Babore observes.
The RAID) technology used by Fujifilm in providing the service will afford the hospital faster access to data and images, should restoration be necessary. In these instances, data and images alike can be sent from the main data center and/or the secondary facility, at a rate of 9,953.28 megabits per second, making use of multiple OC-192 connections.
At the same time, the vendor will build a temporary, stand-alone server loaded with the CRMC data and deliver it to the site of CRMC’s choice. In addition, anomalies are identified and rectified early because the service allows for round-the-clock monitoring of network and application availability; bandwidth, CPU, and memory use; and storage performance and use. “The upshot is that if we lose everything, we will be back online within 72 hours—which is a very reasonable amount of time, and right up there with U.S. government Incident Command System models,” Gates says (those models call for a recovery time of 96 hours).
The solution will also facilitate business continuity and uninterrupted patient care in the event of difficulties with CRMC’s PACS. Physicians will be able to retrieve images and data stored at the data center from any remote location.
Although Babore and Gates have clearly not witnessed the service in postdisaster action—and hope to keep it that way—they believe that it was the right choice for CRMC. As Babore puts it, “This is a true backup and disaster recovery solution, in which we are already 100% confident. With the weather as crazy as it has been in the past two years or so, it’s critical.”
Julie Ritzer Ross is a contributing writer for Radinformatics.com.