When Disaster Means No Recovery

Does turning to a PACS vendor for disaster archiving add certainty and ease to the recovery process when a mishap occurs and images must be retrieved or a PACS database must be reestablished? One New York hospital’s radiology department thought so. The PACS administrator confirms that events have proved it right so far. Hudson Valley Hospital Center (HVHC) in Cortland Manor, NY, is like many midsize hospitals that moved quickly to get into the digital age. In 2003, the hospital’s radiology department installed its first PACS. Today, according to Erik Lundberg, manager of radiology and CT systems, the only film that the hospital processes is for a limited amount of mammography, and even that is about to go digital. When HVHC chose its PACS, it was selected, in part, Lundberg says, because the it was, at the time, one of the few Web-based offerings on the market (the hospital wanted a Web-based solution). Once the PACS was installed, administrators at the 128-bed hospital, which serves a string of Hudson River Valley towns about an hour’s drive north of New York City, quickly turned their thoughts to disaster recovery. The World Trade Center attacks had left no illusions about invulnerability by the time the HVHC PACS was installed. “We knew we wanted an off-site disaster-recovery storage facility. We’re pretty close to the Indian Point nuclear power plant, as well as Manhattan,” Lundberg says. A mishap or terror attack at Indian Point could eliminate or make inaccessible any on-site disaster archive, along with the original PACS data. As it turned out, HVHC didn’t have to look far for the solution to its disaster-archiving needs. Its PACS vendor FUJIFILM Medical Systems USA, Inc, Stamford, Conn, was just rolling out its own off-site disaster-archiving option for Synapse customers. HVHC was one of the first to sign on, and the outcome has been a happy one, Lundberg reports. He says, “I think Fuji made our initial PACS entry cost-effective, and the ongoing arrangement has been cost-effective. The product with the disaster archive is stable and it’s been robust. The service has been great. We have no plans to change.” Lundberg knows whereof he speaks. Twice now, he says, the department’s PACS has gone down, once when the local RAID had a glitch and a second (and more serious) time when there was a very rare happening in which two drives failed at once due to a RAID backplane failure. The first time, he says, HVHC was able to recover its missing PACS files simply by downloading them over the Internet from Fujifilm’s disaster-archiving facility in Denver. Because the images and patient data were stored by Fujifilm directly from HVHC’s PACS, a direct substitution for the corrupted files was all that was needed. What Fujifilm had stored at its disaster site conformed exactly to the Synapse PACS database at HVHC, and no complicated interfacing or manipulation of the replacement files was necessary. They fit right in where the originals had been corrupted, Lundberg says. The second mini-disaster was more serious. This time, the hospital’s PACS database was completely compromised and a simple Internet download could not solve the problem. Part of Fujifilm’s disaster-recovery contract specifies that it will, within 72 hours, airlift a backup PACS server to any client suffering sufficient damage. That is exactly what happened with HVHC, according to Lundberg. The hospital made do with files recovered via Web until the server arrived. When it did, it was simply plugged into the hospital’s PACS in place of the old PACS server. After a simple installation, HVHC’s PACS was back up and running. The server that Fujifilm had sent was an exact replica of the original server, preloaded with the PACS database, the patient information, and all the stored images. “We still have the server,” Lundberg says. Integration Reynold Yordy is business manager of managed services for FUJIFILM Medical in the United States. He oversees Fujifilm’s disaster-recovery unit, among other duties. Fujifilm got in the disaster-recovery business, Yordy says, because it saw a need for storage that was integrated with the Synapse PACS and the databases that its customers were compiling. “We felt there was a need for more than simply off-site storage and the hope that a disaster wouldn’t occur, which is what a lot of vendors were providing. There was a lot of off-site storage, but it wasn’t integrated, so there was a big gap in disaster recovery. In health care, hope simply is not a strategy. We built our solution assuming disasters will happen,” Yordy says. Fujifilm now is providing disaster services for Synapse PACS customers (as an optional feature). Once the Synapse PACS has been installed, he adds, the feed to the disaster site is automatic. “There is no added equipment necessary. We establish a secure link to our center in Denver,” he says. The customer’s Synapse PACS automatically recognizes the link and feeds images and data to the recovery site. “It’s more than connectivity,” Yordy says. “It’s not just about DICOM images, it’s the entire database, including all relevant data. What we send to the recovery site is just like their original system.” What Yordy calls the front line, or immediately accessible copy of the archive, is stored just as the customer stores data on-site in a RAID linked to a storage-area network. With an Internet connection, the customer can retrieve specific files as needed from Fujifilm’s Denver archive with mouse clicks. Fujifilm storage also includes added safeguards. The system stores three copies of the customer’s files. There is the primary RAID storage and there are two copies stored on tape, one kept in Denver and the second sent to Fujifilm’s own off-site disaster repository managed by a secondary vendor. Thus, Fujifilm itself is backed up for its own disaster recovery, Yordy says. The primary copy is kept for a period defined in the customer’s storage contract, as is the third (or legal) copy. The secondary (or clinical) copy is kept for 7 years. The retrieval system has been perfected to the point that when a customer’s PACS can’t get an image stored at the hospital, it will automatically default to the disaster-storage site in Denver to retrieve it. “The tech could then call us, and we would replace the data lost on their set,” Yordy says. Because HIPAA mandates that patient information remain private, Fujifilm encrypts all data stored on tape for secondary and tertiary disaster storage, Yordy says. The primary recovery file must be immediately accessible, so it is not encrypted. Any data transmitted between Denver and the client sites, however, are sent through secured virtual private network links that can’t be accessed by outsiders, Yordy says. “That secures communication between us and the facility, whether we’re sending to them or they’re pulling the data. When anything is shipped outside the data center, it’s encrypted. We encrypt before we ship off site,” Yordy says. The Fujifilm disaster-recovery site and the devices used to implement recovery were all designed, Yordy adds, to give the client peace of mind. “When we built our system, we built with the idea that disasters will happen. We wanted to make sure the customer could recover quickly, and that all the data were restored in the shape they were already in. We also wanted a predictable cost,” Yordy says. Pay as You Grow To keep the cost affordable, Fujifilm uses a per-study method of billing. The customer pays an up-front fee based on the estimated number of imaging studies it will be storing. “It’s a one-time flat fee per study,” Yordy says, “with a study defined as what a facility would use to bill by.” The initial fee is based on average modality use. If the customer exceeds the limit, then another small fee is added. Once the one-time fee has been paid, the study (or studies) can be accessed as needed without additional charge. The stored data are accessible for the life of the contract or the legal lifetime of the imaging study itself. The advantage of this system, Yordy says, is that a hospital or radiology group with a Synapse PACS can contract for disaster storage only as needed. “They spend money as they make money, basically,” he explains. Yordy says the Fujifilm is a great success. A number of its 400 Synapse installations in the United States have already signed up for disaster archiving, and many more customers are considering adding disaster recovery services or additional Fujifilm managed services, such as turnkey hosted and managed PACS. “We have definitely tried to come up with ways not to be just a disaster-recovery service, but to be a business-continuance service,” he says. “We know there are a lot of customers out there with that second copy of their data sitting there right next to the first copy.” Good Fit For HVHC, Lundberg says, Fujifilm disaster archiving has not only been a success, but has been implemented and managed in a spirit of cooperation. Storage volumes are one example. According to Lundberg, the hospital’s radiology department is staffed by six radiologists in a local group under contract to the hospital. The radiologists have used the PACS to stay abreast of imaging growth at the hospital. Lundberg’s assistant Danielle Tortorici says that in 2006, the latest year for which figures are available, HVHC completed about 87,000 imaging studies. That was up from about 81,500 studies in 2005 and about 76,000 studies in 2004, she says. As the imaging volumes have risen, Lundberg says, Fujifilm has been more than fair. “There is an agreement, up front, about how big this thing is. Is it a boat or an aircraft carrier?” Lundberg says. “Fuji has been flexible. I’ve killed them with stuff, at times, and they haven’t complained. They don’t nickel and dime you to death. Those conversations just don’t happen.” Moreover, he says, when there is trouble, the disaster site responds quickly. “They navigate the bureaucracy to make things happen. You always get the feeling that upper management supports them.” How would he sum up the relationship? “The product is very stable and the people try to do the right thing,” he says.