Scalability testing of the PACS for the future

From here to there

As diagnostic imaging becomes even more complex, so, too, does the business of running a hospital. Margins are low, competition is high and hospitals are consolidating just to survive. Next-generation imaging solutions are emerging to take the industry to the next level. Industry visionaries have coined the term PACS 3.0 to describe the system of the future with patient-centric data and the fulfillment of anytime, anywhere access. But these visionaries have put the industry on notice that PACS 3.0 simply can’t be achieved without the ability to scale and interoperate with other systems. The burning question in the industry should be: how do we get from here to there?

A robust system is essential, one with low overhead and increased scalability for large customers as well as small customers who cannot afford to make a mistake in an unforgiving economic climate. A balanced PACS ecosystem is also a necessity to take radiology beyond the enterprise level. Lastly, interoperability with third-party systems is an absolute must.

How far off is the industry from true interoperability? Mike Tilkin, CIO of the American College of Radiology (ACR) sited evidence of progress from the Fast Healthcare Interoperability Resources (FIHR) HL7 inititiative, the increasing availability of web DICOM solutions and the formation of the CommonWell Health Alliance vendor group.

In a previous interview with HealthIT Executive Forum, Tilkin explained, “These initiatives should be supported by technology vendors that validate real-life scenarios with practical test protocols that prove scalability and the ability to interoperate with the overall enterprise-wide imaging ecosystem.”

Taking the challenge

 McKesson decided to conduct a large scale test to prove the scalability and stability of McKesson Radiology and McKesson Enterprise Image Repository with real life-scenarios and practical test protocol that illustrate the image volume load eventually experienced by McKesson’s largest customers. In this new architecture, the client- server interaction fundamentally differs because all communication now goes through web servers.

The protocols for the scalability test were developed through the beta phase of NYACK & St. Luke’s Kansas City during the McKesson Radiology 12.0 Beta phase, more emphasizing on the real-life scenarios on the protocols through actual customer interactions. As such, it encapsulated workflow for a small volume community hospital to larger volume enterprise site. The changes were tested internally by the Performance Test Team and then through the actual load experienced at St Luke’s, but the in-housing testing only scaled the system to 600K exams per year, which aligned with the study volume for the initial customer uptake. McKesson Radiology would have to support higher volumes when larger facilities upgraded to version 12.

In anticipation of a growing customer uptake of McKesson Radiology 12.1 in facilities with exam volumes greater than 600K and in anticipation of the ever increasing exam volume expected from larger health systems, McKesson conducted a scalability test using a 5 million exam volume at the Microsoft Testing Labs in Issaquah, Washington. Similar scalability testing had previously been conducted at the HP labs in 2009, and McKesson wanted to establish new scalability and performance baselines, given the development of the new McKesson Radiology architecture.

The process

Preparations started with the data and included tweaking a 15 million study record test database with 8.4 million patient records and populating it with more than two terabytes of image data. This included adding the PACS users and other supporting data, adding 20,000+ studies with medical images as well as corresponding reports and scanned documents. The virtual machine templates, automated scripts, a cold backup of the database, and all 2 TB of image data were packed up and delivered to Microsoft for deployment in their test lab.

McKesson worked closely with the Microsoft team to design the test infrastructure, ensuring sufficient storage capacity and network bandwidth to support the volume of data traversing the system. All workstations and servers, except the database server, are all virtual machines running in Microsoft test lab, which allows for rapid deployment, configuration and scaling of server/workstation resources. The virtual machines for the workstation and servers are configured to our standard hardware specification.

At Microsoft, the test environment was set up with the McKesson systems and equipment including:

•             550 McKesson Radiology workstations

•             50 load-generating machines

•             25 McKesson Radiology Application  Servers

•             40 TB of NAS storage

•             20 TB of archive storage

•             One physical Oracle Database server

The test protocol began with 150 to 200 medical imaging studies of varying modalities being sent from virtual devices, this corresponds to approximately 60 GB/hr of image data being sent or approximately 5000 inbound HL7 messages per hour being sent from virtual devices. The nearly 670 GB/hr of image data was being read and requested by the end users and the studies were marked reported then reports and scanned documents were attached to the image data sets and the studies were archived and retrieved.

Finally, QA issues were identified and resolved. During the process, the frontend, automated workflow used the McKesson Radiology Station 12.0 user interface and simulated 150 technologists and 390 radiologists’ concurrently accessing the system by working through automated scripts for radiologists and technologists that simulated four different workflows.

Key scalability results

Within two days of the actual testing time, McKesson Radiology 12.0 was running the full 5 million exam load. The system ran for 240 straight hours. Benchmark results collected throughout showed no degradation in performance for most tasks. Overall health of the system was good with no memory leaks or runaway CPU usage. Throughput of storage and network were on par with the expectation of data volume and no bottlenecks were identified.

This test case demonstrates McKesson Radiology’s ability to scale and perform for both small and large customers using real life scenarios and practical test protocols. It also illustrates McKesson’s commitment to Better Health 2020 by ensuring the reliability and scalability of the system to perform in mission critical enterprise environment.