How to Reduce Healthcare Operating Costs & Achieve Optimal ROI in a Smart Epic Infrastructure
Kishore Esanakula
The Epic and NetApp Journey to Optimization
Healthcare costs have risen dramatically over the past few decades, and they continue to rise. Many healthcare organizations still run on technologies that are not optimized to reduce costs and operational overhead. To meet the challenge of infrastructure optimization, NetApp continues to innovate and to enhance its industry-leading solutions. Gartner recently recognized NetApp (Gartner Magic Quadrant 2019) as a Leader in Primary Storage that far exceeds other storage vendors in its solution execution and unwavering pursuit of its vision. In 2010, NetApp and Epic, a leader in electronic health record (EHR) systems, started a journey with laser focus and a commitment to optimize the use of Epic infrastructure and to reduce operational overhead for healthcare organizations.Epic Capabilities at a Very High Level
Epic provides a highly customizable health information system for small, midsize, and large healthcare organizations. At a high level, Epic offers capabilities such as patient scheduling, registration, eligibility, inpatient, ambulatory clinics, emergency room, trauma center, radiology, lab information system, portals for patients and for providers, utilization management coding, billing, claims management, population health management, referral management, provider administration, enterprise master patient index (EMPI), clinical research management, procedural clinics, patient registries, patient cohorts, risk scoring and stratification, descriptive analytics, diagnostic analytics, predictive analytics, prescriptive analytics etc. With Epic, your organization can coordinate care across a community of providers irrespective of EHR used.In addition, Epic offers a comprehensive set of tools that enable your healthcare organization to manage inpatient and outpatient pharmacies. With Epic’s capabilities, your organization can also realize a telehealth vision in areas like specialist referral services, patient monitoring, peer-to-peer consultation, and ongoing patient care. And Epic provides interoperability (SOAP, JSON, HL7, X12, NCPDP, and FHIR standards) capabilities that enable your healthcare organization to perform standards-based health information exchange out of the box.
Epic also supports Lucy, a freestanding personal health record (PHR) system that does not depend on any EHR. And Epic offers the App Orchard platform, a healthy ecosystem for vendors to enroll, standardize, implement, and showcase EHR-agnostic provider and patient web and mobile applications.
Epic High-Level Architecture Components
Epic uses an N-tier or multitier architecture, with a thick client, end users access the application via Citrix. All core capabilities (including patient and user identity store) are driven by an Intersystems Caché OLTP database (DB). There are multiple instances of this DB that serve functions as Prod (OLTP), failover in a HA configuration, reporting (RPT), disaster recovery (DR) and several test/train instances. Both Caché Prod and HA instances use common data storage media for the database files.The RPT Caché database acts as the source database for the Epic Cogito analytics suite of capabilities, although certain handpicked low-impact reporting workbench reports run on the OLTP DB Caché Prod. Epic is currently bound to Predictive Model Markup Language (PMML), from a machine learning model portability and execution standpoint. Right now, Epic is well positioned to take advantage of GPU-based machine learning models: Epic Cogito Analytics includes the Clarity DB and an OLAP DB called Epic Caboodle.
The following diagram shows Epic’s high-level architecture, with NetApp®storage.
The Epic Caché DB runs on a virtualized instance of Red Hat Enterprise Linux 64-bit. From a storage standpoint, the data files are the lowest-level components. If the Prod database fails, Epic fails over to the HA pair, provided that the primary database data files and storage are available and in good standing. If the primary Prod and HA databases and the data files fail, Epic has a DR failover member that can take over as the primary database. The Epic DR database uses a separate set of data files and storage. In the unlikely event that the Prod, HA, and DR databases are not usable, Epic RPT can be used as read-only copy. And Epic has one more level of failover implemented, in the form of printable reports called Business Continuity Access (BCA) PC. The Epic Caché Prod OLTP database is the most critical Epic component.
And given the criticality of Epic environments to the Healthcare organization, Epic constantly searches for supporting technology that is reliable and innovative. To establish new leaders in infrastructure platforms, Epic carries out in-house performance testing by using various platforms throughout the year. Historically speaking, Epic in-house test results are not public information, which puts customers at a real disadvantage when they want to choose a platform.
The performance results that NetApp and its partners publish are intended to empower healthcare organizations like yours to make informed choices as budgets continue to be cut. For a sizing guidance for your healthcare organization, see the NetApp technical report, the NetApp Sizing Guidelines for Epic.
Our Starting Point and Roadmap
On an average workday, clinicians deal with structured data, unstructured data, relational data, and nonrelational data. Health information systems like Epic also usually demand various file systems and protocols in order to access and utilize this data. When we consider the volume of data generated by healthcare organizations on a daily basis, it is clearly critical that the underlying storage technology satisfies the performance and usability demands. With our joint laser-focused approach, we have achieved performance results that reflect Epic’s and NetApp’s commitment to affordable healthcare for the current and for the next generations.At the heart of this performance is the FlexPod® platform, a joint solution from NetApp and Cisco that enables healthcare organizations to consolidate storage for Epic in a single cluster. This consolidation means fewer nodes and fewer disks than any other vendor’s system requires. You can provision and deploy storage for complex applications in fewer steps and with ease. And with a FlexPod platform, you also get one-stop FlexPod Cooperative Support from Cisco and NetApp; you don’t deal with several vendors for any troubleshooting that you might need. A number of capabilities are featured in this solution however we will focus on the ONTAP All-Flash Array (A300 and A700 models tested) and ONTAP Adaptive Quality of Service (AQoS) features.
As we started our journey, we realized that a good engineer always tests to see the limits of a system and understand impacts of unplanned situations. So, to really test the NetApp advanced quality-of-service (AQoS) policy, we used a runaway (with no AQoS policy) Epic data seeding and generation instance. Success! The results proved that the AQoS policy and guarantee cannot be bullied by a rouge Epic instance. And our claims are not restricted to just laboratory results. The results that we achieved in the lab have also been proven by real-world Epic installations on FlexPod with NetApp AFF A300 all-flash storage systems. And the FlexPod platform with AFF A700 storage helps organizations operate some of the world’s largest (measured by the number of patient encounters) Epic installations.
As I mentioned, our efforts to optimize Epic infrastructure are laser-focused. That’s why both Epic and NetApp have committed themselves to use only Epic-provided standard performance testing and measuring tools to generate data loads and to measure storage performance. For this performance testing, Epic provided data seeding and the load-generation Perl scripts, dgen.pl and RampRun.pl.
Performance Testing Platform
Epic and NetApp have chosen a proven winner in converged modular infrastructure in the form of the FlexPod AFF series, specifically AFF A300 and its big brother, the AFF A700 system. The FlexPod infrastructure is a modular, converged, virtualized, scalable (scale-out and Scale-Up), and cost-effective platform. With the FlexPod platform, you can independently scale out compute, network, and storage to accelerate your application deployment. And the modular architecture enables nondisruptive operations even during your system scale-out and upgrade activities.AQoS is the industry-leading NetApp capability that really stood out in its superiority during performance testing. AQoS guarantees performance and protects core workloads from runaway workloads. AQoS enables your healthcare organization to get the most out of your hardware with confidence. Gone are the days of single-use infrastructure platforms that result in hoarding of unused assets.
The effectiveness of AQoS has been tested in carefully tailored labs, with oversight by Epic and NetApp engineers. By using policies, AQoS supports storage tier SLAs. NetApp uses machine learning (ML) capabilities in NetApp Active IQ® Unified Manager (formerly OnCommand® Unified Manager software to come up with optimal AQoS policies for workloads. This approach effectively eliminates guesswork and moves toward fact-based AQoS policy definition, which helps you achieve policy-governed and well-managed storage I/O budgets. These are some of the components that add the “smarts” to a smart Epic infrastructure.
The test platforms were NetApp ONTAP A300 and A700 All-Flash Arrays. Overall testing requirements are detailed below, however a key metric required was to meet write latency of 1ms or less, which these platforms beat handily. In Addition, many additional capabilities like multi-protocol access, deduplication/compression and cloud integration are inherent to the platforms. As organizations look to incorporate cloud technologies, the A300 and A700 platforms offer native capabilities using the NetApp Data Fabric and Fabric Pool capabilities.
The following table lists hardware and software configuration used for FlexPod implementation with NetApp AFF A300 and AFF A700 platforms respectively.
For this performance testing, the A300 used virtualized Red Hat Enterprise Linux VMs on a VMware hypervisor. AFF A700 testing used RHEL on bare metal servers. Below are the Epic test hardware and software components. 
How the Test Results Compare with Epic Performance Requirements
Following are standard Epic system requirements:- Epic OLTP transactions are written to the Caché database in bursts. Each burst is initiated by the Epic system at 80-second (80s) intervals and must complete within 45s.
- 99% of the read latencies should be <60 milliseconds (<60ms).
- 99.9% of the read latencies should be <200ms.
- 99.99% of the read latencies should be <600ms.
Deploy a Platform That Outperforms Epic Requirements
The test results prove how the FlexPod AFF A300 system not only outperforms Epic’s required standards, but also sets a new benchmark that competitors can only wish for in the years to come. The AFF A300 can comfortably run 90% of the Epic environments that are in use today. The larger AFF A700 platform is needed for only the largest (the remaining 10%) Epic deployments that leverage the Epic Community Connect capability to the fullest.
The innovative NetApp AQoS capability can help simplify and add “smarts” to your Epic deployment by consolidating Epic and non-Epic workloads on the same FlexPod system while ensuring the critical Epic workloads perform as intended. NetApp AQoS policies can tame even the wildest runaway workloads, guaranteed!
Take the First Step: Learn More
Get started on improving your Epic infrastructure, reducing your operational costs, and ultimately enhancing patient care. Find out more about how FlexPod can optimize your Epic environment:
Kishore Esanakula
Student for life, Kishore is welcoming new ideas from all walks of life. He takes pride in creating proof of concepts and pilot solutions to help socialize concepts by example. Furthermore, he has more than 14 years of experience designing and implementing solutions for business problems with alignment to enterprise strategy, as well as more than 9 years of experience in Health Care domain (payer, plan design, retail/mail-order pharmacy, hospital system and Clinics) performing Architecture and application development functions.
