- The Department of Defense’s (DoD’s) MHS GENESIS EHR system implementations at Initial Operating Capability (IOC) care sites across the Pacific Northwest are not operationally effective or suitable, according to a newly-released Initial Operational Test and Evaluation (IOT&E) report.
“MHS GENESIS is not operationally effective because it does not demonstrate enough workable functionality to manage and document patient care,” wrote OT&E Director Robert F. Behler. “Users successfully performed only 56 percent of the 197 tasks used as Measures of Performance.”
The April 30 report evaluated the MHS GENESIS EHR implementations at Fairchild Air Force Base, Naval Health Clinic Oak Harbor, and Naval Hospital Bremerton in Washington State from September through December 2017. The DoD Program Management Office (PMO) postponed an evaluation at the fourth IOC care site due to the high volume of problems officials found at the first three sites, the report found.
“MHS GENESIS is not operationally suitable because of poor system usability, insufficient training, and inadequate help desk support,” wrote Behler. “Survivability is undetermined because cybersecurity testing is ongoing.”
As part of the evaluations, PMO conducted a series of integrated tests on MHS GENESIS and worked to address all problems uncovered throughout testing — particularly any problems that affected patient safety. Clinicians and administrative staff felt that most of the EHR system’s critical deficiencies were potential patient safety threats, user surveys showed.
In addition to problems with the EHR system’s ability to facilitate clinical documentation, officials also found fault with physician workflows.
“Poorly defined user roles and workflows resulted in an increase in the time required for health care providers to complete daily tasks,” wrote Behler. “Some providers reported that they needed to work overtime and were seeing fewer patients per day due to delays caused by defects in MHS GENESIS.”
Furthermore, officials explained that users “questioned the accuracy” of the health data exchanged between external systems and MHS GENESIS. Problems with health data accuracy could pose a threat to patient safety.
“Users generated 22 high severity incident reports (IRs) that the testers attributed to interoperability, including interoperability of medical and peripheral devices,” noted Behler.
Users also ranked MHS GENESIS poorly for EHR usability.
“Users gave MHS GENESIS usability an average score of only 37 out of 100 on the System Usability Scale (SUS), well below the threshold of 70 that indicates acceptable usability,” stated Behler in the report.
According to officials, users reported that more experience with the system did not improve EHR usability.
Augmenting problems with poor EHR usability was a notable lack of sufficient EHR system training. EHR system training is an imperative aspect of the implementation process—a lack of training has been associated with persistent billing problems and conflict between health system management and staff.
“Training was insufficient to overcome usability problems, and a lack of documentation forced users to develop their own operational workarounds,” wrote Behler.
“User survey comments from the three IOT&E sites reported similar problems that 2 included undocumented and inconsistent workarounds, excessive system latency, inaccurate patient information, badly assigned user roles, poor user training, uneven assistance from on-site trainers, and lack of visibility of the status of trouble tickets,” he continued.
From January 2017 to November 2017, users at IOC care sites submitted 14,383 help desk trouble tickets — more than help desk employees were able to effectively resolve.
According to the report, instances of EHR downtime were common.
“System outages indicated that the end-to-end system and supporting network did not have sufficient availability to support operations at the four IOT&E MTFs,” wrote Behler. “Users reported increased lag times when other IOT&E sites went live, suggesting the current system and supporting network configuration will not support the hundreds of additional sites planned for MHS GENESIS.”
Additionally, officials found that several essential system capabilities were either non-functional or missing completely. The report determined patient referral requests were not processing, lab results were not showing, and oral surgery apps were failing to launch.
“Pharmacists, in particular, found the system difficult to use,” wrote Behler. “They were working extended hours due to longer prescription order workflows. Pharmacies averaged fill times of 45 minutes or more for prescriptions that previously averaged 15 to 20 minutes. Pharmacists had to employ manual processes to fill orders due to interface problems.”
Manual processes — also known as EHR workarounds — were identified among the top 10 patient safety concerns of 2018 by an ECRI Institute report earlier this year.
Users also reported inefficiencies related to system log-ins. Log-in times were inconsistent, with some users reporting log-in times of three minutes, while others stated it took as long as 20 minutes to log into the system.
“Users reported their first login of the day usually took the longest,” stated Behler. “MHS GENESIS system timeouts and users having to change user roles further exacerbated the problem.”
Log-in inefficiencies also contributed to problems related to EHR system credentials. Users normally unauthorized to access certain parts of the EHR system gained access to information and functionality they should not have had access to complete certain tasks, Behler wrote in the report.
Based on observations during the first three evaluations, OT&E officials recommended the Under Secretary of Defense for Acquisition and Sustainment delay further deployments until JITC completes the IOT&E and PMO resolves any remaining problems.
Officials also recommended PMO continue to improve EHR system training and clinical documentation for users. Behler suggested PMO work with users to document, reduce, and standardize EHR workarounds.
Additionally, officials said PMO should improve interoperability, monitor system reliability and availability throughout the EHR system lifecycle, and conduct further operational testing at the next MHS GENESIS deployment to evaluate revised EHR system training and go-live process improvements, among other recommendations.
PMO will conduct an evaluation at the fourth DoD IOC care site within the year.