Cybertrol Engineering Helps Major Hospital Campus Upgrade to Virtualized DCS System

Just as health care has advanced dramatically in the last two decades, so has the physical size of many major health care institutions. Some of these institutions have developed into large campuses that not only include patient treatment facilities, but also administrative offices and a complicated building infrastructure to support these vital activities.

Such is the case for one major Minnesota-based health care campus. The hospital system’s utility plant provides steam for heating and chilled water for cooling to eight buildings on the campus – all managed by a distributed control system (DCS) implemented in 1993.

After 20 years, the system had become outdated and presented significant obstacles to utility management. The hospital wanted to replace the outdated system with technology that allowed it to improve efficiencies and better manage energy usage.

With the advent of new systems and advanced technology, it was becoming increasingly difficult for maintenance personnel on the hospital campus to maintain and support the aging DCS. Local distributors no longer carried the needed spare parts, and the hospital had to bring in maintenance personnel from other cities for significant repairs or issues.

Management couldn’t upgrade to a newer version of their existing system because the former distributor could no longer provide the needed parts. Maintenance personnel were purchasing replacement parts from secondhand markets. This created vulnerabilities - if a replacement for a failed part wasn’t available, heating and cooling operations would be at risk.

Supporting the legacy system had become increasingly expensive. Only a few engineers in the state were familiar with the legacy system, so the hospital had to pay for one to come from another city to provide maintenance.

The aging DCS produced one report daily, with no data on historical trends or the possible impacts of any system changes. The lack of insight limited utility management’s ability to gauge the system’s efficiency or energy use.

The hospital knew what it wanted from a new system – EtherNet/IP connectivity enabling remote support, enhanced reporting and a virtualized server system for ease of computer maintenance. It also wanted redundant processors and a redundant I/O network to help reduce the risk of system downtime.

At the time, the hospital was already using Allen-Bradley CompactLogix programmable automation controllers (PACs) to operate its electric chillers. Utility management wanted to bring that ease of programming to the entire system. The hospital worked with Cybertrol Engineering, a Rockwell Automation Solution Partner, to replace the existing system with a virtualized PlantPAx process control system from Rockwell Automation.

Cybertrol engineers selected the PlantPAx system because of its many advantages for redundancy, scalability and cost. Once installed, the system does not require licensing for additional loops, thereby lowering the cost of ownership. Furthermore, a local distributor within the hospital’s metro area could supply any needed replacement parts.

Installation of the PlantPAx system needed to occur with zero downtime to overall production and no interruption to patient activities. The Rockwell Automation library of process objects – a pre-defined set of code for all objects used in process applications – eased the job of programming the new system. Through the library, Cybertrol engineers could rapidly develop code using pre-defined device lists.

“We knew that the key to avoiding downtime was to engineer a solution that would keep the old system running while we migrated existing wiring to the new solution,” said Mark Visness, engineering manager for Cybertrol. The challenge was that the new controls had to eventually end up in the same cabinets, as none of the field wiring or conduit was being replaced.

Cybertrol engineers custom-built 14 new chassis that fit into the existing marshaling cabinets. They temporarily hung each new chassis on the door of the open enclosure and powered them up on a device-level ring network. Plant maintenance staff then swapped over I/O from the existing panels to the new chassis, point by point. Then the old DCS hardware was removed and the new chassis were relocated – live but “finger safe” – into the enclosures. This solution allowed equipment to be cut over without modifying field wiring, while other equipment controlled from the same cabinet remained running.

The PlantPAx solution includes multiple levels of redundancy on an EtherNet/IP network to minimize, and greatly reduce downtime. The system’s device-level ring topology provides redundant pathways for communications. All devices communicate via the EtherNet/IP network on the ring. If a network cable fails, data can travel in the other direction on the ring.

The virtualized system provides redundancy using primary/secondary HMI servers. If one server fails, the secondary server takes over immediately. In addition, the system utilizes redundant data interfaces to the historian. In the event of a failed connection to the historian, data is buffered until the connection to the historian is restored, and then forwarded to the historian.

The virtualized approach to the system also Cybertrol engineers with the ability to access the system and all servers from a remote location, resolving issues faster and reducing support costs for the hospital.

Management gained a wider range of insights and data into plant performance with production intelligence and historian capabilities. The historian collects and archives data from plant data sources and equipment. The system then correlates and aggregates the data through production intelligence software, and produces real-time dashboards and reports, such as daily production totals, volumetric totals and energy usage.

Utility management and Cybertrol can access the reports on-site or remotely, and Cybertrol can also develop custom reports that utility management can view in a Web browser or receive by email. Operators can see the effects of any changes they implement in the plant, calculate changing energy use and monitor other plant conditions.

The new modern DCS system improved the facility’s performance, enabled remote support and lowered the hospital’s cost of ownership – without ever fully shutting down the plant.

“Downtime was never an option for the hospital, for clear reasons,” Visness said. “With the PlantPAx modern DCS solution, we were able to implement the system in phases with no interruption to the hospital’s critical heating and cooling services.”

The system’s scalable control platform does not require additional licensing for added components. Additional I/O points can be added without licensing costs, significantly lowering the total cost of ownership over the lifetime of the system, compared to the previous DCS.

System support and maintenance is also less costly and more reliable. Replacement parts are stocked by a local distributor. The facility maintenance staff, already familiar with Rockwell Automation equipment, can do more maintenance themselves. For additional support, the Cybertrol engineers can access the system remotely – turning potential half-day projects into 20-minute fixes.

Utility managers also gained opportunities for further improvements. With more real-time and historical data on performance delivered via the EtherNet/IP network, the utility staff has more actionable ways to solve any issues quickly, reduce energy use and increase efficiencies throughout the facility.