Oil Giant Improves Cybersecurity, Lowers Manpower Costs by Automating Data Collection From Thousands of Control Devices
Forests of wooden derricks sprang up in a rural California valley when oil was discovered there in 1899. Today, those towering structures are gone, replaced by the more familiar low pump jacks, plus steam-injection systems necessary to coax the crude from the earth. Most of the valley’s “black gold” is heavy crude, and its viscosity much like molasses.
In 2013, a major California petroleum producer pumped 177,000 barrels per day from the valley. The company’s California business unit comprises six separate oil fields.
The typical well site includes steam generators – essentially huge, high-volume, high-pressure boilers – used to apply the pressure and heat necessary to force heavy crude above ground.
Some sites incorporate cogeneration systems, which use natural gas turbine/generator sets to produce electricity that supports site operations. Cogeneration in the California valley produces enough energy to supply the power needs of over 1.5 million homes.
Such complex processes and so much capital equipment demand a huge number of control assets. The company’s operations oil-field process control network (PCN) utilizes around 2,800 control devices distributed among the six oil fields.
For years, the quantity of devices, diversity of its valley operations, geographic distance between them and absence of an effective technology had complicated the company’s effort to create a comprehensive system for identifying, tagging and monitoring all its control devices.
“They had a manual database of this type of information, but it was incomplete,” said Joe Zaccaria, reliability program manager, Rockwell Automation. The oil and gas company has a large installed base of Rockwell Automation control devices, as well as equipment from multiple other vendors.
“The oil company had spent a lot of time and money putting people in the field to document part numbers, quantities, etc.,” Zaccaria explained. “But this manual collection method couldn’t keep up with such a large and dynamic PCN installation, which includes several generations of mission-critical assets.”
The manual collection process itself carried risks for the company. Employees in the field could encounter hazards ranging from rattlesnakes to desert environmental conditions to the potential for exposure to hydrogen sulfate gas (H2S).
Concerns about a newer and potentially more devastating threat – a cybersecurity attack – loomed even larger.
The tipping point came in late 2011. A corporate SCADA audit identified cybersecurity risks related to the lack of reliable control-asset data keeping by the valley business unit. The audit was prompted by a new, companywide, cybersecurity policy based on the ISA99 standard for industrial control systems.
“Until the early 2000s, control systems and devices were essentially immune to cyberattacks for a number of reasons,” Zaccaria said. “One, the networks weren’t as open. And two, there were few hackers. Now, plenty of them are looking for industrial vulnerabilities, especially where they can cause considerable collateral damage.”
Outsiders weren’t the only potential threat. Numerous system components were missing and considered stolen from control cabinets across the valley.
The oil giant’s California business unit turned to Rockwell Automation for a technological solution to centralize information gathering and continuously monitor the company’s assets. Rockwell Automation was already piloting a new Logix-based application that would do exactly that.
“Over the course of several months, we went through discussions with their IT department, their SCADA department and other key decision makers,” Zaccaria said. “They all agreed our new approach was the best method because it was safe and secure, and applies to Allen-Bradley® and third-party devices.”
At the heart of the Rockwell Automation solution is an innovative software-as-a-service technology. The new diagnostic reliability (DR) system automatically identifies, interrogates and monitors control hardware by continuously scanning the PCN network.
Proprietary code encrypted into the Allen-Bradley SoftLogix™ 5800 controller directs the DR system to locate and tag every device on the PCN – including programmable logix controllers and everything connected to them in the rack and subcomponents – using native industrial protocols to talk to each device.
The DR system also retrieves detailed information about each piece of equipment, including but not limited to the part number, series version, serial number and firmware version. The system exports all that information to a structured Microsoft SQL Server®, and displays it to the end user utilizing FactoryTalk® VantagePoint® software from Rockwell Automation. This provides the oil company with the most essential tool for maintaining a cybersecurity strategy around a master inventory of all the devices on the PCN.
The availability of a comprehensive and accurate control asset master inventory is essential to the effective creation and sustainability of a PCN cybersecurity strategy.
Using FactoryTalk View SE HMI software, the DR system also takes a deep dive into each asset to understand its status, health and parameter changes.
The Rockwell Automation team has continuously added more tracking capabilities to the DR system, including scans for security vulnerabilities. For example, the system continuously checks the key switch status on processers. When a key switch is open, a hacker could potentially change the programming – an attack that could cause downtime and production losses or, at worst, create safety and environmental hazards.
The DR system also reports if a device was moved from one location to another – or if it goes missing.
If the DR system shows that a device that was formerly on the network is gone, the system produces an exception report. Company personnel receive this report and follow up to see if the device has been removed or if it’s some type of failure or fault.
An exception report is just one of many created in the FactoryTalk VantagePoint software, based on information culled from the DR system. The VantagePoint software provides context to the data with Web-based dashboards, allowing different users to view actionable information important to their roles.
In addition to the key switch status reporting, maintenance personnel can see if the processor has a low battery and dispatch a field employee to replace the battery before a failure occurs. The DR system data also is used to produce monthly scorecards detailing the PCN’s overall performance and reports on network anomalies.
Two Rockwell Automation field-service engineers work on-site at one of the oil producer’s California campuses, helping the company’s team troubleshoot and correct issues identified by the DR system, as well as build out its capabilities. “We’re embedded with the customer to continuously develop more decision-support tools,” said Ray Spangler, field service engineer, Rockwell Automation. “The DR system is expanding as we explore, identify and implement new ways to use it.”
One example: The company implemented a yearlong maintenance contract with the Rockwell Automation team to manage, monitor and modify the system as needed. The Rockwell Automation team is currently developing an asset-management and life-cycle extension plan that will help identify repairable equipment, enable the migration of obsolete devices, and identify out-of-date firmware.
The PCN compliance program continues to evolve based on new objectives that are prioritized and incorporated into the service program, expanding the DR system’s capability. The oil company’s business unit is already in full compliance with the new corporate cybersecurity policy based on the ISA99 standard for industrial control systems. By moving from a manual database to automated real-time data collection, the company can also now perform proactive maintenance. This is helping lower the cost of manpower in the field and yield more barrels of oil per day.
While it is too soon to calculate the full potential of the DR system, one anecdote provides a hint. Recently, the DR system identified a fault on one oil-well processor. “They were losing about 15 barrels a day because that well wasn’t pumping,” Spangler said. “But the DR system pinpointed it, and the company got someone out there to fix it.”
The oil giant is so impressed by the preliminary results that it’s considering using the DR system as a corporate standard. In fall 2013, the company’s chairman reviewed the project when it was just in its infancy. “His comment to the team was, ‘How fast can we take this global?’” said Matthew Hermans, global product business manager, Reliability Services, Rockwell Automation.
Soon after, Rockwell Automation experts were asked to conduct a presentation explaining the DR system to the company’s global SCADA management team. Rockwell Automation is now collaborating with the company to install the DR system internationally.
“Different people in different industries have spent tens of thousands of hours trying to figure out how to automatically convert this kind of information into operational intelligence,” Hermans said. “Today, our customer is using data derived from the DR system to help overcome long-standing business obstacles in a matter of days, if not hours.”
