MANAGING and regulating automation device configuration code change - whether it be for a PLC, a human-machine interface (HMI) or even a configurable valve - poses an ever-increasing challenge for industrial automation engineers.
As the sheer number of configurable devices on an operating site increases, and site automation systems become more interlinked and network access points more ubiquitous, so too does the risk of unmanaged code change.
To date, managing code change has most often been addressed (if at all) by keeping manual records of the current code version, plus the code edit details and their authors for every change at each configurable device. Such systems are completely “open loop” and highly reliant on site personnel good practice.
On a busy operating site, with many hundreds of configurable devices and the pressure and cost implications of plant downtime, retaining such records accurately is difficult. The practical reality is that such manual systems offer only a rudimentary safety net - access to controller code remains largely unpoliced and software changes are generally not recorded in a form that can be audited.
Code as an asset
FOR many plant maintenance engineers, automation code configuration is regarded as being in a separate world to the science of asset and maintenance management - the tools, techniques and strategies used to minimise maintenance, repair and operation (MRO) costs, and maximise plant uptime.
Conventional asset and maintenance management strategies tend to focus on physical plant: it’s all about large rotating equipment, production lines, utilities and so on - rather than software and code, right?
Not quite, according to Jeff Sladecek, Rockwell Automation ’s business manager for Rockwell Software - Asia Pacific. Sladecek believes that this definition is too narrow, and that configuration code is an essential and often undervalued part of the plant asset big picture.
“Companies don’t often assign a dollar value to their PLC, HMI or drive code until it becomes a problem,” he said. “The problem occurs when uncontrolled changes in the code cause the machine, line or even entire plant production to shut down. It’s then that they learn the true value of code, and that code - like any piece of physical plant - is an asset that should be factored into any asset management and maintenance strategy.”
For this reason, code change management is one of the four operational elements of Rockwell Automation’s Rockwell Software Maintenance Automation Centre (RSMACC) collaborative asset management and maintenance tool.
In manufacturing facilities, security of code and the validation and verification of code have very real cost and plant throughput implications. Poor or failed code can stall operations, impacting on plant uptime and operational equipment effectiveness (OEE), and causing unnecessary maintenance costs.
In the worst cases, it can lead to contaminated or “off-spec” product, or personnel and machine safety hazards. In highly-regulated industries, such as pharmaceutical, food and beverage or underground mining, the lack of an auditable code “trail” could also lead to costly non-compliance fines and legislative costs.
RSMACC drives solutions to four key plant maintenance areas:
• Change Management manages and polices device configuration code.
• Network Health monitors and manages enterprise-wide control network assets.
• Enterprise Online Condition Monitor integrates condition monitoring data to provide real-time analysis and correction of maintenance problems.
• Automated Asset Manager links automation with CMMS to automate parts of the maintenance system, reduce MRO costs and improve OEE.
In structure, RSMACC mirrors the ubiquitous plantwide nature of contemporary industrial automation topologies, using three core building blocks: Microsoft SQL Server 2000, Rockwell Software FactoryTalk and Rockwell Automation’s Integrated Architecture.
Microsoft SQL Server 2000, an industrial-grade relational database designed to process high volumes of critical data, makes up the repositories for the data RSMACC stores and provides. FactoryTalk, a manufacturing information integration strategy, couples with Integrated Architecture to provide the targeted routes to and from this database.
While the potential benefits of RSMACC extend beyond code change management, this is most often the entry point to RSMACC for most manufacturers.
“Everyone gets excited about change management - it’s an immediate and growing problem for manufacturers all over the world,” Sladecek said.
In physical architecture, RSMACC Change Management is founded on two key existing elements on the customer-side: up to 500 “devices” (PLCs, HMIs, drives and so on), plus the plantwide communications network (Ethernet/IP, ControlNet, DeviceNet, Data Highway Plus (DH+) and so on) that interconnects these devices.
Overlaid on this device/network background are RSMACC’s two main physical “building blocks”: the server plus a network of clients (field-distributed work stations from which RSMACC can be accessed).
From a software perspective, core software provides the functionality of “event log”, “audit” and “security server”, while two additional software modules - the “archive” and “verification” modules - provide the all-important change management functionality.
Once the RSMACC core software, server and clients are in place, the foundations are established to grow the RSMACC system into any or all of its four functional areas.
IN the Asia-Pacific region, the key driver for change management is regulatory compliance - most particularly in the highly-regulated industries of pharmaceutical and food and beverage, where compliance with regulations, such as the US Food & Drug Administration (FDA) 21 CFR Part 11 electronic record/electronic signature specifications, might be required.
Sladecek believes the growing pressure across the Asia-Pacific region to maximise plant uptime and throughput is also an important driver behind the move to more refined change management.
“Asia-Pacific manufacturers, like those in the rest of the world, are feeling the economic pressure - ‘If my line is down I’m not making money’. RSMACC Change Management helps minimise this downtime,” he said.
A further challenge in the Asia-Pacific is the high number of technically qualified personnel on operating sites - many potentially with access to code editing facilities. This access to code, unmanaged and untracked, creates the potential for problems on the plant floor.
There is also the issue of plant automation evolution. Much of Asian-Pacific industry is now at an “automation agglomeration” stage. Courtesy of plantwide networking, what were once disparate islands of automation are now integrated into a seamless total-plant automation system.
As a result, code change manifested in one area of the plant can have far-reaching - and sometimes unexpected - outcomes.
Security and audit trails
SLADECEK is quick to point out that while there is region-wide enthusiasm for “change management” tools, not all change management solutions are what they appear to be.
“So much of what is on the market today doesn’t offer a complete collaboration tool. Usually what they provide is simple ‘backup archive and file data verification’. These are just simple documentation tools,” he said. “RSMACC Change Management provides true process verification. It confirms that what is supposed to be out on the plant floor is actually out there.”
According to Sladecek, policing and controlling access to code change is a top priority for most manufacturers. “The number one source of all code problems is security breaches. RSMACC Change Management addresses this by providing a rules-based security system that can be fully customised,” he said.
A further area of distinction is event audit functionality. Legacy change management tools attempt to achieve “audit” functionality via simple “before and after” code comparisons. These “after-the-fact” comparative audits do not provide a truly accurate record of all events and are clearly fallible, he stated.
“Before-and-after comparison audits leave huge holes in the audit trail, and actually give the user a false sense of security,” Sladecek pointed out.
He cites the example of a temporary force on a controller output that would most often be overlooked by the “before-and-after” comparison audit. “RSMACC uses the actual code editor to create the audit trail,” he said.
People, procedures and places
RSMACC Change Management is not an “out of the box” product, but an engineered solution that is tailored for each specific site.
“A significant element of any RSMACC deployment is a detailed review of the site’s existing automation infrastructure - specifically network architecture and what we call the ‘security roadmap’. As a result, RSMACC Change Management is offered as a complete solution.”
Site network assessment is often a complex area, demanding careful review. “It’s not just about devices and device counts - we must have an efficient network architecture in place,” Sladecek said. “Studies show that around 70 percent of industrial networks are poorly implemented and installed, so this is a crucial starting point for us.”
Secondly, and more importantly, is a careful review of existing site operations and the development of the site’s “authentication matrix”. “This is all about studying the people, procedures and places that will ultimately be described in the ‘rules’ defined in RSMACC Change Management,” Sladecek explained.
Importantly, once the RSMACC foundation is in place, the important value benefits offered by the package’s three additional operational areas - Network Health, Enterprise OnLine Condition Monitor and Automated Asset Manager - are easily accessed.
“RSMACC really offers the complete site collaboration environment,” Sladecek concluded. “Change management is most often the start point, but is rarely the end for most customers!”