Curbing Rogue Programming In Manufacturing

William "Liam" Durbin is VP, CIO, Six Sigma, GE Fanuc Automation.

There is tension between IT and Manufacturing. Manufacturing lives in the here and now, with product on the floor, trucks at the dock, and metrics that measure to several decimal places. IT often has programs that span multiple quarters and has intangible goals like migrating to a common technology stack. For IT, hitting a project timeline and budget within 10% of estimates is still considered pretty good. IT assumes forgiveness for this sort miss from internal "customers," but manufacturing must meet commitments to THE customer and misses of any magnitude can bring severe punishment. The divergence of thought between the two functions can provide fertile ground for something I call "rogue" programming to creep into manufacturing plants.

My experience in this space comes from three years supporting Supply Chain for GE Appliances. The role later grew to include GE Lighting and later GE Industrial Systems, about 150 plants in all. In my tenure supporting Supply Chain, I fought this battle on many plains. Some battles were won, others were lost. If I were to do it all again, I would want some information going in that I didn't have until the very end. That valuable information is the content of this article.

I call the disease "rogue" programming, but it has other names. It is also called "unsupported software development," "custom software development," "skunkworks," "homegrown applications" and many other names. Rogue development is usually performed in Microsoft Access, Microsoft Excel, or Visual Basic (including VBA) and sits on SQL Server or IIS boxes. Sometimes shop floor PCs are converted to servers by the addition of IIS. Rogue development typically starts small -- as point solutions such as managing a specific group of people (i.e. maintenance) or a specific process (i.e. test stands, or test loops). Sometimes the application stays small and isolated, but it can also grow tentacles into other systems/groups and new functionality until it becomes business critical and complex.

The disease is insidious. So how is it possible to know how badly an organization is infected? Here are some tell-tale signs:

Program dollars are sleeved away by Supply Chain leadership for mysteriously named programs that smell like IT, but no IT resources are tied to the program. Often this bucket of money will have a nondescript name tied to it, such as "productivity tools." Similarly, program money for enterprise software is cut.
Purchases of development licenses, wireless network hardware, laptops for undisclosed purposes by factory personnel.
Purchase of non-standard (souped up) PCs for personnel in Supply Chain. These are typically the development machines and are sometimes doubling as servers.
Non-IT personnel requesting access to enterprise systems or access to the network.
Hiring of IT generalists by Supply Chain. A generalist is typically college-educated person, but with very limited experience in formal IT organizations. A generalist can also be self-taught and has broad (but not deep) IT skills. In other words, they can do some networking, some coding (usually in VB or Java), some server construction, and some database design, but none of it particularly well. They typically have limited or no experience (or interest) in security.
Problems reported to IT on the shop floor mysteriously go away, before IT can offer a response or solution.
Lines between IT and shop floor rogue programmers become blurred. Systems are not recognized by the IT Help Desk. Escalation processes have dual paths.

People unfamiliar with the situation often think skunkworks operations should be easy to control. It has been suggested (in some articles on the subject) that it is as simple as "having a good relationship with plant management," or issuing an edict banning non-IT development. But in my role as Supply Chain IT leader I had both and the rogue development did not stop. I've also heard of the 'hug 'em' philosophy, whereby the IT organization recognizes and partners with the rogue development shops. That too has its share of issues. Principally, the logic fails because the reason that the rogue shops exist in the first place is because they run outside the IT organization. For the rogues to become part of IT would be to cease to exist. Besides, the rogue shops spring up from some real need. The need will still exist. If the IT Leader can hug one group they will either eventually break ranks (secretly or publicly) or a new rogue shop will spring up to replace them. It is just the natural evolution of things.

If the rogue shops are so evil, why do they continue to exist? And what is this "real need?" These questions basically ask -- why is rogue development so common in manufacturing? Here is why.

Manufacturing management encourages it. Manufacturing management holds the perception that development done within the IT organization is slower and more expensive. Hey, let's be honest, that perception is true. Of course IT project management is slower and more expensive, because things done right take longer than band-aid approaches. At the end of the day, Manufacturing management does not want to wait for the right application and they don't want to pay for features that they perceive as "baggage," such as security, single sign-on, consistency with a technology stack and virus protection.
Shop floor applications are inherently customizable. Manufacturing aplications need to be customizable so that they can change as the factory changes. This fact fuels growth of power user positions that are half manufacturing and half IT (but allegiance is to Manufacturing). Common shop floor systems (such as Cimplicity) have sandbox development environments where people can try different things. From Cimplicity power user to VB programmer is a very short hop. So when Joe Poweruser finds a need that is beyond the reach of Cimplicity, he picks up a book on programming in MS Access and a rogue programmer is born.
Rogue programmers are loved by the factory floor. These guys are the Jeff Spicoli's of the world. They can fix anything with their "ultimate set of tools". They mock the IT organization and win points by implementing spot solutions that ease pain instantly. They wear their disdain for IT like a badge of honor. Word of their great deeds makes it to manufacturing management and they earn bonuses and trust.
Rogue applications work. That's right -- they work. Contrary to what the IT trade rags indicate, the case for stopping rogue development is not made by the piles of cases where a rogue application grew too large for its environment and started gobbling up resources and costing money. Far more often, the solution works. In my three years supporting Supply Chain, I would have made a big deal out of a rogue program that flopped, but none really did. At worst, they failed to deliver on benefits promised, but since the perception is that this development is "free" most people just move on and say it didn't cost them anything anyhow. Or they continue to tweak the application in perpetuity until it serves some small need. The rest of the code and functionality is just left dangling.
Rogue programs are pinpoint accurate. This is simply because the solution provider (rogue) either once worked on the shop floor, still does work on the shop floor, or stood next to someone on the shop floor as the code was being written. Fact is, they tend to hit their mark. Because they are point solutions, they are seldom reviewed for exhaustiveness. In other words, once the specific discomfort that gave birth to the application is eased, no one ever goes back to consider what was left out of scope and what benefits were left on the table.
Manufacturing management can be short-sighted. The metrics that drive their behaviors are all relevant by days and months more than quarters and years. A point solution that can quickly improve the yield of a single process is valued more than an enterprise system that can mean a wholesale change to the way they do business. It is difficult for a manufacturing manager to find time thinking about the long-term strategy because all their money and time is spent trying to make the current period's commitments.

From the bullet list above, it may seem that I am actually a proponent of rogue programming. Then again, the term "rogue" is seldom used as a compliment in business so the other side of the coin is certainly due for discussion.

So let's get to it.

Rogue software is very difficult to support. Even if it were written in a language in the technology stack, code is usually not documented. There are usually no test plans. Debugging the application, when needed, falls on the shoulders of the sole developer of the application. And that person goes about debugging from memory. If the author is gone or in a new role, the application could be down for extended periods while trying to figure out the code.
Rogue software breeds more rogue software. Because rogue software is difficult to interface with, the cost to implement quality, robust solutions goes up accordingly. The price margin between custom and robust software seems to widen. And manufacturing management is further seduced by the lure of cheap solutions, unaware that they are rowing the boat in the wrong direction. Once custom software becomes pervasive, it becomes almost impossible for a well-intended manufacturing leader to hold up his or her hand and say "enough!" because the cost to remove it is now astronomical.
IT cannot help with outages. The IT leader will be held accountable for the systems that operate the factory floor, but he/she will not have the visibility or the right skill set on the team to support them. So when something breaks and the manufacturing VP calls his IT Leader, the IT Leader may have to answer with "it depends" or "I don't own that system." This is both awkward and dangerous.
Security. In this context, security includes single sign on, role-based access, password enforcement and virus protection. To shop floor personnel and many Supply Chain managers, these concerns are inconsequential, while the coding required to implement is often very complex. As a result, these functions are usually ignored in custom software. It is wasted effort trying to make the case for eliminating rogue software based on these concerns because no one outside of IT will care. But over the long run, the existence of applications that have no security and access control puts the business more and more at risk to intrusion and virus attack. No single application will sink the boat, most likely, but allowing one application to ignore this important functionality is akin to the camel's nose being allowed inside the tent.
Rogue applications get in the way. Like no single cancerous cell will kill a person, no single keystroke of rogue development will kill a business. In fact, rogue development often delivers short-term benefit to businesses with specific business problems. The problem is not just what rogue development is. The problem is also what it isn't, and how it takes up valuable space. The cancer analogy holds -- cancerous cells displace healthy cells. They take up space where healthy cells are supposed to be. Even if rogue development were truly free (which it isn't), the space becomes occupied and the difficulty of getting to the "right answer" goes up enormously.

Suggestions for control and risk mitigation

The following suggestions are based on my three years of experience supporting the supply chain in GE Appliances, Lighting and Industrial Systems. Rather than suggesting that having a great relationship with Manufacturing management is the panacea, I suggest that it is a prerequisite to additional actions that should be taken. There is no single cure-all, but there are some measures that had an impact for me.

The IT organization supporting manufacturing must:

Have standard procedures. IT cannot be hypocritical, condemning rogue programming while doing their own development in 'willy nilly' fashion. IT should celebrate successful implementations that followed the standard procedure. Get Supply Chain leadership to endorse, or at a minimum recognize, the standard procedures. When rogue programming is discovered, the first step is to channel the activity towards the standard process. This brings the effort to the surface and forces it to be recognized.
Have a technology stack. Like the above, it is tough to crack down on certain types of hardware when it has not been stated what the preferred case is and why.
Put the sonar on HIGH. Police activity on the network. Get sourcing involved to track purchase orders and credit card expenditures on IT-related items. Stop purchases that are not approved. While stopping rogue programming is difficult because it goes underground so easily, most businesses willingly give the IT organization approval authority on all computing purchases so that IT can coordinate the buy. IT must not be afraid to use this authority.
Get good at what IT does. IT bears the responsibility of not taking away the quick fix and offering nothing better. The added baggage of a robust project management process will never be as fast as rogue works. But IT must close the gap as much as possible. IT must resource smartly to keep costs down. In short, IT must work their tails off so the team and processes are not such easy targets.
HAVE A ROADMAP! Sounds easy, but many IT leaders don't do it. They stay mired in the day-to-day (much like their manufacturing clients do) and never stop to think about what they want to look like in a couple of years. The plan needs to be a physical document, at a minimum containing one systems map for the entire supply chain. Find some of the more forward-thinking members of Supply Chain and decide what systems are necessary to change the game. More than likely it will be a simpler, more robust, more supportable environment than what exists today. More than likely it will have a handful of systems (no more than four), including: Data collection (PLC Layer), Data Aggregation and Quality Layer, MRP, and ERP. This systems map, when complete will be very attractive. It will be something that Manufacturing will WANT. The IT leader should complete it, print it in poster size, and put it on a wall somewhere.

The importance of this plan is not just that it becomes an active project roadmap, per se. Rather, it becomes a means to say, "No!" to things that are clearly the wrong thing to do. If a client identifies a need, the IT leader can find the void on the systems map and see the plan to address the need long term. The probability that he will suggest a rogue solution drops dramatically because he can see the right way to fill the space. And he can see (especially if the IT leader demonstrates) how a rogue system will plant roots and hog the space, making it expensive and time consuming to remove when the robust system is ready to be implemented. The manufacturing manager will see how his rogue solution presents a roadblock to the systems environment of his/her dreams. The IT leader can use the systems map to prevent and diffuse rogue solutions. It works.

The systems map can also serve in the budgeting process. If the elegant solution is three years in the making, the final answer can be peeled back a year at a time to see what funding is needed in the current year, next year, etc.

As long as there are functionality gaps, creative minds and untapped resources, there will be rogue development. In small doses, it keeps the IT organization honest and gives IT a window into the needs on the shop floor. When manufacturing and IT leadership work in tandem, the need for rogue manufacturing is mitigated -- and the business will ultimately reap the rewards of that collaboration

William "Liam" Durbin is VP, CIO, Six Sigma, GE Fanuc Automation.