Are you addicted to hand coding? Are you aware that hand-coded custom point-to-point integrations are bad for you and in the long term always results in an integration hairball that is overly complex, consumes way too much of your maintenance budget, and is temperamental in operation? And the production surprises may be the worst. These are the data dependencies no one anticipated that jump up to impact your customers or suppliers to the point where it becomes a business crisis. In short, it knocks you off your feet and you can’t function in order to get your normal work done.
You know hand coding is bad for you, but you keep doing it. Like an addiction or compulsive behavior, you do it not because you want to behave that way, but because you feel you have to.
The good news? By following a twelve-step program to recovery, software vendors and enterprises can adopt repeatable practices that will put the industry on track for an integration revolution.
Untangling the integration “hairball”
As an IT manager your business users are breathing down your neck to deliver a new capability. You provide an estimate of the time and cost to do it right and the business sponsor cuts the budget in half and says get it done, so you turn to your project team and give them the leeway to cut a few corners.
As a software engineer the project manager comes to you and says, “What is the quickest way to connect these two systems?” Your knee-jerk reaction is to reach for your tool-of-choice, which might be PL/SQL, Talend, Java, or even COBOL because it is your comfort zone. You know it’s not the right tool for the job, but it’s your comfort zone and you can make it fit and build the integration point fast. Besides, you think to yourself, it really won’t be a maintenance issue in the future since my code is such high quality and the requirements will never change.
We all know what the result of this behavior is. 100 percent of the time organizations end up with an integration hairball. There are no counterexamples when you are addicted to integration by exception. Integration by exception is an approach to defining solutions by drawing clear boundaries around a narrowly defined problem and excluding external factors. Environmental or contextual issues outside of the project boundaries are not considered and hence the solution is coherent within the scope of the project but may be less than ideal in the broader context.
There is an alternative however. There are many examples of integration teams and entire IT organizations that have broken the pattern and are in recovery. The alternative approach is to start with a big-picture context and use an Integration Competency Center (ICC) as a permanent service-oriented cross-functional team to target a specific domain to focus a transformation effort on; the detailed integration solution is still focused strictly on the needs of each project, but always within a broader enterprise or data governance context.
But, hand coding is an addiction that can be hard to break. One of the common manifestations of the addiction is the Not-Invented-Here (NIH) syndrome. NIH is an “antipattern” that describes a persistent culture that avoids using or buying into tools or methods because of their external origins. The business and market pressure to rollout new projects fast and to continually cut costs feeds the addition further. And if someone proposes a solution like a centralized ICC, the immediate reaction of the addicts is to push back because of fears that the central team will be yet another ivory tower group that adds bureaucracy and is disconnected from the real needs of the front line users.
A 12-Step program to recovery
So how then can one recover from the hand coding addiction? Wikipedia defines a 12-step program as “a set of guiding principles outlining a course of action for recovery from addiction, compulsion, or other behavioral problems.” The twelve-step program was first developed by and for people addicted to alcohol in the 1930s. Since then it has been adapted and applied to other forms of addiction such as drugs, gambling, shopping, or even workaholics.
We can generalize the 12-step program into six major stages for recovery:
- Admit that you can’t control your addiction. The first stages is to admit you have a problem and recognize that it won’t go away by itself. One way to do this is to shine a light on the problem by quantifying it; count the number of production incidents resulting from code changes; analyze how much money you spend on maintenance and what the trend has been in recent years; critically look at how many projects have been late, over budget, or even cancelled because of complexities that were poorly understood; or simply count the number of integration points between internal systems and externally with your customers and suppliers. These numbers can often serve as a powerful shock to get IT leaders on the same page and to acknowledge that something needs to change.
- Recognize a greater power that can give strength. This is a tough stage since figuring out who or what your greater power is is a personal choice that may not be obvious at first. To generalize it, your greater power is that thing that gives you the confidence to stand by your convictions in the face of opposing forces. In other words, it is your fundamental principles and beliefs and the persistence to stick with them. It might take some time to develop and internalize your principles in a way that is practical in your day to day activities. A good way to start is by talking to other recovering addicts, reading books and articles, or attending conferences or training classes about leading practices.
- Examine past errors with the help of a sponsor. Your sponsor should be an experienced IT specialist, and ideally someone who is a recovering addict him/herself. In this stage you need to critically look at your current integration infrastructure as well as corporate policies, practices, organizational structures and metrics. It is really hard to see some of the problems since learned behaviors can become so ingrained that they blind you to alternatives. The best person to help you see the light is someone who has done hand-coding themselves and has managed to break out of the pattern.
- Make amends to correct past errors. This is another tough step since it means cleaning up, or refactoring, existing integrations without necessarily adding new functionality. This effort of course requires resources—sometimes just people resources but it may also require a capital investment—but in any event no business wants to spend their limited resources for an effort that doesn’t appear to deliver incremental benefits. The solution to this conundrum is to learn how to articulate the business value of an integration strategy. It is not easy, but other recovering addicts have done it and it is so rewarding when you do.
- Learn to get things done with a new code of behavior. Doing things “the right way” doesn’t have to be slower or more expensive or bureaucratic. Achieving an approach that is better, faster, AND cheaper (i.e. no compromise) than traditional hand-coding requires a new mode of working. It is not just about what tools or technology you use—it’s also about HOW you get the work done and effectively integrate the people, policy and process elements in a sustainable fashion.
- Help others that suffer from the same addiction or compulsion. This is an important stage for an addict because you are never cured—you are always in recovery. And the best way to stay in recovery is to teach others. By teaching others it helps you to internalize the lessons from “doing it right” and thereby reinforces the underlying principles. This, in turn, gives you the strength to continue your own recovery. If you stop helping others, it is easy to relapse and fall back into your old compulsive habits.
Obviously, the preceding comparison of hand-coding to other forms of addiction is meant to prove a point—not to actually compare the severity of hand coding to substance abuse. To be sure, substance abuse issues are insidious illnesses that can have terrible consequences for individuals and families. In no way do I intend to diminish or undermine the seriousness of drug addiction.
My point is that the 12-step program has worked to change the lives of millions of people in a very positive way. Switching from hand-coding point-to-point integration practices to sustainable and repeatable integration practices also requires fundamental changes to deep-seated behaviors. There are powerful lessons that we can learn about how to change our own integration-related behaviors from people that have had to deal with much more profound changes. If you adopt the Lean integration principles and stick with them, you can be assured to not only have a bigger impact on the success of your organization, but also have more fun doing it.
John Schmidt is vice president of Global Integration Services at Informatica Corporation where he advises clients on the business potential of emerging technologies and directs the company’s Integration Competency Center Practice. Schmidt’s latest book, co-authored with David Lyle, is Lean Integration, An Integration Factory Approach to Business Agility. The book compiles the current collection of leading practices that can enable organizationns to make the change from hand-coding to sustainable integration practices. To learn more about the latest book, visit www.integrationfactory.com.