As business continuously evolves, so do the supporting pillars of business, including technology. It is one thing to create a system capable of enabling that evolution but quite another to have a software architecting process that can maintain a dynamic system over time. This article discusses the symptoms of business evolution and provides an overview of an architecting process that allows it.
First, we need to understand a few concepts. The first one, of course, is software architecture. Architecture is often confused with either the design of a system or the technological choices used to build it. A system is a set of components that interact with each other following a set of rules and policies. Architecture is the structure of those components, their properties and the rules controlling the interaction between the parts.
Architecture is the actual structure, the current system, and not its design on paper. Architecting is the process of designing, implementing and maintaining that architecture that is built to deliver a business value. Every system has architecture, even if it is not actually described on paper.
Ralph Stacey coined the term “shadow system”, which is the “set of interactions among members of a legitimate organizational system that fall outside that legitimate organizational system.” We borrow that “shadow” qualification to use the “shadow architecture” term to describe the architecture that is real but unofficial.
Given the architecture is the current structure of the system, it may be different from what the official documentation says it is. It is not because the original implementation was wrong or didn’t match the design but, rather, because the actual architecture has changed, becoming something different that is not yet documented. Let’s analyze the symptoms of shadow architecture.
Every component, screen, report and process in the system is designed to fulfill a business need and deliver value. If the system does not deliver for some reason, then it is a failure. Not delivering means:
- The system is not producing the expected value.
- The system may produce some value but at a high cost.
- The system may produce some value but the user is not using it.
- The system may produce some value but not how the user can use it.
If the system does not deliver, then how does the user fulfill the task and the system itself deliver? Ultimately, the job must be performed, forcing the user to fill the gap of what is possible with the system and what is needed.
The user will, unwanted, wind up finding small spreadsheets at every computer that will deal with the data — processing it the way the user intends. The processes may have different steps, and there will be data channels that are not those of the system. An example is flash drives exchanged between multiple computers. In short, the user will have a different set of components with interactions and rules that create a shadow architecture.
Studies about the shadow architecture phenomenon have identified three possible causes.
- Unfulfilled business needs. The main cause of shadow architecture is that the business value is not being delivered. The system does not provide what is needed to complete the job. Or, if it does, there is a better way of providing it than the one implemented.
- Forced domain anti-pattern. This is a special anti-pattern that is very common: the design of the system uses the wrong domain or metaphor and thus the solution or the process does not fit naturally into the user’s experience. The most common is when the solution is built in the solution space, meaning it is using the information technology metaphors and processes. The user needs to learn IT concepts to use the system (i.e., databases, records, backups, etc.). The outcome will be a defective communication channel because of using the wrong concepts for the job.
- Invisibilized talent. Here we have non-participating stakeholders that will not give feedback or are not taken into account for the design of the solution. One either talks to the tactical level that tells how the system should work at operational level or talks to the operational level asking what the strategic level needs. In the end, the system may be delivering operational IT, non-tactic or non-strategic solutions.
This reality will result in significant negative effects on the value delivered — including the unofficial shadow architecture — with additional cost in control and support. There may even be competing solutions! Additionally, there will be a dysfunctional domain where users are forced to learn about IT, using unnatural methods.
The evolution game
Is shadow architecture a bad thing? As a consequence of the problems described above, it is. Fortunately, it also gives those involved rich material that they can utilize to fix and improve the solution.
When identifying a shadow architecture, there is a real structure that is an evolutionary step forward from what was originally designed. Every change, every new functionality and every component not used presents an opportunity for improvement. But the overall strategy is to allow evolution in a controlled manner. Here is a list of things to do to maintain that control:
- Keep track of information, processes and executions.
- Keep an eye on symptoms; they will direct you to the variations.
- Document and learn from shadows.
- Use techniques like domain nurturing to create domain awareness and interoperability looking forward to a business value-delivering system.
- Use contextual interviews for domain design and to analyze variations.
- Absorb and combine shadows. Keep them under control; make them official. Make stakeholders part of the process.
Put an “evolution cycle” in place by following these simple steps:
- Establish an architecture evaluation. This process will check on architecture, goals and actual value delivered. It should produce a report of what is out of place.
- Identify combinations (different processes that are combined by the user) and variations (processes that are executed somewhat differently from what the system establishes).
- Identify a) replacements (components that are not used or whose delivered products are discarded completely or in part, replaced by a user-created component such as a spreadsheet) and b) deletions (components nobody uses and whose value is irrelevant or useless for the business goals).
- Add newcomers and new components identified by current needs.
- Evolve: implement the changes. This will change the current working software architecture and improve it, in the process making it the new official solution.
This cycle should be repeated every six months or so, resulting in new changes that emerge based on the needs and the results of the current period’s architecture.
There is a lot more to how systems change than just the concepts. This article covers just the tip of the proverbial iceberg.
 R.D. Stacey, Complxity and Creativity in Organizations, San Francisco: Berret-Koehler Publishers, 1996.
 S. Behrens and W. Sedera, “Why do Shadow Systems Exist after an ERP Implementation? Lessons from a Case Study,” in PACIS 2004 Proceedings, 2004.
William Martinez, a software architect and R&D manager at the nearshore software engineering services company, Avantica Technologies, works with venture-backed startups, the Software 500 and industry in development, testing, high-end productivity evaluations, SOA technologies and mobile platforms. He is a REST evangelist and is currently writing a book on the subject from the architecture perspective. He also leads the IASA Costa Rica Chapter (International Association of Software Architects) and is a professor at the University of Costa Rica. He has been a technical reviewer of popular books including Fast SOA and Rest in Practice, and co-wrote the Planning and Evaluation of IT Projects book for UNED.