The place of architects in a business organization

[Author : Kinshuk Adhikary - software plumber. LinkedIn : http://in.linkedin.com/in/kinshuka Blog : http://me-plumber.blogspot.com/ Email: kinshuk-in@yahoo.com ]

Introduction : Due to a profusion of roles and titles in the software world, there seems to be some confusion about the architect, and the place and roles of such people in businesses. This write-up is mainly for business people who may appreciate a perspective that is a mix of the technical and the management aspects of this problem. Disclaimer : The reader must clearly understand that everything written below could be inapplicable, that everyone's experience is different, and a lot depends on the type and nature of the system and the business. My own experience has been in business products and applications, development, launch, adoption and modification cycles. You will need to form your own opinion on the correctness of everything said below. Understanding architecture : The simplest definition that I can come up with, cobbling several smart people's views and misquoting them, is - "Architecture is an abstract description of the system in its various working phases and in the various stages of its life-cycle". In the largest sense, the "system" is the entire organization, seen as an information machine. Quite naturally, the architectural responsibility becomes not individual small systems, but policies, standards and quality aspects that apply to every system in the organization. The most important external responsibility of the architect is communication. Without clear communication, both to technical and non-technical teams, the inward responsibilities i.e. the abstraction and the description , lose their meaning. An architect needs to constantly distance oneself from the actual "concrete" implementation of the system. Too much contact with concreteness disturbs the abstraction that is extremely needed. At the same time, at a private level not always talked about, the architect needs to be extremely hands-on in certain "pilot" areas. Too much distance from actual implementation makes the architect quite unable to reach out to the real implementers who are (a) the design folks (b) the coding folks. Another disclaimer : As I said earlier, the term "architect" has been extremely misused and misapplied, especially in the last 5-6 years. While writing this, the architects I have in my mind are those I have met and worked with "more that six years ago". Phrases like "an architect should not do any actual implementation" can get extremely distorted if your picture of the architect is not the same as mine.

The kind of architects I am talking about were (a) very highly technical, up to date and extremely knowledgeable in all aspects of their work (b) very busy all the time, far more "work" than they could handle (c) visionaries, almost mystical - I do not know how to express this. Very often, the systems they created were nearly perfect, and brought to the organization and stakeholders extreme successes and large amounts of money. The committee, the council, the guild : It is extremely unbalanced to have a single architect in charge of a large mission-critical system. Because of high costs, organizations often think a single architect is all that is needed. But a single architect could be a disaster for the system or the product. Everyone in the organization must get used to talking in terms of "the council of architects", a group, a committee. All actions must be attributed to this group. There usually is a Chief Architect, a key spokesman who enjoys the confidence of top managements. But architectural decisions are taken by "the group", during internal meetings, in consultation. Why is it unbalanced to have a lone architect ? Because of too much uncertainty, too vast a field, the need to discuss and thresh out between peers. And a hundred other reasons, I will write some of them below. - Profile of the individual : Most architects I have known have been a mixture of immense egos and extreme humility. It is part of the training and the drive that is needed to keep up to date and on top of everyone else. In a group of peers, a moderating effect sets in, and many ego-driven unidirectional mistakes are avoided. As every architect is aware, the number of things one does not know exceeds the known by factors of 10s, or 100s. - Peer understanding : This has a spurring effect on an architect's abilities, it is all about bouncing things with others who resonate on the same frequency, are of a similar background, although of slightly different specializations. - Policy and guidelines : Beyond the immediate system, the work is a lot about policy. One cannot set a policy without a very holistic understanding, almost impossible for one person to do. When does an architect work in a silo ? : It is also possible that an organization has many architects each working in a silo. This is typically the case when the org has many business units more or less independent, and each catering to a different class of customers and user groups and lines of business. However, sooner or later, as the "enterprise" view of the organization gains priority in the minds of top management, a council does get formed, all systems in the organization follow some core principles, initially very very few. The lesser the better : An architect's work obeys this more often than not. Too much detailing is bad. Too descriptive is bad. It is more important to "hide" the complexity behind abstract black boxes. (purely my own arbit statement) - Good architecture seems to be very cognizant of Godel's incompleteness. For a system to be both complete and consistent over time, its

complexity has to be achieved rather slowly, the relations between its parts and the parts themselves cannot be written in stone too quickly. At the same time, the complexity is all there, for whoever wants or can dig deep enough. It exists as a mass of "maybes", not as "musts". The best things are incomplete "shells", into which the other participants, the design folks and the developers, can put in their best notions. Important to appreciate - 3-year paradigm shifts, 6-year eras : Good architectural styles (the decision driving large system efforts) are those that can reasonably withstand the flux of technology. Technology changes very fast (ugh!!) - a paradigm change happens every 3 years, and an era changes every 6 years (era == my own term, two paradigm shifts, if you are not in touch, you are in oblivion and it is too difficult to catch up). Lets say that in 2000, JDBC was everyone's staple. In 3 years ORM changed how everyone looks at data persistence. Now it is the age of Hadoop in the cloud. So which is your particular cave ? The problem is not that old things will not work, but that applications, systems built using them will get slowly isolated. When several eras have gone by, the great old stuff will now become heavy weights making it unable to do the nimble things that everyone around will be doing. That is another reason architect's are both enthusiastic as well as careful of technology. And why individual architect's can be too prone to "push" only those things they have had personal experience in, and why a council becomes necessary. It is of course needless to state that an evolutionary progression of knowledge, right up to the here-and-now, is a necessary toolkit for every architect, excluding nothing by personal preferences, including as much as is humanly possible. [P.S : An architect of my acquaintance had to take a 15 day vacation, just to study ORM/JPA/EJB3 in which he was feeling out of depth. He was about 50 years old, and was cursing the day he took up this profession, and was warning other young techie enthusiasts who all wanted to be architects]. Sir Christopher Wren : Was a good mathematician, had extremely good mechanical skills, was a good water color painter (or is it sketches, I don't remember, look up Wikipedia), and enjoyed the confidence of the patrons who wrote the cheques. All in all, an ideal architectural scenario. So who is this "design" guy ? : Enter the next participant in the process of creation. This is the person the architect finds it easiest to talk to (other than other architects, of course). They bridge the gap between the abstract and the lab experiment. The design person is a terrible realist, who deals with artifacts of all sorts, UML and what not, and the latest APIs. The whole focus is on solving a problem in the best available way within the description the architect has provided. Sometimes both are the same guy.

Architects are often wrong. That is obvious, if you are not about to make mistakes - you are probably doing nothing of any value to anyone. For example, EJB1 was most probably a mistake. But EJB3 is most likely not a mistake. No one really knows 100%. (Side note : Personally, I get very worried when a Project Manager says - oh, the project will take 6 months and 17 days, and there are only six risks, and here is the mitigation. Does it really work that way ? Does this guy know anything at all ? These definitive numbers are the warning signs of complete ignorance about complexity. If the project does get done in 6 months and 17 days, then you can be quite sure it is not the same project that was initially discussed, but a much diluted, much changed version, whose internals only the developers know which they guided as they wished :-) It is the design people who go a little more concrete, do more pilots and initial studies, and more or less assure that things are most likely going to work out when the developers write the final code. More about design folks and developers in a later discussion. It is sufficient to say here that design and development are in their own rights equally important to the effort. The ridiculous hierarchies that emerge in organizations (developers are way below, designers are somehow "above", and architects are the "cream de la creme" - these things one wishes did not exist). For the business : Why is an architectural council even needed ? The basic need is "direction". There is a huge technology flux out there, you need some ways to withstand it. Or the best point of entry into it, given the stage of evolution the organization is in. Do you really need SOA ? Why should you go into the cloud ? Cutting across the jungle of jargon, you need to evaluate the suitability of various alternatives that best suits the business as it is now and as it will be in 3 years and 6 years. If you are in the client-server era, what is the best way to jump in ? [I first learned about the cloud nearly 3 years ago, from an architect I respected a lot. He himself did not know enough about it back then, I guess, but did his best to explain.] Of course, solving the immediate problems, conceptualizing a solution, overcoming the usual architectural constraints of performance, scalability, extensibility, achieving standards and quality, consolidating across systems - these are other (and sometimes minor) needs that architects fulfill. Most applicable when the organization is seen as an information machine, whose competitiveness depends to quite an extent on the systems that process information. But then today, such a view is applicable to nearly all organizations. There are still many who believe that systems purchased from vendors without an internal effort will solve all problems, but the jury is still out on this one. Most are embracing data analytics, or soon will. "The architect" was interestingly described in Matrix-3, but of course business needs some help, maybe like this write-up, to distinguish between a movie and the real world. Otherwise it is easy to fall into the trap that the concrete is the real. It is not so. It is the math that is the real. Ever heard of "high frequency stock trading" ?