Deja Vu All Over Again

Thus, history repeats itself. MIS, as it was called a generation ago, thumbed its nose at all of the new-fangled personal computers that started mysteriously appearing on desktops throughout the corporate world. Those new microcomputers were there for one specific reason: business was tired of white-coated, non-responsive arrogance from its centralized computer department, so used the new systems to get things done.

(From Cloud Computing Journal, “Cloud:Biz & IT Not on the Same Page”)

A new study conducted by Forrester Consulting for BMC Software has received a lot of attention this week. Although the full results will not be available until the end of April, the findings published in a March 21 press release indicate that business leaders are using cloud computing as a way to bypass IT constraints. Just as the proliferation of smart phones and tablets has generated a substantial guerilla movement to force support of those devices in spite of IT opposition, cloud computing seems poised to join in. Unfortunately, as the quote above points out, this is reminiscent of the days when PCs began their invasion of the corporate space and we all know how that turned out.

Computer Weekly’s “Executives using public cloud to dodge IT rules” highlighted the initial results of the study:

• Three out of four CIOs believe that business executives view cloud computing as a means to circumvent IT control.
• Although only 36% of businesses have policies allowing for use of public cloud resources for mission critical work, 58% are doing so, with a planned increase to 79% within the next two years.
• IT executives see the unmanaged use of cloud services as a factor adding complexity to their operations.

Earlier this month, Steve Ranger writing for TechRepublic noted that even IT staff can be the source of covert cloud computing. Some are using cloud resources to meet business needs and submitting expense reports for the costs.

Shadow IT is a fact of life. Frequently, it can be a valuable source of innovation. However, when major business functions drop off the grid, it should be seen as a red flag. When your customers are going out of their way to avoid you, you have two problems: first is the loss of trust on the part of the business unit that sees you as an impediment and second is the potential loss of support from upper management. Unless there is a legal or regulatory issue, circumventing IT rules will likely get tacit support if the unit is making money. Complaints from IT may even convert that unofficial tolerance into official approval.

Like a turtle in the middle of a busy road, CIOs face a dilemma with no easy answers. The CIO must answer to both individual business units and the enterprise as a whole. Frequently, the needs of one group may not align perfectly with the rest of the company. Too little control allows costs to spiral out of control while too much stifles innovation.

Collaboration, making the business a partner in meeting their needs, will be far more likely to work than attempting to dictate and veto. Fostering innovative efforts puts IT in a better position to mitigate risks than a prohibition policy which encourages hiding of projects until it’s too late for IT to stop them.

A little over thirty years ago, rogues began sneaking non-standard IT solutions into enterprises to meet needs that IT did not. It’s been a rocky road, but we all know how that turned out.

Knowing Where You Stand

“What is it about the Apple Store that’s just so great?” is the question that opens the UDig blog post “How to Staff Your Organization Like an Apple Store”, by Kyle Lagunas. In the article, he observes:

Apple is running a seriously smooth operation in their retail stores. Each employee has a distinct role to play, understands that role, and does his/her part to deliver the level of service we’ve come to expect from this powerful brand. All of this requires serious alignment of brand, business goals and people process.

Although Kyle is discussing staffing, the characteristics of Apple’s operation that he highlights are extremely relevant to the development process. Understanding your role within a process where the objectives of all roles are aligned to produce customer satisfaction is critical to success. Consider the following:

Once upon a time, there was a development organization that had unhappy customers. Those users would spend countless hours defining solutions to their needs. The tomes describing these solutions were then duly turned over to the organization. Product managers, responsible for serving as the liaison between the business and the technical staff for a given part of the application, would then review the voluminous output of the business users. After filing their documents in the trash can, the Product managers would produce their own detailed solution to what they perceived the customers’ problems to be. These documents would then be deposited on the desk of a developer working on the application module controlled by that Product manager. The developers, finding the requirements document to be technically deficient, would then implement their own interpretation of what the users wanted. The code would then be deposited in the laps of testers. Those testers, unconstrained by the visions of the users, product managers, or developers, would then proceed to enter bug tickets based on how they thought the application should work. At this point, an iterative cycle would occur: developers would “fix” bugs according to the testers’ tickets, subject to the developers’ adjustments; tickets that made it past testing would then be sent back by the project manager to be reworked according to the product manager’s vision; the reworked tickets would then be rejected by the testers, returning the cycle to the beginning. At some point, enough of the players would give up and the release (perhaps “escape” would be a better term for it) would make its way into the hands of the end users. The customers, having received a buggy product, late and with little resemblance to what they had asked for, were dissatisfied.

Amazingly enough, the story had a happy ending. A structured, iterative process was adopted. The end users worked with analysts (the re-titled product managers) to turn business needs into coherent requirements, analysts worked with architects to understand the architectural implications of changes and enhancements, architects worked with developers to design the technical expression of the requirements, and testers ensured conformance to the requirements. Issues were bubbled up to the appropriate level: analyst and users dealt with the functional and quality questions; developers and architects handled the purely technical ones. Communication and collaboration were promoted. Usurping another’s role was prohibited.

Although they did not feature in the story above, the DBA and infrastructure roles were included as well. Pairing responsibility with skill is as important in these areas as in those noted above. A key role that was introduced, invisible to end users, but vital nonetheless, was that of the release management specialist. I’ve previously detailed the importance of release management in the post “Do you have releases or escapes?”.

What did not change is as important as what did. The same people involved in the old dysfunctional process defined and refined the new process. What changed was their understanding of their roles and how those roles were to come together to produce quality software. Having each role, from user to tester, acting as partners in the process made it work. Each role had both authority and responsibility over its own area while deferring to the other roles in their areas. Far from being constrained, each role was empowered by this as it allowed them to concentrate on their own area of expertise, all in the pursuit of consistently producing software that met the needs of its users.

What’s driving your architecture?

In a recent post titled “Architectural Drivers”, Davy Brion listed the factors that influence his architectural choices. Additionally, he makes a number of important observations, such as the wisdom of avoiding unnecessary complexity and the fact that there is no absolute “right” or “wrong”, only trade-offs. His efforts inspired me to put together my own list of drivers, several of which are the same or at least related to his. Additionally, I include some discussion of how they affect the architecture of a given system or solution.

Functionality

Functionality being the first consideration should not come as a surprise since the name of the site is “Form Follows Function”. Are the user interactions task-oriented processes, basic CRUD operations, or a combination of both? Are those interactions real-time, asynchronous, or a mixture? Is the system stand-alone, server-based, peer to peer, or a hybrid? How much, if any, of the desired functionality is available from third party providers?

The requirements that describe the system’s behavior should be a major determinant of the solution’s application architecture, data architecture, and infrastructure architecture. It is important to look at the requirements in a holistic manner. Although those of the current iteration of the current release will be foremost in mind, some thought should be given to those that will be coming later to avoid painting yourself into a corner. It is also important to understand that the functional requirements will also influence many quality of service requirements. For example, the nature of the processes will affect the authorization scheme, the ability to process asynchronously will affect performance and scalability, etc.

Data Profile

The characteristics of the data to be managed will be a major driver of the architecture. Does the system deal with structured data, unstructured data, or a mixture? If the data is structured, does the functionality infer the need for a relational database or one of the NoSQL variants? Is the data sensitive (either wholly or partly), requiring extra security? How many records/documents will the system handle per day? How large are the units of data? Will the volume of data grow linearly or exponentially? Is there a need for archiving or near-line storage? Does the data lend itself to caching, either in whole or in part? Will there be a need to import data from a prior version of the system or from other systems to be replaced by this one?

This driver will have a major impact on the data architecture, infrastructure architecture and application architecture. Additionally, the characteristics of the data will affect a wide range of non-functional requirements: availability, backup requirements, certification, compliance, disaster recovery requirements, performance, response time, scalability, and security.

Audience

Who uses/will use a system significantly influences its architecture. Is the system intended for users internal to an organization, its customers, vendors, or some combination of these? Does the system interact directly with users, provide services to other applications, or both? Does the system have to support multiple platforms, either as a host or client (e.g. mobile)? Is accessibility an issue?

The answers to these questions will particularly impact quality of service concerns such as interoperability, portability, security, and usability. Likewise, this aspect is a significant driver of the infrastructure architecture.

Usage Characteristics

Question around how heavily the system will be used are as significant as the issue of who will be using it. Numbers of concurrent users and operation counts will have considerable impact. Another consideration will be the pattern of that usage: business hours versus around the clock. Will it be steady or spiky? If there are spikes, will they occur on a daily, weekly or monthly basis?

This driver will primarily affect quality of service requirements such as performance, response time, and scalability. If the usage pattern is twenty-four hours a day or nearly so, this will also affect maintenance and backup considerations too. The impact on infrastructure architecture is obvious, but it will also drive application and data architecture as well.

Business Priority

The nature of the system in relation to the enterprise, ranging from mission critical to minor, is a key architectural driver. Gartner’s application classes, as described in my previous post “Application Lifecycle Management Revisited”, may be useful in classifying the business priority of a system as well. Time to market can also be a component of this driver, particularly for systems of innovation. The business owner’s tolerance for risk will also fall under this category, with greater tolerance leading to less emphasis on the associated quality of service requirements. A system of innovation with a business owner having a high tolerance for risk will be a good candidate for quickly accumulating technical debt.

Business priority will drive application architecture, data architecture, and infrastructure architecture. Non-functional requirements associated with this driver include availability, backup requirements, disaster recovery requirements, maintainability, performance, response time, scalability, security, and supportability.

Regulatory and Legal Obligations

Regulatory and legal compliance can have as wide ranging an impact on application, data and infrastructure architecture as functionality and business priority. The salient feature of this driver, however, is that there will be little flexibility around it. If a regulation applies, then the solution complies (or else). This is not to say that there will be no gray areas with this driver, just that they will generally be questions of whether the system falls under the jurisdiction of a particular law or regulation rather than whether or not to comply if it does. HIPAA and Sarbanes-Oxley (SOX) are but two examples of the myriad potential legal obligations that could apply. Other examples of legal issues include licensing compliance (for both commercial and Open Source components and code), copyright, trademark, and patent issues.

Common quality of service requirements associated with this driver include accessibility, auditability, certification, compliance, and security. This is not to say, however, that others could not be impacted by legal and regulatory obligations.

Architectural Standards

If an enterprise has formal architectural standards, obviously they will affect application, infrastructure, and data architecture, including the full range of non-functional requirements. Absence of formal standards, however, should not be mistaken for absence of constraint. The existing environment (e.g. hardware and operating systems, development team background, support team background, etc.) will act as this type of driver in the absence of formal standards or where such formal standards are ignored.

Non-functional requirements associated with this driver will typically include accessibility, extensibility, maintainability, security, supportability, testability, and usability.

Audit Requirements

Arguably, audit requirements could have been lumped under functionality, but I listed it separately because of the variety of sources that these requirements can come from. Laws, regulations, the organization’s internal auditors, and even the business users may require tracking system activity. Audit information can also be valuable in determining usage volume and patterns, making operations another potential stakeholder for this driver.

Audit requirements will primarily drive application and data architecture. Obviously, auditability is the primary non-functional requirement affected, but performance, response time, scalability, security, and supportability can all be impacted as well.

Reporting Considerations

Like audit requirements, reporting considerations could be considered a sub-topic of functionality. I list it separately, however, because this driver can be both internal and external to the solution. Reporting that is integral to the solution can affect application, data, and infrastructure architecture (e.g. maintaining a dedicated reporting database to offload reports traffic is a common tactic). Enterprise reporting considerations can likewise impact the architecture of the solution, whether because of the need to support ETL jobs or to participate in master data management, as a supplier, consumer, or both.

This driver can impact quality of service requirements such as auditability (in situations where even reads must be logged), availability, backup requirements, performance, response time, scalability, and security.

Dependencies and Integrations

In “Managing Dependencies”, I outlined a number of ways in which the application architecture is affected by dependency management practices (see “Dependency Management – Anti-Patterns as Art from The Daily WTF” for some humorous, but true examples of how poor dependency management can affect architecture). Likewise, “Using Extension Methods for Message Transformation” and “Strict Versioning for Services – Applying the Open/Closed Principle” cover some of the effects of integrations (using services) on application architecture. Integrations and dependencies may also affect infrastructure (e.g. when connections come from systems external to your network or when using file drops) and data architecture (e.g. when using shared databases).

Non-functional requirements potentially impacted by dependencies and integrations include availability, disaster recovery requirements, interoperability, performance, response time, scalability, security, and supportability. Extensibility could also be potentially affected by this driver. A key consideration in the “Using Extension Methods for Message Transformation” and “Strict Versioning for Services – Applying the Open/Closed Principle” posts is structuring the system to avoid situations where two or more teams must conduct parallel development and releases to avoid breaking integrations.

Cost Constraints

I list this driver separate from business priority because it’s a fact of life that funds may be limited, no matter how critical the system may be. Salaries for development and operations staff; licenses for operating systems, database servers, and other dependencies; servers, storage appliances, network hardware and capacity all cost money (and this is far from a comprehensive list). These costs must be accounted for to properly understand their impact on the solution as a whole given that all aspects of the architecture are vulnerable to this driver.

Initial State

While this is the last driver in my list, it is by no means the least. New development may be subject to fewer constraints than changes to existing systems, but it isn’t without issues. Existing architecture can be both a limit to your freedom and also a base to build on.

In my opinion, these twelve drivers provide a framework for making architectural decisions for technology solutions, whether built, bought, or rented. It is important not to infer the importance of any one driver from its position in the list; that ranking will be a function of the priorities of the business owner(s) plus the requirements of the enterprise as a whole. Establishing the priorities will assist in balancing the conflict between favoring the simplest possible solution and providing sufficient flexibility to grow and change as time goes on. Additionally, these drivers provide a framework for ensuring that decisions are driven by need and not fads or dogma.