Eyeballing Performance

What does slow code look like?

Tony DaSilva recently tweeted:

“Jeez, this code sure looks slow” is hardly helpful and just not quantitative enough for effective decision-making.

Tony’s tweet reminded me of a time where I had to explain to a coder why the data access classes of a particular performance-sensitive application used a DataReader to fill POCO data transfer objects (DTOs). After all, we could have just used one line of code to fill a DataSet; that would be much faster. Patient soul that I am (or pedantic, it depends on who you ask), I took the time to demonstrate how one line of code that we write may involve a lot lines of code within the library we’re calling. In fact, filling a DataSet involves using a DataReader, thus filling DTOs from a DataSet involves iterating the results of a query twice. The size difference between the DTOs and the DataSet when serialized was a bonus lesson.

Some performance issues, notably those involving redundant work, might be detected by inspection, assuming that the redundant work is visible. In the example above, it wasn’t. Many performance issues will only become visible via profiling. More importantly, without profiling data, the relative significance of the issue can’t be determined. Saving a few microseconds in a particular section of code isn’t going to be much help if several seconds are being lost to network or database issues. This type of ad hoc response is symptomatic of more than one performance analysis anti-pattern. Performance profiling and tuning requires a holistic approach to be effective.

It’s not just about better performance, it’s about better performance in the areas that make the most difference.

Shut up, salute & soldier on?

Leadership and management are currently hot topics, with the #NoManager movement among the hottest of the hot. My detailed opinion on flat organizations/holacracy is a post for another day, but one aspect that I fully agree with is the differentiation between leadership and management. They can and should coincide, but they don’t always. Most importantly, the number of leaders should exceed the number of managers. To re-state a point I made in “Lord of the Repository”, the best managers develop their team members so that the team is never without leadership, even when the manager is away.

Tony DaSilva’s recent post on the subject, “In Defense of Hierarchy”, spoke to some benefits that derive from hierarchies. One of those benefits identified was “orderly execution of operations”, supported by the following quote:

Imagine if students argued with their teachers, workers challenged their bosses, and drivers ignored traffic cops anytime they asked them to do something they didn’t like. The world would descend into chaos in about five minutes. – Duncan J. Watts

For each of Watts’ examples, his point is, in order, wrong, possibly wrong, and correct. I have experience that speaks to all three.

My career in software development is my second career; my first was in law enforcement, serving as a Deputy in a county Sheriff’s Office. One of the positions I held during my tenure there was Assistant Director of the Training Academy. My role was split between administration, instruction, and supervision of students (limited strictly to their time in training, I didn’t hold a supervisory rank that would apply beyond that). My position was then and has always been, that any trainer or teacher that cannot tolerate respectful, appropriate challenge is unworthy of their position. A student probing and testing the information being presented was something to be celebrated in my opinion, not discouraged.

An alert went out on the radio one afternoon that there was a fire in one of the housing units of the jail. After a quick run to the location, I found that the supervisors had succeeded in getting the fire knocked down, removing the person who had started it and detailed staff to evacuate the other inmates to a smoke-free secure area. However, the remaining staff were milling about without protective gear or spare fire extinguishers should the embers flare back up. While waiting for someone with authority, I took it upon myself to direct individuals to get the equipment that was needed. Once someone arrived and assumed control, I then headed back to normal duties.

While talking about the incident later on with a co-worker, they happened to mention that they were really shocked when I ordered the Major to go retrieve an air pack and he did so (n.b. the Major in question was the third ranking person in the department and five levels higher than me in the hierarchy). Needless to say, I was just as shocked. I hadn’t been paying attention to much beyond getting the situation safely under control and the Major hadn’t objected, so I didn’t notice the real-life inversion of control, though my colleague certainly did.

That incident illustrates several things about leadership. First, is the point I mentioned above that leadership and management/authority are separate things. I had no official authority, but exercised leadership until someone with authority was in a position to take over. Second, is that my unofficial authority rested on the trust and acquiescence of those executing my orders. I would argue that, far from undermining their trust, my openness to challenge in non-emergency situations made them more likely to follow me in the emergency.

So, to return to Watts’ examples – teachers should be challenged (appropriately), cops should be obeyed (until the emergency is in hand), and both you and the boss should be able to flex based on the whether the current situation requires a teacher or a cop.

Participative leadership is more likely to engender trust and buy-in. Smart leaders (be they managers, architects, team leads, etc.) aren’t looking for passive followers, they know it could cost them. As Tom Cagley observed in a his post “It Takes A Team”:

While a product owner prioritizes and a scrum master facilitates, it takes a whole team to deliver. The whole team is responsible for getting the job done which means that at different times in different situations different members will need to provide leadership. Every team member brings their senses to the project-party, which makes all of them responsible looking for trouble and then helping to resolve it even if there isn’t a scrum master around.

Software Development, Coding, Forests and Trees

Some responses to my post “Why does software development have to be so hard?” illustrated one major (in my opinion) aspect of the problem – for many people, software development is synonymous with coding. It’s certainly understandable that someone might jump to that conclusion. After all, no matter how many slides, documents, diagrams, etc. someone produces, it is code that makes those ideas real.

Code, however, is not enough.

Over the last seventeen-plus years that I’ve been involved in software development, great strides have been made in languages and platforms. Merely look at the plumbing code needed to write a Hello World for Windows in C should you need convincing. Frameworks for application infrastructure, unit testing and acceptance testing are plentiful. Coding and coding cleanly is far, far easier and yet, people still complain about software.

While poor quality code can sink a product, excellent quality code cannot make a product. No matter how right you build a thing, the customer won’t be happy if it’s the wrong thing. The Hagia Sophia, Taj Mahal, Empire State Building, and many others are all breathtakingly magnificent structures that would utterly fail a customer who wanted (not to mention, budgeted for) a garage. We still fail to adequately understand the needs of our customers and the environments they work within. This is an area that desperately needs improvement. This is not a technical issue, but one of communication, collaboration, and organization. Neither customer nor provider can impose this improvement unilaterally.

Understanding the architecture of the problem is critical to designing and evolving the architecture of the solution, which is yet another area of need. Big Design Up Front (BDUF) assumes too much certainty and never (at least in my experience) survives contact with reality. No Design Up Front (NDUF), however, swings too far in the opposite direction and is unlikely to yield a cohesive design without far too much re-work. Striking a balance between the two is, in my opinion, key to producing an architecture that satisfies the functional and quality of service requirements of today while retaining sufficient flexibility for tomorrow.

Quality code implementing an architectural design grounded in a solid understanding of the customer’s problem space is, in my opinion, the essence of software development. Anything less than those three elements misses the mark.

Lord of the Repository

In Robert “Uncle Bob” Martin’s “Where is the Foreman”, he advocated for a “foreman” with exclusive commit rights who would review each and every potential commit before it made its way into the repository in the interest of ensuring quality. While I am in sympathy with some of his points, ultimately the idea breaks down for a number of reasons, most particularly in terms of introducing a bottleneck. A single person will only be able to keep up with so many team members and if a sudden bout of the flu can bring your operation to a standstill, there’s a huge problem.

Unlike Jason Gorman, I believe that egalitarian development teams are not the answer. When everyone is responsible for something, it is cliche that nobody takes responsibility for it (they’ve even given the phenomena its own name). However, being responsible for something does not mean dictating. Dictators eventually tend to fall prey to tunnel vision.

Jason Gorman pointed out in a follow-up post, “Why Code Inspections Need To Be Egalitarian”, “You can’t force people, con people, bribe people or blackmail them into caring.” You can, however, help people to understand the reasons behind decisions and participate in the making of those decisions. Understanding and participation are more conducive to ownership and adoption than coercion. Promoting ownership and adoption of values vital to the mission is the essence of leadership.

A recent Tweet from Thomas Cagley illustrates the need for reflective, purposeful leadership:

Is the leadership style you employ a conscious choice? It should be.

— Thomas Cagley (@TCagley) March 5, 2014

In my experience, the best leaders exercise their power lightly. It’s less a question of what they can decide and more a question of should they decide out of hand. When your philosophy is “I make the decisions”, you make yourself a hostage to presence. Anywhere you’re not, no decision will be made, regardless of how disastrous that lack of action may be. I learned from an old mentor that the mark of a true leader is that they can sleep when they go on vacation. They’re still responsible for what happens, but they’ve equipped their team to respond reasonably to issues rather than to mill about helplessly.

In his follow-up post, “Oh Foreman, Where art Thou?”, Uncle Bob moderated his position a bit, introducing the idea of assistants to help in the reviews and extension of commit rights to those team members who had proved trustworthy. It’s a better position than the first post, but still a bit too controlling and self-certain. The goal should not be to grow a pack of followers who mimic the alpha wolf, but to grow the predators who snap at your heals. This keeps them and just as important, you, on the path of learning and growth.