Above All Else, Sustainability

From the principles of the Agile Manifesto:

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Extreme measures are always doomed beyond a limited time horizon. Whether we're talking about diets or software development practices, without sustainability you have nothing.

Initiatives to increase productivity, improve quality, lower costs, or any other "good thing" simply don't matter if they're not sustainable.

A common topic in Agile circles is "sustainable pace", which usually focuses on the futility of working overtime, typically over 40 hours a week. I would argue that sustainable pace is about more than just the number of hours clocked in a work day or work week. 

Is the work emotionally sustainable? Do people hate working here? Do people end the work week feeling a sense of accomplishment? Do people feel like their leaders have reasonable expectations? Are they constantly battling reality?

Sometimes the degree to which an enforcer must continuously apply pressure to get people to follow a process indicates how sustainable the process is. Powering through is not a sign of discipline, it’s a sign of delusion.

Ignorance of reality is unsustainable. Coercion is unsustainable. Surveillance is unsustainable.

Can I and the people around me keep this up forever? If not, it's time to pause and reflect. Above all else, sustainability.

Agile as Progressive Enhancement

When I was coming up as a young web developer. the concept of progressive enhancement was kind of at the hipster vanguard of web development. This was in the phase of the web where JavaScript and CSS feature parity amongst browsers could not be taken for granted like it is today.

The idea of progressive enhancement was that you started by making your web application work on an essential informational level with semantic markup, then you layered on presentational and dynamic niceties with CSS and JavaScript respectively, if available to the user agent. At each layer, or each stage, you left the user with something usable and valuable that stood on its own without the other stuff.

Learning about the concept of progressive enhancement early in my career colored my interpretation of the Agile philosophy forever. The iterative nature of Agile methodologies, with "Working software [as] the primary measure of progress" is essential to how I approach making software for users to this day. In my mind's eye, I still visualize iteration as progressive enhancement. We start with the essential core of value, and ship that. If the core of the idea resonates with real users, then we iteratively layer on enhancements to the core, shipping to users at each stage.

In order to work this way, you have to accept that quality and feature completeness are not the same thing. At each layer, we're delivering a scope with high quality always, but with no ambition of completeness. We're delivering shippable increments of software as layers over a core idea of value that we've already shipped. We build out our product backlog as all the nice things we could layer over the core, as we think of them, but they are not essential. One of the amazing things about working this way is that users are delighted by it. The features just keep getting nicer before their eyes.

Legibility

I recently came across a blog post called Seeing Like a Software Company by Sean Goedecke on Hacker News. The post hooked me right away by introducing to my vocabulary the term "legibility": 

By “legible”, I mean work that is predictable, well-estimated, has a paper trail, and doesn’t depend on any contingent factors (like the availability of specific people). Quarterly planning, OKRs, and Jira all exist to make work legible. Illegible work is everything else: asking for and giving favors, using tacit knowledge that isn’t or can’t be written down, fitting in unscheduled changes, and drawing on interpersonal relationships.

One of the advantages that small companies have is that they can achieve greater speed compared to a large company by eschewing legibility. When you know everyone else in the company by name, or maybe you all work in the same room together, you don't need standardized processes to go about your work, in fact, they just slow you down. Work happens through what the author calls illegible backchannels:

An engineer on team A reaches out to an engineer on team B asking “hey, can you make this one-line change for me”. That engineer on team B then does it immediately, maybe creating a ticket, maybe not. Then it’s done! This works great, but it’s illegible because the company can’t expect it or plan for it - it relies on the interpersonal relationships between engineers on different teams, which are very difficult to quantify.

One of the struggles that small companies face as they grow into larger companies is that they can't get by anymore without legibility. Maybe they've grown to hundreds of employees or they're working with people distributed across multiple timezones.

The fact of growing up as a company is that the loss of speed in individual tasks is outweighed by the predictability of process. As the author writes:

The processes that slow engineers down are the same processes that make their work legible to the rest of the company. And that legibility (in dollar terms) is more valuable than being able to produce software more efficiently.

I feel like this is a hard concept to sell people on who are used to working in small companies. In the long run, going slower is actually better for the company. Writing things down and organizing the work in a more structured way reduces the chaos left in the wake of localized, marginal speed-ups.

The author concedes that illegible work is necessary in small doses even in large companies. In cases of show-stopping production bugs, for example, large companies create temporary sanctioned zones of illegibility in which they gather together a strike team of experienced people to swarm on a critical issue until it's resolved. They then return to legibility.

Legibility is a massively useful concept that I will carry with me. It explains so much about how companies function internally, in often counterintuitive ways.

The Financial Architecture of Software

Conway's Law is foundational to software engineering. It says:

Organizations which design systems...are constrained to produce designs which are copies of the communication structures of these organizations.

If you've ever had any job in the software industry, you've almost certainly seen this play out in the architecture of your codebase. Group A works on Thing X and Group B works on Thing Y, so Thing X is in one [project, repository, web service] and Thing Y is in another [project, repository, web service]. You can read the organizational structure in the structure of the technology. It helps to make sense of why things are the way they are. The silos of communication are reflected in the solutions.

But I recently came across this interview on InfoQ with Ian Miell, who I've mentioned on this blog before. He has interesting things to say about the sociology of software development, and in this interview he talks about a book he's writing about the financial architecture of software:

The material aspects of the world ultimately determine the software we build and specifically the decisions we make at a grand scale about the software that we build. 

I remember being frustrated and confused as a junior engineer, not understanding why certain things are prioritized in a company and others aren't. Obvious best practices from the industry at large don't get traction in certain places, what seem like universal good things like "quality" don't seem to matter, or why values spoken about at the company level seem unevenly distributed and applied.

Ian explains...

When I have a code base, I want to make sure all the tabs are correctly aligned before I do anything else that might take time, or I wanted to make sure the naming paths are consistent. But you get larger scale problems where engineers say, "No, we have to fix this now because it's a huge thing". And you say, "Well, I get it, but that's going to take us a month and in that month people are going to be looking at me as the leader of this team and saying, 'What have you delivered?' And I can try and explain to them what I've done, but it's not going to apply".

Hard work is not always rewarded. It depends on who sees the work, what their individual priorities are, and how much money they control. My advice to my younger self would be to pay close attention to what your specific boss thinks is important, and show yourself working on those things. The next level is understanding what your boss's boss thinks is important and by what criteria they're judging your boss; then you'll really understand what kind of work is rewarded.

As Ian describes...

Who owns the budget is a fascinating question. ...I start the book with a story about a very successful engineer, I call her Jan. She becomes a leader of a small group of engineers and she builds a platform in her spare time with those engineers, like a Kubernetes platform. And she does it within a central IT function and she builds it and she takes it, she shows it to her manager and the manager's like, "Well, this doesn't help me hit my goals for the year. This doesn't help me get a promotion, this doesn't help me get a pay rise". And she's nonplussed like. "I'm trying to deliver faster, cheaper, better. That's what the company is, we're supposed to be agile. I'm trying to help the whole business".

And it might sound naive, but a lot of people think this way. I certainly did. Where you're thinking, well, I'm doing what's good for the company, but the system is not structured in that way. It's not designed to be run that way because people are complicated and their interactions are complicated, so you silo things and you have different budgets and so on, and the end result is that you are slaving away at the central IT function, trying to make the business better as a whole, but it's not accounted for.

Understanding where the money comes from sheds light on the technology decisions, where quality matters a lot and where it's almost an afterthought, and why different teams are more scrutinized than others. In the end, follow the money.

Slow Is Smooth, Smooth Is Fast

If you want to move fast, don’t make speed your goal. Make smoothness your goal. Fast naturally follows smooth.

Every software organization wants to be fast. And fast has a certain look to it. Buzzing Slack channels, video calls, people huddled around a screen or whiteboard. It's very easy to conflate "high touch" work practices with speed.

As I mentioned in my previous post, one of the huge benefits of sprint-based development is that sprints act as a delivery protector. A bulwark against constant churn.

The principles of the Agile Manifesto talk repeatedly about change, continuous X, face-to-face conversation. Some teams and leaders really latch onto these concepts. Let's change the requirements every day; let's discuss every decision face-to-face.

One of my favorite principles is:

Working software is the primary measure of progress.

All the churn that looks like "Agility" is canceled out if you don't deliver on a regular enough cadence that your customers notice. They don't see anything else, they don't measure you by anything else.

The passé methodology of Waterfall became passé because it didn't respond to change well enough. People were wed to process instead of reality. Why spend a year building something that no one actually wants, right?

But I firmly believe there is such a thing as being too responsive to change. There is such a thing as too much communication. Remember, your customers only see delivery. They only see what pops out the other end.

There is a benefit to following a well-defined process from idea to delivery. Process is perhaps a dirty word in Agile circles, but I think in some ways being anti-process is being anti-delivery. Every time we circumvent a process, we create communication overhead. Each team member impacted must be notified and followed up with to make sure they know about the circumvention. 

Any change is only good if the ripple effects created by the change are worth suffering. I reject the interpretation that Agile implies that more change is better. We have to always balance change with delivery. In fact, we have to balance anything that takes time and effort for a software team with delivery.

Sprints Balance Consistent Delivery With Optimal Value

Value, value, getting value to users. Work on the most valuable thing, always. You could probably summarize the Agile philosophy as "deliver value to end users faster." What's the next most valuable thing we could work on, get it done, and get it out to production.

With a sprint cadence, we commit to a strategy for two weeks, we work as a team through the items in our sprint backlog, and we retrospect at the end to talk about how we did. There's always the chance that some circumstance around the business could change mid-sprint that may change the team's idea about what is most valuable to work on in the moment. Since sprints balance consistent delivery with optimal value, we keep our heads down and finish the chunk of work we agreed to deliver at the start of the sprint.

This requires leaders and engineers to get used to telling people outside the team, "That's a great idea! We'll put it in the product backlog. We can start working on implementing this idea in less than 2 weeks from today!"

As arbitrary as the sprint cadence can seem, I see the sprint as a delivery protector--a built-in reason to say not right now. The items we're working on today may only be 80% of optimal value, but we're going to finish them dammit. In no more than two weeks we can shift our strategy 180 degrees if necessary, but not today.

Sprints Need a Cooldown

Teams vary in the amount of handwringing their leaders do about sprints that aren't perfectly rightsized. What if we bit off too much work at the start of the sprint and can't finish it all by the end? What if we overestimated the work and didn't fit in as many tickets as we could have?

I've worked with teams where having work incomplete on the last day of the sprint was no big deal, it happened regularly, and no one was too concerned about it. And I've worked with teams where the team committing to a certain scope of work was sacred, and tickets carrying over was a grave error that we fretted over and vowed not to repeat.

The teams that prefer the overstuffed sprint are trying to maximize velocity and reduce downtime of the team members. Teams that treat the sprint timebox as sacred are trying to maximize predictability and often striving for a kind of "standardization" across teams. Some teams will even wonder why they're bothering with these stupid sprint things and opt out for a methodology like Kanban where people just grab work items when they finish the previous one, and toss out the timebox.

I have always believed strongly that whether you're using Scrum or Kanban, and working in discrete timeboxes or not, that team members need a regular cooldown. Just as an athlete cannot sprint indefinitely and must rest, I believe the same is true of software engineering teams.

As you might imagine, this puts me in the camp of "fill the sprint conservatively so that we don't go over." This, of course, means that many sprints will end without the theoretical maximum amount of trackable work items packed into them. And to that I say: Good. That's the point.

I believe that implicit in the social contract of teams is this: If you want people to sprint, then they must be allowed to cool down. That's the deal. Continuous sprinting means burnout. 

Quoting a previous post of mine, here are a few things engineers need to do that are not represented by tickets in a sprint backlog, and are perfect for filling the cooldown period at the end of a sprint, at their discretion:

• Training
• Preparing for a presentation
• Proof of concept / demo of an intriguing tool, framework, technology
• Updating documentation that's been bugging you
• Spikes into performance improvements
• Cleaning out your inbox
• For companies that do some sort of periodic "goal setting" for each employee, let people work on goals from their list

If we set a strong sprint goal, bite off a chunk of work that the team is confident of completing, and we get it all done without rushing a day or two before the end of our sprint, that's a good thing. The team feels a sense of accomplishment, the business gets a valuable increment of working software that fulfills a real need, and the individuals get some time to cool down and shift their attention to some low-intensity stuff that nevertheless needs doing before the next sprint starts again.

For me, the cooldown at the end of the sprint is what sustainable pace is all about. We go hard toward an important goal, and if our planning and teamwork are on point, then we know we'll have that down-shift at the end.