Prototypes Live Forever

Code always lives longer than you want it to.

So often as developers, we're writing code under the gun. Just get it working. We have a deadline. We'll come back and fix this later.

Something I've learned through experience is that prototypes go into production. No one wants them to, but they do, 99% of the time.

That thing that you wrote that you rushed, that you're embarrassed to put out into the world—that thing will be in production, probably for a long time. Future developers, maybe long after you've moved on, will be reading it over and over, wanting to change it but being too afraid because it works, and it's been like that for a long time.

Code ossifies, the temporary becomes permanent. That next sprint where you're going to come back and clean up that code never comes. 

What does that mean for the daily practices of the software engineer? Do the little things now that can make your code better. Before the end of the sprint, before you've marked the ticket as "Ready for QA", before you've told anyone it's "done", before the pull request is merged. Give that variable a better name. Write one more (or just one) unit test that covers the basics. Split this class into two. Write a comment. Leave one more bit of feedback on that pull request. 

There are generations of developers that are counting on you. They'll be living with the shortcuts of today for years to come. I promise.

RIP Fred Brooks

Fred Brooks passed away this month. I first wrote about him on this blog in April 2007, right after I had finished reading his seminal book on software engineering, The Mythical Man-Month. I was about 2 years into my career as a software engineer at that point. How time flies!

I thought it might be interesting to pull out my dog-eared copy of the book and reflect on a few of my favorite concepts and how they've borne out in my career 15 years on.

The Mythical Man-Month

First off is the titular concept from the book, in which Brooks's Law is coined.

Adding manpower to a late software project makes it later.

I will say that in my career, I've only seen one project where bodies were hurled at a late project to hasten its completion. Interestingly, that was also the latest project I've ever seen in my career. 

Thankfully, my experience has been that Brooks's Law has permeated amongst the managers I've worked with—if not in name, they at least intuitively understood that they couldn't just add more people to a project to get it done faster.

The Second-System Effect

I've seen this play out several times in my career. The idea that we'll replace the hacky system we developed before we knew better with one where we take our time and "get it right" this time.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used "next time." Sooner or later the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.

It's easy to forget that just because we have a lot of ideas for improvements that not all of them are worth doing.

No Silver Bullet

I would dare say that most software engineers who have heard of Fred Brooks, found their way to him via his essay No Silver Bullet.

There is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.

Not only are there no silver bullets now in view, the very nature of software makes it unlikely that there will be any.

Personally, the message of No Silver Bullet is so ingrained in me as a software engineer, that I don't consciously think about where it came from very often, although it colors my world view constantly.

Every time I see a headline on Hacker News about Framework X vs. Framework Y, or Language A vs. Language B, I understand intuitively how little they can possibly impact the success of a software project.

When I wrote my post Stack Chasers vs. Evergreens in 2018, I don't think I was even aware that I was channeling No Silver Bullet that I had read over a decade earlier. It was in my bones at this point that technological solutions for improving the art of software engineering are inherently, severely limited.

Brooks's treatment of essential complexity vs. accidental complexity had such an impact on me, that I reference it when approaching challenges in my own life that have nothing to do with software engineering or my career at all.

The Mythical Man-Month is holding up amazingly well in 2022. I can't think of any other nearly 50-year-old book written about computer software that is still so relevant and insightful.

Thank you, Fred! R.I.P. And get a copy of The Mythical Man-Month if you haven't already read it. 

What Is a Staff Engineer?

What's next for someone who has been at the Senior Engineer level for several years or more? It used to be that you had to go into people management to level up. But nowadays, more and more companies are offering a level above Senior called Staff, that is not inherently a people management role. So what is Staff? 

There's an interesting post over on the GitLab blog where the author talked to several Staff Engineers in the company and identified the common threads in their roles:

• Proactive problem identification
• Cultural stewardship
• Knowledge sharing
• Technical empowerment
• Broader impact beyond the team

I would summarize this list as doing technical work that crosses multiple team boundaries. A Staff Engineer would be doing whatever it takes to make cross-team initiatives that need an expert software engineer successful.

It's hard not to come across Will Larson's work when searching for definitions of Staff Engineer. He literally wrote a book called Staff Engineer that is all about this very topic. Here is Will's answer to the question, "What do Staff Engineers actually do?"…

The most straightforward answer is that Staff engineers keep doing much of what made them successful as Senior engineers: building relationships, writing software, coordinating projects. However, that's a misleading answer. Staff engineers do those same tasks, but whereas previously they were the core of their work, now they're auxiliary tasks. Their daily schedule varies a bit by archetype, but there's a shared foundation across all archetypes: setting and editing technical direction, providing sponsorship and mentorship, injecting engineering context into organizational decisions, exploration, and what Tanya Reilly calls being glue.

Will talks more about archetypes of the Staff Engineer in another article which is what introduced me to his work (Hacker News discussion here). He's identified four different kinds of Staff Engineers that are commonly found in the wild:

• The Tech Lead guides the approach and execution of a particular team. They partner closely with a single manager, but sometimes they partner with two or three managers within a focused area. Some companies also have a Tech Lead Manager role, which is similar to the Tech Lead archetype but exists on the engineering manager ladder and includes people management responsibilities.
• The Architect is responsible for the direction, quality, and approach within a critical area. They combine in-depth knowledge of technical constraints, user needs, and organization level leadership.
• The Solver digs deep into arbitrarily complex problems and finds an appropriate path forward. Some focus on a given area for long periods. Others bounce from hotspot to hotspot as guided by organizational leadership.
• The Right Hand extends an executive's attention, borrowing their scope and authority to operate particularly complex organizations. They provide additional leadership bandwidth to leaders of large-scale organizations.

Read Will's article for deep dives into all the archetypes, complete with examples of what their schedules are like and interviews with actual Staff Engineers who he sees as fitting these different archetypes. It's very enlightening, and I can't really imagine a better way of getting insight into the real-world nuts and bolts of that title.

My takeaway is that a Senior Engineer sort of maxes out as an expert software engineer. The Staff Engineer is about taking engineering expertise and expanding the scope of one's influence, which could mean overseeing a group of people, facilitating communication between groups of people, or solving problems involving multiple codebases. And maybe this goes without saying, but you're not likely to see the Staff Engineer title existing outside of large technical organizations, where the scope of engineering activity requires this sort of role.

Remote Hiring and the Paradox of Choice

Oh, how I love remote work. And oh, boy, I've written a lot about the topic on this blog. We all know the reasons why remote work is amazing; there's no need to rehash at this point.

But I haven't written much about the flipside, particularly the process of being hired for a remote job. If you consider how much more awesome working from your home is compared to a typical open plan nightmare, then a similar multiplier exists in the other direction for how much more frustrating a job search is for a remote job compared to a local in-person job.

Just as the abundance of remote jobs is a dream from a job seeker's perspective, the abundance of the talent pool from a hiring company's perspective leads to some annoying behavior.

There are clear comparisons to be made to the shift to online dating from "traditional" dating. As the pool of eligible participants increases, the value of each individual participant drops. And just as the modern phenomenon of "ghosting" is common in the world of online dating, it's also common in the world of remote job hunting.

It's way more common in a remote job search to go through multiple rounds of interviews with a company and then never hear from them again, for example. Or for recruiters to hound you on LinkedIn until you agree to a phone call, gush about the perfect fit for a role, and then never contact you again about that next meeting with the hiring manager. Follow-up emails go into a black hole. If you get feedback at all, you find out you were rejected for some seemingly minor and arbitrary shortcoming. That's just how it is when the potential talent pool for a job opening expands across a country or across the globe.

Just as I mentioned earlier a multiplier for how much better a remote job is to a traditional job, you will experience general flakiness from hiring companies at a similar multiplier. There's just something about the abstraction and lack of physical proximity between the participants that amplifies this behavior.

And believe me, it's not just employers who are making this difficult, there's no end to the shenanigans that occur on the candidate's side. Both sides are enjoying the benefits while cursing the drawbacks.

So let's file this post under "Advice for a Younger Me." I write this post in the same spirit as posts I wrote years ago like The Many-Worlds Interpretation of Developer Interviews and People Hire Clones of Themselves. The hiring process for software engineers seems so bizarre and arbitrary a lot of the time that it helps to remember that there are reasons for the weirdness that have nothing to do with you, and it's like this for everybody.

It's a brave new world out there. Long live remote work. :)

Swamped to Sustainable

I've written a lot about the concept of sustainability on this blog, and it's something I'm constantly thinking about with regards to software development.

That's why I enjoyed this recent post by Greg Kogan about being "swamped" all the time:

I used to think being swamped was a good sign. I’m doing stuff! I’m making progress! I’m important! I have an excuse to make others wait! …

Now, I’m impressed by people who are not swamped. They prioritize ruthlessly to separate what’s most important from everything else, think deeply about those most-important things, execute them well to make a big impact, do that consistently, and get others around them to do the same. Damn, that’s impressive!

Being swamped isn’t a badge of honor, it’s something to work on.

There were good comments in the discussion of the post on Hacker News, including a link to a manifesto of sorts I wasn't previously familiar with called Sustainable Development.

Sustainable Development is a set of principles for software teams to use to operate in healthy and productive ways, and for the long term.

Software teams practicing Sustainable Development follow guidelines that benefit them in three areas:

Physical: They work in ways that allows them to maintain good physical health.

Emotional: They work in an environment that supports their emotional health.

Cognitive: They work in ways that encourage creativity and support the intellectual nature of software engineering.

I could see this being used as a framing device for discussions in a sprint retrospective.

Development teams wanting to adopt Sustainable Development simply write down the practices they wish to embrace, and then commit to following them. Each practice should benefit the team in at least one of the three areas of sustainability (physical, emotional, or cognitive).

It's often hard getting people to speak productively in retros, and I feel like a good place to start is asking the question to the group: What did we do in this last sprint that doesn't seem sustainable? What can we do to increase the sustainability of our sprints? The suggested sample practices could give the group some ideas on action items to work toward greater sustainability.

I hope every team can agree that swamped is not sustainable.

Documentation Is a Competitive Advantage

People often say that one of the reasons Stripe is so successful is that their documentation is world-class.

Most companies, open source projects, and individual developers find this to be a Herculean challenge. I'd love to know if Stripe has any secrets to keeping their Docs updated. Though, my gut says that they likely just put in the work. Their Docs are incredibly important to their success.

- cdevroe, Hacker News commenter

Thanks to Stripe and a few others, developers' expectations have changed. Now they expect thorough, navigable documentation presented in a clean, error-free UI. If you deliver anything less, few developers will even bother with your API.

- ReadMe Blog

It really is amazing how much of a difference clear, up-to-date documentation makes to set one product apart from its competitors. 

But I like to think about the impact documentation has on one's career as a software engineer. I've written extensively on this blog about the importance of written communication in the context of remote work.

There's so much value one can contribute as a software engineer that goes beyond writing code. 

Some examples:

- You're working on a ticket where the acceptance criteria are ambiguous in some way. You ask the product owner for guidance on the parts that aren't clear. Instead of just moving on with your work on the ticket, go back to the ticket and either update the acceptance criteria to what you discussed with the product owner, or leave a comment on that ticket that documents what you discussed and how it affects the requirements.

- You ran into some error when running the system that you've never seen before. You ask in the team chat for help. A teammate has seen this error before and pings you with the steps on how to get past the error. Before continuing on with your work, write a bug ticket in the team's backlog with as much info as you can while the problem and solution are still fresh in your mind, and post a link to the bug ticket in your team chat.

- You discover while following a process documented in the team's wiki that some of the steps are out-of-date or not relevant anymore. Take some notes on what you found lacking in the wiki page, and then come back and edit the page yourself to make it clearer.

Most engineers don't do these things, and as such, they're great ways to stand out for engineers that make a point to do them. The work of an engineer involves so much learning—valuable information that is often lost when the feature is added or the bug is fixed, and we move on without documenting what we learned along the way, artifacts that won't be captured in the code alone.

Just as products can stand out with great documentation, so can engineers.

New People Write the Best Documentation

Have you ever heard that the best way to learn a subject is to teach it?

I've had the experience several times when I've joined a new team and started ramping up, that their documentation is out of date and skips over important information.

We know there's always that awkward transitional phase when a new engineer joins a team that's existed for a while, because they're not immediately ready to take on "real work", even if they're an experienced engineer in general. 

This is a perfect opportunity to improve the documentation! In fact, I think a great first assignment for an engineer new to a team is to update the onboarding documentation. They'll see exactly what's missing as they try to follow along, in a way that veterans on the team never will.

When experienced folks are tasked with writing documentation, they don't need the documentation, so they will tend to hand-wave and make leaps in explaining things because they're not truly starting from scratch like a new person is.

Documentation written with a fresh perspective is so helpful, and of course, living documents are best. Establish a virtuous cycle where each new person leaves the docs a little better than they found them.

Let developers do "off the board" work

When sprints are well planned and sustainably paced, it's natural to have developers who have finished all of their committed backlog items with time to spare. In occasions like this, I've seen teams where the Scrum Master—or some other authority figure—reflexively looks for additional items in the backlog to "pull into" the sprint to keep the developers busy. I think this is a mistake, and I'll explain why.

For Agile teams that are organizing their work in sprints, it's traditional to have some sort of visual board or central place that anyone on the team or anyone interested in the team's work can go to see which items are being worked on and the progress of the items. Some people get uncomfortable with the idea of work happening off the board, where it's not officially recognized as work that counts. They want to see each developer with at least one assigned item on the board that is in progress right up until the end of the sprint. That's how we know that we're getting maximum velocity and value from the team, right?

But I think it's important to normalize the idea that people need time to work on things that are "off the board" for a couple of reasons:

1. People always have work to get done that is not directly related to a team goal.
2. Continuous sprinting is not sustainable.

Here are a few legitimate things to work on that don't belong on a sprint board:

• Training
• Preparing for a presentation
• Proof of concept / demo of an intriguing tool, framework, technology
• Updating documentation that's been bugging you
• Spikes into performance improvements
• Cleaning out your inbox
• For companies that do some sort of periodic "goal setting" for each employee, let people work on goals from their list
• In general…things that don't involve QA

I think that last point is important, because people will often talk about downtime being useful for addressing technical debt. But that only works for these end-of-sprint downtime scenarios as long as you're not adding to the regression testing burden of any QA folks that are still testing development work completed during the current sprint. A big technical-debt-reducing refactor is not something we want to check in casually without QA time reserved for making sure it hasn't blown up anything (unless your codebase has truly superb automated regression tests in place).

It is incumbent on the developers to identify their own priorities that they can pull from when they have extra time. Don't ask for more backlog items if you have work in mind that's off the board. For Scrum Masters, the go-to move should not be to pull in more backlog items without asking if the developer has any off-the-board work that can keep them busy.

Apart from recognizing the reality that developers have work to do that isn't directly related to a product backlog, it's important to reflect on the idea of sustainable pace and what "sprinting" really means anyway.

What does the word sprint mean outside the software world? It's a short, high intensity run. In the real world, you don't finish a sprint and then instantly start sprinting again. You have to catch your breath and take a pause before you can sprint again. Otherwise you'll collapse. It's important that software teams do this too—take a breather before the next sprint begins.

Smooth, predictable iterations necessitate a sustainable pace and regular buffers. It's okay if we have time each sprint that is not maximally stuffed with points.

Penny-wise and Pound-foolish

What a time to be a software developer! I'm always amazed that even in an industry with high salaries and intense competition for talent, companies find a way to shoot themselves in the foot with cheapness.

There's a great thread on Hacker News today where people are chiming in with their experiences of company cheapness around basic things like coffee and bathroom cleaning. The common lesson being that when you see a company attempting to cheap out on basic things, it's time to leave.

Early in my career I read Joel Spolsky's A Field Guide to Developers and it painted a picture for me of how good tech companies treat their developers. And it taught me early on to pay attention to the anti-patterns of companies that were less enlightened.

At one of my first jobs, I asked for a book about a new technology that was important for projects I would be working on. I'll never forget the development manager making me feel bad that in a time where the company was trying to save money that I would ask for a $30 book on a technology they wanted me to learn. Keep in mind this was a company that was paying me tens of thousands of dollars in salary in addition to full benefits. It's in small moments of cheapness that you learn a lot about a company.

In the early years of .NET, when Visual Studio was less packed with nice productivity features, the 3rd-party extension ReSharper was more or less ubiquitous amongst respectable developers in that space. It was always very telling to see which .NET shops assumed you would want that tool and provided a license on your first day, and which ones treated that extra cost of a few hundred dollars with suspicion.

In fact, the way that employers view licensing costs for software of the daily-use variety is always enlightening. One company I worked at had developed an impressive build system around a popular source control system you've definitely heard of. At some point someone in management decided they didn't like paying for an open source product that could be had in a free edition, so they downgraded to a "community" edition of the source control system that was free (as in beer), requiring the company to allocate developer hours to re-implementing features of the paid edition that they relied upon heavily but no longer had in the free version, not to mention training everyone to use the new bolted-on system in their daily workflow.

Training is another area that speaks volumes about an employer. Companies that believe in the importance of training will make it as easy as they can for you to get that training. How onerous is the process to get training courses approved? Companies that want to provide simple, default training options will subscribe every developer to Pluralsight, Safari Books Online, or other popular all-you-can-eat educational resources. Some companies are aware of these resources but will try to save money by buying only a certain numbers of licenses, so that employees have to keep bugging each other to release a license. Some companies will force employees to sign up using their own credit cards and then submit expense reports every month to get reimbursed. These are money saving measures that put barriers up and ensure that most people will simply stop caring about getting training because it's too much of a hassle. Employees aren't dumb—they know the company had ways they could have made these things easier, and chose to go the penny-pinching route that shifts burden to the employees.

These are just a few examples I've personally witnessed, but there are myriad ways that companies turn off developers with moments of cheapness. In a period of unprecedented mobility among developers, where employers routinely pay six-figure salaries to entice and retain their people, it's a bad time to be penny-wise and pound-foolish.

Ron Jeffries on Prediction

It is common practice to make a list of essential features, think about them for a while, and then decide that they define the next release of our product. The next question, of course, is “when will all this be done?”

The answer is that no one knows. We could do a lot of work to improve our not knowing, and in some areas and at some times some of that is worth doing, such as when there’s a large contract waiting to be bid. But when we’re in the business of developing solutions for internal or external customers, we do best to provide small amounts of value frequently, not wait for Big Bang releases that seem often to recede indefinitely into the future.

…

You’re trying to get into a frame of mind where you think “If we just did this one little thing, Customer Jack could actually use this”. Then, do that little thing and let Customer Jack try it. We want to move as quickly as we can to continuous delivery of value.

We want to make the value of what we’re doing so visible that our Product Owner and other stakeholders can’t wait to get it out there. Then … we’ll be doing the right thing, with, or without, story estimates. 

Ron Jeffries - Story Points Revisited

Bring Engineers Problems, Not Solutions

I read a great article posted to Hacker News this week called Effective Communication With Software Engineers by Simón Muñoz.

He made some wonderful points that rang true for me, as a software engineer, that I wanted to discuss here.

What resonated most with me is this idea that as engineers, we want to know the high-level goals of the product we work on, why they're important, and be given the leeway to design solutions.

You don’t present a solution to an engineering team; you bring them a problem to solve. The best engineers won’t willingly accept implementing a solution if they have not participated in defining it.

I've written about this idea before in my post Goals, Not Tasks. Engineers don't want to be treated like highly-paid instruction followers—they want to understand business objectives.

And not only do we want to know what the high-level objectives are, we want to know why they are important to real users of the product. Simón continues…

Starting with the problem is not enough. You have to demonstrate how solving this problem fits with the vision and strategy of the company. Your team also has to understand why you want to solve this particular problem and not any other.

It's so much more motivating to work on a product when you can see that real people are benefiting from it, and you're not just fulfilling some manager's dictum. Simón mentions the importance of proving to the engineering team that the work they are doing is "moving the needle"…

Your engineers want to know if their work impacts the organization. A huge smell that they are working in a feature factory rather than as product engineers is when you don’t define how to measure the success of an initiative.

We don’t measure success by the number of story points that the team manages to get each sprint... We measure success by moving the needle on the metrics that matter to the business.

If you don’t obsessively communicate the importance of these and can’t get your team interested in them, you’re losing much of their value.

We engineers want to know what are the important problems to solve, why are they important, and be given the freedom and respect to invent solutions to those problems.

Finish Things

In Lean, there's a lot of talk about reducing work in progress (WIP). It's interesting to me that even though it seems that Lean concepts have permeated the software industry, so many teams struggle with finishing things.

Speaking from the point of view of a developer, I can say that there's a real drag on morale when it feels like no one particularly cares about receiving the value of my work. Obvious bottlenecks in a team's process that persistently go unaddressed are not just a waste of money, they're a reason for people to check out. Add to this malaise the tedium of fixing the continuous merge conflicts that come when we create code branches that we can't merge for a long time. Ugh.

Some common impediments to finishing things:

• Slow code reviews
• Exhaustive manual testing
• Distracted product owners

Slow code reviews

I've long been of the opinion that most code reviews are hardly worth the time spent on them by the reviewer and not worth the interruption to the flow of work that they cause. It's really hard to do good code reviews and it's a major bummer to wait a long time for someone to review your code only to get a rubber stamp. I like to call this "Quality Theater", where a checklist of procedures are half-heartedly (yet religiously) performed that don't actually improve the quality much. I would encourage teams to look at metrics in whichever code review system they use to see how long code reviews are taking, and how many significant changes to code occur as a result of the reviews. Be honest if they're resulting in long delays that usually end in a rubber stamp.

Exhaustive manual testing

If the team has dedicated QA people, is it taking a long time for work to make it through testing? Why? Is there a large amount of manual testing going on for every story? Can we automate more of the tests? Are stories hanging around in the "QA in Progress" stage waiting for clarification from the product owner about an edge case that's not really central to the story? Are stories being rejected by QA because of a small issue that does not detract from the value of the story?

Distracted product owners

Is the product owner plugged into the project fully and making it a priority? Are we waiting a long time for them to accept the work we've done? Are stories being held up because of ambiguous requirements where the team can't agree if they were met or not?

So how do we finish things faster? Other than recognizing the impediments above and having some earnest discussions about why they're occurring, there are some general concepts that apply broadly.

The common thread is in identifying what is our unit of "done"?

• Make stories that are small increments of value
• If a story is being held up because of a tricky aspect, consider splitting off the hard part and finishing the easier part cleanly
• If we discover a story has a subset of the requirements that is contentious or poorly defined, can we finish the obvious part now and put a story in the backlog for the wishy-washy part?

The idea is to focus on finishing things. What is slowing down our process of getting units of value to "done"?