Software Development Methodologies as Collective Fictions

The failure of methodologies to accurately model reality is a common source of cynicism amongst boots-on-the-ground developers. I can recall many conversations with teammates where we’d end a meeting and then chuckle in hushed tones with each other about how “wrong” we were doing Scrum and how clueless our managers were.

In his article My 20-Year Experience of Software Development Methodologies, Ian Miell makes the point that methodologies are necessary as “collective fictions”—a thought I personally find comforting.

As an industry, we tend to look back on Waterfall as hilariously naive and antiquated, but to Ian…

…the waterfall process was a ‘collective fiction’ that gave us enough stability and coherence to collaborate, get something out of the door, and get paid.

He goes on to discuss his experience with Rapid Application Development at a startup:

We were small enough not to need a collective fiction we had to name. Relationships and facts could be kept in our heads, and if you needed help, you literally called out to the room.

…

We got slightly bigger, and customers started asking us what our software methodology was. We guessed it wasn’t acceptable to say ‘we just write the code’.

…

Turns out there was this thing called ‘Rapid Application Development’ that emphasized prototyping. We told customers we did RAD, and they seemed happy, as it was A Thing. It sounded to me like ‘hacking’, but to be honest I’m not sure anyone among us really properly understood it or read up on it.
As a collective fiction it worked, because it kept customers off our backs while we wrote the software.

Ian then experienced Agile, with all its foibles:

The few that really had read up on it seemed incapable of actually dealing with the very real pressures we faced when delivering software to non-sprint-friendly customers, timescales, and blockers. So we carried on delivering software with our specs, and some sprinkling of agile terminology. Meetings were called ‘scrums’ now, but otherwise it felt very similar to what went on before.
As a collective fiction it worked, because it kept customers and project managers off our backs while we wrote the software.

As someone who’s been making software professionally for over a decade now (not as long as Ian!), I have arrived at much the same conclusion, that software development methodologies have value regardless of their literal reality. Organizing principles and shared terminology alone are worth adopting something...anything.

Accepting the inevitability of these collective fictions is like taking a weight off of one’s shoulders. It just saves so much angst.

Bosses like to give politically-correct names to things that their people are already doing, and that’s fine. The software industry’s current fixation on Agile concepts is a reflection of the way that a distributed workforce would inevitably do things. Ian explains:

If software methodologies didn’t exist we’d have to invent them, because how else would we work together effectively? You need these fictions in order to function at scale. It’s no coincidence that the Agile paradigm has such a quasi-religious hold over a workforce that is immensely fluid and mobile.

We’ll always be working under some “methodology” and that’s okay. Just keep making good software, and don’t worry about what people are calling it.

Inclusive Culture, Minimum Culture

Our startup has a very progressive culture…as long as you’re white, childless, in your twenties, politically liberal, and love craft beer, hackathons, and indie rock.

My own personal experience working for a big, boring, multi-national company matches what Rich Armstrong recounts in his great article on Hacker Noon about inclusive company cultures:

As I said, to this day, my team at J.D. Edwards was the most diverse I’ve ever worked on. My first boss was an African immigrant, second boss was a forty-something mom. Our team measured high on nearly every dimension of diversity — gender, race, religion, age, parental status, national origin, sexual orientation, disability status, veteran status.

Rich goes on to talk about how that experience contrasted with his subsequent experience with the monoculture of a hip startup:

When our office culture is focused on business rather than socializing, we reduce the number of ways in which we all have to be the same. When we do that, we allow diversity to flourish. If your culture expects people to work long hours or hang out off-hours, the strain on the people who are different, in whatever way, is increased, and your ability to retain a diverse work force is reduced.

He then discusses his experience at a startup, Fog Creek Software, that managed to retain the inclusive culture of his big company experience:

...the culture was mostly about the business of software, how you build it, how you sell it, how you support it. If you were excited about that, you automatically belonged. You didn’t need to stay late, or drink alcohol, or play Rock Band, or play board games, or not have kids to pick up, or go to church, or not go to church, or do anything except show up 9-to-5 and care a lot about good software.

…the lesson I’m taking for myself going forward is this: if you want to build an inclusive culture, build a minimum culture. Build it around professionalism, boundaries, and work-life balance. Make sure your senior staff walks the walk, and spreads the word.

I found this last bit so poignant, as it’s really sort of counterintuitive—perhaps the reason why so many tech companies fail at trying to bolt on “inclusiveness” to a monoculture after the fact.

For a company that really wants to have a diverse pool of employees, you have to back off the idea that they’re all going to be best buddies, and don’t insist that their personal identities are enmeshed with a corporate culture.

Job Ad Red Flags: Calling All Ninjas!

I’m not sure this really needs to be said, but these words in a job posting are an instant buzzkill:

• Ninja
• Rockstar
• Jam sessions
• etc.

Basically, if the ad sounds like it was composed out of words that immature young men would think are cool, you lost me.

These may be controversial, but I’m also wary of mentioning things like “hackathons” and “kegerators”. Hey, I love working as a software developer, and I love beer, but it seems kind of weird to think I want to hang around the office after hours. (In fact, let’s kill the office altogether—I’ll take the savings in my paycheck, please.)

Don’t Trap Your Clients in the Bikeshed

As software developers, one of the best things we can do for our clients (internal or external) is to restrain from bombarding them with questions about trivial and easy-to-change aspects of their requirements.

When I find myself in doubt about how some aspect of the software I’m developing should work, I try to identify if the requirement is “bikesheddable” before stopping my progress to reach out to the client.

For example, the client has a requirement for a feature like “when an administrator grants access to a new user, the user should be notified by email.” I’ve seen developers (typically more junior) block their progress on a feature like this until the client gets back to them with a definitive answer on what title to use for the email.

I define a question about the requirements of software to be bikesheddable if it is…

1. Trivial
2. Likely to generate back-and-forth and delays
3. Easy to change later

If it’s bikesheddable, pick a sane default and continue on with the feature. When the client sees the feature deployed, they are of course most likely not even going to notice the choice you made for the aforementioned bikesheddable thing, but if it happens to matter, by way of #3 above, you can easily change it later.

Get the broad strokes right in v1, and the fine details in v2 and beyond.

Job Ad Red Flags: Fad Obsession

This red flag may be considered a subcategory of the previous red flag I wrote about in my post highlighting technical minutiae.

Remember when everyone was talking about how jQuery was unmanageable without a framework like Backbone.js? And then three seconds later it was embarrassing to be using Backbone.js, and startups would talk sheepishly about using it and how they should really rewrite using Angular? And then three seconds after that, they were embarrassed to be using antiquated technology like Angular, and how they couldn’t wait to start their rewrite using React?

As Dan McKinley put it in his classic post, Choose Boring Technology (2015):

Let's say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while.

…

If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens. If you choose to use service discovery tech that's existed for a year or less, you just spent one of your innovation tokens.

…

Adding technology to your company comes with a cost. …if we’re already using Ruby, adding Python to the mix doesn't feel sensible because the resulting complexity would outweigh Python's marginal utility. But somehow when we're talking about Python and Scala or MySQL and Redis people lose their minds, discard all constraints, and start raving about using the best tool for the job.

…

The problem with "best tool for the job" thinking is that it takes a myopic view of the words "best" and "job." Your job is keeping the company in business, god damn it. And the "best" tool is the one that occupies the "least worst" position for as many of your problems as possible.

When I read a job ad where the focus is on an intricately presented list of the most bleeding edge languages, frameworks, databases, text editors, chat tools, and the like, I have to wonder what other trivial things they’re spending an inordinate amount of cycles on instead of delivering software to their customers. 

Job Ad Red Flags: Technical Minutiae

Ever see one of those developer job postings that reads like the hiring manager sent out a mass email to every developer in the company just saying something like, “Please send me a list of every technology you’ve ever used at this company,” and then combined every list together to form the ad?

Looking for an experienced software engineer.

Qualifications:

.NET 2.0, .NET 3.5, .NET 4.0, Visual Studio 2005 Professional, Visual Studio 2008 Premium, Visual Studio 2015 Enterprise, ASP.NET 1.0, ASP.NET 2.0, jQuery 1.7, jQuery 2.2.4, PowerShell 1.0, …

And then give no details about the work environment, philosophy, or any sort of non-technical human consideration?

I completely understand if your company builds all of its software on Microsoft technologies, that you probably aren’t looking for Java developers. Or if you want to avoid hearing from Ruby on Rails developers when your project is embedded C. Mentioning the general constellation of technologies that you need someone to work with is a good idea for everyone involved.

But focusing your outreach effort to potential hires on a laundry list of technical minutiae shows a lack of understanding about what’s important. The hardest problems to solve in most software organizations are not technical problems but human problems. Talk mostly to the human being you’d be hiring and the other human beings they’d be working with.

Below is a great example of hiring for a technical role in which the writer takes a human tone first. In this case the company Stripe is hiring for a Web Developer role:

We’re looking for someone with:

• 3+ years of experience as a web developer
• Experience building public facing websites that work elegantly across commonly used browsers
• An extreme attention to detail and deep empathy with design
• An interest in helping startups
• Advanced knowledge of modern HTML and CSS

Perks of working on this team include:

• This is a new and very small team with lots of opportunity to help shape the future
• Working closely with Stripe’s world class design and marketing teams
• High impact role as we spin up our customer acquisition machine
• An opportunity to focus on fit, finish, and polish of frontend output

The only specific technologies mentioned here are HTML and CSS, which, of course, any person developing for the Web would need to know.

The copy is mostly written to a human being who likes building high-quality websites, rather than a list of the technical details by which Stripe has implemented their website. (Check out Stripe’s careers section for more great stuff like this.)

If there’s one thing I’ve learned in my career as a software developer, it’s that technology is almost certainly the least important aspect of software development. Hire a human being, not a list of technologies.

Job Ad Red Flags: “Passion”

There’s no doubt that people who enjoy a pursuit will generally get better at it than people who don’t. And there’s no doubt that people looking to hire employees prefer to hire ones who will take the work seriously and care about the quality of their work.

Unfortunately, it’s become all too common to abuse the idea of “passion” in recruiting efforts.

There is a certain kind of hiring manager who screens for people (often young, inexperienced people) who are willing to burn themselves out for the benefit of a company.

As Hacker News commenter s73ver puts it:

In job adverts, "passion" is often shorthand for "willing to work unpaid overtime."

Avdi Grimm has this to say, in his classic post on the subject of passion…

If what you really mean is “we seek people who are enthusiastic about programming”, say so. But consider why you are saying so. Is it because you want people who will do a good job because they care? Or is it because you expect the long hours to be their own reward?

No product or company deserves your passion. You can choose to throw your passion into anything you want, but no project inherently deserves it. ... What you are passionate for is something deeply personal, and it should flow from your most closely-held values.

I think ultimately, it’s just important to always remember that humans are not obligated to sacrifice their personal lives and happiness for the good of a company. Any job where you’re made to feel that, as a matter of course, your obligation as a person is first to the company, and then to yourself, is probably a job you should leave (or never apply in the first place).

The Past Needs You

There was a great article on Hacker News the other day called Maintainers Make the World Go Round which discussed the popular obsession with innovation and the chronic lack of interest in maintenance.

…innovation-centric narratives…have misrepresented engineers and scientists as inventors and designers when in fact the majority are “mainly concerned with the operation and maintenance of things.”

…most computer programmers maintain, refine and extend legacy systems while most scientists apply well-established science in routine ways. “In the software industry, about 70% of investment goes into maintenance…”

Innovation speak is a massive abstraction that lets its author destroy most of the world.

…by the standards of the past the present does not seem radically innovative.

Most new software projects will not be as valuable as software that already exists. I've noticed that in my career I've been paid better and treated with more respect when I've worked on maintaining and keeping up existing software that is important to the organization with a lot of investment, as compared to jobs where we were constantly starting new applications from scratch.

Greenfield projects can be exciting, but running software probably needs you the most.

Opportunistic Programming

As a developer, I’ve been frustrated at times when I felt that a product owner or manager was forcing me to spend an inordinate amount of time on a “pet feature” of theirs while glaring deficiencies elsewhere went unaddressed.

And to be fair, product owners and managers surely get frustrated at developers who spend too much time chasing some technical purity that no customer will care about instead of shipping.

I touched on this in a previous post—the importance of developer morale to the health of a project, and how it’s unwise for management to dictate the development team’s priorities all the time without considering their preferences and input.

Salvatore Sanfilippo (creator of Redis) introduced the term opportunistic programming:

There is a way to stress simplicity that I like to call “opportunistic programming”. Basically at every development step, the set of features to implement is chosen in order to have the maximum impact on the user base of the program, with the minimum requirement of efforts.

This really gets to the heart of my malaise when I’m being pushed to spend considerable time and effort on some aspect of a project when my intuition tells me it won’t pay off, and no one can adequately explain otherwise.

Salvatore goes on:

Often complexity is generated when there is no willingness to recognize that a non fundamental goal of a project is accounting for a very large amount of design complexity, or is making another more important goal very hard to reach, because there is a design tension among a fundamental feature and a non fundamental one. It is very important for a designer to recognize all the parts of a design that are not easy wins, that is, there is no proportionality between the effort and the advantages.

Let’s be real: code sucks, and adding more of it without a good reason is bad news.

The Many-Worlds Interpretation of Developer Interviews

A recent post by Yegor Bugayenko titled Why I Don’t Talk to Google Recruiters highlighted a common misconception about careers in the software industry, certainly a misconception that I had in the early years of my career.

Yegor described a very unproductive interview experience he had with Amazon, where he was subjected to the dreaded whiteboard hazing:

Some programmers who didn't know a thing about my profile asked me to invent some algorithms on a white board for almost four hours. Did I manage? I don't think so. Did they make me an offer? No.

…

If [the Amazon recruiter] would have started her email with "We're looking for an algorithm expert," we would never have gotten any further and would not have wasted our time. Clearly, I'm not an expert in algorithms. There is no point in giving me binary-tree-traversing questions; I don't know those answers and will never be interested in learning them.

I stopped nodding my head long enough to write a sympathetic tweet:

Like many developers, I fantasized about working for companies like Google—a company with cool products that I admired and with legendary perks for its employees. It took me a long time to understand that Google selects for a certain kind of programmer, one that I fundamentally was not.

As Jon Galloway explained in a post back in 2008, one that was highly influential on me at the time:

Steve Yegge works for Google. Steve and I are both professional programmers; in fact we both work on web applications. Yet, our jobs are no more related than if we were both doctors in different fields - say, a podiatrist and an orthodontist. For example, if you work for Google, you're going to be concerned with how to make your applications run against huge data sets using large, unreliable computer clusters. None of those are priorities in applications I work on, since I'm generally working against a relational database which aren't running at high volume or scale. So, Steve is thinking about how to build Google-scale applications running on Map/Reduce, while I'm working on comparatively small projects running against relational databases.

…

I still believe that Steve's questions would be useless in hiring someone to work on my current projects. I don't care if a candidate can check if a high-order bit is set, to the point that I might be a little turned off by an applicant whose answer revealed they were a bit-twiddler. I've hired bit-twiddlers before, and while they interviewed well, they weren't much help in shipping applications. Worse, we're wasting valuable time talking about hexadecimal formatting when we should be covering things like database access and rudimentary knowledge of HTML and HTTP.

What it comes down to is that the “software industry” is comprised of many kinds of people, working for many different companies, working on very different things, with very different priorities.

Don’t beat yourself up if you don’t spend your free time on Project Euler or if the idea of working through the linked list algorithms in a book about technical interviews makes you want to slit your wrists.

Most software jobs outside of Silicon Valley are not with high profile tech companies that hire for these kinds of traits. Most companies are interested in hiring people with experience in solving real, everyday business challenges with the technologies they’re invested in.

Find your world.

Small Diffs and Code Reviews

I enjoyed reading Dan McKinley’s recent post, Ship Small Diffs, in which he explains some of the benefits that fall out of committing code in very small amounts, perhaps a dozen lines.

My favorite benefit of doing this is the way it gives code reviews a fighting chance of being useful.

Dan writes:

Submitting hundreds of lines of code for review is a large commitment. It encourages sunk cost thinking and entrenchment. Reviews for large diffs are closed with a single “lgtm,” or miss big-picture problems for the weeds. Even the strongest cultures have reviews that devolve into Maoist struggle sessions about whitespace.

Looking at a dozen lines for mistakes is the sort of activity that is reasonably effective without being a burden.

In my experience, code review is one of those “best practices” that teams tend to adopt in a sort of “put a check mark here” way where they’ve heard it’s a good thing to do, and lots of good tech companies do them, but they never take the time to do them in a way that actually adds value to their development process.

Submitting a thousand-line diff that traverses many files for code review is almost worse than useless. No one has the time to rebuild the mental context of the person who wrote this pile of code, to a point where they could even hope to intelligently criticize its design.

A code reviewer who receives a diff like this will 9 times of out 10, pull out their LGTM stamp and continue on with a productive activity. If you’re lucky, maybe they’ll notice a variable name is misspelled or something. If you’re unlucky, the reviewer will turn into the world’s most inefficient linter, and point out all the places you used double-quotes instead of single-quotes.

Unfortunately, to get any real benefits out of code reviews, the team must internalize some lessons about breaking down large changes into minimally viable increments. It’s not as simple as just saying, “Yep, we do code reviews.” and patting yourselves on the back.

But…when you get there, the benefits go so far beyond effective code reviews.

Different Strokes for (Remote) Folks

There was a recent article on the popular topic of remote work that made the rounds and generated a lot of discussion.

I cannot trust employers to provide me with an adequate work environment, and this holds me back from doing the best possible work for them.

…

In the case of working from home/remote work, some employees do not do their best work from home, or simply don’t like it. That is fine—but you should trust your employees and treat them like adults. Let them make the call for themselves.

- Yan Lhert

There’s nothing that engenders respect within me for a company more than when they give me autonomy over my own working conditions. And I, like most people, give the greatest effort to a company I respect. A vibe of “We trust that you know what’s best for yourself” is extremely motivating.

It’s so interesting reading through the comments on Hacker News when articles about remote work come around. I could summarize most of the back-and-forth like so:

Person A: I work best like [this], and I think it’s ridiculous you like to work like [that].

Person B: Well, I work best like [that], and I think it’s ridiculous you like to work like [this].

Jeez! It’s almost like we’re all adults who have strong preferences for the way we like to work.

The solution to this conundrum begins and ends with Autonomy.

Here are some old sayings that capture the sentiment:

• To each their own.
• Whatever floats your boat.
• Live and let live.

One Workplace Design to Rule Them All is doomed.

I’m a believer in hybrid workplaces. The company has an office (or offices), but no one is forced to work in the office every day on a compulsory basis. And the office itself is a hybrid: sections are open plan for the people who prefer that environment, and there are quiet sections with a partitioned “cube farm” or small private rooms. Choose your own adventure.

There are two caveats I feel I must mention.

1. The employer must actually care about having high-performing employees. There are plenty of “body shops” out there who don’t necessarily care about this, and likely can’t differentiate high performers anyway.
2. Employees must understand the common theme throughout the working world that the highest performers get the most latitude. It’s up to the individual to take that autonomy with good grace, and kick so much ass that no one would dream of messing up your flow.

Different strokes for different folks.