About Me

Until recently, I was a senior backend developer and tech lead at ecobee. Currently I am on a nice sabbatical, which I’d highly recommend to everyone. I worked on the device backend team, supporting the server platform for smart home devices in over a million homes. I’ve been working in software development for about eight years now. Most of my experience is with on-premise Java applications, and shifting an ever-growing production footprint into Go and Google Cloud. I also mentor senior engineers with BestPracticer.

I wrote this several weeks before I began my sabbatical, attempting to distill everything I had learned – or lessons I’d had painfully reinforced – into some bite-sized principles. Most of these aren’t novel observations as much as they are the sorts of things that stand out after my experiences putting theory into practice. Often I was precisely the person who made the mistakes that I am criticizing.

This is “v2021” because I intend to write more of these over time. It’s always interesting to look at where you are and compare it to where you were; and especially against where you thought you were, which is almost certainly different than where you acutally were. Memory is reconstructive, and you will often recall a different past than you felt at the time. Basically this is an exercise of journaling in public.

Foundational Beliefs

  • Whatever endangers trust endangers everything
  • “Make it easy to do the right thing” is indeed a powerful guiding principle
  • Conway’s Law always wins
  • Manage less state, always and everywhere
  • “Every moment you spend outside of production is a moment spent learning the wrong lessons” - Beau Lyddon, https://twitter.com/mipsytipsy/status/1112240468887248896
  • Often the best solution to a problem is to stop it from existing at all. Don’t try to handle bad data better; stop generating it in the first place. (paraphrased from somewhere, but I can’t remember the source)

Corollary Beliefs

  • Always. Be. Deploying.
  • Observability is worth its weight in gold. And probably some extra silver on top of that.
  • All alerts endanger your on-call team’s psychological safety. Terminate alerts and their causes with extreme prejudice.
  • The biggest problems all live near domain boundaries
  • The only true agile methodology is the one that evolves from your team’s retrospectives
  • Like diets and exercise routines, the best tool for any task is the one your team actually wants to use
  • Napkin math that gets you moving is often better than waiting for perfect data
  • Performance tuning outside of your SLOs is almost always a mistake
  • Do not comment in PRs about style, nits, formatting, or lintable things – automate that
  • Code reviews are primarily to make sure everyone understands the code, and only incidentally to prevent issues
  • There is such a thing as being too clever. And the bar for that is probably way lower than you think.
  • Your staging environment is a palace of lies. Tear it down regularly.
  • Do not underestimate the value of a good manager. Or a good team.
  • The purpose of seniority is to shield, grow, and advocate for people who are less senior
  • A team that doesn’t train juniors is doomed to fail in a very painful way for everyone
  • Writing clearly in plain language is a superpower. It heals more problems than you could ever realize.
  • Sometimes a “fact” that everybody’s always known is just straight up wrong. Other times a fact that everybody’s always known is just as true as they always said it was, and you are going to get very hurt in challenging it. The correct response to both is to document why these beliefs exist.

Cynical Beliefs

  • “Turn it off and on again” is unreasonably effective for solving production incidents, even for the most complex issues. But you’d better be sure your system can handle turning back on again. Do you hear that herd a-thundering?
  • A Slack message to @everyone will probably solve a mysterious production incident 10x faster than your own investigation. First check within the house. (And then be nice to the people that caused the issue! Or else they might not speak up next time.)
  • The world runs on long-forgotten cron jobs
  • Everyone knows temporary solutions are permanent solutions. You will still promise yourself this time will be different, and you will still usually be wrong.
  • There is a reason every sysadmin you know can recite that DNS haiku
  • There are critical services you’ve never heard of that are maintained purely via DMs. You will usually only discover these at least a month after one of the DM recipients resigns.
  • Everyone claims to be against gatekeeping, and yet somehow gatekeeping keeps showing up everywhere.
  • A flaky test suite should be deleted on sight, even for legacy services you’re afraid to touch. Search your feelings, and you know that they aren’t really preventing issues from reaching production. They’re just dead weight, being ignored. It’s more honest to get rid of them first and put higher priority on replacing them.
  • A workplace that regularly rewards heroism is showing a deeper rot. They should not need so many heroes.
  • It takes years to eliminate the effects of brilliant jerks. And over their tenure, they hire others like them, compounding the problem.
  • Customer support knows more about your product than anyone, and yet they are probably systemically devalued. Including by you.

For Protection

  • Toxicity thrives in private spaces. Whenever possible, clash with toxicity in public. This also builds trust with the right people.
  • The more spaces you create, the more will become de facto private. It’s worth consolodating communication channels.
  • None of the above should suggest that private spaces are inherently bad. They are often crucial survival tools.
  • The people that benefit most from you hiding how much you are paid are the people that want to pay you less.
  • Whenever a higher-up says something like “we’re a family here,” they are lying. They know that. Make sure you do too.
  • It is always hard to say no, but it can be necessary. You must make peace with this.
  • It is entirely possible (likely, even) for people and organizations to be lovely to you and nasty to others.
  • For their own survival, the marginalized people around you are sensitive to extremely subtle signals in how others behave. Trust that every one of those signals have been learned in the hardest of ways. Respect them.
  • Literally the only experts about your system in the entire world are all on your team. Trust that you will do the right thing, and probably no one on Earth could have done better at the time. Forgive yourselves when you make mistakes. (And if your organization doesn’t support that, leave.)

For Yourself

  • LOG. OFF.
  • Write a brag document
  • Be humble. You are surrounded by smart, motivated people. Consider the possibility that you might not be the smartest person in the room.
  • Seriously, log off.

Quotes I Repeat Often

  • Less is more / YAGNI
  • Donald Knuth: “Programs are meant to be read by humans and only incidentally for computers to execute.”
  • Goodhart’s Law: When a measure becomes a target, it ceases to be a good measure
  • Hanlon’s Razor: Never attribute to malice that which is adequately explained by stupidity an honest mistake
  • The Zen of Python: Explicit is better than implicit. Simple is better than complex.
  • “Teams are immutable. Every time someone leaves, or joins, you have a new team, not a changed team.” – https://twitter.com/richardadalton/status/569275411508682752
  • “Write tests. Not too many. Mostly integration.” – https://twitter.com/rauchg/status/807626710350839808
  • “Gofmt’s style is nobody’s favourite, but gofmt is everybody’s favourite.” – https://twitter.com/bitfield/status/953395343353315329

Thrown-In Technical Opinions

  • Never take a job involving Infinispan.
  • Java in 2021 is really nice. But it’s difficult to appreciate the improvements when most jobs are about maintaining legacy Java applications. I imagine this will happen to all languages as they approach Java’s age.
  • RxJava is so concise and beautiful. Until you have to debug it. That beauty has a high price.
  • The Spring framework is a weed in your codebase, it will only spread. Dependency injection is probably all you really want anyway, so use a dedicated DI framework like Dagger or Guice.
  • Go is pretty good. Though its pain points all make me appreciate Java a little more.
  • Python has never been my primary language, but it has been the first language I reach for when prototyping from the moment I learned it. That’s amazing.
  • Especially in terms of developer experience, software development can be categorized pretty well as pre- vs post-Ruby on Rails. In some sense, we are all Rails developers now.
  • Ansible is the worst provisioning tool except for all the rest. Terraform is alright too.
  • I wish this industry had standardized on a better DVCS than git. A thousand and one ways to shoot yourself in the foot in the bikeshed.
  • SQL is the only trillion dollar language. Learn it, even if you don’t love it. I love it though.
  • Kubernetes is really nice, but things like Cloud Run are a way better place to start.
  • Little’s Law applies in more places than you realize
  • Microservices are bad, monoliths are alright, SOA is good.
  • Vim is so much more useful than just being that text editor you use when ssh’d somewhere. (Though I am also not yet modern enough to use VS Code.)
  • Anybody who can draw a flowchart can understand technical problems. It’s not magic.
  • Generic high praise: Redis, SQLite, Grafana, Prometheus, PubSub, Dataflow, Honeycomb