The High Price of Cheap Engineering

I have come to believe in the value of good foundations. Without a good foundation, a beautiful mansion built on a crumbling cliff will soon fall into the ocean. Not only that, but it will cost an absolute fortune to go back and fix what should have been done right in the first place.

When startups try to save money with cheap engineering, they pay a high price for that decision. Let’s start with a hypothetical case study of two companies: Company A and Company B. Company A is an established hardware engineering company, while Company B is a promising device startup.

Company A is also looking to build out a new product. While they’d love to build something quickly, they’re willing to take the time to build something right the first time.

Company B’s CEO, a first-time entrepreneur named Danni, just completed a round of fundraising that netted $4MM. Danni’s hope is that she can make it last at least 9 months, but longer is obviously better. Although the company doesn’t have a product yet, her vision is compelling, and she’s eager to get something built as quickly as possible.

Both companies need to hire an engineer to build out a promising product, and they now face a major decision.

Invest upfront in quality, or settle for mediocrity?

Both companies need to decide whether to invest in an experienced, highly qualified Senior Engineer with a successful track record. If not that, they could hire a hard-working newly-graduated computer science student.

A Senior Engineer charges several hundred per hour. The new grad Junior Engineer, looking to get his foot in the door and excited about equity upside, will work for around $80k/year. The Junior Engineer is not just cheaper, but a trusted and well-liked friend of the CEO.

If you were in this situation, who would you hire?

Company A decides to go with the Senior Engineer.

Company B, looking to save money and extend runway a few months, goes with the Junior Engineer.

This is a big decision that will have a significant impact on a company’s long term viability. It’s not just a question of “good” or “bad” code, it’s a question of architecting an appropriate foundation for the company’s core product. Does the engineer have the right experience, training, and skill to set the company up for long-term success?

How cheap engineering up front leads to chaos later

Although many CEOs convince themselves that early code “is just a stopgap” that they “can fix down the road” because right now they “just need to ship”, the truth is that early architecture is going to be with the company until it starts over or falls flat.

Unfortunately, or fortunately depending on how you invest, the code that is initially written at a startup calcifies and remains with the company basically forever. We have worked with dozens of startups, and we consistently see the same thing: new design almost never replaces old design. Instead, new systems are built around and on top of legacy systems. Think about it: if it seems challenging to spend the time and money to write quality code upfront, those challenges and costs will only continue to grow as features get added and the client base grows.

Let’s take a look at Company B, now three years down the road

As it turns out, business for Company B is booming! They’re scaling up and getting real customers and good data. Unfortunately, there’s a major problem: they’ve pivoted the company since they first started, but kept as much of the code as possible with each direction change. It doesn’t help that the early code was written poorly and is only comprehensible to Mike, the Junior Engineer, who built it in the first place.

Every time the company adds a new feature, a lack of tests lead to regressions where old features break. This leads to a fire drill for the engineers who get bombarded with frantic emails and Slack messages at all hours. At this point it barely matters if the engineers are highly skilled or not. The environment is chaos, and everyone is running around like crazy trying to keep it under control.

Because of the constant fire drills, they have to hire more engineers. And because the customer base is so angry about features breaking, they have to hire a team of customer success specialists to pacify the customers. Additionally, a lack of automated testing necessitates an increasingly sprawling quality control team who run manual regression tests with the hopes of finding broken stuff before the customers do.

Now several years in, Company B has never found the chance to take a break and build things right from the ground up. The software feels like a house of cards, so they’re afraid to add the features their customers are begging for. The chaotic work culture means they can’t attract or retain quality talent. They are stuck in a doom loop.

Practically speaking, cheap code is incredibly expensive

Now, Company B’s “cheap” code costs them hundreds of thousands per year in engineering, customer support, and QC to maintain, while driving away talent and delivering an increasingly irritating experience to customers. And the costs of this product will just keep rising.

In other words, their product has become their problem, and now serves as an anchor holding them back. Failure is just around the corner.

Now let’s return to Company A

Company A, as you may have already guessed, hired Senior Engineers like Geocene. As a team, Geocene has learned the lessons of cheap engineering many times. Whether we’re designing software internally or for our clients, we never “save” money with cheap engineering.

To prove this point, take an example of where Geocene put our money where our mouth is. A few years ago, Geocene needed to build a complex product for our own internal use. This wasn’t a client’s project—it was intellectual property that we would own. We could have tried to learn how to build this system ourselves, but our internal engineers were not experienced with this particular technical subject area. So, we hired a senior backend engineer with about 15 years of experience.

Now, this was not a small investment. The engineer’s work cost us nearly a half-million dollars, which was a significant portion of our entire early budget.

Plus, the first half of that half million dollars was spent planning! He didn’t write a single line of code! He also spent a lot time writing tests, so when future code was created it wouldn’t break existing features, and if it did, we’d know immediately what happened and could catch it before our customers did. We also wanted to be sure the product would work even with 100K sensors pointed at it, so we built a synthetic fleet and pointed it at the system to test it.

It took a person-year of effort before we had anything we could show a customer.

After launch, with real data pointed at the system, we then spent another 6 person-months refining the code to fix issues that naturally show up at that point, which we were able to address easily thanks to the number of tests built into the code. Then, without much time or additional money, our product began to work. And that was it – our product just…kept working, and when we needed to add new features, it was easy.

Although the upfront cost was high, the maintenance cost for our product plunged to about $250 per month within two years. In a wild twist, years after spending $500K internally on our own product, we began working with a client who had built almost the exact same platform. However, this client used Company B’s “cheap” engineering strategy. At that point, the client was 5+ years in and spending about $750k/year on an 8-person team of frazzled engineers, customer service reps, and overseas QC, all frantically working to keep the entire project from bursting into flames.

Your product shouldn’t be your problem. Just the opposite—it can be a rocket ship pulling your whole company to the stratosphere.

Geocene is your device engineering partner

Now, our point here is not that we’re so great and that everyone else is doing it wrong. Trust us, the only reason we know what not to do is because we’ve done it the wrong way ourselves!

But now that we’ve learned from it, we hope to help other startups avoid the pitfalls associated with low-cost upfront engineering, because we’ve seen too many companies fall into the void of cheap engineering, never to return.

If you’re a startup thinking about going cheap with your engineering talent, please, please, please reconsider. Sure, it’s objectively cheaper dollar-wise over a short time horizon. But believe us when we say it’s much more expensive down the road.

Even more important than the cost is the fact that dysfunctional code creates dysfunctional companies. Walking through the offices of these companies, you’re reminded of being at the DMV. Nobody wants to be there, everyone is exhausted, the product is beating everyone up, and they can’t get good talent.

The good news is that there is a better way. We’d love to help you lay down a solid foundation for your product that benefits your company year after year. We’ve been there, we know how to do it well, and we know we only succeed when our clients do. Our work isn’t about flash - it’s about solid foundations, clean code, good long-term decisions, and capital spent on just the right portions of the project.