You Must Revise Your Infrastructure Spending

Seven signs that you are on the wrong path and tips on getting back to track.

Picture this: Your employees are working hard to deliver new features to customers. The user base is growing very nicely. Based on the positive user feedback, the management has new feature ideas, and the board is happy to see the increase in profit. The company is investing more in its software engineering teams, which are getting bigger budgets.

It holds true for almost any other software company with a good product to sell. That's good, isn't it? What's wrong with that? Well, not much. At least while the company grows dynamically. My favorite mother-in-law has an excellent saying:

A company either grows dynamically or sinks irretrievably.

This is especially true for IT companies that are operating on elevated costs. Of course, what high is objective and depends on many factors, though one thing is common: they shouldn’t have to pay as much as they do.

How do you know that you are on the wrong track, and you need to invest a bit into taking care of what you have done so far? Here are seven signs and suggestions.

1. Excessive Spendings

Let’s start with the most obvious, though often overlooked sign, elevated spendings. Why is it so easy to overlook or ignore this sign if it’s so frequent? Mostly due to lack of monitoring and the human nature. We tend to find reasons for questionable things, even if we shouldn’t. The issue is realized when the company starts to stagnate, or the bills are going way above the budget.

The solution is relatively easy, but requires investigation. The first step is to go down the rabbit hole and track down all cost drivers. This can include checking all billing items on the bill, checking the infrastructure usage, or literally anything that tracks the resource usages.

By knowing which components of the infrastructure contribute the most to the bills, several actions can be taken. From this point, it is highly dependent on the infrastructure what to do next. However, making sure that the right-sized computing resources are used, correct VAT settings are applied on billing accounts, or dangling resources are cleaned up easily can result in tens of thousands of dollars savings annually.

2. High Maintenance Costs

It is challenging to pinpoint the cause of increased maintenance costs, as it can be somewhat natural in the case of complex systems. Many times, the processes the teams are following are inefficient. In other cases, the system is poorly or overengineered. Also, it can happen that not the right tools are used or not the right team is doing the maintenance work.

In this case, all problems need a unique solution. There is no one fit for all. A good first step is to revise the processes in collaboration with the teams that are participating in maintenance. In many cases, they are not raising issues with the processes, or they don’t even realize that their processes are inefficient.

Then, check for resource usage. Are the resources utilized as they should have been? Is there any under or over provisioned computing instances?

If you found no resource utilization issues, the processes in place are efficient, but the maintenance cost is still high, it may be worth investigating if you are using the right technology. For example, setting up Kubernetes for a website just not makes sense.

3. Inefficient Processes

How can processes influence IT expenses? If you don’t know the answer to this straight off, you need to get in touch with me.

A company without efficient processes is doomed. That’s true for its teams as well. No, we are not talking about micromanagement here. It is about defining the way a team works and keeps itself accountable to other teams.

If the infrastructure team (or any other teams) has no processes for regular, scheduled, and most importantly, uninterrupted maintenance time, no matter how clever the team or well established the infrastructure, it will get rusty.

The solution is introducing dedicated maintenance time with a plan the team can stick to. However, everyone in the company should respect this maintenance time. Even the CEO should not intervene in that. Why? Because otherwise it is a wasted time if the team is dragged back to do the regular feature shipping or bug fixing.

Of course, there are scenarios when this cannot be avoided (like an outage), but that should be rare to be able to count on one hand during the course of the year.

4. Lack of Standardization

Is the “not invented here (NIH) syndrome” familiar to you? NIH is basically the tendency for management or teams to reject any idea that did not originate within the organization.

In the beginning, custom tooling may cost less and comes with less maintenance, though after some time, this trend will turn to the opposite. Training people for in-house solutions, writing extra documentation, and not using the benefits provided by using standardized solutions, which are mostly open-sourced.

Nowadays, open-source communities provide solutions to almost anything a company may need. These communities are taking care of bug fixes, shipping new features, maintaining the documentation, keeping the tooling up-to-date, and so much more.

Think about Docker or Kubernetes. If a company had to ship the containerization or orchestration environment, how many teams would have to work on that for how long? What about Prometheus or Grafana? These products are maintained by hundreds of developers every day. Does your company have these spare resources?

Letting the fear of not inventing everything go, and starting using standardized and open-source tooling, will definitely help lower the costs. Sure, not all open-source products are completely free. Open-source, per se, doesn’t mean it is free on any scale with no restrictions. However, still saves a company a lot of effort, time, and money.

5. Obsolete Technology

The mantra of “what works doesn't need to be changed” is not true in this case. Sticking with outdated technology means more fixing and upkeep. When parts get old and the vendors stop lending a hand, you end up shelling out more just to keep things running.

Technology moves fast, and old systems can't always keep up. This can force companies to splurge on pricey upgrades or come up with clever hacks to make sure everything still plays nice together, all adding to the bill.

Without dusting off and regularly patching old technology, the company-wide open to hackers. Also, it can really put the brakes on your team's productivity. Slow interfaces, laggy processing, and compatibility issues with newer tools can turn work into a slog.

The solution to the problem is simple, but time-consuming. Start upgrading the old tech stack. It comes with planning, resource allocations, focused work, increased maintenance time, but keep in mind: at some point, the tech debt must be paid.

Acknowledging and being conscious about the tech debt is not a shame. It is simply responsible thinking. Nothing is wrong with that.

6. Uncontrolled Growth

Let's talk about the wild side of growth — the uncontrolled kind. When things start booming without a leash, it's like the wild west out there.

Uncontrolled growth tends to spread like wild weeds. It's like trying to manage an overgrown garden, where you spend all your time cutting back excess instead of nurturing what's important. When growth happens too fast, the company can find itself scrambling to keep up with rising demands, whether it's for customers, supplies, staff, or even office space.

Quality control and peer review becomes shaky amidst unchecked expansion. With all the focus on getting bigger, there's a risk of letting craftsmanship slip or dropping the ball on multiple fields.

For the infrastructure, this means bad decisions, such as renting cloud computing resources that are not needed, forgetting about unused temporary instances, over-provisioned instances, poor resource utilization and so much more.

Dealing with uncontrolled growth requires a joined effort of multiple teams, and the issue should be mitigated together. While mitigating the issue, make sure to look out for bad practices, over provisioned instances, dangling resources, and so on. These can be cleaned up easily.

7. Overestimated Growth

In contrast to uncontrolled growth, the other side of the coin is overestimated growth. When a company is releasing a product, it is often believing that it will get a huge traction from day one. However, be honest, it’s not the truth.

The estimated user base is communicated to the engineering teams, who are designing the infrastructure accordingly. And here is the key word: accordingly. It is not their role to (always) second guess the management’s expectations, so they will scale the systems up as needed.

The upscaled resources evidently result in higher bills. By simply revising the actual user base, the system can be scaled down as required, lowering the bills. It must be noted that this requires some sort of monitoring and reporting. Without knowing the exact usage of the systems, this cannot be done.

Conclusion

Running a software company is like steering through a constantly changing environment. Success comes from balancing new ideas and careful management. As your customer base grows and profits increase, it might be tempting to expand fast. But there are risks you need to watch out for that could mess up your plans.

Dealing with overspending, high costs to keep things running, and inefficient ways of doing things needs more than just looking into the problem. You have to really work at making things smoother. Using standard methods and getting rid of old ways of doing things can make your company more effective. And if your company is growing too fast, you need to rethink your strategy.

Success isn't just about moving forward. It's also about managing your resources wisely and creating a workplace where being able to change and plan ahead is important. By keeping a close eye on things and being ready to change direction, software companies can keep growing and coming up with new ideas sustainably.