(2023-10-29) I Accidentally Saved Half A Million Dollars

Nikhil Suresh: I Accidentally Saved Half A Million Dollars. I saved my company half a million dollars in about five minutes. This is more money than I've made for my employers over the course of my entire career because this industry is a sham. I clicked about five buttons.

A few years ago, this company decided that it wanted to create an analytics platform. We, of course, did not do any data science, because the organization did not require any data science to be done - what they actually needed to do was fire most of the staff in every team, leaving behind the two people who actually had good domain knowledge, then allow them to collaborate with good engineering teams to build sensible processes and systems. Instead, they hired a bunch of Big Firm Consultants.

I'd ask for some sort of compute to perform some machine learning, or even set up data pipelines.
It never worked. Instead, we were told that we just had to wait for the Advanced Analytics Platform (AAP) to be deployed

By the next December, I had left the organization and the AAP was still nowhere to be seen.

We skip ahead three years. The AAP is finally ready to launch.
Four engineers leave the company in the same week, and I speak with the directors because I know they need a real engineer in and they can't find them.

My conditions - a big pile of money and they had to put me on the AAP team because they're the only team that gets actual toys to play with.

It's an insane dumpster fire spiderweb of technical debt and it's only like one week old.

The entire thing is stitched together by spreadsheets that are parsed by Python, dropped into S3, parsed by Lambdas into more S3, the S3 files are picked up by MongoDB, then MongoDB records are passed by another Lambda into S3, the S3 files are pulled into Snowflake via Snowpipe, the new Snowflake data is pivoted by a Javascript stored procedure into a relational format... and that's how you edit someone's database access.

the kind of operational incompetence that allows you to waste so much money on processing <1TB of data per day that it dwarfs your team's salary.

The next thing to realize is that this platform never really had a chance of making any money for the organization.

The deal is that we pretend the whole team is doing something or other, and we stay within budget because the organization can't afford to spend infinite money on this social fiction.

However, the budget for our database costs was being drastically overrun.

Some quick facts: 1. We use Snowflake as our database, which charges you based on the size of the computer you use to run your queries. 2. You only pay for computers while they're on. 3. We probably run a few thousand queries per week

The computers are set to idle for 10 minutes after every query. I noticed this about a month into joining the team, and suggested we uh... don't have the computers run for like two orders of magnitude longer than they need to for every query.

months later, they finally give me a card that says "Discovery: Optimize Costs". Now I have to optimize costs so that I have something to say at the next standup, and fortunately I know just the thing! I'll test my hypothesis that this is all a sick joke, and I'm going to push the button that I secretly think should obviously have been pushed.

At 4PM on the last day of the week, I ping a chat full of good engineers and no managers to make sure I'm not about to nuke everything, then just do it.

I return to work the following Monday. I suspected that this would save a bunch of money, and guess what, our projected bill dropped from a million to half a million dollars, and everyone is losing their fucking minds.

My team has spun this as a huge cost saving, when really we just applied a fire extinguisher to the pile of money that we had set alight.

While my managers are very happy, they quietly suggest it may be unwise to roll out the changes to all the computers (I only did a few to be safe) because it would oversaturate the department to hear about us all day.

And invite unwelcome questions. The subtext is that if we do this all slowly enough, it might seem like it took a lot of effort instead of just clicking buttons that I said had to be clicked almost a year ago.

Every day, I dread someone asking me to explain what the change was, because I will have to fucking yeet some managers I like under a bus, but they can't resist talking about the change non-stop because it is the closest some of them will ever get to impacting the bottom line.

And many of them are actually decent managers, it's just that this whole department, like many departments, is some sort of weird political PsyOp to get executives promoted. It's cosplaying as a real business and the board thinks the costume is convincing.

By identifying a handful of good engineers and going totally rogue, we outperformed the entire department pretty effortlessly. The competent people are there, just made totally impotent by the organization, and I'm still convinced that this place is probably better than the median organization.

I have even more meetings now because everyone wants to talk about how we saved the money. I had to make a PowerPoint. Kill me.
I would have been better off not doing anything. Let that be a lesson to you. Do you hear me? I applied myself for five minutes against my own better judgement, had the greatest success of my career, and have immediately been punished for it. Learn from my mistakes, I beg of you.