How Intuit decided to think smaller when operating on the cloud

Soon after making the decision to get out of the data center business and into cloud computing about five years ago, Intuit realized that it was going to have to change nearly everything about how it built and managed its application infrastructure to operate in this new world.

At one point in its history, the company behind Quickbooks and TurboTax ran both the largest SQL Server database in the world as well as the largest Oracle database in the world, according to Alex Balazs, a longtime Intuit tech executive who was tapped as the company's new chief technology officer Monday. It decided to "lift and shift" this setup into AWS for the sake of simplicity and expediency, and found out pretty quickly that it wasn't going to work.

"When you own your own data center, you tend to be more monolithic as to how you think about problems, you tend to think about vertical scale," Balazs said in a recent interview conducted before his promotion. "And in the cloud, you think about horizontal scale."

That led Intuit on a path to database sharding, microservices, and multiregion deployment across the world to support the company's financial products, many of which are subject to spikes in demand around the close of a quarter or the frenzied early days of April when tax bills are due in the U.S.

"For us, that's what the journey was; truly understanding that moving from this largely monolithic, coarse-grain world going up into the cloud (is) expensive, it's hard to maintain, it's hard to failover, and then decomposing it into smaller pieces," Balazs said.

The bill has arrived

Intuit's services run primarily in AWS, except for Credit Karma, which the company acquired in 2020 and runs on Google Cloud. Because it has such a close relationship with the federal government around tax preparation and filing, the company still maintains a small colocation center to adhere to certain security practices around those applications, but otherwise everything else is now in the cloud.

"We meet regularly with AWS both at an executive level and our frontline level to make sure that we're leveraging the best of everything as it relates to AWS," Balazs said. "We've learned a lot and we've taught AWS a lot about how to operate in the cloud world where it's multi (availability zone) and multiregion and how you deploy across all that different strata of infrastructure capability."

One of the things Intuit learned was that it needed to spread accountability for cloud-computing costs much more broadly across its organization.

Inside most companies during the data-center days, the CTO or CIO was accountable for managing hardware spending and there was no real incentive for developers and operations teams to use that hardware efficiently. That was especially true at Intuit, which needed to buy enough capacity to handle those high-traffic weeks of the year and otherwise watched its data-center infrastructure idle at 5% to 10% utilization, Balazs said.

"The seasonality of the businesses just forces us to be much more explicit in general, about how to manage your infrastructure and how to take it seriously," he said.

After it shifted to the cloud, Intuit set up a system based around dashboards, targets, and consumption alerts that moved that responsibility to the leaders of teams that were responsible for managing the microservices it implemented after breaking down its monolithic applications.

"The only way that you can really drive accountability is when you actually have to pay the bill," Balazs said. "It gets you to really work on the efficiency of your designs."

Information superhighway

But as part of the decision to break up its massive infrastructure into lots of smaller pieces, Intuit realized it also needed to give its developers a standardized platform for building and deploying new code across the company.

It borrowed a concept from Netflix known as the "paved road," which gives developers incentives to follow a well-worn path when writing new code rather than doing whatever they wanted, which can make it much harder to manage the interactions between microservices.

"Engineering teams are heavily incented to adopt the paved road, because they're being measured against the performance of what you can get on the paved road," Balazs said. Intuit allows some teams to deviate from that paved road if there is a clear tech or business case for adopting the path less traveled by, especially when it comes to programming languages, but they still have to match those performance goals.

Now those teams are playing around with generative AI tools in hopes of improving productivity. While Intuit still considers itself in the experimental phase when it comes to tools like GitHub's Copilot, some interesting applications are starting to pop up.

"There's a lot of different places where you can actually use natural language to accelerate the pace at which things are going," Balazs said. "Things like, can you auto generate the documentation of an API? Or can you actually even auto generate the API itself by inspecting the services themselves?"

But Intuit developers that want to participate in the generative AI era are forced to use its own application runtime called GenOS, which automatically builds unique security and privacy features into Intuit applications developed with generative AI when they are deployed.

"You can't go natively to any of the GenAI providers," Balazs said. Unlike most of Intuit's development rules, the path to apps with generative AI "is a walled road."