The limiting factor—how to design an AI software factory for speed | Geoff Charles (Ramp CPO)

Hello everyone. First off, it's been absolutely amazing to see the speakers before me. I want to give one more round of applause for the incredible amount of content and preparation I'm inspired. I've written my speech three times, I learned how to do storytelling, and so here's a story for you.

At Ramp, we're known about velocity. We love speed. I love speed. I love speed so much that I once signed up to race cars at Le Mans. I'm just kidding. Lemons. 24 hours of Lemons. It's a race where you buy a $500 car and you race around for 24 hours, hoping to win.

Unfortunately for me, my car did not make it. I crashed. I crashed hard. I was okay. The car, unfortunately, was not. Now, this crash was entirely my fault as the driver. But in real professional racing, the driver is rarely the problem.

In fact, the driver is only 15% of the impact on the race. The real impact is the interaction between the driver and the car, between the car and the team. In other words, winning isn't just about driving faster.

The best drivers in the world can only perform up to the level of the system. Winning the race is about removing the bottlenecks around the driving. That's the key point I want to drive today. Take, for example, the pit stop.

In the 1950s, it took 67 seconds to change tires. Today, 1.8 seconds. How did 67 seconds turn into 1.8? Certainly not by asking the mechanic to work 37 times harder. All you leaders, you know what I'm talking about.

They found the bottleneck and they iterated to remove them. Specialized functions, better technology, more practice. And F1 is relentless. 90% of the parts of a car change every year. Only 10% of the 16,000 parts of an F1 vehicle carry over to the next year.

Now, I want you to also think about: what if 90% of your code changed every year? Because that is the bar. Why am I telling you all this? We're in a race. Speed is everything. Execution is everything. And we're in a race to build with AI.

For the race car, it starts when the car comes in for new tires and the clock stops when it leaves. For us, the clock starts when there's pain and it leaves when the customer has a product that solves it.

AI simply removes the bottleneck, but moves it. And the best team, the winning team, is the team that can find the bottleneck faster, remove it, and move on to the next one. Now, we all know the product development lifecycle.

We identify, we define, we spend a long time building, and then we improve. Now, the engineers. Fantastic engineers. They've been able to automate a lot of their work. And coding is now much easier. So here's a tip: be more like engineers.

Be as lazy as engineers. Because the reality is that the bottleneck has now shifted to us. We need more things to define, we need more things to collaborate, we need more things to coordinate, more things to ship, more things to test, more things to release.

The bottleneck has shifted to us, and so we need to invest in our own factory. The speed of this loop determines the outcome of the race. And the question becomes: how quickly can you identify the bottleneck and redesign the factory around it?

What I'm going to show you today are five of these steps and how Ramp has gone far in iterating and automating through it. The first step is identify. Identify the problem. Identify the pain,right? The issue is that that pain lives somewhere.

And there's a lot of pain. A lot of data. Gong, Zendesk, LogRocket, a survey, a very angry email to an executive. The problem is that context lives all over the place. Too many silos, too many opinions. And so the first bottleneck to identify is how to sift through the noise.

And LLMs are great, but a 1 million token window? That's less than 0.5% of Gong transcripts at Ramp. So we started small. We created a hate channel, and every day it posts all the lovely things that our customers say.

And yes, these are real quotes.

That got out of hand really quickly.

Yes, I know. I am not proud of it. No one's perfect. And so we had to redesign from scratch. We built an actual customer insight agent. That agent pulls from all the data sources at a company. It uses traditional ETL and pipelines.

It uses actual vector search. It has clustering around the same context. It understands our product. It understands our teams. It understands the features that we have. And it's accessible to the rest of the organization. Once we had this working, we asked ourselves, how do we get this to as many hands as possible?

And so we, again, experimented. A Slack agent where you can ask questions. A nice HTML dashboard that people can log in and read. My favorite one was the podcast. The Hate Podcast. Great way to start your day. You listen in.

100 customers yelling at you about all the ways your product is broken. The concept is simple,right? Get people access to this data as fast as possible. That doesn't mean don't talk to customers. It means you actually identify exactly which customers to talk to.

Because the data is traceable. Okay, once you've found what to focus on, the next step is to define your product. How do you go from a concept to a fully scoped idea? And so we turn to AI,right? And the AIs all kind of look the same.

They ask you, what do you want to build? And you stare. You have this super intelligent being asking you what you want to build. Is that really the best question to ask? Is that really the best starting point?

Right? The goal with AI is to connect it to your systems. That is how you're going to make AI extremely, extremely actionable. And so we did that. We built our own AI agent called Glass, and we gave it all the context it needs to do the work.

It connects to all our systems. So it understands both the data with Snowflake. It understands our user research. And it can nail exactly the job to be done that our customers are asking for. That gives us specificity with both qualitative and quantitative data.

Glass also understands our product strategy. It understands how we define product specs. It understands our code base. And so instead of a PM bothering an engineer asking them, is this possible? Will this break something? What am I missing?

You ask AI. AI can be your best thought partner. So now AI is your tech lead. And finally, you can build a prototype. Because it understands our product principles. It understands our design system. It understands our code base.

You can actually define and build a prototype that actually works within your product. Your engineering team don't want any of these individual things. They don't want your prototype. It's useless. They don't want your long spec. That's useless. They want the combination of qualitative and quantitative data to convince them that this is a problem.

The actual requirements that they can use for their coding agents. And a prototype that they can actually get inspired by. That is the next contract. So now you have a strong foundation for which to go. The next step is to build.

I mentioned that building is no longer the bottleneck. Obviously, it's a bit more nuanced than that. When we focus on one bottleneck, the next one appears. The first bottleneck was coding. Everyone should have a coding agent at this point.

We built Inspect. Inspect is our own coding agent. And the reason why we built it ourselves was because we wanted to have a strong harness. And we wanted to make sure that our coding agent really understood our code base.

It works inside of Slack. It's fully provisioned. It can run under 5 seconds. And it can return an actual product in a deploy preview so that the actual product manager can actually use and interact with the code that's being built.

Anyone at Ramp can now code. Inspect has a million sessions. 75% of our PRs is actually built by Inspect. And a thousand of those PRs over the last month were submitted by a non-engineer. One tip here, though, is that coding agents are really good when you have a strong architecture and a strong code base.

This expands how much of your company can actually use AI. You now are shipping a lot of code. The next step is code reviews. Becomes the next bottleneck. Engineers are just constantly combing through a ton of code. And so we focused on that.

We built ReviewBuddy. ReviewBuddy fully understands our code base. Fully understands our actual quality checks, our security concerns. It finds theright human reviewer to loop in. And it also understands the context. All the prompts that led you to build that code.

So it can actually audit how you used AI in the first place. So the review and the collaboration is now grounded in visible context. 93% of our PRs are now automatically handled by ReviewBuddy so that the highest quality engineers are spending time on the 7% of PRs that truly matter.

That have a lot of questions around, are we building this correctly? Are there any risks? The next bottleneck is testing. Previously, PMs. I'm sure you're familiar with this. We spin up a QA environment. We try to tweak different fixtures.

We click through the screens. We make sure that the engineer didn't skip out on that one feature we really liked. We automated that. We built Testo. Testo is a browser-based QA agent. It spins up the product in 100 different combinations based on the actual production data that we have access to.

And it runs in the product like a user. We give it instructions. We say, hey, pay an invoice, but amortize it. And it clicks around and actually gives us feedback. Both blocking feedback, bugs, issues, blockers. And honestly, extremely thoughtful design and qualitative feedback around things that were confusing.

Things that break our design system or break our design language. And that goes back into the product development lifecycle. Testo in the last 30 days caught 425 bugs.

I think these are bugs better caught by us than our customers. And they sent it back through the loop to fix. So now you're shipping a lot of product. The next step is to coordinate. How do you make sure that everyone in your company is in the loop?

Things are moving so fast. You're shipping so many different things. The next bottleneck becomes human attention. We've all been there. PMs being inundated with notifications. And so our job becomes coordination. I see a few people coordinatingright now. Too much process,right?

And it slows down the builders. Too little process and everyone is confused. And so human attention becomes that bottleneck. How do you solve this? Well, every question is an API. Every question is an API. And you architect a system.

We build gadgets. Gadget essentially understands the intent of these questions and connects it to the formal record. The roadmap in Notion. The specs in Notion. The customer calls in Notion. Slack. Linear. Our tickets. The principle here is simple.

For you to empower an agent, the agent needs to be able to understand and read the organization. The organization needs to be legible to your agents. I'll give you an example. A question is like, what's the status of this project?

What's the status of this launch? Are we on track? An agent handles that. It understands the full picture. It understands who owns the next steps. It answers with evidence. But not only that, the agent updates our roadmap. The agent gives status updates.

The agent pings people who are late on their deliverables. There's a lot of fun ways you can actually scale. Or another question. Sales. Asking questions about, what is this feature? How do I sell it? Is it available in Brazil?

What's the use case? What's the price? All of these things are fully understood. And Gadget, if it does not understand, routes it to theright person. But we can take a step further. A launch. Gadget understands the product. And so it can write the help center article.

It can help write the blog post. It can help write the customer email. So on and so forth. Again, start one place and you'll see how far it expands. 85% of questions that are being asked to PMs now are fully answered with AI.

And those that are not are answered and fed back into the system. So you're shipping a lot of products now. Everyone's fully aware and coordinated. Everything is perfect. But that's just the start,right? You are constantly improving on the product.

And the loop starts again. Now, PMs, we naturally gravitate towards the small reactive things. The things that we have high confidence in. The things that are easy. That are visible. The dopamine hit. Oh, we've solved something. I think that is a huge mistake.

Claire, I think, had a great talk around how to be ambitious. You're not ambitious because you're focusing on small things. You need to automate your way out of these small loops. And so we did just that. For most small things, an AI fully runs the loop.

It routes it to theright team. It matches it with your backlog and linear. It dedupes. It counts. It ranks. It runs the plan. It has a human in the loop through Slack around, hey, is this ready to go?

I'm ready to code. It writes the code. It runs through test and CI/CD just like we talked about. And when it launches, it runs through the knowledge base. These are small loops. And humans are in the loop only just barely.

Thousands of little loops with thousands of little things. So you can focus on the big ones. Thanks to these autonomous loops, 60% of UX issues that are identified by either a customer or a salesperson or a CX person or ourselves are fixed within 24 hours.

So at this stage, you're probably asking yourselves three questions. The first question is, okay, how do we know we're moving faster? Right? How do we move faster with quality? After all, we talk about product velocity. How do you measure it?

It's certainly harder than a lap time at a racetrack. Well, some things are not easily quantifiable. Niki Lauda in 1974 took a lap with a Ferrari car and told Enzo, this car is a piece of shit. It drives poorly.

It handles poorly. It brakes poorly. Got out of the car. Went to the engineer. Fixed it. That is not dissimilar to the culture at Ramp. I'll start there. But the point I'm trying to make here is that you will know if you're moving fast enough if you actually hire a driver that knows what speed looks like.

And you put them in charge. And you actually allow them to challenge you. That might be hard for some leaders, but that's what we need to do. To be challenged by people who know what great speed looks like.

The second question might be, what about our budget? We have limited resources. We have limited resources in terms of engineers or simply the tokens. How do we compete with companies with no constraints? In 2006, Audi had a bad car.

A very slow car. And so they asked themselves, how can we win if we can't go faster than anyone else? Their answer, fuel efficiency. Fewer stops. More time on the track. And they won them all three years in a row.

So constraints force you to choose a dimension in which you can be world-class. So embrace your constraints, but not your bottlenecks. Find the one thing that you're really, really good at. Find that one bottleneck that you think can 10x your company.

And just start there. And the last question, something that we've talked a lot about today, is the future role of product. Are we automating our jobs? What's going to happen if these loops are running without PMs? Well, the first thing I'll say is, I think it's amazing for product managers to be out of the loop for a lot of things that engineers can drive.

This helps us get a ton of leverage. And I don't think we should be scared about that. Our jobs, though, I think will evolve in three ways. There are three tracks that I see for product managers. The first track is the technical track.

And something that I don't think we are investing nearly enough. It's the PM that's building the actual factory. The technical product manager that is identifying bottlenecks and removing the drag in your organization. They're not shipping products for the customer.

They're shipping the product that helps you build the product for the customer or helps AI do so. The second role, arguably one of the most important roles, certainly the one that gets the most limelight, is the tastemaker. It's the driver.

Ultimately, you need someone who's holding the steering wheel and doing things that AI cannot do and holding the bar for what great product, great taste looks like. And finally, the GM. I think PMs will absolutely evolve and expand their role to not just the product organization, but the marketing organization, the sales organization, the growth organizations, the operational organizations.

PMs will become GMs. And they will own the actual business outcome and lead the entire function. So we walked through a lot. I'll recap three main takeaways. The first is that speed is not about just coding faster. It's identifying and removing the bottleneck in your build process.

And everyone has one. Scientifically, there is always one. Two, as you actually solve that first bottleneck, the next one appears. And your job is to move through them as fast as possible. And lastly, us as product leaders, we need to obsess a little bit less about the product that we're delivering and a little bit more about the factory that helps us build products faster.

Now, I just walked through a lot here. A lot of great things that we built. And I did so because the reality is that everything you've seen here is outdated. Just like F1, teams will copy each other. I want you to copy us.

I want you to actually hopefully surpass many of the things you see today. And I'm sure people here have gone much further. The reality is that it's all about the next season. And so I'll leave with this. Enzo Ferrari said, the best Ferrari I ever built is the next one.

The best product you'll ever build is the next product that you'll launch. That begins in the factory. That begins in the software factory that you build today. I hope you copy us. I hope we get to copy you.

Let's share more together as a community around how we are building these products so we can deliver more for our customers. And I hope you all stay safe out there. Thank you.