Jason Richards

Welcome to Orbit, the Hg podcast series, where we talk to leaders of technology businesses and hear how they have built some of the most successful software companies across the world. I am Jason Richards, Head of Portfolio Technology at Hg, and I am delighted today to be joined by Russell Kaplan, founding president of Cognition. We'll hear more shortly, but Cognition's Devin product is an AI software engineer that's capable of completing programming tasks and collaborating with human developers. Cognition is well known to us here at Hg, we have over 20 portfolio companies currently engaged with cognition, leveraging Devin.

Russell is fresh off the stage from his high-impact keynote here at Hg's Digital Forum. Russell, thanks for being here. Really appreciate it.

Russell Kaplan

Thanks for having me.

Jason Richards

Before we get started and talk about Cognition, a number of our listeners may not be familiar with yourself and your background, so I wondered if you could perhaps introduce yourself for a second.

Russell Kaplan

Yeah, absolutely. I'm from New York originally, I grew up loving programming as a little kid, and I got really excited about AI in high school. I was taking all these classes on Udacity, sort of like an online courses website, and they had a prize for, if you took the most online classes and recruited the most number of people to do it, you could win a trip to California, to Silicon Valley, and get a ride in what was then called the Google Self-Driving Car project. And so I took a lot of classes and I recruited a lot of people to take a lot of classes and I was lucky to basically get this trip and I got to visit Stanford and go to the Bay Area and take a ride in the Google Self-Driving Car. I remember it at that time, it was just doing loops around Google's campus and it almost worked perfectly until it slammed on the brakes when a low-hanging tree branch was in the way.

So at that moment I was like, "Okay, this AI thing seems really cool, I want to work on this." So I ended up going to Stanford. I was in the vision lab there. I was a researcher advised by Fei-Fei Li, working on steganography and other kind computer vision techniques. I was trying to decide, "Should I go do a PhD or go into industry?" So I figured, "All right, I'll do the master's program and then I'll also go work in applied machine learning and see which is more fun." And I went to Tesla on the autopilot team and I had much more fun at that than writing research papers. So I went to Tesla full time.

I was working on the vision neural network. So specifically I did a lot of our multitask learning, our large-scale distributed training infrastructure and working on the intersection of sort of applied ML and systems there. And then from there I ended up leaving. I started a company doing real-time computer vision that ended up getting acquired by Scale AI, where I led the machine learning organization, and about a year ago, left to start working on Cognition. It's been a really exciting journey since then.

Jason Richards

Fantastic. I mean, some great experiences during that time. And so how have they sort of... Any sort of pivotal moments that laid the foundations for what you're doing at Cognition now that you reflect on and think, "That's been really key for the mission, what we want to achieve and how we're going to take ourselves forward"?

Russell Kaplan

Yeah, so for me, one of the reasons I joined Cognition and started working on this problem was previously working at Scale was a really good vantage point into how the AI industry is evolving. And for those who don't know, Scale, it's a data for AI company. So they do labeling for a lot of the foundation model labs to make models better in post-training. And because of that leading the machine learning team there, I had very good visibility into all the data pipelines of all these different foundation model labs. And what became really clear is that these models are just going to get better at everything.

But in particular, they're getting really good at coding. And there's a few reasons for that, but one of the core ones is that coding is one of the few domains where you can automatically improve via self-play relatively more straightforwardly. You can have a model write code and then you can actually run the code on a computer and you can see, "Does that code work or not?" Or you can have a model write code and write tests and see, "Are these compatible or not?" And because you have that automatic feedback loop, you can scale data acquisition much more aggressively than another domains. And so I became very bullish on, "Hey, this coding thing is going to get really good." And just having loved programming since I was a kid, it was a really exciting problem.

Jason Richards

Well, because had that competitive programming background. So I guess that sort of interest, that passion coupled with... This has got... It sounds like boundaries where you're going to be able to tell quite quickly the impact of the new processes, the way you're leveraging the models, the new technology. So it sounds like... It's a great use case, really, for the foundations of Cognition and how you've leveraged technology.

Russell Kaplan

And I think AI always sort of works better in more narrow artificial domains at first than in the more general complicated domains. And yeah, my background in competitive programming, I did something called American Computer Science League in high school, which is a very niche, not common thing, but we won. Yeah. But we won states my year and you kind of had to do a mix of programming and then also sort of handwritten problem solving. And then in college my focus was mostly on hackathons. So it's a bit more of a creative domain where you're not necessarily algorithmically solving a set of seven questions, you're more building something cool.

And so I started Stanford's hackathon called TreeHacks, and we really tried to encourage an environment of just ship cool stuff. And AI, it's so good for that. And so it's been really, really exciting to see. We're at the point with coding and most things in machine learning progress where the moment you can kind of codify it as a benchmark, then you can hill climb it and make it much better. And so I think that's obviously now happened in competitive programming. I think it's happening increasingly in more creative programming as well too.

Jason Richards

And I think sort of that depth of technical knowledge understanding is really cool, really key. But what we are finding, and I think that's what's going to be helpful for our listeners is as we are entering into... Already in, but as we're progressing in this era of AI and AI enablement, any sort of behavioral attributes that you think have been really important to you... Like we talk about the importance of being inquisitive and curious and brave and those kind of things. So from the more of the innate behavioral aspects, how would you describe what's helped you drive forward in the manner and the pace at which you have, but then perhaps also what you think about and what you look for in others?

Russell Kaplan

Yeah, I think curiosity is a big one. In many ways, the pace of progress now is so fast that if you're not curious about it, by the time you look up, the best way to do things are going to be totally different. And so that's, I think, what part of what drew me to machine learning in the first place, was just sort of curiosity about how does intelligence work and how is it possible to maybe one day sort of simulate this in Silicon. We see this a lot with Devin too, where it doesn't really matter if you're a junior engineer or a senior engineer or a staff engineer. What matters is, are you curious and leaning into what is possible in the future? And if so, regardless of your seniority level, you're going to be able to make these sort of modern AI tools do amazing things.

Jason Richards

What about that bravery aspect? Because us, I think... I think it's an era where we don't want people to be constrained by the norm, right? Break the habits, think about how you can do things differently, whether you want to call it visioning or whatever, but what's your view on that and the criticality of being brave right now?

Russell Kaplan

Interesting. In many ways, one of the reasons I think coding has taken off as the killer use case of generative AI and agents is that it's a little bit safer than other areas. If you're deploying AI that's directly customer facing, for example, that's nerve wracking, right? Because now you have one system go wrong and the customers are going to be upset. There's a lot of risk. But it turns out that most software engineering organizations are already quite robust to occasional errors, whether it's from an AI or a human. That's why we have a software development life cycle where people submit pull requests and people review them. So in some ways, because of the sort of nervousness, especially in larger organizations about really rolling out AI in a big way quickly, I think that's why coding is going so fast, is that it's a little bit safer. It's a little bit safer to do. But as an individual, I think the real risk is not doing this stuff. The real risk is that your head's down, you're not thinking about how your workflow is evolving, and then you look up, and it's completely changed.

Jason Richards

Yeah, I think the other flip side of this, which you sort of just touched on, is this isn't about being reckless either. So with this power comes responsibility because things happen much faster with a great degree of autonomous sort of action. You can't just be hands off. You have to still have the responsibility, I guess, for what you are engaging with what you are asking for, and ultimately, what you're creating.

Russell Kaplan

That sort of human sign-off sample approval, it's probably going to be around for a while. Devin, for example, when it submits a pull request, it always says, "Hey, Devin has created this pull request, but actually Jason wrote the request to make it, and here's the full trace of that log." And in some ways, actually, the AI agents are more auditable and more traceable than maybe this sort of autocomplete style coding systems because you have very clear delineation between, "All right, which lines of code did the human write and which lines of code did the AI write?" And to your point, I think especially the large organizations that we work with, they're very sensitive about this because they want to see, "Hey, what's the audit trail of what we did and how can we justify it? How can we stand behind it?"

Jason Richards

To be at the leading edge of where we are, I think there's a real importance of having accessibility to talent. And so within your role, that sort of responsibility for providing that leadership and that leadership philosophy, alongside Scott, I guess, across Cognition, so... Yeah. How would you describe that leadership philosophy where you have this new world, this highly talented people sort of massively empowered through the technology? What's your leadership philosophy?

Russell Kaplan

Talent is everything. And recruiting is the most important thing we do at Cognition. I think we take building the team extremely seriously. And we think that in the sort of modern world of AI progress, you can do extraordinary things with very small teams. And I think it's a lesson that's sort of been beating over my head in my career a few times. I remember in my first sort of full-time job at Tesla, I finally at one point got invited to the weekly skip level meetings where Elon would go to each engineer and say, "What are you doing? And how can I help you go faster?" And the very first time I was sort of invited to one of these meetings and we were going around and saying what we were doing and how we could go faster, it became my turn and I shared what I was working on.

I said, "I think we really could use some more people to help build our internal tools because our internal tools are kind a disaster right now. And it can be a lot more productive if we kind of had a few more folks." And then he immediately said, "Stop. That's the wrong answer. The answer is never more people." Actually, exceptional outcomes are accomplished by small teams of highly technical people, and you never need more people to accomplish a goal. And I was like, "Oh, okay, okay, okay. I got this wrong." What was ironic was that my next meeting, it was a meeting of just...

Russell Kaplan

Which was ironic was that my next meeting, it was a meeting of just me and Elon and my manager Andrej. And the three of us were sitting down to talk about Tesla's recruiting strategy. And Elon said, "We have to get the best people. It's so important for us to go recruit new, good people." And so I kind of took that, I think lesson pretty directly.

And at Cognition, we spend extraordinary amounts of effort trying to go find and recruit the best possible folks in the world to the point where we've had people from unusual backgrounds, people who maybe are quite junior in their career and we'll go fly and meet their family and talk about, "Hey, is this the right opportunity for your son or your daughter to join here?" And we view it as probably the most important thing we do.

Jason Richards

Interesting. And that's a lovely example you gave where Elon had that influence and impact. Are there any other sort of examples where perhaps individuals have helped shape your way of thinking or something as critical as that? So the answer's not more people kind of thing. Is there anybody else you would say has helped shape what you do?

Russell Kaplan

I think I've been really lucky to work with great leaders in the companies I've worked at. And so even at Tesla for example, my manager was Andrej Karpathy and he's an amazing sort of AI leader and really well-known for all of the content and work he's done. But what I really took away from him, I think the first thing, my desk was next to his, and so we'd be working on the autopilot system all the time. And whenever I looked over at his desk, he was always just labeling data. It was crazy. I was like, "Aren't you the manager of the whole team? We have a whole data labeling team." I wanted to ask him, why are you just spending so much time just labeling the data, looking at these? He's like, "The data is the ceiling on the performance of the entire system. If we have inaccurate labels, it's going to bleed downstream into inaccurate models and it's worse than that. It's not going to be easy to catch."

And so this focus on getting the details right at the root level of the system, especially when it comes to data, it became sort of a big focus for me. And that's one of the reasons that when we sold our startup, we went to Scale because Scale was building the data layer for AI. And the importance of the quality of data, the inputs to your system defining the outputs of your system, I think it's even more true today. And that's what we see with Devin.

Jason Richards

It's interesting. We're seeing something similar. So that attention to detail, but also ensuring that the basics are in place and are there because obviously Devin is out seeking that information and is largely, I guess going to be responding to what it reads, what it observes some of the sort decision-making. So our view is as the leaders, we also have to have some attention to that detail and make sure that there's not fundamental gaps that might send Devin off on a wrong path or something like that. I guess that's important.

Russell Kaplan

Definitely. And I think that's the sort of the benefit and the challenge of having a team of AI engineers is you can get so much done that are you really going to review all of it carefully? But maybe for certain systems, you want to and it's important. And I think what we see is that the job of the human engineer, it's really becoming the tech lead manager where you're managing a team of AI junior engineers and then you're more of a reviewer and a coach, but getting the details right then becomes the central.

Jason Richards

I think what people will be really intrigued at is you are clearly an AI-native organization. What's a day like in your organization? Just give us a feeling for how your organization works. And I think what's really interesting as well is, and it's not that Cognition is a competitor to us. Clearly, we see it as an important part of how we are evolving and growing ourselves, but there may be other organizations that are fast-paced, well-funded startups, AI natives that could become competitors to HD organizations. So how should we think about what you guys are, how you're functioning, maybe the pace of what you do? Those kinds of things.

Russell Kaplan

So typical day for us. We're almost all engineers, and so people come to the office late in the morning and then they stay really late at night. So we're on a bit of a shifted schedule. And usually the first thing you do when you come in is you see, all right, what did my Devins do last night while I was sleeping? You've probably kicked off a bunch Devins for different tasks and different things that have to get done. Some of them might be small improvements, some of might be big architectural changes. So you're probably reviewing a set of, here are all the changes that I asked for and what am I doing?

And once you kind of get that in place as an engineer, we organize in different pods and we sort of deliberately scramble them every week so that every engineer gets more exposure to different parts of the system, and we can constantly adapt and evolve the product based on the edge of what's possible. That's been a really important principle for us, is being willing to just rethink everything from first principles the moment we make a new discovery or some external AI system gets better or something new becomes possible. Because the right form factor for AI software engineering, it's a function of the edge of what's possible from an intelligence perspective. So we're managing these Devins, we come in, we review what's going on, we merge some stuff, and then we kind of have these very flexible different engineering pods where different people are working on different things. There's a lot of bottoms-up creativity ideas of what people want to work on. We also have a lot of customers who share feedback on what they want, and we try to strike the right balance of doing both of those things.

People are basically just coding late into the evening themselves, working on the hard stuff, delegating Devins for the medium and the easier stuff, and then reviewing and getting stuff done. And I think the really fun part that we have is that these small teams can get extraordinary amounts of work done now with the leverage of AI. And so we find that there's a huge benefit to having only dozens of people, but hundreds of Devins because you also get rid of the communication overhead, you get rid of the bloat, you don't need as many meetings. There's all these second order benefits too.

Jason Richards

Certainly one of the things we've seen is the whole thing around geographic boundaries, time zones just largely become irrelevant. What would be really useful, and I'd love to come back in a minute to how you guys are actually leveraging Devin yourselves, but I'm conscious perhaps we haven't really explained yet to the listeners what Devin is. So maybe if you don't mind, could you provide an overview of Devin?

Russell Kaplan

Absolutely. So Devin is an AI software engineering agent. And so you can think on the sort of spectrum of AI developer tools, maybe the first wave that really popularized them was GitHub Copilot. And GitHub Copilot's an amazing tool. It's almost like autocomplete on steroids. You're in your editor, you're writing code and you're getting real-time suggestions with now chat, et cetera. Devin is sort of at the other end of the spectrum of autonomous software engineering agent, so it's more asynchronous than synchronous and it's more about delegating entire tasks that then you give to an AI system and then Devin goes and gets them done in the background while you're working on other things. And then Devin comes back and says, Hey Jason, this is what you asked me to work on. Not only have I written the code, but I've also tested it out. I know there was an error over here. I fixed that. I've iterated on some more and now I have a complete pull request. So that's how we plug in with developers or teams and people use Devin to ship all sorts of things.

Jason Richards

Yeah, cool. And is there a particular skill set that's required in order to leverage Devin, get the most out of Devin?

Russell Kaplan

So we designed Devin to be used by engineers because we think what's going to happen is that engineers today are going to be able to get way more done per person, per unit time in the future thanks to AI. But AI is still not perfect. And so you still have sometimes that last 10%, that last 5% that a human really needs to do. And then the same for the more complex tasks. What we had a customer describe Devin to us as, "An infinite army of junior engineers," which I think is a reasonable heuristic for what it's like.

Jason Richards

On the junior engineer aspect, I guess what's the future? Do you see that as being where we are and that's going to evolve and expand potentially becoming more and more sophisticated? I guess that's the natural progression?

Russell Kaplan

I think so. Our main research goal as a company is get Devin promoted to senior engineer.

Jason Richards

Yeah. Yeah. I love that. I love the idea of having Devin promoted through its various stages, but I guess you've got similar to how you might think about the progression of a new entry-level junior engineer, what you are willing to enable that person to do, and then progressively how you move them up ranks. Probably not dissimilar, right?

Russell Kaplan

It's actually sort of a shockingly relevant analogy. We have a lot of customers who they ask us, "Well, how should I think about Devin for this? Or how should I think about Devin for that?" And the answer is almost always, "Well, how do you think about working with a junior engineer and team for this and that?"

And I'll give you a specific example when I was at Scale. The engineering leadership team, we would use these leveling guides for promotions and career development. And the first row of our leveling guide at Scale was scoped in terms of the outcomes that you could achieve as an individual. And so our expectations on an L-III engineer at scale were that you could take to find tasks and go and get them done. At the L-IV level. You could take defined outcomes and get them done. And that would involve breaking them into tasks yourself.

And at the L-V level, you should be able to define your own outcomes. And at the L-VI level, you should be able to define the outcomes for your team. And L-VII may be outcomes for your whole organization. So there is this kind of progression in the human engineering ladder of scope complexity that you can handle. And it's actually the same in many ways of how we think about AI engineering progress, which is right now Devin is more L-III. You give it tasks, then it gets the task done. But when can we give outcomes to get the outcomes done? That's what we're really focused on.

Jason Richards

One of the things I loved in the summit presentation that you gave just now was this concept of, yes, you can give Devin tasks, but you can also invite it to become part of a dialogue, a discussion, help us evaluate certain situations and for it to lend its power to those maybe architectural conversations and those kinds of things. I find that really quite interesting. That's really, really cool.

Russell Kaplan

Yeah. And I think it goes to an important point for us, which is what we see in the organizations we work with is Devin is being used by the existing engineers to get way more done, right? Organizations who use Devin, they have not said, "Okay, great, now we have Devin. Now we don't need our human engineers." They actually said, "Great, now we have Devin. Now our massive backlog and roadmap of things we wanted to get done that was going to take a year previously, how can we get it done in three months instead?"

And a big part of that is, as you said, there's this collaborative element too. So using Devin to index all of the existing code and understand it really well, being able to ask questions on top of it, being able to brainstorm new architectural ideas or new features. How should those actually be built? And then once it's reasonably scoped in this more collaborative process, great. Now you actually can delegate a lot of the details for Devin to go implement. Yeah.

Jason Richards

And you touched on a question I was going to pose, which there's things in the press from other people around. We'll no longer need engineers at certain levels and potentially ever, kind of thing. It doesn't sound like that's the philosophy or the approach of cognition. Certainly, at this stage, it sounds like it's empower your teams to do more, do more faster, that kind of stuff. Is that a good way of summarizing that?

Russell Kaplan

Yeah. I think the reality is that we've been saying, "Oh, we're not going to have these jobs anymore because of AI or automation" since the beginning of the industrial revolution. And it's true that sometimes a lot of tasks get automated from a certain job, but then usually what happens is that job evolves a lot. I love the example of bank tellers, and how after the ATM came out, there actually were more bank tellers in the future than less because it lowered the capital cost of opening a retail bank branch, so you could actually now open a branch with much fewer tellers. But then, the nature of the job of the human teller changed, there's more client-facing component, more customer service component, it's less about the actual handing out of the money.

And so, I think the same is definitely happening in software engineering, where back in the day, we had punch cards, and we've automated punch cards once we got to assembly, and then we automated assembly once we got to interpreted languages. And so, at every stage, you could say, "Hey, okay, are all the assembly programmers out of work?" No, we still have a few assembly programmers, but most of them have now evolved to code in these high-level languages, and AI agents like Devin, it's probably the next, maybe most significant yet, order of magnitude transformation of that.

So I think in the future, yeah, we're going to have a world where way more people are shaping the creation of software than what's happening today. Whether we call them software engineers or not, I think that might end up just being a semantic distinction. Probably not going to be mucking through all the implementation details the way we do right now with that many people in the future, but way more people, I think, are going to be steering the creation of software than today.

Jason Richards

What about quality? Quite rightly, when we're talking about the need to have attention to some detail, not be reckless, those kinds of things, and clearly when you're delivering things to the customer, the view, I think, maybe two things here. One is your view on evolving customer expectation, and then also how we can ensure quality and maybe your view on quality increasing, I guess, as we go.

Russell Kaplan

I think this is one I'm really excited about. I think a lot of AI-generated output gets a rep of, okay, this is low quality. It always starts as low quality, and then usually in the blink of an eye, or a blink of a sigmoid, it becomes superhuman quality in any one domain, and then we just expand the domains of what we handle. I remember back when I was a computer vision researcher in college, we had ImageNet. That was the big challenge, and it was obviously a huge deal when neural networks were first applied to ImageNet, and suddenly you could see the progression at some point, where now AI systems are massively superhuman image classifiers versus actual humans.

And so, it's probably going to be the same thing for code, where right now, some systems you can look at AI code and say, "Oh, it's not as good as how I would write it." But if you just extrapolate from the trend lines, this stuff is going to be superhuman quality very, very soon, and in some domains, it probably already is superhuman quality in terms of coding. And so, my hope for the future is that we end up in this world where software is good by default. Most software today is very bad. Even if you spend a lot of money as a company, it's just really hard to find good software engineers and it's really hard to build great software, and so most of the products we use today, they're filled with bugs, they're slow, they're hard to interact with, and it's almost the exception, not the rule, that we are interacting with great software. And I think one of the potential promises of this AI engineering era is that humans are shaping what we're building, but maybe we have much more good software by default.

Jason Richards

And so, that extension then of the customer expectation, I guess I should expect that things are going to get better, easier to use, maybe fast iterations of solutions being made available to me, that kind of stuff, what's your view on that?

Russell Kaplan

Oh, yeah. Customer expectations are going to rise dramatically, and this is why I think it's very short-sighted when people talk about, "Oh, is SaaS dead?" Or, "Are there no longer going to be software companies? Are we no longer going to have humans steering software?" What's actually going to happen is just expectations are going to rise tremendously on what we expect out of our existing software systems. Any feature or company or service you interact with, oh, it doesn't have a native experience for every surface, that's fast, that's performance is fully integrated, that's going to be unacceptable.

And I think you can see this already today, if you just look at the backlog of work to do, for example, the HG portfolio is shipping all sorts of amazing software in all these different industries, but there's way more that they want to ship than they have bandwidth to ship. So I think in a competitive market, you end up in a situation where there's still a lot of SaaS companies, but the expectations on them are orders of magnitude higher, and we all benefit as consumers as a result.

Jason Richards

We talked a bit about the evolution of the engineering software, engineering role. One of the things that we see as well is what I'll call upstream positive impact implication on product management. What's your view on how that whole ecosystem might evolve, so not just the role of the software engineer, but that whole from ideation through to delivery? Maybe you could talk to that for a second.

Russell Kaplan

Yeah. This is a really exciting topic, because so much of the process today of developing new products, it's predicated on the fact that the software creation part is slow and expensive. And so, you have very detailed planning and you have meetings to align stakeholders to figure out that we're building the right thing and you have all this user testing, you have so much infrastructure to just make sure that your very precious software creation time is maximally well spent. And in a world where you actually have software abundance, where it's cheap and easy and fast to create great software, I think probably the rest of the organizational cadence has to evolve too.

You mentioned product managers, I think product managers will be extraordinarily empowered to just ship the prototype yourself, don't write the document, just show exactly what you want and work with an AI to make that happen. I think it's going to be a really exciting time to be steering the creation of software.

Jason Richards

I think it's important to have that wider lens, because I think the danger is, otherwise, you can create a bottleneck, where you have this really high-performant engineering capability, but you just can't get enough into it fast enough, so that's one of the things we're really cognizant of.

Russell Kaplan

An analogy I think about here is if you look at inflation in the US and internationally by sector, you can see there are some sectors that technology has led to massive disinflation, like buying plasma screen TVs is way cheaper, and you have other sectors where technology hasn't been allowed to disrupt or innovate or improve, and so you have spiraling costs in healthcare and education, things like this. And what happens is that basically, the percentage of the economy by spend is just growing towards all of the non-software-disrupted parts of the economy, because you have this massive disinflation in areas where you're allowed to innovate. And I think the same is going to be true inside an individual organization, which is if you just keep doing things the same way, but now software creation is 10x faster, 10x more efficient, the bloat of everything else is just by far the bottleneck, so you have to evolve that part too.

Jason Richards

Yeah, interesting. If we extrapolate out now, in terms of just generally what's happening in the industry, where things are going, the pace of change, what is it that excites you about where we are now?

Russell Kaplan

Yeah. There was an interesting paper from Metre recently that was trying to measure how fast we're progressing on AI progress for solving tasks, and they had this very nice, interesting graph on the time that it would take a human to do a task that could then now be done by AI at 50% reliability. If you look back in the GPT-2 era, you could say GPT-2 would do tasks really only that could take humans like a few seconds. Then by the time you're at GPT-3.5, it's maybe on the order of a minute or so, and four, and now Claude 3.7 Sonnet and so on, it's getting higher and higher, and they posit a doubling time, almost like a Moore's Law for agents, of seven months. So every seven months, the length of tasks that now can be done by AI at 50% reliability is doubling every seven months.

And then, within software specifically, they measure, it's a bit noisier, the measurement, but they extrapolate a trend line of 70 days, which is crazy to have a doubling time of 70 days. But it kind of lines up with our first-hand experience with Devin too. Devin is getting so much better, so fast, that a year from now, if we had five doublings in task complexity, it's not unreasonable to think that might happen, and if it does, the world looks dramatically different.

Jason Richards

That takes me back to the questions about leadership, leadership philosophy, the traditions of how you might plan your strategy in a software organization aren't going to work with that kind of timeframe. So how do you guys do that? So your forward-thinking aspects of cognition, how far out your strategic horizon is, that kind of stuff, I'd love to know more about that.

Russell Kaplan

Yeah. I think we're in a world where it may be the case that tactics matter a lot more than strategy in the future, and actually, how fast you execute becomes the defining thing, because the error bars are so high on predicting the future right now that it's really hard to say, "Oh, we have our master plan for the next three years and we're going to execute on and get it done." I think there's certainly some things you can extrapolate on, but for us, speed is the strategy. We've got to be fast, we've got to be shipping, and we've got to be making our product better, faster, than anyone else.

Jason Richards

Yeah. Well, I guess be decisive as well. You can't sit and wait things out. By the time you've done, things have gone. So I imagine it's that evaluate, understand where you are, be decisive, make a decision, go. Is that the kind of approach?

Russell Kaplan

100%. And if you look even back to the very first thing we put out publicly for Cognition, it was a demo of Devin, it was about a year ago we released it, and at the time, it went unexpectedly mega viral. And I think the reason was people didn't expect that it was possible yet, at that time, to actually build a software engineering agent that you could give a task to and it could decompose and do end-to-end. It was just barely possible, it was just barely possible. It was a very hard engineering effort, just barely possible. Now, it's easier to build that, but there's a new frontier of things that are just barely possible. And so, we want to be riding the edge of what's barely possible at every moment in time and pushing that frontier forward.

Jason Richards

And how much does Devin contribute to the considerations the decisions, it's all coming back to how you guys are leveraging Devin itself. Is Devin involved in supporting you in those apps?

Russell Kaplan

Yeah, this is the really fun part, I think, of what working on Devin is, Devin is the number one contributor to Devin, by a lot. You have people talk about recursively self-improving systems, and we actually already have them. Devin is a recursively self-improving system. We still hit approve on the pull requests and we're still in the loop, but it is a substantial difference in the development cadence and philosophy, I think, of a company.

Jason Richards

And I didn't come back to it when I meant to, but the thing about the answer is not throwing more people at this. I don't know if you're willing, but the ratios that you have between your staff and then the number of Devins that are working for you in effect, is that something you're willing to share with us?

Russell Kaplan

Yeah. I think it's about probably 12 Devins per person or something like that. We probably have on the order of a few hundred Devins, and around 30 people.

Jason Richards

And you're finding that the people overseeing that, that they're perfectly capable of keeping those instances engaged, busy, off doing their autonomous task.

Russell Kaplan

It really depends, because there's this baseline load of tasks that you do. But a lot of what's really interesting about Devin is that it also enables, to the earlier point we were talking about, if you rethink from first principles how to do software development in an AI agent ready era, you actually come to some surprising conclusions. Yeah, certainly we have people keeping Devins occupied, but we also have these workloads now where you burst up to spin up hundreds of Devins or thousands of Devins, and then they just get one big task done in parallel, and then you spin them down.

And I'll give you an example from one of our customers, it's a large financial services customer. They realized that they could hire Devin temporarily inside the CI loop of their development... Because we have an API, so you can use Devin via API or via the user interface. And so, when new code gets shipped, they spin up a team of Devins and that team of Devins is QA-ing the new code, it's checking the test coverage, it's adding tests if they're missing. Doing all this work that a traditional QA engineering team would do, and then it spins down at the end of that. And so, you're getting much more flexible and different looking software development patterns too.

Jason Richards

Yeah, that's true. I think that whole on-demand aspect, to be honest, I hadn't really given that any consideration, but you can see the real value of that now as you said, if you've got tasks where it's a case of it is capacity, turn it on, get the task done, turn it off kind of thing. You can see that's really pretty powerful.

Russell Kaplan

Yeah, it's a very exciting time to be a software engineer, I think. I think it's probably the most exciting time to be shaping... Really, software engineering ultimately, it's just about telling your computer what you want it to do. And I think it's getting easier than ever to do that.

Jason Richards

Any further advice you might give to our CEOs or CTOs that are in software organizations and now faced with this era of GenAI, and both from that enablement and leverage of internal optimization capabilities? But also, thinking about product and product iteration evolution, any further advice you might give?

Russell Kaplan

The first main piece of advice is, you just got to try this stuff. You have to try it, you have to get hands-on and you have to see it. I would say we actually have CEOs at companies, or very senior engineering leaders at the companies we work with. And they're finding, software creation is fun again now for me. Because their calendar is so filled with meetings that they, historically, it's been very hard to find maker time. But now, you have your team of Devins working in the background while you're in your meetings and you can check in, you can have a lot more maker time.

So I would say, get hands on and actually try this stuff out and get a firsthand sense of what does this feel like and what does this look like. And then for the organization, I think it's really important that we realize that we're going to be able to ship so much more. It's an abundance mindset and not a scarcity mindset. We're heading towards software abundance. I think sometimes there's fear that, "Hey, are we all about to lose our jobs?" I can tell you very directly what we see from all of our customers, is that what they actually do with Devin is they pull in their engineering roadmap dramatically, because now the ROI of hiring one human engineer to oversee this stuff is much higher than it used to be. And so, people are shipping dramatically more, I think. I think, that's a really important takeaway.

And then lastly, I would say in any large organization, there's a distribution of how enthusiastic people are to try new things. How curious are they, how interested are they, how much do they want to do things the same way? I think that's the thing that's really important is, how do you shift more and more people to be, "Hey, we've got to try new stuff and we've got to be willing to adapt, because the stuff is coming."

Jason Richards

And the ROI aspect, maybe just as we close. It'd be useful, I think, just to give that quantification of this magnitude of impact that you'll see in your sales, but also in the customer base.

Russell Kaplan

The average ROI we hear from customers, is between 8x to 12x, which is a big difference. And the way that's calculated is, if you look at one hour of human engineering time and what you could output of that, how much could you output of one hour of human engineering time instead spent telling them what to do, and then reviewing Devin's work after it's done. And that output comparison for the use cases that Devin is good at, it's about eight to 12x. And so, people look at that and say, "Hey, wow, this is really easy to quantify, how much time we save or how much faster we shipped as a result of that." And I think, probably this human time savings is the right ROI.

Jason Richards

Yeah. Attribution's always difficult. So, we found initially in some of the early stages when we were doing things like coding assistance and trying to attribute the value, is very difficult. So, the model you've just described, I think, is one that we're leveraging ourselves.

Russell Kaplan

I think part of the reason that the attribution has been hard in the past is that it's like, the first wave of GenAI tools are very horizontal development. Oh, let's give everyone a GitHub copilot, and people maybe are 10% faster. And there's a little bit of hand waviness, but it's certainly helping people and people love those types of tools. What's interesting about Devin is what we're seeing now in the market is, it's actually getting rolled out use case by use case, more than, "Oh, hey. Everyone have this and do whatever with it." Because it's these specific use cases, these use cases where you can break down the use case into many tasks that are doable by an AI junior engineer, that are really good for Devin. And so, if you're working on a large refactor or migration or modernization project that was going to take you two years and now it takes you three months, and one 10th of people time, it's just a huge win.

Jason Richards

Perhaps, if you were to go back to your beginning of cognition, is there anything now upon reflection you would do differently?

Russell Kaplan

I think we would have pushed to release the product to everyone even earlier. When we started, we were working with large enterprises mostly. And the reason for that was that we wanted Devin to be really good in real world code bases. There's so many amazing AI demos out there of building a new website from scratch. But most software engineering work today, it happens in the context of real existing code bases, existing environments. And so, our bet was, we've got to be able to work with the largest organizations upfront to solve it.

And I think that's been true, and architecturally it's been really valuable for us that we started that way, because now we can... We actually work in large organizations as well as the small startups, and there's a lot of value in working with large companies too. But the pace of product improvement and feedback iteration velocity that we got once we released Devin to self-serve, available to everyone in December, was so high that I wish we did that sooner.

Jason Richards

And do you think that's a common thing that our listeners should be thinking about is, don't be afraid to get stuff out early? Now clearly, not so early that it's not ready, but don't be afraid, get it out there, because there's a ton of stuff happening and if you don't...

Russell Kaplan

Yeah. And I think we're fortunate when we released when we did. We knew we had a product that we could see people... It was successful already in large and medium and small companies, and we were confident that people could use it themselves. But yeah, I think for everyone working on software, because the pace at which you can ship new things is so fast now, there's even more alpha in just getting out there as soon as possible.

Jason Richards

But we really enjoy working with you guys, and we're really enjoying the benefits we're getting from Devin already. Thank you very much for your time and your insights, and also being part of the summit, is fantastic. All of our 160 odd attendees certainly, I think, have learned a lot today, so thank you for that. Our listeners, is there a way of following you that our listeners could maybe... What would your advice be?

Russell Kaplan

Yeah, thank you so much for having me. And yeah, I think, people should just try out Devin. Go to devin.ai and give a spin for yourself. And then, my name is Russell Kaplan, so I'm around.

Jason Richards

Great stuff. Thanks, Russ.

Russell Kaplan

So great to see you.

Jason Richards

Cheers.