Don't get me wrong, ultimately the developer is in control.
They are the ones who decide whether this code is good.
They're the ones that will run and verify that it is working as expected.
They're the ones that will ship.
What the Generative AI based tools like Code Whisperer are helping with is you don't have
to do a lot of reading documentation pages.
They're just saying, hey, this is stuff that is easy to get.
You as an application developer should be focusing on creating value for your customer
by doing higher level things.
Sandeep Pokkunuri And Doug Seven have been developing software
and building developer tools for decades.
Sandeep is a Senior Principal Engineer at Amazon
and Doug leads the engineering team responsible for Code Whisperer.
They each have an impressive resume as software engineers and have seen the industry evolve
over the years.
Today they're at the forefront of some of the biggest challenges that we're working
on across the company.
I wanted to sit down with them and see what kind of insights they may be able to share
about where Generative AI is playing a role in their jobs
and where it might go in the future.
Doug, Sandeep, thank you for meeting with me here today.
We're going to talk a bit about sort of the tech behind how we are helping developers
with Generative AI.
But can you first tell me a bit, what is your role within Amazon and in this world?
Sure.
So I'm the general manager for Code Whisper, which is our large language model product
for developers.
And I came here by way of about two decades in developer tools and focused on developer
productivity and how to help developers do what they do faster, better, more fun.
Did you used to be a developer yourself?
I have been a developer for a very long time, which is how I got into it.
I spent a lot of time writing code and figuring things out.
Sandeep?
I've been a developer myself for twelve years at Amazon.
Actually, today is the 12th year of completion.
I worked on distributed systems, products, DynamoDB, SQS over the past six or seven years
close to now.
I've been working in the machine learning organization, building various services like
Lex and Voice ID.
I'm actually working on large language models myself now.
So, yeah, we hear a lot about all this Generative AI stuff and large language models and things
like that.
And the word language in there suggests that it's all about text -- writing poetry or new
articles or things like that.
What are we doing using this technology to help developers?
Well, language isn't all about text, right?
That's just one expression of language.
But certainly when you're a developer, you're writing code that's a form of text.
And so if you think of the process a developer goes through, I'm going to write some code,
I'm going to think about what I'm doing.
I'm trying to solve a problem, right.
The idea of backing that up with a large language model and say, hey, let me understand what
you're doing.
And from what I understand of that, let me infer what I think you want to do next and
suggest that to you and give you that suggestion in the form of maybe I'm just going to offer
you the completion of the line of code you're working on.
You're writing a method signature, and I'm going to give you the parameters that you
want to fill in.
But didn't we have this completion already in Ides and things like that for particular
signatures, for example?
Yeah, code completion has been around for a long time.
Right.
And the evolution of code completion from something as simple as I type a class name,
I hit a period, and then we're just going to iterate the methods and properties that
are available and list them as a really simple form of code completion.
The evolution of that to not just say, here's the properties and methods that are available
to you, but to say, I think I know what you're doing, let me suggest you even more code that
would help you complete that task.
It's almost like continuous pair programming.
Yes, exactly.
Your peer here is not a human, but it’s...
We phrase it as your AI coding companion, right?
Yeah, it's just that it's like we're sitting next to each other, we're writing code, we're
solving this problem.
And he doesn't need to read the documentation.
He's already read it all.
So where does the inference happen?
On your laptop?
Or do you need to be connected to the Code Whisperer backend?
Inference is just one part of the story.
The full story is more complex.
For example, on the IDE, the plugin is doing a lot of work.
It is seeing, okay, what programming language is the developer using?
Where are they in the current context?
Are they opening a function?
Are they trying to finish a comment?
Are they trying to write a block for loop or an if condition or something like that?
It figures out the exact time where you might need a recommendation, a code recommendation.
That logic is embedded in the plugin wherever it is, and then it makes an API request.
And even when it shows you one recommendation, it is still working.
So all of that logic lives on the service side.
And of course, we also have some cutting edge response features such as reference tracker.
All of those also reside on the service side, trying to help the developer make the best
decision for their customers and their applications.
So tell me a bit about sort of how are these models created?
I mean, it's not all the text in the World Wide Web, I mean, because that won't help
you as a developer.
So what sits inside the model?
Generally when we train large language models, we collect a lot of data from the public Internet.
We clean it up and make sure that we train these models such that they understand the
vocabulary and the structure of the language.
How do you make meaningful sentences and paragraphs in the language?
If you look at sort of the imperative programming languages, let's say you have example code
that you've found in Java.
Would the model be able to translate that into C plus plus?
So you don't need to have the C plus plus code originally into the model?
Yeah, the models that we build, the transformer architecture absolutely allows for that.
So very soon we will be seeing automatic translation from one language to another.
Especially some of the legacy languages of the older times.
They want to upgrade to a newer language or even the more recent languages.
You want to go from one language to another because your development team is more familiar
with it or it's more efficient.
For example, Rust is quite popular these days for high performance applications.
So absolutely it's going to be possible with large language models.
So I always thought that as engineers or as programmers, we have one of the most creative
jobs in the world.
You can go to work every morning and create something new, and it's fun.
Does this take the fun away?
The way I look at this is the idea behind Code Whisper is if you and I were going to
sit down and write an application together, you bring to the problem a knowledge set,
I bring to the problem a knowledge set, and together we're going to solve this problem
and figure it out.
And you might have some suggestions for how to do things that I wasn't aware of.
I'm like, oh, I didn't ever think of doing it that way, and vice versa.
And so Code Whisper and these generative tools work largely in the same way.
We're just going to suggest things and sometimes you're like, yes, that's exactly what I would
have done, but now I don't have to type it.
And other times it's like, oh, well, that's interesting.
I maybe wouldn't have done it that way.
One of the most interesting things for me was the ability to approach something that
I'm not familiar with.
So in my case, I wanted to just try something and I wanted to go use an API that I didn't
have a lot of experience with, and I wanted to use a programming language I hadn't really
worked in before just to see what the experience would be like.
Okay, so there's a lot of work that goes in there.
A tremendous amount of work.
And it's truly augmenting my skills as a developer because quite a few of those things I would
maybe be myself not be aware of.
I love coding, okay?
The part of the job that I do that is the most fun is actually writing code.
But to me, my job is actually a lot of creation.
It is a creative profession, right?
So it's a lot about brainstorming with the product managers about what we want for our
customers, what is the desired customer experience, what makes our customers delighted, right?
And then the implementation part is, okay, how do I convert that into designs?
How do I make sure that this is highly available, highly scalable, all of that.
And then finally, the last part is actually writing code.
I don't measure my self esteem based on the amount of code that I write.
I measure my self esteem based on how happy the customer is.
Some of my favorite comments are when we talk to people who are like, oh, this is bringing
the fun back.
Because you think about the day in the life of a developer, and the process a developer
goes through, like I said, fundamentally you're problem solving.
A part of your day is sort of mundane.
A really trivial example is, oh, I've got to write a class to represent a data object.
That's just like, I'm going to spend the next three or four minutes typing gets and sets
to represent the things that it needs to do.
Or I can just type a comment that says, a class to represent this data object and I'm
going to start generating that code and I'm going to be done with it in like 30 seconds.
So that's the way you interact with it.
Basically, you give it a regular text prompt and it will go and try and find out whether
it can help you with that.
There's essentially two ways.
One is, as I'm writing code, so like I was saying earlier, I'm writing method signature
and it's understanding what I'm doing and it's inferring from that that I'm going to
maybe want some parameters or here's what the function is going to look like.
And so as I'm writing code, it's kind of completing the code, sort of code completion.
The other is, before I'm writing the code, I'm documenting my intent.
Here's what I want.
I'm going to write a comment that describes what I want and the language model can understand,
can look at that comment and say, okay, I understand what you're describing, and then
it'll go through and start producing that code.
Okay.
Like foundation models that are trained on enormous amounts of text, Code Whisperer is
trained on billions of lines of code.
When combined with the context in your IDE, Code Whisperer can write entire blocks of
relevant code for a wide variety of application types and use cases.
This can help you with common tasks like writing unit tests or working with the AWS APIs or
adding boilerplate code to your application.
But it can also help speed up uncommon tasks or solve complex problems that may have taken
you much longer without a coding assistant.
Let's say you're writing a lambda function and you're inside the Lambda console, Lambda
editor, and you say, hey, I just want to read a message from the Kinesis stream and I want
to send an SMS to the customer through Twilio. Right?
So that's your top of the Lambda function comment, right?
So from then you just say def read message of something.
And then from the context, Code Whisperer can figure out that, okay, this person is
trying to read a Kinesis message.
Let me read it and let me parse it and let me pick the interesting thing and it'll fill
for me.
And if I need to change something, I can just do the last bit.
The last mile, I'll take care.
Don't get me wrong, ultimately the developer is in control.
They are the ones who decide whether this code is good.
They're the ones that will run and verify that it is working as expected.
They're the ones that will ship.
What the generative AI based tools like Code Whisperer are helping with is you don't have
to do a lot of reading documentation pages.
They're just saying, hey, this is stuff that is easy to get.
You as an application developer should be focusing on creating value for your customer
by doing higher level things, not boilerplate undifferentiated heavy lifting.
So you're saying the fun part of being a developer is not reading the documentation?
Yeah, absolutely.
Reading documentation is not the fun part of being a developer. For sure.
You've been using Code Whisperer probably much longer than we have.
So what is it that you really like about it?
Right.
To me, the most compelling part of Code Whisperer is the reference tracker feature.
It was launched with it.
On the day it launched, it was there.
So the idea is that you're training on a lot of public code and it's possible that the
models, the large language models, they may repeat something that they have seen at training
time and the person who is using the assistant, they may just accept your recommendation and
move on.
But that may not be the ideal thing to do because there may be a license associated
with the repository from where the training data was procured and the person who is using
that code should know, oh, this belongs to a certain license, then there are obligations
that I must meet and so on so forth.
And the developer may choose to say, hey, I looked at the license, I'm good with it,
I'll proceed or say, oh, I don't want to pick any software that looks like this license,
I'm going to just edit it myself.
Or pick a different recommendation from the list of…
Or your company made.
Yeah, exactly.
Exactly.
This changes life for developers dramatically.
So does this mean that the skill sets of developers are going to change?
The requirements?
I mean, you no longer need a four-year computer science degree to actually do these things.
We're making the developer more productive.
We're helping them do the same things faster.
They still have to know what they're doing.
They still have to be able to look at the suggestion they're getting and understand
what it's doing.
And saying, yes, that's what I want, or maybe, yes, that's what I want, but I just want to
change this one or two things.
To some degree, I always equate this to mathematics class. Right?
As you're learning mathematics, you have to learn the fundamentals.
You have to learn addition, subtraction, multiplication, division.
And then you move on to learning some basic algorithms and some basic algebra capabilities.
And eventually you get to a point where your teacher says, okay, you can bring a calculator
to class now, and you're going to use that to speed yourself up in doing the things that
you already learned how to do by hand.
And that's what Code Whisperer is.
It's the calculator for a developer.
Sometimes it's being looked at as that this is a paradigm shift, but I think it's much
more in the tooling space than it is in sort of the shifts we solve with object orientation
or functional programming or things like that.
Where do you see this go?
What is the Holy Grail?
Yeah, the paradigm shift is going to happen not in the core programming software development
process.
We are traveling on the same road.
Instead of going on a bicycle, you're going on a Ferrari or something.
That's what we are doing here, right?
It is a huge change in how developers work.
And Generative AI has become so important in our conversations and everything we're
doing about how is this going to affect what we do, that we want to get this into as many
hands as possible, get as many people the ability to use this tool and get the productivity
gains and do more.
It is part of our democratizing AI story.
Usually these productivity tools, big companies can pay for them, for their developers.
But at the same time, there are a lot of app developers and freelancers who are just beginning.
They don't have big companies to pay for these licenses and all that.
They're just starting to build a mobile app.
They want to do a quick POC, get feedback from their customers.
They should be moving at the same pace as a person working for a very big company who
can afford those licenses.
You guys are building amazing tools and I hope that we can build a lot more free things
to make our developers much more successful.
I'd like to thank Doug and Sandeep for joining us.
Their perspective on how generative AI will change the way we work as developers
has enlightened and energized me.
I can't imagine how much more productive I would have been had I had tools like this
when I wrote my first program over 40 years ago.