Hello and welcome to C++ Weekly. I'm your host, Jason Turner. I'm available for code
reviews and on-site training. In this episode I'd like to clarify my position
on using AI-based code generation tools and give you some best practices based on my experience.
[Music]
Now before I get into this episode I would like to bring your attention to the video description
below where you can learn more about this channel such as the t-shirts that I have available,
such as the C++ Weekly, maybe_unused and constexprt t-shirts, the fact that you can
support me on Patreon if you choose to and the set of Best Practices and Puzzler books that I
have been working on - available both as eBooks and print books. So getting back to the matter
at hand - as we all know, AI generators such as ChatGPT are becoming incredibly popular right
now. It's the fastest growing industry that we have seen in a very long time,
and you can use these tools to generate code as I demonstrated in a recent episode of C++ Weekly.
Now I also did a two hour investigation into the kind of code actually generated by ChatGPT in my
Meetup, so of course I want to take a moment to suggest that you find a C++ Meetup in your area
and get to know some of your fellow developers in your local area. If you happen to be in the
Denver area then be sure to check out the Denver Metro C++ Meetup. You can find us on Meetup.com,
and we have been meeting for something like seven or eight years now continuously - even over covid
we moved to remote and now we're back in person again. Okay, so all of that out of the way,
I have spent a lot of time playing with this and I think some people
might accuse me of advocating for using AI code generation tools,
but they have a lot of pitfalls that we should be aware of. Let's just go ahead and dig into these.
So the first thing that you really must do... Should I do these as numbers? Let's
do these as numbers. You have to start with the assumption that the answer that the AI
code generator gave you is wrong. It's going to appear very correct
and that is going to be part of the problem here, and it seems in many cases the more
wrong the AI generator is, the more confident it will be in its answer.
This makes the tool particularly dangerous if you don't know any better or aren't paying attention,
so you need to assume that the answer is wrong even though it appears to be correct.
Now this one is a bit bold of me to proclaim, but these AI generation tools generally work
by indexing huge amounts of information on the internet, and it is entirely possible with any
AI generator - image, video or text for it to generate something that is nearly identical
to a piece of copyrighted work, so you have to assume that the answer is stolen from somewhere,
and you need to be cautious of that when working on copyrighted code. Now the next thing that you
need to do is tell the AI what guidelines and best practices you want it to follow.
This might look something like,
"always use f-strings if we're in python" or
something like "prefer lib {fmt} over iostreams," and you always have to nudge the AI Tools in these
directions. They're not always going to follow the rules that you say "always follow these rules," so
this kind of unfortunately brings us back around to point 1 - you still have to assume that the
answer is wrong, no matter how many guidelines you gave the tool, and you must remember
these are going to represent the average code quality across the entirety of the internet,
effectively. So you might think, "Oh, well, obviously it's going to use C++20
best practices because that's what I asked it to do," but really it's starting from a basis of
probably a large swath of C++98 or C++11 examples on the internet. If you nudge it in the direction
you want it to go you might get most of the way there, but you have to assume that it's wrong,
or you know, I don't want to say lying to you, because it's not really an intelligence,
so to argue that it's "lying" might be a bit of a stretch, but I mean,
it's lying to you. So with these things in mind, if you assume that you cannot get a
correct answer from it and it will be highly confident in the wrong answer that it gave you,
how should we use it? Let's say there's a few guidelines that I have learned in places where
I have actually found these tools to be incredibly valuable. One is - this is a smart rubber duck.
Now if you're familiar with the concept of rubber duck programming, this is where you just explain
the problem to some sort of toy, or your dog, or a stuffed animal or something like that,
and in the process of describing the problem you get your own thought processes (to) work better
and you're able to come up with a new solution. Now if you describe the problem you're trying to
solve to something like ChatGPT it might point you in a very wrong direction, but still inspire you.
So we're basically using it for inspiration, for problem solving. Closely related to these two,
I kind of want to phrase this as a "second set of eyes." For example...
so example: you might take some C ++ compiler error, paste it into ChatGPT and ask it,
"What went wrong here?" and it is entirely possible that it will actually be able to
decipher this error and give you a direction to look in, or again, even if it's wrong,
it's giving you, perhaps, some new place to look. So this kind of smart rubber duck,
I'm really going to double down on this, I think that this is a good way to use these tools.
Finally the thing that I want to mention is this idea of a quick start. This is a quick start for
new technology, and I'm going to demo this to make a point. I know from my experience already that
some of you are going to say, "That's ridiculous because I have to assume its answer is wrong," and
some of you are going to get what I am showing, so let's just go ahead and try to do a quick example
here. Let's say there's some new technology that you've always wanted to learn, but you just simply
can't find a good example, and I will state from personal experience I have actually used ChatGPT
in this specific way to save myself dozens of hours of work. So I might say something like,
something like this. I just want to learn Tkinter. I want to have some sort of message box. I want a
modal dialog. I just want to have an idea how these things work, so I just asked it for this
example. I have two buttons - one called Quit, and it appears to actually quit; and the other
called Hello World and it calls this function called display_hello_world, which pops up a new
message box with Tkinter and it appears correct. I must at this point assume that it is wrong,
following my own rules, however this just got me 98% of the way there. Is this best practices for
how to use Tkinter? I have literally no idea whatsoever, but if I just want to get started
with a project, I'm there. I'm there almost instantaneously. Let's do something like this:
So I want a C++ function that generates this Fibonacci sequence,
and I can just glance at this already and say I would have made this constexpr,
but I know this is correct. I know enough about this problem domain to know that this is correct,
and arguably, this is C, not C++, but it is the classical definition. Now let's just try something
that I've never tried before, and we could try something like
ask it if it has an optimization for it. So it created an unordered map version that memoizes
the results. This is not the solution that I would have gone for personally, but fascinatingly it's
extremely close to the example that I had on the function cache on C++ Weekly not that long ago.
Again, it shows how to print out the results. But again,
like having to kind of nudge it in this direction might be something like:
So now I'm really nudging it in a direction here. Here we go, so I was able to ask for a linear
version that does a for loop, and I would say that is not a bad solution. That looks probably
pretty close. Now this could have been done much better with the std::exchange algorithm,
so looking at this through the eyes of someone who regularly does code reviews, I would say
you should have been using std::exchange. This should be constexpr, but that's a good try. You
are initializing the variables properly, and your conversion to lib {fmt} looks a hundred percent
correct. It's going to print the first 10, and then it defines catch2 and does a requires for
the first 10 elements to make sure the Fibonacci sequence is correct. So this now, let's just say
hypothetically you had no idea how to use catch2, you didn't know anything about C++, but you knew
something about the Fibonacci sequence, then this is going to give you that leg up in the learning
curve of C++. Now I do know C++. I knew what questions to ask it. It even generated a CMake
script for me. I would say that this was very useful, although if I don't already have catch2
or {fmt} available, then I'm going to be sad and confused at this point, but otherwise it looks
pretty darn good. So I could now say I understand this example. It does what I want it to do.
I could ask it to translate it to R because I don't know anything about R whatsoever,
and now I can see a linear version of this - what at least a testing library that exists for
R and how I might execute this. So this, again, is where I would say that this really shines,
but I want to clarify again that at this point I have to assume that this R version is incorrect.
I would have to actually try to execute this, see if it does what I want it to do, and probably sort
out a couple of compiler or runtime errors in the process of doing that. So there you go,
uh, this quick start for new technology that you don't know anything about, a good rubber duck,
a second set of eyes on your problem, asking it things like, "Can I optimize this code?" - I had
people in my Meetup tell me that they use ChatGPT for that all the time, and I've actually learned
a lot from it, but always remember these primary rules. The answer is going to appear correct. The
AI is going to be very confident about its answer, but you must assume it is wrong. You must assume
that has actually been stolen from somewhere, so you're using it for inspiration primarily.
So thank you for watching this episode of C++ Weekly. I hope you
enjoyed this. Be sure to subscribe and like the video, and I will catch you next week.
[Music]