I’ve written, that the right way for humans to rise to the challenge of AI is to “each aim to join a new republic of power users”. So I’ve finally got around to practising what I preach, just a little bit.
I’m going to use AI tools to help me stand up a little web-based fitness tracking app. This app is a fairly stupid project of no real value except as a way to get some practice using AI tools.
For this episode, I’ve used ChatGPT (i.e. v3.5) to advise me about how write the app. So far I’ve done little more than stand up a simple “hello world” web-server in something like a realistic development environment. Yet this is enough for me to have some observations about the process. These observations are not particularly surprising or insightful, but they are worth recording, if only to see how things change over the next year or three.
It understands programmers well
By which I mean it is amazingly capable of unpacking and responding helpfully to the customary inarticulate grunting of the software profession. For example:
AR: What about `response_class=HTMLResponse`?
GPT3.5 understood that I was objecting it’s previous suggestion of hand-hacking a content-type header when it could decorate the calling function instead. The long-winded text that went with the code-suggestion included the actual insight:
… By adding
response_class=HTMLResponse
as a decorator argument for theread_root
function, we specify that the response should be treated as an HTML response.
It’s long-winded
This isn’t just a programming thing, ChatGPT will never answer with a short sentence when a whole essay will do. I think that’s because it was trained by Americans.
In the programming context, that means questions don’t yield answers, they yield answers embedded in paragraphs of text and coding examples. I can’t comment on the quality of the text, because I barely read it. The code examples are pretty good, but more on that later.
It’s talented at spitting out code
I say spitting out code, not coding. I expect ChatGPT would be bad at solving some non-cargo-cultable algorithmic problem. But for the dumbshit API gluing that is most of our work, it’s a great first guesser. It pulls in all the bits that are needed and stitches them together in mostly correct ways. It even explains what its doing, though I wish it wouldn’t.
It doesn’t check it’s work
ChatGPT is not a source of correct solutions, it’s a source of very good first guesses. As such it’s much better than a human, it’s first guess will generally be syntactically correct and cover all the obvious bases. They might even work.
But it will still happily make errors at one or two levels up. For example I asked it to redirect unauthorised requests, and it happily set up a redirect loop because the user had no authority to see the target page. But then eyeball the error either.
The way to think about this is that you will always need to iteratively evaluate and fix your code, but ChatGPT makes each iteration faster and more effective. It will quickly spit out a good first guess, and than make good improvements in response on your vague off-the-cuff suggestions.
It’s not just code
Prior to writing a line of code, I had a long and fruitful discussion with the thing about the different design options, tools, and what off-the-shelf components to put in which part of the stack. It’s a great way to quickly get up to speed on the bland-common sense general opinion of what the state of the art is some boring part of the field.
The Verdict
ChatGPT is a good programming consultant. It’s helpful, and mostly pleasant. Yet I’m not sure it actually made me any faster. I could have banged this app out quickly by simply Googling and cargo-culting StackOverflow. While ChatGPT does sort-of automate that, the manual workflow was already hard to beat.
The next step is obviously to try GitHub Copilot, even though it gives me the willies.
I said above that programming with ChatGPT makes each iteration of your coding cycle faster and more effective. The limiting friction is the mismatch between your real code and the toy example code that ChatGPT is spitting out. Not only do you have to manually cut & paste, but you have to manually adapt the result to you real code (changing variable names etc).
Setting aside the willies, an interface which read your real code and offered you a side-by-side diff in the IDE would cut out nearly all this friction, and really might supercharge the edit-check-improve cycle.