“Any sufficiently advanced technology is indistinguishable from magic.” — Arthur C. Clarke
“The world is made of language.” — Terence McKenna
INTRO
As an emerging tech analyst and cargo-coder with an eye on the AI/ML/Data Science/Analytics space, OpenAI’s NLP models first caught my attention back in 2018 when the non-profit released a v1 of their breakthrough natural language model — GPT-1. As larger and more capable versions of the model were released (GPT-2, GPT-3 and now GPT-3 Codex), I couldn’t wait to see what all the hype was about first-hand.
Back in August, I got an invite:
After a quick look over the Docs and some “Hello World” examples, I was off to the races. It didn’t take long to start blowing my mind:
For the non-coders out there, don’t be shy — forget the syntax, and focus on the basic steps in the process:
- Describe the code that you want DaVinci to generate.
- Set DaVinci’s hyper-parameters (response length, intensity, penalties, etc.)
- Wait for DaVinci to generate a response/code completion.
- Take the code from DaVinci, and run it.
- Debug errors (install reqs, libraries, dependencies, file path conflicts, etc.).
- Re-run the code until it executes.
- Rinse & Repeat (…and Go 7 Layers Deep (h/t Kevin Kelly) (link)
While it might seem insignificant, there is an infinity of implications hiding in that simple loop — you just have to know what to look for.
A simple coding task like dimming an LED might’ve taken me 30 minutes (or more) to code from scratch, but it took Codex less than 3 seconds.
These time savings start to amplify massively when considering more complex programming tasks. What at first may seem trivial, could ultimately prove revolutionary.
Transformers, Generators, GPTx & OpenAI
No — not the transformers and generators that power the electrical grid — transformers & generative models are software inventions that use Machine Learning models based on Neural Network architectures that mimic the human brain. GPT-3 is a particular type of AI model known as a “Generative Pre-trained Transformer” which has been trained on an enormous dataset that it can call upon to generate realiatic text and natural language response and completions.
The original concept, known as a Transformer, was developed at Google Brain’s Toronto office by researchers Aidan Gomez and Nic Frosst. These researchers, alongside a third cofounder, Ivan Zhang, recently raised $40 million to commercialize the technology through a company called Cohere.
“The technology that undergirds Cohere’s natural-language processing models was originally developed by the Toronto-based Google Brain team. Two of that team’s members, Aidan Gomez and Nick Frosst (along with a third cofounder, Ivan Zhang), started Cohere two years ago to further develop and commercialize the models, which are delivered to customers through an API.” — Fast Company
Preceeding Cohere by a number of years was OpenAI (founded by Elon Musk and others) that was created as a research organization (now for-profit) to democratize AI for the benefit of humanity. Sam Altman, Wojciech Zaremba, Ilya Sutskever, and Greg Brockman are the lead scientists and chiefs at the company.
Arguably, OpenAI’s crowning achievement to-date has been it’s series of GPT models — GPT-3/Codex being the most recent. As Eric Schmidt said in his interview on the Tim Ferriss Show recently, these code generation models are major inflection points in AI research in the last three years (2019–2021) (AlphaGo and Halicin being the other two).
Knowing a Big Idea When You See One
The reason this is so big, to me at least, is that it starts to transform the value proposition of being a creative generalist. In a world that rewards specialization and an atomized division of labour, being a generalist has traditionally been a risky move. Even most generals & generalists have a few areas where they have specialized knowledge.
Most programmers know maybe a handful of languages, 10–100 methods and functions, and a vocabulary of 100,000 words.
GPT on the other hand knows at least 10 programming languages, 100–1,000k methods & functions, and has a vocabulary as large as the entire English language corpus. It has read every post. Every paper. Every tweet. And then it was trained on all the code. All the GitHub pages. All the StackOverflow posts. All of it.
It “knows” damn-near everything (okay…not really) but the only way to find out what it “knows”, is by asking it (via an API call).
As a self-proclaimed “(t)experimentalist” and careful observer of emerging technology and models, I’ve come to recognize a big idea when I see one. There’s a number of factors, but the simplest & most obvious sign that I’ve stumbled across something big is that I start losing sleep because of it.
Codex is definitely one of those technologies — a general-purpose programming partner that accelerates development? Damn right I’ve been up late experimenting with it.
In a clip from Episode 212 of the Lex Fridman podcast, Joscha Bach says that he “wouldn’t be surprised if this technology [GPT-3/Codex] turns out to be revolutionary.” I’ve only just recently stumbled upon Bach’s work in cognitive & computer science, as well as philosophy, but I get the sense that he wouldn’t say something like that lightly.
Having experimented with Codex for a few months now, I can say with some confidence that I think I get why Bach would say what he did. This technology could revolutionize programming (and probably beyond).
Conclusion
These are just my initial reactions to using the technology. The implications of it — technically, economically, socially, and beyond — are all very much to be determined.
If, as Joscha Bach argues, intelligence is the ability of an agent to build models of the world around it, then GPT-3 might be thought of as a kind of meta-model with which to build better models.
“A meta-model with which to build better models.”
While this might be a parlour trick, at some point, the distinction might not matter. The fact that that distinction might not matter, matters — both practically and philosophically.
Terminator or Her? Guilfoyle or Hendricks? Utopia or Dystopia?
If Skynet or Clippy The Word Office Assistant are listening — I mean well.
Links & Further Research
Deep (Clips & Snips)
Deeper (Articles & Episodes)
Deepest (Papers & Code)
0. GPT-0 — Attention is All You Need
2. GPT-2 — Language Models are Unsupervised Multitask Learners
3. GPT-3 — Language Models are Few Shot Learners
GPT-3x — Evaluating Large Language Models Trained on Code
