3 remarkable things that happened in AI this week: Google saves millions of lives, “LCICL”, and Et Tu, Stack Overflow?
Google is going to save hundreds of thousands of lives. No, seriously.
What happened: Google introduced AlphaFold 3, the next iteration of its AI tool that predicts molecular structures and interactions. This is the latest iteration in the AlphaFold modelling series, which dates back to 2018, and has been used to predict protein folding as well as the shape of other molecules, including ligands (nope–me neither). The model is trained on a library of hundreds of thousands of known protein structures and is–according to people who understand these things–incredibly accurate, far better than previous tools.
Why it matters: Identifying the structure of an organic molecule–not just its composition, but its actual shape–is critical for understanding how complex molecules, particularly proteins, will interact. This has widespread applications in medicine and biology. No, I mean, seriously widespread applications, enabling an entirely different approach to drug discovery. AlphaFold 2 has won countless prizes and been cited tens of thousands of times in scientific papers. Unlike many of the topics I cover here, this isn’t a theoretical advance that will someday contribute in a small way; this will save lives starting in a few years.
OpenAI and Stack Overflow agree to monetize your work
What happened: Stack Overflow, a website where developers ask each other coding questions, announced that it will allow OpenAI to use its content to enhance the coding abilities of its models. It’s not entirely clear how this will work, but presumably OpenAI will train on Stack Overflow’s data (consisting of questions and answers from developers). The announcement also states that OpenAI will “provide attribution to the Stack Overflow community within ChatGPT,” which probably means that ChatGPT’s answers to coding questions might link back to the relevant Stack Overflow pages (which is one of the site’s terms and conditions for partners, although the OpenAI deal is large enough that it might well get to rewrite some of those).
Why it matters: all sorts of reasons. First, this will almost certainly make ChatGPT better at coding, which is interesting in its own right, and also because it looked at one point as if OpenAI might cede the coding-support-LLM space to specialist models. Stack Overflow has almost 60 million coding questions and answers which contain snippets of code with reasonably clear topic labels (from the questions) and very clear quality scores (through user upvoting); it’s a great data source for model training. Second, this is the first time (as far as I know) that OpenAI has agreed to provide backlinks to sources, similar to Perplexity and other tools; it’s a very different user experience and might point the way towards a slightly more open approach. Finally, Stack Overflow has something of a throwback, open, democratic ethos reminiscent of the earlier days of the internet, even though it’s a for-profit company owned by a listed European conglomerate. Cutting a deal with OpenAI to allow both companies to profit off the knowledge that users have contributed for free has made a number of those users angry, although it’s hardly the first time a company has sold access to user-generated content (and it’s not even Stack Overflow’s first such deal).
LCICL: a terrible acronym for a great new tool
What happened: okay, so the paper calls it “long-context ICL,” and I’ve abbreviated it to LCICL, which I’ve decided rhymes with pickle. ICL, an acronym I didn’t make up, stands for “in-context learning,” which means you include a handful of examples in your LLM prompt. This works pretty well. A new paper (and explanatory tweet) takes this further, using the new long-context-window LLMs to provide hundreds or even thousands of examples, and they find that it works better still, with performance improvements only leveling off after several hundred examples, and a significant net performance gain.
Why it matters: obviously more accuracy and better performance from LLMs is good, and LCICL is a lot cheaper than spending seven trillion dollars to move from a zillion parameters to a billion gazillion parameters. I might be too bullish, but I think it will take its place along fine-tuning and retrieval-augmented generation (RAG) as the tools available to LLM users to create the “last mile” models for their specific use cases. Also, the acronym is fun to say. Do you not like it? Because I can stop this car right now.
3 remarkable things is a more-or-less weekly roundup of interesting events from the AI community over the past more-or-less week
