Even within the context of the breathless pace of innovation in AI, Meta’s and Microsoft’s July 18 joint announcementof the release of Meta’s Llama 2 foundational model – the second variant of its large language model Meta AI (Llama) model – was attention-worthy.
Why so, one might ask? The answer lies in a possible future where AI large language models (LLMs) are dominated by a very few, privately held, players who have both the technical skills and computational power to build the leading so-called ‘foundational’ models. That is, the models upon which other downstream tasks such as chatting or information retrieval or document summarization rest. Open AI has created and maintained clear performance leadership with closed-source solutions whose algorithms, training data and performance are – despite the company’s name – generally not shared publicly. Anthropic provides modest competition but are also closed. Conversely, credible open-source solutions have struggled to maintain pace. Therefore, anything that suggests an open-source foundational model might approach the performance of the leading closed solutions is significant. And potentially disruptive.
Why then, does Llama 2 suggest credible open-source solutions might just be at hand? First, the technical basis for Llama 2 is well documented, in contrast to closed models such as the GPT-3.5 and GPT-4 models from Open AI or the Claude models from Anthropic. Importantly, Meta released details of its use of reinforcement learning human feedback (RLHF) and proximal policy optimization (PPO) to fine-tune the basic model for specific use cases such as chatbot operation. These techniques are critical to approach the performance of leading models such as GPT-4 and – given the associated labor and computational costs – are often prohibitively expensive for small organizations. The same approach was used to enhance model safety and helpfulness. That is, to guide Llama 2 away from abusive or biased behavior. This relative transparency and commitment to maximize benign characteristics has allowed Meta to incentivize academic researchers to join an ‘open innovation’ community committed to further enhancing Llama.
Second, the performance of Llama 2 is credible. Meta compared the performance of the released 7B (7 billion weight), 13B, and 70B model variants with contemporary open large language models such as MPT (from Mosaic ML, a US startup) or Falcon (from the Technology Innovation Institute, funded by the UAE government) across a variety of benchmarks ranging from natural question-answering tests to logical inference. Llama 2 significantly outperformed the comparable open models. When compared to leading closed models, the results were more mixed: performance was close to that of GPT-3.5 but still trailing GPT-4, sometimes significantly. And in other areas, such as the size of the supported context window, which dictates the size of the documents that Llama 2 can process, it lags well behind some closed models. Accordingly, Llama 2 narrows the open community ‘gap’ to GPT-4 but does not eliminate it. Nevertheless, Meta is betting that its approach to open collaboration will provide the innovation needed to reach GPT-4 levels of performance.
A further tailwind to innovation around Llama 2 is provided by the community licensing terms surrounding the use of the model. AI model licensing terms can be arcane. Not so for Llama 2, where Meta grants users a liberal world-wide license to deploy, and modify, the Llama 2 model for both research and commercial purposes. A deployer must seek a license from Meta only if usage exceeds a hefty 700 million monthly active users. This would apply only to the largest commercial deployments.
Finally, Meta announced Llama 2 in combination with Microsoft. Microsoft is well known of course for investing in, and deploying, Open AI GPT models. Nevertheless, supporting Llama 2 allows Microsoft to diversify its Azure model catalog, play in both the open and closed AI domains and see, over time, who wins. For Meta, the visible support of Microsoft for Llama 2 moves forward its apparent goal of disrupting the technical and economic AI ‘walled gardens’ upon which Open AI, Anthropic and others are betting.
The game around the future of AI is truly afoot.
