Introducing Stability AI FreeWilly Language Models

July 25, 2023: Stability AI has unveiled its latest groundbreaking technology in the form of two new large language models (LLMs) called FreeWilly1 and FreeWilly2. These powerful models are designed to excel in intricate reasoning, understand linguistic subtleties, and answer complex questions, particularly in specialized domains like law and mathematics. Let’s break down the critical aspects of these models in a simple and easy-to-understand way.

The FreeWilly Models: Based on Meta’s LLaMA and LLaMA 2 open-source models, FreeWilly1 uses the LLaMA 65B foundation model, while FreeWilly2 uses the newer LLaMA 270B foundation model.

Smaller and Greener: Stability AI implemented the “Orca” AI training methodology developed by Microsoft. This involves training a smaller model with step-by-step reasoning processes from a larger model rather than just mimicking its outputs. The FreeWilly models were trained on a new, smaller dataset, including synthetic data, using instructions from four datasets created by Enrico Shippole. This resulted in a dataset containing 600,000 data points, only about 10% of the size of the original Orca dataset.

Superior Performance: Despite their smaller dataset and reduced energy consumption, the FreeWilly models perform exceptionally well, outperforming even ChatGPT on GPT-3.5 in some instances.

Promising Synthetic Data: Stability AI’s use of synthetic data offers potential solutions to the “model collapse,” which can occur when models are trained on AI-generated data. The FreeWilly models demonstrated strong performance even when trained with synthetic examples.

Open Access and Research: The FreeWilly models are released under a non-commercial license to foster available research and promote access to AI in the community.

Setting New Standards: Stability AI envisions FreeWilly1 and FreeWilly2 as pioneers in open-access LLMs, empowering natural language understanding and enabling complex tasks. These models open up endless possibilities for the AI community and inspire new applications.

How to Access: Researchers and developers can access the weights for FreeWilly2 as-is, while FreeWilly1’s weights are released as deltas over the original model.

Introducing FreeWilly1 and FreeWilly2 marks a significant leap in AI technology, providing powerful language models that can understand and respond to complex inputs with nuance and sophistication. As AI continues to evolve, these models are expected to play a prominent role in shaping the future of artificial intelligence.