Introducing Code Llama: Powerful Language Models for Efficient Coding

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Sep 24 2023

In a paper published in the journal Meta AI Research, researchers introduced the Code Llama family of Large Language Models (LLMs). These models provide cutting-edge capability for jobs involving code. A few use cases for such jobs are infilling and comprehensive input support. Their ability to take instructions without prior examples makes them efficient coders. The family consists of three model types: fundamental models, Python-specific variations, and instruction-following models with various parameter sizes. The models are available for study and commercial use under a permissive license.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Background

The rapid advancement of LLMs has enabled their application in various domains. These models can do a variety of things and can understand spoken language. They work best when trained on large datasets that are relevant to particular fields and can connect normal language with specialized topic knowledge because of their ability.  Formal interactions with computer systems have proven to be useful for LLMs. These interactions include activities like program synthesis, code completion, debugging, and document generation.

The key focus of past studies on developing code using LLMs was the training of these models with code data. This training often utilized tools like AlphaCode, InCoder, and StarCoder. Code Llama takes a unique approach, starting with the Llama 2 foundation model, and then undergoes pretraining on a combination of general text and code data. The comparative analysis shows that Code Llama outperforms models created from code data performance and stands out among other code-generation tools. It achieves this distinction by utilizing "infilling" to enhance context-aware text completion and further demonstrates its prowess by handling extended input contexts and optimizing instructions.

Proposed Method

Training and Specialization

The primary parts of the Code Llama model family include Code Llama, Code Llama - Python, and Code Llama - Instruct. The size of each variant—7B, 13B, and 34 B—is determined by the needs for code production and comprehension. Code Llama designed the 7B and 13B models for code infilling in an IDE. In contrast, the 34B model in Code Llama focuses on code generation without the infilling aim. Code Llama trains its models on a large dataset with 500 billion code-heavy tokens. Based on the Llama 2 foundation model, these models inherit its weights. The modified models enable longer input contexts, improving efficiency and precision.

Code Llama

Python models can generate code in three sizes: 7B, 13B, and 34B. These models generate Python code, unlike other code generation models. They begin by initializing from Llama 2 models when they are training. They receive more training using a dataset of 500 billion tokens from the Code Llama. In the beginning, infilling is not one of their training goals. After the initial instruction, they undergo fine-tuning to better manage longer contexts.

Infilling and Long Context Handling

Code Llama specializes in code infilling by predicting missing sections. Applying causal masking improves this feature and enhances context-aware text completion capabilities in the 7B and 13B models. Code Llama introduced Long Context Fine-Tuning (LCFT) to handle long sequence processing challenges. This stage enables the models to process sequences of up to 16,384 tokens. This improvement enhances their long-range capabilities without increasing training costs. This is accomplished by altering the rotary position embedding's rotational frequencies present in the Llama 2 foundational models.

To enhance performance, Llama 2 adjusts the rotational frequency of position embeddings. The study found that Code Llama models perform well on sequences of up to 100,000 tokens. The Code Llama family's unique properties and training methods excel in coding.

Testing Results

Code Llama undergoes testing through two benchmarks: Python code generation and multilingual evaluations. These assessments show the model's performance, especially when tailored for larger contexts. Long-context fine-tuning improves the handling of prolonged sequences. Yet, there can be a modest decrease in performance for shorter contexts. However, all Code Llama models now come with long-context capabilities. The ability of a model to handle lengthy sequences is essential when dealing with actual applications. This underscores the significance of a model's ability to adjust to lengthier sequences.

The ablation study shows that Code Llama models outperform scratch-built models. This superiority is particularly notable when utilizing code data for fine-tuning. Also, Code Llama - Instruct shows coding advancements while still being beneficial. Self-instructed data can improve model performance and ensure proper code formatting.

Conclusion

To summarize, there are three different sizes and three versions of the Code Llama family. These models support infilling and huge contexts while prioritizing real-world utility. Although specific benchmarks can impact models, they outperform in real-world scenarios. The Code Llama - Instruct models aim to provide zero-shot instruction capabilities; however, there is still work needed in managing context and nuance.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.