1. Introduction

Natural Language Processing (NLP) has empowered machines with language understanding and generation capabilities. The advent of the transformer architecture and the subsequent investigations into pre-trained language models have revolutionised NLP. Transformers enable massive parallel training and scalability across downstream tasks. A dominant paradigm emerged in which pre-training is conducted on large text corpora. Transformers followed by task-specific fine-tuning yielded significant improvements over word-based representations. In this line of research, GPT-2 started the trend of applying the transformer-auto-regressive architecture for text generation tasks. The success of BERT and RoBERTa triggered investigations into the multi-purpose capabilities of transformer-based architectures for different tasks regardless of the input and output structure. An intensive period of inquiry followed with numerous improvements in architectural designs and training strategies emerging for every large pre-trained language model. The popularity of large pre-trained language models sparked concern regarding their extensive computational and environment cost, multi-modality compatibility, and generalisation performance. Several venues have been explored with the aim of reducing pre-training costs with distillation techniques or improving training efficiency with architectural adaptations. Large pre-trained language models are continuing to evolve. The release of huge models with billions of parameters provoked a game of size in the community raising questions regarding the challenge of scaling models up and beyond the current paradigm limitations, the justifications of spending on extensive resources, and the recruitment of more data with its corresponding challenges (Fan et al., 2023).

Meta’s Large Language Model (LLM) family, LLaMA (Large Language Model Meta AI), was recently made open access. In contrast to the trending (GPT-3-like) paradigm of training autoregressive models on web text corpus (e.g., Common Crawl), LLaMA-7B was studied for training a pretrained masked language model (MLM) transformer for academic text (e.g., arXiv, PubMed) and then for fine-tuning it, using supervised and reinforcement learning from human feedback (RLHF), to obtain a dialogue model called LLaMA-7B-Chat, targeting alignment with respect to user intent (Leivada et al., 2023).

2. Evolution of NLP

Natural Language Processing (NLP) is a subclass of Artificial Intelligence (AI) that enables machines to read, comprehend and interpret human language. NLP is concerned with the development of algorithms and models that can process, analyze, and generate natural language text and speech (Khanmohammadi et al., 2023). The history of natural language processing begins with the development of the first language translation programs in the 1950s. These programs depended on the substitution of words in sentences and their comprehension was based on replacements in the target language. The introduction of statistical methods in the early 1980s represented the beginning of more sophisticated language models and posed a sweet challenge for the development of more robust models. The emergence of neural networks and deep learning in the early 2010s led to significant advancements in NLP and the proposal for a novel architecture that combined convolutional and recurrent networks for language modeling. The development of the transformer architecture in 2017 by Vaswani et al. represented the most significant breakthrough in the field, with the publication of Attention is All You Need.

Riding on the capabilities of the transformer architecture, a new breed of models emerged, referred to as Large Language Models (LLMs), some of them trained with hundreds of billions of parameters and trillions of tokens of textual data. BERT from Google Research, GPT-3 and GPT-4 developed by Open AI, and ChatGPT the well-known conversational agents are some of the most famous LLMs in the NLP field. Language modeling, machine translation, and sentiment analysis are among many natural language processing tasks that have been shown to perform remarkably well with these models. Their development has revolutionized language processing and opened new possibilities for humans to interact with machines using natural language.

3. Meta’s AI Language Model

Large language models (LLMs) have undergone improvement in their ability to imitate human behavior (Barua, 2024). In particular, LLMs may exhibit remarkably autonomous and intelligent skills: they can obtain perceptions, make plans (for better or worse), and act on those plans through conversation, vision manipulation, and code. This growing concern raises profound issues about the capabilities of these models, with scientists, ethicists, and science fiction writers sparring over the implications. Meta’s AI language model called “Llama” also belongs to this class of generalized LLMs.To explore Llama, four generic, non-technical prompts about it are presented as a self-Q&A between its authors.

A pretrained BART (lewis et al. 2020) model of around 400 million parameters was used for all experiments. BART is a denoising sequence-to-sequence pre-training model where a model learns to generate uncorrupted sequences given corrupted input sequences. Starting from the initial BART checkpoint, the model was fine-tuned on clickbait detection with 0.2% of the training set for all experiments to allow substantial comparison while keeping the training cost reasonable (Sejnowski, 2022). The total number of training steps was 358 and the batch size was set to 8.

3.1. Development and Features

The sharp rise of Llama has been remarkable, with the Llama 1 model quickly rising to prominence after its release in February 2023. This can be partially credited to the open weight availability of each of the models in the family. However, snowballing just model availability wouldn’t be sufficient for the current wave of interest in Llama; instead, its companion open-source training code, data, and instructions are important contributors. Put simply, Llama 1 was made available as a scalable research platform where researchers could opt to fine-tune the model, train from scratch, and investigate the model-building pipeline at large.

One prominent example that perfectly demonstrates the importance of Llama’s multiple families of open offerings is the 7B-parameter Llama 1 model. Although other models of similar size (e.g., 7B GPT-2 and 7B Ada) existed before, no other public models adopted similar datasets, architectures, or training settings. As such, the 7B Llama 1 model served as a reference point with many comparably sized models being proposed and behaviorally evaluated against it in the subsequent months after the introduction of Llama 1. This reinforces the notion that, in terms of development velocity, it is not so much the absolute model size but the model-building pipeline at large and its components that matter (Fan et al., 2023).

3.2. Applications in NLP

Natural Language Processing (NLP) is a branch of artificial intelligence that deals with the interaction between computers and humans using natural language. Large language models have energized and revolutionized the field by improving translation quality and enabling zero-shot transfer, text completion, and text generation tasks such as chatbots and virtual assistants (Barua, 2024). A plethora of large language models has come into play in recent years, each developed by different groups and having its own unique capabilities and underlying design.

Llama, developed by Meta, belongs to the transformer language models category and was introduced as a family of foundational large language models. Meta emphasizes that it trained Llama on publicly available datasets in line with its principles of openness and avow openness to adversaries (Pahune and Chandrasekharan, 2023). The 7B parameter Llama-7B language model was released with the intention that it would be used for research use cases such as exploring its capabilities, limitations, and biases through peer-reviewed publications, as opposed to commercial endeavors.

4. Impact on Industry

The impact on the industry is multifaceted, potentially affecting business practices, research and development, and the future direction of the field. If open research and model developments continue to be strong, it may influence the kinds of tools or systems that are broadly used within companies, research labs, and the public. Alternatively, a small number of systems trained by a few actors and made commercially available could create a situation in which many organizations become dependent on a subset of huge, expensive, and potentially proprietary technologies (Abdalla et al., 2023). This could have the scientific effect of creating a “black box” on the state of models, as well as potentially increasing inequality due to smaller organizations and developing countries lacking access. There is a tension between rapid advancement in the state of the art versus support for broader needs such as safety, reliability, interpretability, and inclusiveness.

In the past few months, there has been an influx of these systems, and interest in using them as tools for research and application has become widespread. In many cases, there is not yet a clear understanding of any limitations or pitfalls related to their use, nor best practices for using them. Findings from the review and analyses could help explore how companies and researchers are approaching these technologies in practice, the kinds of questions and challenges being explored, and how they fit with current NLP tools and practices. In doing so, it could help promote better understanding, support, and use of the technologies within research and industry.

5. Future Trends and Challenges

Meta has made great strides in refining instruction-tuned foundational Large Language Models (LLMs) and producing open and reproducible LLMs at scale, beginning with LLaMA (Joublin et al., 2023). As part of the effort, Meta has trained and released LLaMA 2, a collection of three state-of-the-art open LLMs tuned for dialogue use cases, trained on publicly available datasets, improving upon an earlier model that tested well in a private setting. With this model release, the research community has the opportunity to study the strengths and limitations of these models, and consider the impact different circumstances may have on their safety, competitiveness, and deployment.

To facilitate the use of LLaMA 2 and better align with Meta’s safety and ethics commitments, a discussion campaign called “LLaMA 2 Red Teaming” was organized, bringing together researchers with diverse cultural backgrounds and expertise in AI ethics, mechanical turk labeler worker safety, toxicity mitigation, and potential uses of LLMs in law enforcement (Barua, 2024). This poster reflects on the red teaming experience with LLaMA 2 and emphasizes its collaborative aspect, inspiring future research on red teaming LLMs (and AI systems more broadly) and other models released as part of the pursuit of responsible AI. Meta is open to academics/researchers demoing or testing LLaMA and welcomes feedback about fairness, robustness, safety, evaluation, and any broader social implications, especially regarding their deployment.

References:

Fan, L., Li, L., Ma, Z., Lee, S., Yu, H., and Hemphill, L. “A Bibliometric Review of Large Language Models Research from 2017 to 2023.” 2023. [PDF]

Leivada, E., Dentella, V., and Murphy, E. “The Quo Vadis of the Relationship between Language and Large Language Models.” 2023. [PDF]

Khanmohammadi, R., M. Ghassemi, M., Verdecchia, K., I. Ghanem, A., Bing, L., J. Chetty, I., Bagher-Ebadian, H., Siddiqui, F., Elshaikh, M., Movsas, B., and Thind, K. “An Introduction to Natural Language Processing Techniques and Framework for Clinical Implementation in Radiation Oncology.” 2023. [PDF]

Barua, S. “Exploring Autonomous Agents through the Lens of Large Language Models: A Review.” 2024. [PDF]

Sejnowski, T. “Large Language Models and the Reverse Turing Test.” 2022. [PDF]

Pahune, S. and Chandrasekharan, M. “Several categories of Large Language Models (LLMs): A Short Survey.” 2023. [PDF]

Abdalla, M., Philip Wahle, J., Ruas, T., Névéol, A., Ducel, F., M. Mohammad, S., and Fort, K. “The Elephant in the Room: Analyzing the Presence of Big Tech in Natural Language Processing Research.” 2023. [PDF]

Joublin, F., Ceravola, A., Deigmoeller, J., Gienger, M., Franzius, M., and Eggert, J. “A Glimpse in ChatGPT Capabilities and its impact for AI research.” 2023. [PDF]