LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Aug 13 2023

In an article recently submitted to the arXiv* server, researchers addressed the need to evaluate the performance of Large Language Models (LLMs) across various Natural Language Processing (NLP) tasks in different languages. While multiple frameworks existed, they often posed challenges for customization based on particular tasks and datasets. Introducing the Large Language Model Evaluation Benchmark 1 (LLMeBench1) framework, initially designed for evaluating Arabic NLP tasks using OpenAI’s Generative Pre-trained Transformer (GPT) and BLOOM models, the framework could be effortlessly tailored for any NLP task and model, irrespective of the language.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

LLMeBench1 offered zero- and few-shot learning options, allowing the addition of a new custom dataset within a span of fewer than 10 minutes and utilizing the user’s model Application Programming Interface (API) keys for task evaluation. The framework underwent testing on 31 distinct NLP tasks employing 53 publicly accessible datasets across 90 experimental setups, involving about 296K data points. The intention was to make the framework open-source for the broader community, and a video showcasing its functionalities was accessible online.3.

Background

The rapid rise of sophisticated LLMs, driven by in-context learning (ICL), has garnered significant attention across academic and industrial realms. These models, employing the ICL approach, enabled diverse applications, including addressing mathematical reasoning challenges. However, to accurately assess their potential, a thorough evaluation against state-of-the-art benchmarks was essential. Comprehensive evaluation not only revealed advantages and limitations but also guided human-LLM interactions and their application in critical domains like healthcare and finance.

Several initiatives evaluated LLMs on standard NLP tasks, such as the Holistic Evaluation of Language Models (HELM) project and the BIG-Bench initiative, even extending to low-resource languages. Evaluating these models across various tasks presented challenges in terms of effort, costs, and complexity.

To overcome this challenge, the present paper introduced LLMeBench, a versatile framework designed to evaluate LLMs comprehensively. LLMeBench enabled diverse LLM assessment, seamless task, and dataset integration, and supported zero- and few-shot learning. With features like automatic example selection, caching, extensive logging, and varied task recipes, LLMeBench served as an open-source benchmarking solution. It empowered experts and newcomers to explore LLM capabilities across NLP tasks, enhancing their application in the field.

Related work

Efforts to evaluate the performance of LLMs on standard NLP tasks were initiated following the introduction of ChatGPT. Many studies explored this field, conducting comprehensive assessments of LLMs for English and multilingual evaluations. Initiatives like BIGBench evaluated various tasks, including those for non-English low-resource languages. Several frameworks, including Evaluation of LLMs (EVALs), OpenICL, and PromptBench, were developed for evaluations, each with specific focuses and methodologies. In comparison, the LLMeBench approach stood out for its customization, support for zero- and few-shot learning, caching mechanism, and implementation for diverse datasets, LLM models, and tasks.

Proposed method

In the present study, the architecture of the LLMeBench framework was deliberately designed to streamline the integration of common elements across various experimental setups. Its primary objective was to establish a consistent structure for input and intermediary results, irrespective of the specific task under evaluation. This was accomplished by implementing a well-structured pipeline mechanism, effectively leveraging key-value dictionaries to facilitate the seamless flow of data among different process stages.

The LLMeBench framework comprises four core modules, each serving a distinct purpose within the evaluation workflow. The journey begins with the Dataset module, acting as the point of entry for individual input samples (Si), which are directed into subsequent stages. Once an input sample is received, the Asset module takes charge, generating prompts and transmitting them to the Model module for processing. This step ensures that the input data is appropriately prepared and formatted for subsequent stages. The Model module, responsible for prompt processing, employs the designated LLM model to generate responses based on the provided prompts. The resulting responses are then returned to the Asset module for further post-processing.

Following the completion of input sample processing, the Evaluation module comes into play. This module is dedicated to assessing the performance of the LLM model's responses, employing pre-defined metrics to measure the quality and effectiveness of the generated outputs. Throughout the entire process, the Benchmark Driver module orchestrates seamless communication between the different modules, ensuring efficient data flow between Dataset, Asset, Model, and Evaluation. Additionally, the framework incorporates a cache system to store processed output results. This caching mechanism enhances efficiency by preventing redundant computations, storing outputs for potential post-processing, and contributing to an optimized evaluation process.

Experimental results

The LLMeBench framework underwent a thorough evaluation encompassing various Arabic NLP tasks and datasets. This evaluation involved extensive experimentation, employing both zero-shot and few-shot learning methods with state-of-the-art LLM, including GPT-3.5-Turbo, GPT4, and the 8-bit version of the BLOOMZ 176B model. The assessment considered task-specific metrics from existing literature, covering 31 NLP tasks categorized by ACL tracks, 53 associated datasets, and 3 models in 2 learning setups, with all task recipes available within the framework.

Conclusion

To sum up, this study introduced LLMeBench, an open-source benchmarking framework designed to streamline the process of agile LLM benchmarking. LLMeBench offered a customizable infrastructure through a modular design, enabling the integration of new tasks, datasets, and model APIs. The framework included efficient caching mechanisms that reduced the time, costs, and effort associated with task evaluations.

At the time of the study, LLMeBench encompassed 31 predefined recipes covering a range of NLP tasks, with the potential for expansion to include new tasks, datasets, and LLM models. Continuous updates are planned to make the framework a valuable resource for researchers and industry practitioners involved in LLM evaluation and benchmarking. The research community is also encouraged to actively contribute to this collaborative effort. The aim is to extend the framework's capabilities by incorporating more tasks and languages, fostering growth through community engagement. Future enhancements encompass cross-validation datasets, models with various configurations, and improved approaches for few-shot selections. The goal is to enhance accessibility by enabling the seamless utilization of both offline and online models for inference purposes.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.