For event understanding, the objective is to fathom the content and interconnections within textual events. This encompasses intricate information extraction tasks: event detection (ED), event argument extraction (EAE), and event relation extraction (ERE). In a recent submission to the arXiv* server, researchers introduced an event-understanding toolkit called OmniEvent, which embodies comprehensiveness, fairness, and user-friendliness.
Study: OmniEvent: A Comprehensive Toolkit for Event Understanding in Text. Image credit: SWKStock/Shutterstock
*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.
Background
The essence of grasping events accurately serves as a cornerstone for human comprehension of the world. Event understanding entails the identification of real-world occurrences within texts and an exploration of their intricate connections. This, in turn, bestows its advantages upon diverse downstream applications, ranging from stock prediction to adverse drug event detection, narrative event prediction, and legal case analysis.
Within the framework, event understanding encompasses three intricate information extraction tasks: ED, EAE, and ERE. The objective of ED is to identify event triggers—words or phrases that conjure up specific events in texts—and to categorize those events. In EAE, the task involves extracting event arguments associated with each trigger and classifying their roles. In ERE, the aim is to discern complex relationships between events, typically encompassing temporal, coreference, causal, and subevent relations. ED and EAE collectively constitute the conventional event extraction (EE) task.
In natural language processing (NLP) research, several toolkits and systems have emerged for event understanding. These efforts typically concentrate on enhancing EE systems for better performance on public benchmarks or robust real-world applications. However, they are often confined to specific EE models and lack versatility for further development. Some user-friendly algorithmic frameworks have been meticulously designed, but their scope is not tailored for event understanding, limiting their support.
Additionally, there is a pressing need for support for large language models (LLMs). Additionally, present systems do not fully address how complicated data processing and evaluation frequently result in inconsistent and biased conclusions.
OmniEvent framework
OmniEvent's overall architecture includes a data pre-processing module for unified pre-processing. Users can employ supported datasets or customize their own. Post-pre-processing, OmniEvent offers a flexible modular framework for model implementation. It abstracts mainstream models into three basic modules that are highly encapsulated. Users can easily assemble models by combining the provided modules or creating their own, thereby reproducing various widely used models.
Comprehensive Support: OmniEvent encompasses the entire event understanding pipeline, covering ED, ERE, and EAE tasks. It offers comprehensive model and dataset coverage. For ED and EAE, OmniEvent includes four mainstream method paradigms. It further implements a unified pairwise relation extraction framework and an antecedent ranking method for event coreference resolution. It also develops a joint event relation extraction model.
OmniEvent boasts a collection of widely-used Chinese and English event understanding datasets spanning general, legal, and financial domains. The toolkit provides pre-processing scripts for dataset conversion and standardizes output space to ensure fair evaluations.
Fair Evaluation: Earlier research highlighted pitfalls in EE evaluation, and OmniEvent addresses these issues. It offers solutions to data pre-processing discrepancies, output space differences, and the absence of pipeline evaluation. Users can specify pre-processing options, standardize output space, and obtain unified sets of predicted triggers for fair comparisons.
Easy-to-Use: Designed for user-friendliness, OmniEvent provides off-the-shelf models for easy adoption. It employs a modular implementation, abstracting mainstream models into basic modules, allowing users to create new models effortlessly. OmniEvent efficiently supports Large Language Models (LLMs) through DeepSpeed integration.
Online Demonstration: OmniEvent also powers an online demonstration system using the T5-base model for EE and the Roberta-base model for ERE. Users can choose language, task, and ontology, with results displayed in the output field. The system provides end-to-end event understanding results, including ED, EAE, and ERE, in the form of a knowledge graph.
Results and analysis
Researchers conducted several empirical experiments to assess OmniEvent's efficacy on widely used datasets. For EE, representative EE models within OmniEvent are evaluated across diverse datasets using a unified evaluation protocol. Results demonstrate OmniEvent's effectiveness, achieving similar performance to their original implementations. Configuration files in YAML format are provided for result reproducibility.
For ERE, empirical experiments on various widely used datasets assess ERE models developed in OmniEvent. It was observed that the results align with or slightly surpass recent works on ERE, validating OmniEvent's ERE models. Configuration files with hyper-parameter settings are supplied for replication.
Experiments using LLMs
OmniEvent supports efficient fine-tuning and inference for LLMs. Multiple models are trained on different datasets, demonstrating LLMs' efficiency and performance scaling with model size. The encoder-decoder model (Flan-UL2) excels on multilingual training corpus (ACE 2005) and RichERE datasets, confirming the utility of LLMs in OmniEvent. The ERE results, possibly influenced by complex output spaces and long contexts, suggest potential avenues for future LLM research in event understanding.
Conclusion
In summary, researchers proposed the OmniEvent toolkit, a comprehensive, equitable, and user-friendly toolkit designed to address the shortcomings. OmniEvent delivers thorough model implementations and covers the whole event comprehension pipeline, unlike other systems. It also facilitates efficient fine-tuning and inference with LLMs. Furthermore, OmniEvent remedies the discrepancies highlighted in the EE evaluation. With a modular structure and a range of pre-built models, this toolkit is user-friendly and highly accessible. Future work includes expanding support for models and datasets.
*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.
AI vs. Human Creativity