Using AI to Advance Air Traffic Control Communication Transcription

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Oct 25 2023

In a paper published in the journal Aerospace, the authors discussed the Air Traffic Controller Communications Optimization and Open Data (ATCO2) project, which aimed to improve air traffic control communications through artificial intelligence (AI). They covered aspects like Automatic Speech Recognition (ASR), Natural Language Processing, language identification, and biasing ASR with surveillance data. The project provided open-sourced data for research and achieved a 17.9% Word Error Rate on public ATC datasets, making it a significant step in enhancing automatic speech understanding for ATC communications.

Prior Research

Developing ASR for ATC communications enhances aviation safety and efficiency. Challenges include domain-specific adaptation, costly data collection and transcription, and noisy Very High Frequency (VHF) audio data. Advanced ASR systems can reduce ATCo workload and improve safety. Past work in ATC ASR and understanding includes research for trainee ATCo training, workload estimation, benchmarking databases, and semi-supervised learning. Open-sourcing ATC-related databases has also been a focus, with various initiatives and challenges in the field.

ATCO2 Project and Data Scarcity in ATC

The critical need for substantial and accurately transcribed data to support AI-based tools, particularly in ATC, underscores. It introduces the ATCO2 corpora to mitigate data scarcity by addressing four significant challenges. Existing ATC-related corpora primarily focus on ASR, while the paper emphasizes the importance of AI tools transcribing and comprehending ATC communication. These tasks include identifying speaker roles and extracting essential information such as callsigns and commands, which the ATCO2 corpora comprehensively addresses with detailed tags.

The second challenge highlighted is the unsuitability of out-of-domain ASR and Natural Language Processing (NLP) corpora for ATC due to the unique grammatical structure and specific vocabulary defined by ICAO. The ATCO2 project collects a substantial volume of ATC-specific data, which is pivotal for developing ASR and understanding engines tailored specifically for ATC.

The third challenge concerns more annotated data in the ATC research community. The ATCO2 project tackles this challenge by releasing a vast corpus that includes over 5000 hours of automatically transcribed data (ATCO2-T set) and 4 hours of manually annotated data (ATCO2-test-set-4h). These transcriptions exhibit robustness, with word error rates as low as 9%, mainly when training ASR engines with the ATCO2 corpora.

Lastly, the paper addresses the absence of a standardized metric for evaluating the quality of non-transcribed data before transcription. While new corpora typically undergo collection and labeling phases with some quality filtering, the ATCO2 project introduces a quality estimation system to assist in selecting high-quality audio files for human transcription.

The paper provides in-depth insights into the ATCO2 system by detailing the data collection pipeline. This system encompasses various stages, such as preprocessing, diarization, ASR, and named entity recognition (NER), all designed to obtain and process ATC communication data efficiently. Researchers present the data collection pipeline as a Python script, worker.py, which offers dynamic adjustments to the logic and data flow of the pipeline. Key components, including segmentation, diarization, ASR, and NER, are encapsulated in BASH scripts with a unified interface, ensuring an organized and efficient workflow.

Furthermore, the paper delves into quality estimation for data transcription, introducing a scoring system based on several metrics to rank the quality of audio recordings in the ATCO2 corpora. This score integrates signal-to-noise ratio, the number of speakers, speech duration, English language detection, ASR confidence, and word count. The primary goal is to select the most intelligible and clean data for human transcription, thus contributing to creating "gold transcriptions" for evaluating ASR and NLP systems.

Additionally, the paper sheds light on the runtime characteristics of the ATCO2 processing pipeline, with a detailed breakdown of processing times for various components. ASR and speaker diarization are the most time-consuming due to their AI-powered nature. The paper underscores the computational intensity of the pipeline by revealing a real-time factor of 4.47, emphasizing the need for efficiency. Finally, the paper provides specific metrics for the average processing time of a five-second recording, offering a comprehensive overview of the challenges of data scarcity in ATC and positioning the ATCO2 project as a valuable resource for research and development in this domain.

Data Collection and Annotator Community

The data collection platform relies on dedicated volunteers capturing ATC communication using their receiver equipment. These individuals include aviation enthusiasts and those interested in technology. The platform's architecture ensures scalability and simplicity, divided into feeder equipment, back-end, and front-end components. The feeder equipment captures conversations and transmits data to the back end, which stores recordings and transcripts and provides interfaces. The front end offers web access to public statistics and documentation. Since its launch in March 2023, the project has recorded ATC communication from 24 airports in 14 countries, providing a robust data source for ATC research. Data annotators also play a crucial role in transcribing and enriching the dataset.

Improving ATC with ASR and NLU through ATCO2

The ATCO2 project presents significant advancements in ASR technology, particularly tailored for ATC communications. The ASR system undergoes training in various scenarios, encompassing supervised data and specialized ATCO2 datasets. Additionally, the innovative concept of "callsign boosting" improves the recognition of aircraft callsigns in ATC conversations, enhancing the accuracy and reliability of transcriptions in this critical domain. Furthermore, the ATCO2 corpora serve as a valuable resource for exploring Natural Language Understanding (NLU) applications in ATC, facilitating the extraction of essential information from spoken communications to enhance safety and efficiency in air traffic management.

Conclusion

In summary, the ATCO2 project offers significant insights into developing a reliable transcription engine and best practices for ATC communications. Training ASR systems exclusively on ATCO2 data yields competitive results, even for challenging accented speech in ATC test sets. The integration of ATC surveillance data greatly enhances ASR accuracy, particularly for callsign recognition. The ATCO2 corpora is also a valuable resource for NLU in ATC, with NLU modules for callsigns, commands, values, and speaker roles. These lessons and the provided ASR and NLU engines represent substantial advancements in the ATC domain, with no current equivalent research or commercial activity based on publicly available data.