Redefining Chemical Language Models: Embracing Invalid Outputs

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Apr 9 2024

In a paper published in the journal Nature Machine Intelligence, researchers explored the capabilities of generative machine learning models, focusing on their capacity to generate novel molecules with specific chemical or biological properties. These models, particularly those trained on simplified molecular-input line-entry system (SMILES) representations, had undergone extensive validation and widespread adoption. However, the issue of generating invalid SMILES strings had been a common concern, prompting efforts to address it.

The researchers presented evidence suggesting that generating invalid outputs was not detrimental but beneficial for chemical language models. They argued that it served as a self-corrective mechanism, filtering out unlikely samples from the model's production. Conversely, enforcing valid outputs could introduce biases and limit the model's ability to learn and generalize effectively. These findings challenged the prevailing view of invalid SMILES as a flaw and framed them as a feature integral to the model's functionality.

Related Work

Recent research has challenged the perception that generating invalid SMILES is a limitation of chemical language models, suggesting it may benefit model performance. Researchers uncovered a self-corrective mechanism that filters low-quality samples from model outputs by demonstrating that they sample invalid SMILES with lower likelihoods.

Removing valency constraints from the SELFIES language provided causal evidence that generating invalid outputs improves model performance. It facilitates the exploration of chemical space and enhances the elucidation of complex chemical structures from minimal data. These findings redefine the role of invalid SMILES and highlight their potential as features rather than limitations in chemical language models.

Dataset Curation Analysis

In the initial phase of the study, datasets were curated from the chemical database of bioactive molecules (ChEMBL) database, with duplicate and unparsable SMILES removed, followed by the extraction of a subset of molecules for training chemical language models. These models were trained on samples ranging from 30,000 to 300,000 molecules, ensuring comprehensive coverage of chemical space.

Additionally, experiments explored the impact of training set chemical diversity and data augmentation through SMILES enumeration. Researchers replicated the training across ten independent datasets to assess model performance variability, creating 180 distinct models for analysis. Moreover, they trained models on the Gaussian distribution-based database of molecular structures up to 13 atoms (GDB-13) database to validate findings across different datasets.

Researchers employed two primary architectures for chemical language models: long short-term memory (LSTMs) and generative pre-trained transformers (GPTs). They trained models to minimize cross-entropy loss and validated them using a variety of metrics, including Fréchet ChemNet distance, Jensen–Shannon distances, and natural product-likeness scores. The evaluation revealed the robustness of LSTMs and GPTs in learning the statistical properties of molecular datasets and generating novel molecules with desired properties.

Exploration of the model outputs revealed an intriguing phenomenon regarding the generation of invalid SMILES. The analysis demonstrated that invalid SMILES were sampled with lower likelihoods than valid SMILES, suggesting a self-corrective mechanism inherent in the model. Furthermore, removing valency constraints in SELFIES representations provided causal evidence that generating invalid outputs could enhance model performance, challenging the prevailing assumption that such outputs are detrimental.

The study extended beyond model training and evaluation to assess the generalization of chemical language models to unseen chemical space and their utility in structure elucidation. Results showcased the efficacy of language models in generating structural hypotheses from minimal analytical data, providing valuable insights into their potential applications in drug discovery and chemical synthesis.

Throughout the analysis, researchers employed robust statistical methods to ensure the reliability and reproducibility of the findings, thereby facilitating a comprehensive understanding of the capabilities and limitations of chemical language models in exploring and navigating complex chemical space.

Model Representation Comparison

The study aimed to compare the performance of chemical language models trained on SMILES and SELFIES representations. Contrary to previous assumptions, models trained on SMILES, which can produce invalid outputs, outperformed those trained on SELFIES in terms of generating molecules that better matched the training set. This superiority was consistent across various training datasets, model architectures, and data augmentation techniques.

The analysis revealed that invalid SMILES were sampled with lower likelihoods than valid ones, suggesting a self-corrective mechanism in the model. Moreover, experiments with modified valency constraints in SELFIES demonstrated that generating invalid outputs improved model performance, challenging conventional beliefs about the detrimental effects of invalid outputs.

The study also investigated how the choice of representation affected the exploration of chemical space and the generalization capability of the models. Models trained on SMILES explored more of the GDB-13 chemical space than those trained on SELFIES, indicating that the latter had limitations in generalizing to unseen chemical space.

Additionally, the ability to generate invalid outputs improved structure elucidation tasks, showcasing the potential of chemical language models to provide accurate hypotheses about unknown chemical structures from minimal analytical data. These findings challenge the notion that invalid outputs are inherently problematic and highlight the importance of considering the broader implications of representation choices in chemical language modeling.

Conclusion

To sum up, the study underscored the pivotal role of representation choices in chemical language modeling, challenging prevailing assumptions about the impact of invalid outputs. Demonstrating the superior performance of models trained on SMILES and the benefits of generating invalid outputs emphasized the need for a nuanced understanding of model behavior and its implications for chemical space exploration and structure elucidation tasks.