Enhancing Glycemic Control in Type I Diabetes with Offline Reinforcement Learning

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Oct 20 2023

In an article recently submitted to the ArXiv* server, researchers discussed improving closed-loop systems for type I diabetes glycemic control. These systems relied on simulated patients, but enhancing their adaptability could lead to overfitting the simulator, especially in unusual cases. To address this, the researchers proposed using offline Reinforcement Learning (RL) agents trained on actual patient data for glycemia control and introduced an end-to-end personalization pipeline. This innovative approach eliminated the need for a simulator while still enabling the estimation of clinically relevant diabetes metrics.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Literature Review

Type I diabetes results from the destruction of beta cells in the pancreas, necessitating insulin therapy. Closed-loop systems automate this process, improving glycemic control. These systems undergo safety assurance testing through virtual patient simulators, while more complex control algorithms may necessitate offline RL to mitigate overfitting risks. However, existing research only covers simulated data, leaving real-life applicability unverified.

Research on glycemia control includes indirect RL to optimize system parameters and complete RL for direct insulin delivery. Some methods rely on simulators, limiting real-world application. Offline RL has shown promise, though real-life applicability and personalized patient data remain challenges.

Methodology

Problem Formalization: In the context of a closed-loop system for diabetes management, the primary objective is to ensure that blood glucose levels remain within the range of 70-180 mg/dL. Achieving this goal necessitates making insulin delivery decisions, which rely on real-time blood glucose data obtained from a Continuous Glucose Monitoring (CGM) device. The decision-making process is modeled as a Partially Observable Markov Decision Process (POMDP), making RL a natural fit for building effective closed-loop systems. The patient's state at each time step, along with action selection and rewards, is central to this process. 

State Construction: The state representation for decision-making includes various parameters like historical glycemia and insulin data, insulin metrics (e.g., insulin on board and total daily dose), carbohydrate metrics, time information (e.g., the current time of day), and physiological metrics (e.g., body weight). These features provide essential information about the patient's condition and history, enabling effective decision-making. These features' careful selection and normalization are critical for constructing a suitable state representation.

Metrics for Evaluation: Researchers used clinical metrics such as Time in Range (TIR), Time Below Range (TBR), Critical Time Below Range (TBR<54), Time Above Range (TAR), Coefficient of Variation (CV), and Mean Glycemia to evaluate closed-loop system performance. Each metric plays a vital role in evaluating the effectiveness of the control algorithm, ensuring that the system aligns with clinical targets and delivers satisfactory glycemic control.

Reward Function Design: Unlike traditional RL tasks with predefined reward functions, designing a suitable reward function for glycemic control is critical and complex. The choice of the reward function significantly influences the behavior of the RL agents. Several reward functions are considered, including binary rewards and those from prior studies. Selecting a reward function depends on the trade-offs between hypoglycemia and hyperglycemia and the tolerance for deviations from the target glycemia level. The choice of the reward function is essential for guiding the agents in achieving optimal glycemic control. Analyzing reward function and performance metrics suggests selecting the most appropriate reward function for the task.

Study Findings

Population models aim to improve the management of blood glucose levels, a critical aspect of diabetes care. Data collection and pre-processing involve real-life data from over 10,000 patients who used the Diabeloop Generation1 (DBLG1) artificial pancreas system. The study includes detailed information about the data selection process and presents critical statistics about the dataset, showcasing its comprehensiveness.

The authors compared offline RL algorithms and their effectiveness in enhancing glycemic control. By conducting rigorous algorithm comparisons, they identified the best-performing model, TD3-BC, which shows notable improvements in key glycemic metrics. The findings reveal the potential of offline RL agents to enhance glycemic control beyond the capabilities of the behavior policy.

Moreover, the population model's ability to handle unannounced meals, a crucial aspect of diabetes management, is also evaluated. The results of these simulations indicate that the population model excels in glycemic control even in scenarios where meals are not explicitly declared, offering significant enhancements   (TIR), time below range (TBR), and mean glycemia. These promising outcomes suggest progress toward a fully automated and efficient closed-loop glycemic control system that can adapt to real-world patient situations.

The best TD3-BC model showed an average increase of +7% in TIR, a slight decrease of -1% in TBR, and a significant reduction of -12 mg/dL in mean glycemia across various simulated patients. Further in silico evaluations, particularly those without announced meals, underscored the robustness and accuracy of RL control.

Moreover, the study showcased the ability to achieve personalization of these RL agents for individual patients within a realistic context. Researchers utilized Off-Policy Evaluation (OPE) methods to recover key diabetic metrics directly, avoiding reliance on hard-to-interpret Q-value estimates.

Conclusion and Future Scope

To summarize, the extensive comparison of offline RL algorithms for glycemic control, based on actual data, revealed their ability to outperform the existing behavior policy. Again, a more rigorous evaluation of the FQE method applied to TIR/TBR/TAR estimation might provide valuable insights.

Enhancing glycemic control, particularly in challenging cases, is challenging and addresses a standard limitation of existing commercial artificial pancreas systems. For future research, conducting an ablation study to assess the impact of manual patient actions within the dataset on offline RL training may be valuable. Additionally, a more rigorous evaluation of the Fuzzy Q-value estimation (FQE) method applied to TIR/TBR/TAR estimation may offer valuable insights.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.