Optimizing Spaced Repetition with Deep Reinforcement Learning

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Jul 9 2024

In a paper published in the journal Applied Sciences, researchers proposed a novel deep reinforcement learning (DRL) approach to optimize spaced repetition schedules, which are crucial for enhancing long-term memory in both online learning and cognitive science.

Their framework targeted optimal review intervals, unlike traditional methods with handcrafted rules or existing DRL approaches focused on daily item selection. Their contributions included a Transformer-based model for accurately estimating recall probabilities, a simulation environment based on this model, and a deep Q-network (DQN) agent to learn optimal review intervals. Experimental results showed that their method achieved a lower mean average error (MAE) score in memory prediction and higher mean recall probabilities across different environments, outperforming all other methods.

Background

Past work on spaced repetition scheduling includes traditional rule-based methods like Pimsleur, Leitner, SuperMemo, and Anki, which, while advanced, lacked flexibility for individual learning patterns. The advent of DRL brought adaptive policies, with notable contributions from Reddy, Sinha, and Upadhyay, and further enhancements by Yang's time-aware scheduler with the dyna-style planning (TADS) approach. Despite their promise, these DRL methods face challenges such as naive simulation environments and ineffective algorithms.

Framework Components Overview

The framework consists of three main components: the transformer-based half-life regression (THLR) memory prediction module, a simulation environment, and a DRL-based spaced repetition algorithm. The THLR module estimates the recall probability of learning items by considering probability history, recall history, and interval history.

The simulation environment replicates a learner's daily review process, capturing inter-day and intra-day dynamics. The DRL-based algorithm uses a DQN with a long-short term memory (LSTM) mechanism to determine optimal review intervals for long-term memory retention.

Spaced repetition optimization aims to schedule learning items to maximize long-term memory retention while minimizing memory costs. Given N learning items, the learning process involves a sequence of learning events represented by vectors, including the item, days since the last review, recall probability, and recall result.

The optimization algorithm determines the optimal interval for each item based on recall outcomes. This problem is formulated as a RL problem, defining state space, action space, observation space, and reward function accordingly. The optimal policy aims to maximize rewards by choosing the best review intervals.

The first component of the framework calculates the half-life of a learning item using a Transformer model, known for its effectiveness in time-series prediction. Unlike previous methods, the THLR flexibly captures temporal dynamics without manually designed state transitions. The model predicts the half-life of an item based on last recall results, probabilities, and intervals, enabling accurate recall probability calculations.

Based on the memory model from the THLR or other baselines, the simulation environment simulates inter-day and intra-day learning phases. In the intra-day phase, the environment processes items due for review within daily time limits, accounting for successful and unsuccessful recall costs. New items are scheduled for the next day, and any remaining items are postponed.

The detailed simulation process accurately represents the learner's review dynamics. The RL-based spaced repetition policy utilizes a model-free, off-policy DQN. An LSTM is employed in the policy network to capture temporal dynamics, training it recurrently.

The DQN algorithm approximates the optimal Q function, using neural networks to maximize rewards by selecting the best actions. Temporal difference error minimization and Huber loss optimization ensure effective policy learning, while LSTM encoding considers the temporal relations between learning events, improving the scheduling policy's accuracy and effectiveness.

Experimental Evaluation Summary

In the experiments, the framework is evaluated in two main aspects: memory prediction and schedule optimization. The THLR model is compared with several baselines for memory prediction using MAE and mean absolute percentage error (MAPE) metrics.

THLR significantly outperforms all baselines in predicting recall probabilities, demonstrating its efficacy in capturing temporal dynamics and improving accuracy. The framework is assessed against various baselines across different simulation environments using average recall probability (ARP) for schedule optimization. The results consistently show that the method outperforms all competitors, validating its effectiveness in optimizing spaced repetition schedules for enhanced long-term memory retention.

Conclusion

To summarize, the paper introduced DRL-SRS, a novel framework using DRL to optimize spaced repetition scheduling for improved long-term memory retention in online learning and cognitive science. It addressed shortcomings of traditional methods and previous DRL models with three innovations: THLR for precise recall probability estimation, a simulation environment replicating daily review dynamics, and a DQN with LSTM for learning optimal review intervals. Future directions include enhancing personalization by integrating individual features for optimized scheduling and incorporating multi-modal learning data to broaden applicability beyond textual inputs.