Our paper A Reference Architecture of Reinforcement Learning Environments, co-authored with Xiaoran (Sharon) Liu (PhD student in our lab) has been accepted for the 23rd IEEE International Conference on Software Architecture (ICSA).
ICSA is a CORE A-ranked conference, the premier venue for practitioners and researchers interested in software architecture, in component-based software engineering and in quality aspects of software and how these relate to the design of software architectures. Every paper at such a high-ranked venue is an honour, but I’m particularly happy for Sharon’s success and the boost this work gives to her PhD. Sharon worked tirelessly to analyze the ins and outs of 18 widely used RL frameworks at the implementation level, to elicit architectural patterns and help us all build better RL software. We hope that this piece will be as valuable to the wider RL researcher and practitioner community as it already has been to us in our research.
Congratulations, Sharon! Your success is well-deserved, and we are very proud of you.
I am looking forward to visiting an important branch of my academic family as this year’s ICSA is hosted by VU Amsterdam and the Software and Sustainability (S2) research group. I spent a short but extremely formative time as a postdoc in S2, and learned a lot about empirical research and software sustainability. See you in Amsterdam!
Preprint available here: https://arxiv.org/abs/2603.06413.
Abstract. The surge in reinforcement learning (RL) applications gave rise to diverse supporting technology, such as RL frameworks. However, the architectural patterns of these frameworks are inconsistent across implementations and there exists no reference architecture (RA) to form a common basis of comparison, evaluation, and integration. To address this gap, we propose an RA of RL frameworks. Through a grounded theory approach, we analyze 18 state-of-the-practice RL frameworks and by that, we identify recurring architectural components and their relationships, and codify them in an RA. To demonstrate our RA, we reconstruct characteristic RL patterns. Finally, we identify architectural trends, e.g., commonly used components, and outline paths to improving RL frameworks.
