Daisy Welham

Benchmarking the Performance of Q-Learning and Actor-Critic Agents on the Control ofWater Systems

Abstract:

Control of water systems is an essential and safety-critical task. Recent innovations in machine learning have been applied to water systems [1], but due to the black-box nature of many of these algorithms, actual deployment in practice is scarce. This paper applies the explainable machine learning architecture of Q-learning to water systems in an attempt to make systems which not only make good decisions, but justify those decisions in a way that is understandable to domain experts. An actor-critic network [2] is also implemented to compare performance on these problems. Several simulated environments of varying levels of complexity are implemented to test how the agents perform on increasingly realistic versions of the water tank control problem. Knowledge from domain experts is used to ensure that the simulated environments resemble actual deployment environments, but some approximations are made to make the problem computationally tractable.

This paper demonstrates cases where Q-learning can be an effective tool for control of water systems, and also highlights cases where a simple Q-learning agent would make bad decisions that result in catastrophic failure. Where several agents are capable of performing well on the same task, this paper benchmarks them and evaluates how appropriate each agent may be for that task.