Statistics

Markov decision process

Algorithm

Machine learning

Physics

Mathematical optimization

Function (biology)

Bellman equation

Artificial intelligence

Markov process

Markov model

Component (thermodynamics)

Partially observable Markov decision process

Political science

Representation (politics)

Action (physics)

Mathematics

Computer science

State (computer science)

Quantum mechanics

State space

Evolutionary biology

Biology

Politics

Thermodynamics

Reinforcement learning

Markov chain

Reliability Engineering & System Safety

Decision-making for engineering systems management can be efficiently formulated using Markov Decision Processes (MDPs) or Partially Observable MDPs (POMDPs). Typical MDP/POMDP solution procedures utilize offline knowledge about the environment and provide detailed policies for relatively small systems with tractable state and action spaces. However, in large multi-component systems the dimensions of these spaces easily explode, as system states and actions scale exponentially with the number of components, whereas environment dynamics are difficult to be described explicitly for the entire system and may, often, only be accessible through computationally expensive numerical simulators. In this work, to address these issues, an integrated Deep Reinforcement Learning (DRL) framework is introduced. The Deep Centralized Multi-agent Actor Critic (DCMAC) is developed, an off-policy actor-critic DRL algorithm that directly probes the state/belief space of the underlying MDP/POMDP, providing efficient life-cycle policies for large multi-component systems operating in high-dimensional spaces. Apart from deep network approximators parametrizing complex functions with vast state spaces, DCMAC also adopts a factorized representation of the system actions, thus being able to designate individualized component- and subsystem-level decisions, while maintaining a centralized value function for the entire system. DCMAC compares well against Deep Q-Network and exact solutions, where applicable, and outperforms optimized baseline policies that incorporate time-based, condition-based, and periodic inspection and maintenance considerations.

Managing engineering systems with large state and action spaces through deep reinforcement learning