Chapter 88: Multi-Agent PPO (MAPPO)
Learning objectives Adapt a PPO implementation to the multi-agent setting with parameter sharing: all agents use the same policy network π(a_i | o_i) (and optionally the same value function), with agent identity or observation distinguishing them. Use a centralized value function V(s_global) or V(s_global, a_1,…,a_n) to reduce variance and improve credit assignment; the policy remains decentralized π_i(a_i | o_i). Train on a collaborative task (e.g. particle env or simple grid) and compare with IPPO (Independent PPO: each agent runs PPO with its own parameters and no centralized value). Explain the benefits of parameter sharing (sample efficiency, symmetry) and centralized value (better baseline, stability). Relate MAPPO to game AI (team games) and robot navigation (homogeneous multi-robot). Concept and real-world RL ...