PPO-Guided Adaptive Loss Balancing for Object Inpainting in Mobile Mapping System

Abstract

Mobile Mapping Systems (MMS) capture spherical panoramic images that often contain privacy-sensitive objects, such as vehicles. Removing these objects and plausibly background reconstruction remains a challenging inpainting task due to the complex and highly structured nature of street-level scenes. Although Generative Adversarial Networks (GANs) such as Pix2Pix have shown potential for object inpainting, their performance is highly sensitive to the weighting of competing loss terms, which are typically tuned manually and kept fixed throughout training. To address this limitation, we propose an adaptive loss balancing framework that leverages Proximal Policy Optimization (PPO), a reinforcement learning algorithm, to dynamically adjust the loss weight coefficients of Pix2Pix during training. At each training epoch, the PPO agent observes a state vector composed of running averages of generator and discriminator losses alongside image quality metrics, and outputs updated loss weights to guide the generator toward perceptually realistic reconstructions. The agent is trained using six reward signals derived from pixel-level metrics, including PSNR and SSIM and perceptual quality metrics, LPIPS and FID, to jointly capture structural fidelity and visual realism. Experiments on real-world MMS datasets demonstrate that the PPO-based framework consistently outperforms baselines across all evaluated metrics, achieving gains of up to +16.86% over the fixed-weight baseline and up to +10.2% over the adaptive-weight baseline.