Conservative Offline RL with Uncertainty-Aware Policy Improvement

Abstract

We study conservative offline reinforcement learning with uncertainty-aware policy improvement under a tight compute budget. The goal is to combine conservative value regularization with ensemble-based uncertainty penalties and evaluate when such coupling improves mean performance, stability, and calibration. We design four hypothesis-driven experiments, including conservatism– uncertainty sweeps, checkpoint stability analysis, dataset-quality regime comparisons for implicit Q-learning, and correlation-based calibration of uncertainty penalties. Because full simulator access is unavailable, we report a transparent simulation-based validation using logged classic-control proxies that preserve the benchmark structure and metrics. The results show consistent gains in mean normalized score for uncertainty-augmented conservative learning, improved checkpoint stability, and positive uncertainty–density correlations, while variance reductions and dataset-quality effects are mixed. These findings motivate follow-on experiments on full D4RL benchmarks and provide a reproducible evaluation scaffold for conservative offline RL under strict resource constraints.