Games and immersive training environments frequently rely on user performance measures to adapt the difficulty of tasks and behaviors, responding dynamically to changes in performance. However, users may maintain task performance while experiencing increasing levels of cognitive load. These high levels of load mean the user has no spare capacity and may fail to get the maximum benefit from the training task. While other adaptive mechanisms exist, they do not account well for cognitive load and thus may not be optimal for training tasks. In this paper we outline a conceptual framework for using real-time measures of cognitive load to dynamically adapt immersive environments. We argue that these measures have the benefit of providing a richer mix of data to base adaption on beyond simple performance metrics, and additionally provide further metrics to assess both the learner and the training material. To this end, a Cognitive Adaptive Serious Game Framework (CASG-F) is presented that draws on frameworks and theories of cognitive load and serious games. We additionally outline the range of potential mechanics and environment parameters that could potentially be adjusted to modify difficulty.