We investigated whether cognitive fitness in late midlife is associated with physiological and psychological factors linked to increased risk of age-related cognitive decline. Eighty-one healthy late middle-aged participants (mean age: 59.4 y; range: 50-69 y) were included. Cognitive fitness consisted of a composite score known to be sensitive to early subtle cognitive change. Lifestyle factors (referenced below as cognitive reserve factors; CRF) and affective state were determined through questionnaires, and sleep-wake quality was also assessed through actimetry. Allostatic load (AL) was determined through a large range of objective health measures. Generalized linear mixed models, controlling for sex and age, revealed that higher cognitive reserve and lower allostatic load are related to better cognitive efficiency. Crystallized intelligence, sympathetic nervous system functioning and lipid metabolism were the only sub-fields of CRF and AL to be significantly associated with cognition. These results show that previous lifestyle characteristics and current physiological status are simultaneously explaining variability in cognitive abilities in late midlife. Results further encourage early multimodal prevention programs acting on both of these modifiable factors to preserve cognition during the aging process.