Studies exploring age-related brain and cognitive change have identified substantial heterogeneity among individuals, but the underlying reasons for the differential trajectories remain largely unknown. We investigated cross-sectional and longitudinal associations between brain-imaging phenotypes (IDPs) and cognitive ability, and how these relations may be modified by common risk and protective factors. Participants were recruited from the 1953 Danish Male Birth Cohort (N=123), a longitudinal study of cognitive and brain ageing. Childhood IQ and socio-demographic factors are available for these participants who have been assessed regularly on multiple IDPs and behavioural factors in midlife. Using Pearson correlations and canonical correlation analysis (CCA), we explored the relation between 454 IDPs and 114 behavioural variables. CCA identified a single mode of population covariation coupling cross-subject longitudinal changes in brain structure to changes in cognitive performance and to a range of age-related covariates (r=0.92, Pcorrected < 0.001). Specifically, this CCA-mode indicated that; decreases in IQ and speed assessed tasks, higher rates of familial myocardial infarct, less physical activity, and poorer mental health are associated with larger decreases in whole brain grey matter and white matter. We found no evidence supporting the role of baseline scores as predictors of impending brain and behavioural change in late-midlife.