Normal brain aging is characterized by declining neuronal integrity, yet it remains unclear how microstructural injury influences cognitive aging and whether such mechanisms differ between sexes. Using restriction spectrum imaging (RSI), we examined sex differences in associations between brain microstructure and cognitive function in 147 community-dwelling older men and women (56-99 years). Gray and white matter microstructure correlated with global cognition, executive function, visuospatial memory, episodic memory, and logical memory, with the strongest associations for restricted, hindered and free isotropic diffusion. Associations were stronger for women than for men, a difference likely due to greater age-related variability in cognitive scores and microstructure in women. Isotropic diffusion mediated effects of age on cognition for both sexes, though distinct mediation patterns were present for women and men. For women, hippocampal and corpus callosum microstructure mediated age effects on verbal and visuospatial memory, respectively, whereas for men fiber microstructure (mainly fornix and corpus callosum) mediated age effects on executive function and visuospatial memory. These findings implicate sex-specific pathways by which changing brain cytoarchitecture contributes to cognitive aging, and suggest that RSI may be useful for evaluating risk for cognitive decline or monitoring efficacy of interventions to preserve brain health in later life.