Education in people’s early lives are positively related to their cognitive function, but its modulating effects on detailed cognition domains, its interaction with leisure activities and the associated brain changes have yet to be investigated. This report used data from 659 cognitively normal community dwelling elderly who completed neuropsychological tests, leisure activities measurement, and 78 of them underwent structural and diffusion MRI scans. We found that: (i) the highly educated elderly had a better cognitive functioning in multi-domains, higher frequencies of participation in knowledge-related leisure activities, and slower age-related reductions of executive function; (ii) the intellectual and social types of leisure activities mediated the association between education and multiple cognitive domains, including memory, language, attention and executive function; (iii) there was a significant age by education interaction on the gray matter volume of the anterior brain regions and white matter integrity; and (iv) the interaction between age and education affected cognition indirectly through white matter integrity analyzed using structural equation model. Overall, our results revealed that high education in early life served as a protective factor in aging that may help to postpone cognitive and brain reserve decline in cognitively normal aging.