Advanced age and apolipoprotein E (APOE) ε4 allele are both associated with increased risk of the Alzheimer’s disease (AD). However, the extent of the joint contribution of APOE ε4 allele and age on the brain white matter integrity, cognition and their relationship are unclear. We assessed the age-related variation differences of major cognitions in 846 non-demented elderly, and brain major white matter tracts in an MRI sub-cohort of 111 individuals between ε4 carriers and noncarriers. We found that: (i) carriers showed a steeper age-related decline after age 50 in general mental status, attention, language, and executive function and performed worse than noncarriers at almost all ages; (ii) main effect of age on anterior fibers, but main effect of APOE ε4 on posterior fibers, and the interactive effect of them existed on anterior and posterior fibers; (iii) carriers showed an accelerated age-related integrity reduction of these fibers compared to noncarriers who had a slight decrease but not significant; and (iv) significant associations of the higher white matter integrity with better multi-cognitive performance in old ε4 carriers. Overall, combining APOE status with age may be useful in assessing possible mechanisms of disease development in AD.