The cause of cognitive dedifferentiation has been suggested as specific to late-life abnormal cognitive decline rather than a general feature of aging. This hypothesis was tested in two large cohorts with different characteristics. Individuals (n = 2710) were identified in the Alzheimer’s Disease Neuroimaging Initiative (ADNI) research database (n = 1282) in North America, and in the naturalistic multi-site MemClin Project database (n = 1223), the latter recruiting from 9 out of 10 memory clinics in the greater Stockholm catchment area in Sweden. Comprehensive neuropsychological testing informed diagnosis of dementia, mild cognitive impairment (MCI), or subjective cognitive impairment (SCI). Diagnosis was further collapsed into cognitive impairment (CI: MCI or dementia) vs no cognitive impairment (NCI). After matching, loadings on the first principal component were higher in the CI vs NCI group in both ADNI (53.1% versus 38.3%) and MemClin (33.3% vs 30.8%). Correlations of all paired combinations of individual tests by diagnostic group were also stronger in the CI group in both ADNI (mean inter-test r = 0.51 vs r = 0.33, p < 0.001) and MemClin (r = 0.31 vs r = 0.27, p = 0.042). Dedifferentiation was explained by cognitive impairment when controlling for age, sex, and education. This finding replicated across two separate, large cohorts of older individuals. Knowledge that the structure of human cognition becomes less diversified and more dependent on general intelligence as a function of cognitive impairment should inform clinical assessment and care for these patients as their neurodegeneration progresses.