Hippocampal atrophy rate has been correlated with cognitive decline and its genetic modifiers are still unclear. Here we firstly performed a genome-wide association study (GWAS) to identify genetic loci that regulate hippocampal atrophy rate. Six hundred and two non-Hispanic Caucasian elders without dementia were included from the Alzheimer’s Disease Neuroimaging Initiative cohort. Three single nucleotide polymorphisms (SNPs) (rs4420638, rs56131196, rs157582) in the TOMM40-APOC1 region were associated with hippocampal atrophy rate at genome-wide significance and 3 additional SNPs (in TOMM40 and near MIR302F gene) reached a suggestive level of significance. Strong linkage disequilibrium between rs4420638 and rs56131196 was found. The minor allele of rs4420638 (G) and the minor allele of rs157582 (T) showed associations with lower Mini-mental State Examination score, higher Alzheimer Disease Assessment Scale-cognitive subscale 11 score and smaller entorhinal volume using both baseline and longitudinal measurements, as well as with accelerated cognitive decline. Moreover, rs56131196 (P = 1.96 × 10-454) and rs157582 (P = 9.70 × 10-434) were risk loci for Alzheimer’s disease. Collectively, rs4420638, rs56131196 and rs157582 were found to be associated with hippocampal atrophy rate. Besides, they were also identified as genetic loci for cognitive decline.