Tongue fungiform papillae contain taste buds crucial for taste and hormone-producing taste receptor cells; therefore, they may be considered as endocrine organs and have important age-associated physiological implications. We examine the cross-sectional and longitudinal trajectories of fungiform papillae density in 1084 participants from the Baltimore Longitudinal Study of Aging using linear regression models and mixed effects models. At baseline, the mean age was 67.86 ± 14.20 years, with a mean follow-up time among those with repeat visits of 4.24 ± 1.70 years. Women (53%) were younger (66.85 ± 13.78 vs. 69.04 ± 14.61 years, p < 0.001) and had a higher fungiform papillae density than men (16.14 ± 9.54 vs. 13.77 ± 8.61 papillae/cm2, p < 0.001). Whites (67%) had a lower fungiform papillae density than non-Whites after adjusting for age and sex. Factors cross-sectionally associated with a lower fungiform papillae density included a higher waist-hip ratio (β = −8.525, p = 0.029), current smoking status (β = −5.133, p = 0.014), and alcohol use within the past 12 months (β = −1.571, p = 0.025). Longitudinally, fungiform papillae density decreased linearly with follow-up time (β = −0.646, p < 0.001). The rate of decline was not affected by sex, race, BMI, waist-hip ratio, smoking, or alcohol use. The longitudinal decline of fungiform papillae density over time needs to be explored further in order to identify other possible age-associated physiological determinants.