Adult hematopoietic stem and progenitor cells (HSPCs) reside in the bone marrow (BM) ensuring homeostasis of blood production and immune response throughout life. Sex differences in immunocompetence and mortality are well-documented in humans. However, whether HSPCs behave dimorphically between sexes during aging remains unknown. Here, we show that a significant expansion of BM-derived HSPCs occurs in the middle age of female but in the old age of male mice. We then show that a decline of HSPCs in male mice, as indicated by the expression levels of select hematopoietic genes, occurs much earlier in the aging process than that in female mice. Sex-mismatched heterochronic BM transplantations indicate that the middle-aged female BM microenvironment plays a pivotal role in sustaining hematopoietic gene expression during aging. Furthermore, a higher concentration of the pituitary sex hormone follicle-stimulating hormone (FSH) in the serum and a concomitant higher expression of its receptor on HSPCs in the middle-aged and old female mice than age-matched male mice, suggests that FSH may contribute to the sexual dimorphism in aging hematopoiesis. Our study reveals that HSPCs in the BM niches are possibly regulated in a sex-specific manner and influenced differently by sex hormones during aging hematopoiesis.