Breast cancer is one of the most lethal malignancies among women, and understanding the effects of host immunity on disease progression offers the potential to improve immunotherapies against it. Here, we constructed an immunity-related gene (IRG)-based prognostic signature to stratify breast cancer patients and predict their survival. We identified differentially-expressed genes by analyzing the breast cancer transcriptome data from The Cancer Genome Atlas. Univariate Cox regression revealed 179 survival-correlated IRGs, 12 of which we used to construct an immunity-based prognostic signature that stratified breast cancer patients into high- and low-risk groups. The signature was an independent predictor for survival and was validated in an independent dataset. We also investigated the correlations between our prognostic signature and immune infiltrates and found that signature-derived risk scores correlated negatively with infiltration of B cells, CD4+ T cells, CD8+ T cells, neutrophils and dendritic cells. Our results show that the proposed prognostic signature reflects the tumor immune microenvironment, which makes it a potential indicator for survival that warrants further research to assess its clinical utility.