Colorectal cancer (CRC) is the third most common cancer in the world. The accessibility of the Cancer Genome Atlas (TCGA) and Gene Expression Omnibus data allows the prognostic evaluation of CRC. Fibroblasts play a key role in the development and progression of tumors while fibroblast-related risk signature in CRC patients has rarely been mentioned. In this study, TCGA data was classified into high-fibroblast and low-fibroblast groups according to the median of fibroblast content. Among 3845 differentially expressed genes between two groups, 14 prognostic genes commonly expressed in GSE39582 and TCGA were identified by LASSO-COX analysis. Then we established a fibroblast-related risk signature in TCGA training group and validated in the GSE39582 testing group. The risk score was significantly associated with the overall survival (OS), and the poor prognosis of patients in high-risk group might relate to the immune cell infiltration in the tumor microenvironment, epithelial-mesenchymal transition, and extracellular matrix related processes. Overall, we proved that the fibroblast-related signature could predict the prognosis of patients which might shed light on the treatment of CRC.