Esophageal cancer (ESCA) is a common malignancy in the digestive system with a high mortality rate and poor prognosis. Tumor microenvironment (TME) plays an important role in the tumorigenesis, progression and therapy resistance of ESCA, whereas its role in predicting clinical outcomes has not been fully elucidated. In this study, we comprehensively estimated the TME infiltration patterns of 164 ESCA patients using Gene Set Variation Analysis (GSVA) and identified 4 key immune cells (natural killer T cell, immature B cell, natural killer cell, and type 1 T helper cell) associated with the prognosis of ESCA patients. Besides, two TME groups were defined based on the TME patterns with different clinical outcomes. According to the expression gene set between two TME groups, we built a model to calculate TMEscore based on the single-sample gene-set enrichment analysis (ssGSEA) algorithm. TMEscore systematically correlated the TME groups with genomic characteristics and clinicopathologic features. In conclusion, our data provide a novel TMEscore which can be regarded as a reliable index for predicting the clinical outcomes of ESCA.