Growing evidence highlighted the tumor mutational burden (TMB) as an important feature of carcinogenesis and therapeutic efficacy in esophageal cancer (EC). Our study aimed to explore the genomic landscape and the correlation between TMB and immune cell infiltration in EC patients with or without radiotherapy. The EC patients were categorized into high TMB (TMB-H) and low TMB (TMB-L) groups by the ESTIMATE algorithm, and subgroup analysis was performed based on receiving radiotherapy or not. Univariate regression analysis indicated TMB and TNM stages as high-risk prognostic factors (Hazard ratio > 1 and P < 0.05). Multivariate regression analysis suggested TMB as an independent prognostic factor (Hazard ratio = 1.051, P = 0.003). Kaplan-Meier analysis showed no significant difference of the overall survival (OS) between TMB-H and TMB-L groups (P = 0.082). However, EC patients without radiotherapy in the TMB-H group had significantly decreased OS (P = 0.038) and increased Tregs cell infiltration (P = 0.033). These results suggested TMB as a prognostic marker for EC patients. Especially for patients who did not receive radiotherapy, the prognosis of TMB-H patients was significantly poorer than that of TMB-L patients, which might result from the different regulatory T cell infiltration.