The catechol-O-methyltransferase (COMT) Val158Met polymorphism has been associated with working memory (WM) in many studies, but the results have not been consistent. One plausible explanation is sex-specific effects of this polymorphism as reported in several studies. The current study aimed to explore the sex-specific effects of the COMT Val158Met polymorphism on WM-related brain function in an elderly sample. We found that Val homozygotes outperformed Met allele carriers on the backward digit span subtest for both males and females. The triangular part of the left inferior frontal gyrus and the left inferior temporal gyrus exhibited higher activation in Met allele carriers compared with Val homozygotes during the n-back task, while the background functional connectivity (bFC) between the left angular gyrus (ANG) and the right ANG was enhanced in Val homozygotes as compared to Met allele carriers. Finally, the associations between brain activation, bFC (among various regions), and WM performance were identified only in specific genotype groups of the female participants. These findings provide new insights into the role of COMT Val158Met gene polymorphism in brain function, particularly its female-specific nature.