The accumulation of amyloid-β (Aβ), considered as the major cause of Alzheimer’s disease (AD) pathogenesis, relays on the rate of its biosynthesis and degradation. Aβ degradation is a common overture to late-onset AD and targeting the impairment of Aβ degradation has gained attention in the recent years. In this study, we demonstrated a rhamnoside derivative PL402 suppressed Aβ level in cell models without changing the expression or activity of Aβ generation-related secretases. However, the levels of matrix metalloproteinase (MMP) 3 and 9, belonging to amyloid-degrading enzymes (ADEs), were up-regulated by PL402. The inhibition or the knockdown of these two enzymes abolished the effect of PL402, indicating that PL402 may reduce Aβ via MMP3/9-mediated Aβ degradation. Notably, administration of PL402 significantly attenuated Aβ pathology and cognitive defects in APP/PS1 transgenic mice with the consistent promotion of ADEs expression. Thus, our study suggests that targeting Aβ degradation could be an effective strategy against AD and the rhamnoside derivatives may have therapeutic effects.