Given the age of your Magic Mouse, I'd say it's a pretty good bet your battery is on its last legs and can no longer hold a charge, so yeah, I'd say it was very much worth your time to replace the battery.
The good news here is that iFixit has the guide you need to replacing your battery; outside of actually defeating the adhesive holding the battery in place, it looks like a fairly straight-forward operation.
[guide|146363|Magic Mouse 2 Battery Replacement - iFixit Repair Guide]
The guide has a link to a source for the battery; it looks like a fairly decent price since many of the other ones that came up on a search were substantially more expensive, although similar prices can be had on eBay or AliExpress.