Pulling the adhesive is a delicate balance between strength and speed. It is important to pull as slowly as possible, with a decent force but not to break it. You can also pull a bit and then relax and wait for the adhesive to pull itself out from under the speaker. Then repeat the short pull and wait.

Even with the above, my adhesives broke each time. A hairdryer can help warm the speaker (and the adhesive under it), but don’t warm the battery next to it too much. Once hot, slowly move the spudger under the speaker and loosen it gradually. It will eventually come off in one piece.