TL;DR? See the bottom.
I'm researching a fix for this. My Steam Controller's rubber thumb stick is also crumbling.
The thumb stick just slides off the joystick. There are no fasteners or adhesives to worry about. If you pull it up, you'll see it slides a millimeter or two before the case gets in the way. I disassembled my Steam Controller today, and the whole thumb stick slid off once the front cover was out of the way.
The same goes for other controllers, according to what I've read online. I just tested this with an old XBox 360 controller that I'm never planning to use again. After opening it, the thumb stick just popped off.
The 360 thumb stick's dimensions are very close to the SC's. The hole underneath is almost identical, and in personal testing it fits nicely on the SC's joystick.
The 360 thumb stick fails in two places. Both might be good enough, with slight modification.
The head is slightly wider than the hole in the Steam Controller's case. I managed to force it through at an angle without breaking anything (twice, since I also removed it), but I can't guarantee that applies to all situations. Anyway, it might be best to get the thumb stick through that hole before sliding it onto the joystick, even if that means closing the case is a little more complicated.
Also, this means that metal or hard plastic thumb stick replacements are probably a bad idea unless you're absolutely sure they will fit in the Steam Controller's case.
The distance between the opening for the joystick rod and the bottom of the thumb stick dome is too short. This causes the thumb stick to sit too high to fit in the Steam Controller's case. I haven't tried it yet, but I think the plastic around the hole can be sanded or otherwise trimmed to a point where the thumb stick will sit on the joystick at the right height. One risk is that there won't be enough hole left to securely sit on the joystick after sufficient trimming. Another risk is trimming it too far by mistake, and then it would sit too low.
Replacement thumb sticks for a variety of controllers are cheap and plentiful online, and they come in pairs so there's a backup if the first one gets ruined. My next step is to look for dimensions for different controller models. Maybe one of them will fit better without modifying it.
Update: Some discussions online say that thumb sticks can ride too high not because the hole starts too low but rather because it's not deep enough. It might take a drill press to modify those sticks.
Update: This reddit thread says that certain 8BitDo thumb sticks fit perfectly. The link in that thread to the 8BitDo store is outdated. I think these might be the same. I just now ordered a pair. I'll update when I've had a chance to try them.
Update: The 8BitDo thumb stick arrived today. USPS delivery was less than a week. It just barely fits on the joystick, and it just barely fits in the case. The Joystick test in Big Picture Mode shows that it produces a full range of motion (-32767 to 32767 in X and Y), and it snaps back to center from all directions. In the event the fit is too tight for someone, I think the worst that would need to be done is some light sanding around the front cover's joystick hole to add clearance.
This is the fixed controller next to a mint spare I'm hoarding.
This is a new 8BitDo thumb stick beside the old Steam Controller stick.
OP's caps could still be useful if they fit a new thumb stick better than one that has been worn down.
Update: After several hours of game play, I can say it's working fine.
Update: Steam Community comment with more details about using the 8BitDo thumb sticks.