This was done a couple years back, but it seemed to work out for him. I'd probably go with a bigger amp (somehow) and then just use discretion on the volume control for a while before changing speakers.
I'm with you... Looking for better sound. I'm going to bypass he Goldwing audio system. In my humble opinion, it's just not that good. I ordered a pair of Sena SMH10 intercoms. I heard a set of them last week (not while riding, however) and it sounded pretty decent. I'm going to install them in The new Bell Mag-9 helmet. I'm expecting a big improvement. If I don't get it, I'm prepared to hack the SMH10's with better headphone speakers.