I had the same concerns when I started reloading and looking into reloading for my G23 but I read from tons of people that put 10k+ rounds of reloads through with no problems. All I shoot is practice/plinking reloads through my G23, all my full power self defense ammo is factory loads so I haven't worried about it, never saw any signs of problems with the reloads.
.40 S&W Club, #153 (G23)
Big Dawg Club, #1530 (G21)
"You never need a gun until you need a gun, and when you do, you need it more than anything else in the world."