There's a lot of debate as to that. Typical police load-out is one in the gun and two more on the belt. Guys who've taken metallurgy classes say a fully loaded magazine's spring should never take a set, but armorers who have to replace springs that have gone bad say otherwise. Being a belt and suspenders kind of guy, I go with the latter "just in case," and suggest a minimum of six magazines per carry gun. Three are kept filled, the rest are empty and "resting."
Swapout every six months or so. Consider putting a little spot of white paint ("Winter") on the floorplates of half the magazines, and yellow paint ("Summer") on the other half. When you "spring forward, fall back" with time changes, attend to batteries in smoke alarms and flashlights, and also swap out the magazines, letting the loaded ones take a rest and filling up the ones that have been "resting."
Whether or not it proves actually necessary with your particular magazines, it seems to give peace of mind if nothing else.
Sporting use may also dictate magazine count. You probably want at least four magazines to shoot a GSSF or PPC match, more for USPSA.