As you noted, we've all seen ammo that was properly stored which worked fine after lying dormant for decades. However, the rule of thumb I've always heard is that "shelf life" of ammo should be considered to be no more than ten years. That presumes storage in a cool, dry environment.
Personally, I like to see carry loads shot up and replaced at least every year, preferably every six months. When it's in a loaded gun we're walking around with constantly, the powder is being jiggled with every step we take. Rounds racked in and out of pistol chambers during routine loading and unloading can begin to "set back." The ammo, like the gun that contains it, is constantly being exposed to temperature changes that invite condensation.