What kind of gun were you using?The difference will be greater if your shooting a LCR 357 compared to a GP-100.
And yes, I believe the difference will become greater with the longer distances. Shooting fast is not good enough if you are not hitting what you're aiming at. If your target is close, well, that's great, but if you target is at 25 yds, and you actually have to use both of your sights. Well, I believe this would also make the time differences greater. Maybe not by too much, but maybe enough to get you hurt.
Additionally, I do believe the 357 is a much better caliber then the 38, and the numbers back that up with shooting statistics. That's not to say a 38 won't get the job done. Compare the energy levels of all loads within both calibers. Compare them to other calibers as well. The numbers never lie.