It sounds like you are using an OWB holster along the lines of a pancake, an Askins Avenger-style, or something similar. It also sounds like there isn't much space behind the holster for the pants belt loop.
In cases like these, the 'easiest' thing to do is put the rig together (minus the gun, of course) before you don your pants. However, that can be kind of 'tricky,' too. It also doesn't address the problem of removing or replacing your holster when fully dressed.
Another method that does, but is a little awkard to explain, is to 'take a seat' as if you were using the john. Of course, you don't have to use an actual toilet for this (a sturdy chair or bench will do), but I think the analogy paints a pretty clear picture.
In that posture, you have a 'top view' of everything, easier access to 3:00 and beyond, and there is no strain (or at least very little strain) on the system. You also have two more or less natural 'rest points' from which you can 'go to work': Pants around the ankles or at the knees (more or less).
Note that the 'knee position' can also be used in places like public restrooms to keep things together and suspended above ground level: If you buckle your belt in the last notch, assume a 'wide stance,' and spread your knees slightly when seated, you can hold you 'gear' at knee-height instead of having the whole works sitting on the floor. You can also more easily hold onto the belt itself (or holster/gun, if you prefer) to assist in the effort.
Lastly, as you probably already know, it's better to avoid severely bending your belt, and it should 'never' be rolled or tightly curled. The 'only' exception is buckling up which normally involves bending the tongue of the belt to an angle greater than 90° but doesn't really harm anything because that section doesn't bear any load.
PS: All your puns are belong to us.