There's a tiny little retaining spring/clip in-set into the retaining washer. I only know because I just went to the trouble to put one
on my 870,

.
Mine didn't come with the swivel when I bought it several years ago and just recently I decided to add a sling so the stock swivel seemed the way to go.
Anyway, if you can get to it that might be the simplest way to get it off without risking damage to the gun. I kind of put mine on wrong the first time so had to work it back off. I did it with two small picks (kind of like a scratch awl but smaller with finer points). I was able to spread the spring with them and get one pick underneath the end of the spring to start working it out.
That's all assuming your swivel is the same as mine, of course!