Definitely sounds like a loose connection, though it could also be bad grounding through the hitch itself.
Usually I put one of those short ball-on-a-loop bungies around my 7-pin round connector after I've made sure all the lights are working - I've never had an issue with any other style but I don't use them often. This keeps the connection tight. We have a trailer at work that gives everyone else problems, but because of this trick I never have an issue.
It may also be a good idea to clean your hitch ball and coupler. Most trailers use this as a 'backup ground' so to speak, and some it is the primary ground. If you have a lot of build-up inside the coupler this ground won't be made and it could cause this issue.