Three ideas:
- Did you try to clean your cache then and when?
- Are those servers which don't have spectator slots perhaps, and which are full by chance exactly when you connect, although they were listed to have some free player slots just before?
- The hang phase could be when exactly the server is changing the map, which also takes a few seconds up to a few couple of seconds. Then it is possible that the connection request is delayed, or in worst case ignored (not received), and then there might be a protocol timeout (default IP timeout is ~2 minutes) and after a retry you're suddenly in?
Of course, if you are waiting, then this takes "ages", but only because your "impatient" to play
I don't think it is related to video card, except your system is not the fastest, and it needs time to load all the 3D information and the textures (especially if it is a modded map) into the GPU memory.
What does the task manager tell you then? busy or idle system? Any disk activities to be seen?