when you say holding, im assuming you mean the GS is actually defending the defender.  i love doing this during my games.  i set myself up on one side of the circle keeping the GK in front of me and away from the goal post and the ball is passed from the center third straight into the circle behind where im standing.  i barely have to move.

how to not let this happen?  your defender cant just stand there and be defended.  they have to work at getting that position between the GS and the goal post.  if they are unable to do this, maybe move off the GS and pick off the passes in as they will get comfortable with all that space.  the idea is not to let the GS get to dictate their position and play.  the GK has to be moving and keeping the GS occupied with trying to get their position instead of just standing there dictating the easy plays.