When you fly directly behind or below someone they cannot see you, and are therefore either unaware of your presence or have reason to worry about your actual position. If you don’t know what to expect of someone else then the uncertainty is probably mutual and you should maintain an extra margin of safety.
The stakes go up when more than two sailplanes soar together, hazardous variables multiplied and safe maneuvering room fractional. The only way you can see all of them at one time is if they’re all in front of you, in which case it’s probable that at least one of them cannot see you…
So the critical issue remains: how close is too close? Some pilots are warier than others about minimum safe separation, and that is their right. But if they’re compelled to leave a thermal because you made them nervous, then it was your wrong. There must be room for others to make sudden, unexpected changes of attitude or direction at a moment when you’re looking away, and then time enough for even a slow response on your part to provide easy collision avoidance. If anyone involved could be startled by a sudden move, then someone is too close. And even if ‘they’ do it, the one who’s too close is you.
Regardless of who technically has right-of-way, you must be able to fly your ship fully well while monitoring positions and movement of other aircraft. Yet it’s impossible to gaggle safely or successfully if you’re watching them instead of your own horizon. Real safety demands that you fly your aircraft first while tracking the position of others in peripheral vision, and in the back of your mind. If you’re not yet capable of dividing your concentration and functioning simultaneously in each mental area, then instead of gaggling, go off by yourself and practice the fundamentals!