Use Voice Mode in Meta Horizon Worlds
Voice Mode allows you to choose how you hear people you aren't following, including the option to not hear strangers at all. With Voice Mode turned to Garble Voices, the voices of people you aren't following will become unintelligible, pleasant sounds.
If you set Voice Mode to Garble Voices, people you aren't following will see an indicator to let them know you can't hear them. If you want to hear what people are saying with Voice Mode on Garble Voices, you can hold either controller near your ear to temporarily hear them.
You can manage Voice Mode from the Sound tab in Settings.