I feel like "It goes without saying" can be like a passive aggressive rhetoric. It is a commentary on the people/person it does not, so subtly or frictionlessly "go without saying"

That quote is a rhetorical statement from a TV comedy. It's an exaggerated grievance, a snarky comment humorous in its original context. Even divorced from the context, there is no mention of "ban[ning]" anything. Sorry for inadvertently stoking your pet peeve, friend. I'm just trying to relax and enjoy this life, dodging drama everywhere I can.

No, it doesn't. It's a rhetorical device. Do you want to also ban "it's obvious/evident/clear/patent that...", "we all know that...", "everybody agrees that...", "it is undeniable that...", etc.? What bugs me, however, is people mindlessly complaining about these things (not the first time I come across this one, btw.). Anyway, I hope that answer your question :-)

P.S. Not 100% confident about the placement of the commas between the quotations there, so feel free to comment on it if you think they're wrong.