Brown, born Royston Vasey, was due to perform at the Swansea Grand Theatre in April next year, but the local authority said it had decided not to go ahead with the provisional booking.
In a post on its Facebook page, Swansea Council said it was refunding 52 advance sale ticket holders and apologised to people who may have been disappointed.
It added: "We have decided not to go ahead with a provisional booking for Roy 'Chubby' Brown after considering how it would work alongside our new programming aspirations and partnerships.
"We want all our communities and staff to feel welcome and we felt this booking was unlikely to reflect our values and commitments to them.
"After fully considering the booking we are refunding 52 advance sale ticket-holders and we apologise to those who may have been disappointed."
In a statement on his Facebook page, Brown's manager apologised to his fans who had bought tickets, and said the comedian was "gutted" the show was cancelled.
But they slammed Swansea Council for its decision to "unceremoniously" drop the performance and compared the decision to Brexit.
In the statement posted on Monday, they added: "Roy is one of the most popular artistes ever to appear in Swansea and the crowds love him.
"Who has made a minority decision for the majority? It's like Brexit, what the people want doesn't seem to matter anymore, it's all about a few individuals making a lot of noise on the council.
"It's about time the majority out voted the minority it's not a crime to have an opinion.
"Every body [sic] seems to be frightened to be honest in case they upset some militant minority soap box protagonist.
"Grow a backbone and tell them to live or let live, everybody's got different tastes not everybody likes marmite but it doesn't mean the people that eat it are wrong."