SAN FRANCISCO (CBS SF) – The nation’s top housing official is catching some heat for comments he allegedly made about transgender people, while visiting San Francisco earlier this week.

According to The Washington Post, Secretary of Housing and Urban Development Ben Carson expressed concern about “big, hairy men” coming into women’s homeless shelters. The Post reports he also said he’s unhappy that society no longer seems to know the difference between men and women.

READ MORE: Cal Fire Confirms Estrada Fire Sparked by Controlled Burn, Holding at 150 Acres; Evacuations Ordered

Carson visited HUD’s office in San Francisco on Tuesday, the same day President Donald Trump was in the Bay Area for a fundraiser in Silicon Valley.

People who were there told the paper that many staffers were shocked.

READ MORE: Car Fleeing CHP Causes AC Transit Bus to Plow Into Oakland Home

Alphonso David, president of the Human Rights Campaign, condemned the HUD secretary’s reported comments.

“From his comments on trans service members to his support for a proposal that would literally permit emergency shelters to turn away trans people who are homeless, hardly surprising that @SecretaryCarson would blatantly dehumanize trans people in his official capacity,” David said in a tweet Thursday.

MORE NEWS: Marin County Judge Tentatively Rejects Cutting Inmate Crowding at San Quentin

A HUD spokesperson responded to the Post, saying “The secretary does not use derogatory language to refer to transgendered individuals. Any reporting to the contrary is false.”