NEW YORK (CBS News) — President Trump’s planned statements on infrastructure derailed into explosive comments about the violence in Charlottesville, Virginia, in which the president defended his Saturday statement that there was hatred and violence, “on many sides.”

Mr. Trump, speaking from a podium at Trump Tower on Tuesday, said the “alt-left” bears some responsibility for the violence in Charlottesville, and “nobody wants to say that,” blaming violence on “both sides.”

READ MORE: San Leandro Police Arrest 2 Suspects -- Age 12 and 13 -- in Armed Carjacking

Trump Again Says ‘Both Sides’ To Blame For Charlottesville Violence
Right-Wing Groups To Hold Bay Area Rallies In Wake Of Charlottesville
Pelosi Urges NPS To Reconsider Crissy Field Right-Wing Rally Permit
Continuing Coverage

“Let me ask you this,” Mr. Trump said. “What about the fact that they came charging, that they came charging with clubs in their hands, swinging clubs. Do they have any problem? I think they do.”

“I will tell you something,” the president added. “I watched this very closely, much more closely than you people watched it and you have, you had a group on one side that was bad and you had a group on the other side that was also very violent. And nobody wants to say that, but I’ll say it right now. You had a group on the other side that came charging in without a permit and they were very, very violent.”

Mr. Trump was accused — even by top members of his own party — of failing to call out the white supremacists at a rally that left one woman dead after a suspected attacker rammed his car into a sea of people. Mr. Trump was pressured to renounce the Nazi and white supremacist supporters behind Saturday’s deadly events, and eventually did so on Monday.

But on Tuesday, Mr. Trump circled back to his earlier comments, defending his original statements by saying he wanted to be sure he had all the facts.

“I didn’t wait long,” Mr. Trump said. “I didn’t wait long. I didn’t wait long. I wanted to make sure, unlike most politicians, that what I said was correct. Not make a quick statement. The statement i made on Saturday, a quick statement, the first statement, was a fine statement. But you don’t make statements that direct to get the facts. It takes a little while to get the facts.”

The president did condemn the suspected driver, Alex Fields Jr., calling him a, “murderer.” Mr. Trump has not yet reached out to the family of Heather Hayer, the woman killed in the apparent attack.

“The driver of the car is a murderer, and what he did is a horrible, horrible inexcusable thing,” Mr. Trump said.

READ MORE: 850 Additional Parking Spaces To Open At Antioch BART Station

But he couldn’t resist revisiting the comments he made Saturday.

“I think there’s blame on both sides,” Mr. Trump said.

The president said he feared the movement to remove controversial statutes was going too far, asking if people would support the removal of statues depicting former presidents George Washington or Thomas Jefferson. The Charlottesville “Unite the Right” rally was originally set up to protest the removal of a statute of Confederate general Robert E. Lee from a public park.

“Those people, all of those people, excuse me, I’ve condemned neo-Nazis, I’ve condemned many different groups,” Mr. Trump said. “But not all those people were Neo-nazis, believe me. Not all those people were white supremacists. Those people were also there because they wanted to protest the taking down of a statue Robert E. Lee.

So, excuse me, when you take a look of some of the groups and you see, and you’d know if if you were honest reporters and many of you are not, many of those people were there to protest the taking down of the statue of Robert E. Lee. So this week it’s Robert E. Lee, I notice Stonewall Jackson is coming down. I wonder is it George Washington next week and is it Thomas Jefferson the week after? You really do have to wonder, where does it stop?”

The president also made remarkable comments as he was leaving the podium, saying he owns a large “house” in Charlottesville. When a reporter asked if he meant his winery, he said yes.

“I mean, I know a lot about Charlottesville,” Mr. Trump said. “Charlottesville is a great place that’s been very badly hurt over the last couple of days. I own actually one of the largest wineries in the United States. It’s in Charlottesville.”

Elaine Chao, the transportation secretary, and Gary Cohn, the president’s chief economic adviser, were forced to awkwardly come to the podium to take questions from reporters on infrastructure after Mr. Trump’s explosive, off-the-cuff remarks.

Mr. Trump is spending the day at New York City’s Trump Tower, in the middle of a two-week working vacation mostly spent in Bedminster, New Jersey.

MORE NEWS: Santa Clara Shoe Store Ransacked In Late Night Smash-And-Grab

© 2017 CBS Interactive Inc. All Rights Reserved.