"I think there are elements inside of Washington, also inclusive in the White House, that are not necessarily abetting the president's interests or his agenda", he added.
He said that if he was still working in the White House, he would not have recommended Trump make the statement he did. "I think [Trump] needed to be much harsher as it related to white supremacists".
"I applaud General McMaster [Trump's national security adviser] for calling it out for what it is, it's actually terrorism".
The former White House staffer's comments come at the same time Trump's daughter Ivanka took to social media to condemn the racist violence that saw one man driver a auto into a crowd of people.
When pressed further, McMaster said he ready to work with anybody who will "help advance the president's agenda and advance the security, prosperity of the American people". "This hate and division must stop, and it must stop now".
Mr Scaramucci criticised the influence of Mr Bannon and his right-wing Breitbart News website, suggesting the President needed to move more toward the mainstream and moderates in order to push through "a very successful legislative agenda". "I think he's also of the impression that there is hatred on all sides", he said.
He did not elaborate on what "that sort of nonsense" was, either, on the back of Breitbart News declaring the Unite the Right rally a "dereliction of American values" and "a disgusting and unwelcome throwback to segregationist ideas and rhetoric at a time when patriotic conservatism is gaining ground" on Saturday night.More news: Record number of migrants arrive in Spain
More news: Trump to back plan that would curb legal immigration
More news: Eight people injured as van rams into diners, driver arrested
In Charlottesville, Virginia on Saturday, white nationalists and counter-protesters clashed in riots that turned deadly, resulting in three deaths and several injuries.
"Can you and Steve Bannon still work together", Todd asked.
He added: "With the moral authority of the President you have to call that stuff out".
Ahead of the livestream, Scaramucci gave his first television interview since his short-lived tenure in the White House, speaking out against Bannon and saying the president "knows what he's going to do with" him.
"I think the president knows what he's going to do with Steve Bannon".
One source said rumors of Bannon's demise have been exaggerated in the past, but that there are serious conversations happening now about whether there is a place for him in the administration going forward.