BALTIMORE (WJZ)–  Local religious and business leaders are criticizing Maryland Senate President Mike Miller after he wrote a letter that suggested Gov. Larry Hogan acted too fast in removing the Roger B. Taney statue.

The fallout continues over a controversial statue that was removed from the capitol grounds in Annapolis.

READ MORE: Two Teenagers Shot, One Killed, At Inner Harbor During Memorial Day Weekend, Police Say

RELATED: Crews Remove Taney Statue From Maryland State House Overnight

Senator Miller questioned the removal of the controversial statue but now he’s said he regrets his decision.

Miller is facing pressure not from opponents but from voters, after he sent a long letter to Gov. Hogan questioning why the controversial statue was removed without public input.

Taney was a Supreme Court justice who supported the controversial Dred Scott decision that upheld slavery.

In the letter, Miller acknowledged that decision created lasting wounds for the country, but the senator also defended Taney, at one point he wrote:

“Unlike George Washington who freed his slaves upon his death, Taney freed his slaves early in his life.”

After the letter, people protested outside one of Miller’s Maryland businesses.

READ MORE: Jarrettsville Young Marines Honor Fallen Heroes, Prepare For Memorial Day

“We were deeply offended by his letter to the governor,” said Bruce Branch of the Maryland Business and Clergy Partnership. “We just think it’s a slap in the face at a time when he was supposed to support us.”

After the backlash, the senate president released a statement to clarify his initial remarks. He wrote:

“I do regret that sharing my historical perspective has distracted from the larger issues we must face together as a nation and from my role to bring unity.”

After the violent protests in Charlottesville, leaders around the country removed Confederate and other controversial monuments.

Protesters in Miller’s district said they want reassurances he will fight for their cause.

“It hurt us in that we cannot look to him as a leader for all people which as I’ve stated, created a moral crisis,” said Rev. Charles McNeill of the National Capital Baptist Convention.

The protesters said they may continue demonstrating outside Miller’s business if they do not like his future actions.

In his most recent statement, senator Miller also wrote that he’s condemned the statements of chief justice Taney.

MORE NEWS: Baltimore City And Surrounding Communities Experiencing High Community Transmission Of COVID-19

Follow @CBSBaltimore on Twitter and like WJZ-TV | CBS Baltimore on Facebook

Ava-joye Burnett