HILLSBOROUGH, N.C. (WNCN) — An Orange County chocolate shop owner is an unlikely voice in what’s turning out to be a high stakes game of free speech – and he is showing no signs of backing down.

Matthew Shepherd says almost every Saturday a group gathers outside his chocolate shop in downtown Hillsborough waving Confederate flags.

“They were calling me names, they’re calling everybody names,” Shepherd said. “They’re instigating, they’re poking, they’re blocking the sidewalk.”

Shepherd had had enough so he called the police. The police told him the group had a right to free speech and could be there.

“So I put on my sign, ‘burn a Rebel flag, get a free chocolate.’ None of the locals want that flag there, none of them. So instead of seeing angry people, I was seeing people, ‘Oh look at this!’ It was lightening the mood.”

But when members of the Sons of the Confederacy shared a photo of them posing with the monument on Facebook, things escalated.

“When they left, I took the sign away and wrote gelatos on it again, you know, it was just for the locals,” explained Shepherd. “Just a little laugh, all it was and then death threats started pouring in from all over Texas and just everywhere.”

CBS 17 reached out to some of the Confederate supporters who were there, but have not heard back.

One of them, Steve Marley, posted publicly on Facebook, “When I’m minding my own business and I get attacked by this guy and he puts out advertisements on his business chalkboard to promote violence on burning my property, then it’s not going to end well.”

While Mayor Tom Stevens would prefer not to have the Confederate flag in his town.

“I firmly believe that showing the Confederate flag in public regardless of any other intent sends a message that re-enacts how the flag was used to intimidate persons of color and is a hateful message,” Stevens said.

He understands it’s protected.

The mayor says the police are investigating the threats and taking them seriously.

“Were committed to doing what we can to reduce that rhetoric, to keep people safe, to welcome people from all background here in Hillsborough,” Stevens said.

On Saturday, locals held a rally in support of Matthews chocolates, so many people bought chocolate that day Shepard sold out and had to close on Sunday.

“I didn’t do this to make money. This was not even in my realm of thinking,” explained Shepherd. “I was in Home Depot the other day and I got two fist bumps and a high five from people I don’t even know. They’re like a way to go, fight the fight, we’re all behind you. I’m like I just put a sign out it was there for an hour.”