A statue is not merely a marker of some neutral historical fact. It honors those it depicts and exerts a subtle moral influence over the society that raises them in its public spaces. This is why the removal of statues commemorating Confederate figures is entirely justified.

Confederate monuments were erected to read back into history the lie that these men acted with honor and out of duty, and that their public service should be commemorated. Most were raised decades after the Civil War to mark the defeat of Reconstruction and restoration of white supremacy to the South and elsewhere in the United States. Confederate statues are idols of the order they sought to preserve and extend, one of white supremacy and black bondage.

Protesters tearing down confederate monuments and public officials ordering their removal is the right step to rectifying the injustice done to the memory of those they sought to subjugate. Yet the recent wave of removals of Confederate statues has been interspersed with protesters tearing down Union memorials.

In San Francisco, a statue of Ulysses S. Grant, described by one observer as “a slave owner too, before the Civil War,” was torn down. Grant served as a Union general and as president he used every power at his disposal to dismantle and suppress white supremacy in the former Confederate states. In Madison, Wisconsin, a statue of Colonel Hans Christian Heg, an antislavery activist who dedicated his life to abolition and died fighting the Confederacy, was torn down, beheaded, and dumped in a nearby lake.

In Washington, DC, protesters announced they plan to tear down the Emancipation Memorial depicting Lincoln hovering over a kneeling freed slave. DC delegate Eleanor Holmes Norton plans to introduce a bill to remove the statue from Lincoln Park, the same bill that would remove a statue of Andrew Jackson. The iconography of the Emancipation Memorial is indeed unfortunate, appearing to depict a relationship of paternalism between Lincoln and the slaves. But the historical pedigree of the memorial should give serious pause to those calling for its removal.

The monument was financed at considerable expense by freed (and free) blacks. It was inaugurated in the presence of President Grant and other representatives of the federal government along with DC’s black community. Frederick Douglass’s speech at the dedication did not shy away from rebuking Lincoln for his flaws: his early plan to remove blacks from the United States, his delay in embracing the cause of emancipation, his restraining of more radical abolitionists in his command.

Nevertheless, Douglass continued, “it was enough for us that Abraham Lincoln was at the head of a great movement, and was in living and earnest sympathy with that movement, which, in the nature of things, must go on until slavery should be utterly and forever abolished in the United States.”

Douglass highlighted Lincoln’s ability to marshal the “earnest sympathy and the powerful co-operation” of those loyal to the United States to accomplish what was otherwise impossible, the destruction of the Confederacy and slavery along with it. Douglass recruited this memory for the cause of saving Reconstruction — as we now should to complete the work of emancipation that was begun but unfinished.

Adding statues to the park — perhaps the original proposal by Harriet Hosmer to have Lincoln flanked by black Union soldiers, shoulder to shoulder as equals — is a good way to counterbalance the memorial without taking away what black men and women in their freedom and from their treasure had commissioned and raised. It better illustrates what Civil War historian Matt Karp calls “the mass politics of antislavery,” in which “the construction of an antislavery majority in the North — the true ‘abolition-democracy’ — was an essential precondition for the Civil War’s emancipatory bond between Republican politicians, Northern soldiers, and Southern slaves.”

It is not wise to reason about the motivation of presumably uncoordinated acts across multiple cities. The small handful of people who have torn down Union memorials aren’t representative of the millions who have taken to the streets to protest. But just as the intention behind the raising of Confederate monuments was a clear celebration of white supremacy, the intention behind monuments commemorating Union leaders and antislavery activists is about the celebration of black emancipation — exactly the intention at the heart of the current wave of antiracist protests.

The Civil War should never be forgotten. Monuments to the defeated white supremacists of the Confederacy should indeed come down. Let a new era of commemoration of the cause of antislavery flourish. Erect statues to Harriet Tubman, Frederick Douglass, Thaddeus Stevens, Charles Sumner, John Brown, Hans Christian Heg, the Wide Awakes, German 48ers, and countless others who fought to vanquish slavery and the confederacy in every city and town in America. We cannot dismantle racism without the mass politics and multiracial solidarity modeled effectively by the antislavery cause. May their memory endure forever.