This is apparently a letter he wrote to his wife on 27th Dec, 1856:
The steamer also brought the President's message to Cong; & the reports of the various heads of Depts; the proceedings of Cong: &c &c. So that we are now assured, that the Govt: is in operation, & the Union in existence, not that we had any fears to the Contrary, but it is Satisfactory always to have facts to go on. They restrain Supposition & Conjecture, Confirm faith, & bring Contentment: I was much pleased with the President's message & the report of the Secy of War, the only two documents that have reached us entire. Of the others synopsis [sic] have only arrived. The views of the Pres: of the Systematic & progressive efforts of certain people of the North, to interfere with & change the domestic institutions of the South, are truthfully & faithfully expressed. The Consequences of their plans & purposes are also clearly set forth, & they must also be aware, that their object is both unlawful & entirely foreign to them & their duty; for which they are irresponsible & unaccountable; & Can only be accomplished by them through the agency of a Civil & Servile war. In this enlightened age, there are few I believe, but what will acknowledge, that slavery as an institution, is a moral & political evil in any Country. It is useless to expatiate on its disadvantages. I think it however a greater evil to the white man than to the black race, & while my feelings are strongly enlisted in behalf of the latter, my sympathies are more strong for the former. The blacks are immeasurably better off here than in Africa, morally, socially & physically. The painful discipline they are undergoing, is necessary for their instruction as a race, & I hope will prepare & lead them to better things. How long their subjugation may be necessary is known & ordered by a wise Merciful Providence. Their emancipation will sooner result from the mild & melting influence of Christianity, than the storms & tempests of fiery Controversy. This influence though slow, is sure. The doctrines & miracles of our Saviour have required nearly two thousand years, to Convert but a small part of the human race, & even among Christian nations, what gross errors still exist! While we see the Course of the final abolition of human Slavery is onward, & we give it the aid of our prayers & all justifiable means in our power, we must leave the progress as well as the result in his hands who sees the end; who Chooses to work by slow influences; & with whom two thousand years are but as a Single day. Although the Abolitionist must know this, & must See that he has neither the right or power of operating except by moral means & suasion, & if he means well to the slave, he must not Create angry feelings in the Master; that although he may not approve the mode which it pleases Providence to accomplish its purposes, the result will nevertheless be the same; that the reasons he gives for interference in what he has no Concern, holds good for every kind of interference with our neighbors when we disapprove their Conduct; Still I fear he will persevere in his evil Course. Is it not strange that the descendants of those pilgrim fathers who Crossed the Atlantic to preserve their own freedom of opinion, have always proved themselves intolerant of the Spiritual liberty of others?
He seems to have opposed slavery in principle, but like so many western people of the time he couldn't see how it could be dispensed with. He didn't speak loudly against slavery but didn't advocate it either. He seemed to perceive it as an inescapable evil of the period in which he lived that would require time to resolve. When the Civil War ended slavery, it seemed to have left the slaves in a disadvantaged position and many racist attitudes persisted.
Many statues of Confederate figures were apparently erected in the 1920s. There are comparatively few statues commemorating black figures. It is easy to see why the statues need to be removed or replaced with something more appropriate or inspiring, but destroying them seems paranoid.