A good read on the subject of the history of the Constitution and slavery is: Lawrence Goldstone, Dark Bargain Slavery, Profits, and the Struggle for the Constitution, New York: Walker & Company, 2005.
Old Walther's views on slavery, although surprising to this modern reader, were typical of his times. Author Goldstone notes that "Southerners argued that divine revelation showed inequality to the order of the universe. The Scriptures, they said, demonstrated that 'an inferior race must live under the domination of the superior' (Elkins, Slavery, p. 36)."
The cite for the Elkins book is as follows: Elkins, Stanley. Slavery: A Problem in American Institutional and Intellectual Life. Chicago: University of Chicago Press, 1959.