Thomas AJ Oliver was called a traitor by some of his neighbors, but it appears more likely that he simply refused to re-enlist when his unit got order to leave Gloucester, Virginia and go to the defend of Richmond late in the war. This speaks to the point that he had never traveled outside of Gloucester county in his life, up to the time of the war. He joined the CSA Army to defend his family, their home, and the local community; he did not sign up to defend others from “far-off” places, and he certainly did not sign up to defend the institution of slavery!

https://www.findagrave.com/memorial/26612328/thomas-andrew_jackson-oliver