It is simply not the case that war, slavery, conquest and colonization came first and the objections only much later.
I suspect that just about every time someone was about to do something evil to another person in the presence of witnesses, there was someone who objected.
These things are not and never have been about "reason" or justice or any genuine belief that the oppressors are doing "god's work"--it is about greed, money, power and control.
I like to think of the example of Lincoln--it makes the US as a political unit look much better to "forget" that Lincoln was not willing to wage Civil War to emancipate anyone--he did so to preserve the union. At least Lincoln was honest about his motives and limits in that regard, but we "need" to remember the situation falsely to feel better about ourselves collectively.