Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets