An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models