An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning