Prompt-free and Efficient Few-shot Learning with Language Models