Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions