Who’s on First?: Probing the Learning and Representation Capabilities of Language Models on Deterministic Closed Domains