Orca

Orca is a 13-billion parameter model developed by Microsoft Research. It addresses the challenge of enhancing the capability of smaller models through imitation learning. Orca learns to imitate the reasoning process of large foundation models (LFMs) by leveraging rich signals from GPT-4, including explanation traces, step-by-step thought processes, and other complex instructions. It is guided by teacher assistance from ChatGPT. The progressive learning approach of Orca aims to improve model perf...