Use the vitals package with ellmer to evaluate and compare the accuracy of LLMs, including writing evals to test local models.

Abstract: Programming has become increasingly important in our society. However, the learning process presents significant challenges, particularly for novice students of introductory courses. From ...