HUMANEVAL-WORKSHOP.GITHUB.IO

With rapid advances in generative models for both language and vision modalities, such as GPT-3, DALL-E, CLIP, and OPT, human evaluation of these systems is critical to ensure that they are meaningful, reliable, and aligned with the values of those who need them. These human evaluations are often trusted as indicators of whether models are safe enough to deploy, so it is important that these evaluations themselves are reliable. Several applications relying on these models have since emerged. Aside from the private sector, even governments are increasingly using generative models such as chatbots to better serve their citizens. However, the community also faces a lack of clarity around how to best conduct human evaluations (and what to even evaluate for). It is thus unclear whether prior established practices are sufficient given the socio-technical challenges posed by these systems. Recognizing the successes and socio-technical challenges associated with these technologies, this..