Apple Unveils Ferret-UI, a Breakthrough AI for Mobile UI Understanding

The new AI model, Ferret-UI, significantly enhances the interpretation of mobile interfaces, promising improved accessibility and usability for iPhone users.

Apple's Ferret-UI, a multimodal large language model, is designed to accurately interpret and interact with mobile user interfaces.
The AI model showcases superior performance, outperforming existing models in tasks like icon recognition and widget classification.
Ferret-UI's advanced capabilities could potentially revolutionize Siri's functionality, making it more intuitive and helpful.
The model was trained using a diverse dataset, including proprietary iPhone screen data and the RICO dataset for Android.
Apple plans to integrate Ferret-UI in its upcoming iPhone 16 Pro, enhancing the device's AI capabilities.