Apple is planning to integrate the smart chatbot ChatGPT into its personal assistant Siri. This integration will provide users with many capabilities, such as the ability to request details about the content displayed on the screen through Siri.
The feature of recognizing screen content is similar to Google’s “Circle to Search” feature, which allows users to draw a circle around any element or highlight it on the phone screen to identify and search for it using the Google search engine. Google has made this feature available on many Android devices in recent months, adding enhancements such as translation support, voice recognition, QR code scanning, and barcode scanning.
Siri’s assistant can now use ChatGPT to answer complex questions and read the content on iPhone screens, iPads, or Mac computers. For example, when watching a video, viewing an image, or browsing the internet, users can now ask questions to ChatGPT through Siri about what is displayed on the screen.
When receiving a question, a screenshot is sent to ChatGPT, which can understand the context and answer what users want. To protect privacy, Siri’s assistant asks for the user’s permission before taking a screenshot and sending it to ChatGPT.
This feature differs from the Visual Intelligence feature, which allows using the camera to get answers to mathematical problems or information about items in the surrounding environment, using internet search, and is available exclusively on iPhone 16.
It is worth mentioning that Apple is also working on developing a feature to understand screen content using its own artificial intelligence models. This feature is expected to be released in another update to iOS 18 next year. This technology will allow Siri to understand context and take actions based on screen content, processing all data on the device to ensure privacy.
It is worth mentioning that the current testing update iOS 18.2 by Apple comes with multiple AI-based features, such as the Image Playground feature, Genmoji emoji generation feature, visual search on iPhone 16, and other features.
