Commentary

Apple Searches For Anything In A Screenshot

Apple iOS 26 is getting a visual search feature by combining Visual Intelligence with Apple Intelligence, giving brands another way to be found on iPhones and iPads. 

The role AI plays gives brands a new marketing channel to help consumers shop for products on their devices by snapping a screenshot.

The search function found on the bottom of the screen can recognize images in similar products in the screenshot. It also can identify events and pre-populate a user's calendar.

The feature is intended to remove the middle step of unlocking the phone, opening Google or ChatGPT, and typing a query or uploading a photo to get an answer. Removing the middle step has been a recent topic of discussion, from all types of companies such ad-tech platforms to search engines.

Billy Sorrentino, senior director of human interface at Apple, said it works automatically with any app in the App Store.

advertisement

advertisement

“Open a social media app, and you see a great jacket,” he said, adding that taking a snapshot of the jacket and one tap on the image search button on the bottom right of the iPhone screen allows the user to find similar images on Google or other frequently used apps like Etsy.

Sorrentino also took a snapshot of a lamp to find other similar products to the one identified in the photo.

There also is an Ask button on the left side of the screen. Taking a photo and clicking on Ask takes the user to ChatGPT to inquire about an object or place. He took a picture of an instrument and asked ChatGPT “what rock songs is this instrument featured in?”

The App Intents framework allows to integrate this feature into their app.

Apple formally announced Visual Intelligence Monday at its annual WWDC event. It has now been designed to use across other apps in the App Store.

This may sound very similar to Google’s Circle to Search feature announced last year now available on Android devices through the Google and Chrome Apps. Google.

Apple also announced an AI-driven live translation function, similar to Google Meet. It works the phone and messaging apps.

Next story loading loading..