Gemini for Android is getting two new features that will allow users to ask questions about their smartphone’s current screen. Dubbed ‘Ask about this screen’ and ‘Ask about this video’, these tools capture a screenshot and process the content of the screen. Then, when prompted by the user, it can answer with relevant information that the user is seeking. The video feature, which only appears to be compatible with YouTube, can also provide video summaries provided there are captions added to the video.
Gemini Now Lets Users Ask Queries About Their Screen
Gadgets 360 staff members spotted the features in the Google app version 15.33.38.28.arm64 and later in several different smartphones. These appear on the top of the Gemini floating window when a user summons the AI assistant. Those who have access to the feature will see a separate rectangular strip with the text “Ask about this screen” and a screenshot icon. If Gemini was activated while playing a video on YouTube, the text is replaced with “Ask about this video” and the YouTube icon.
Gemini Ask about this screen feature
Tapping the icon for ‘Ask about this screen’ captures a temporary screenshot within Gemini, and then the user can ask any query about the screen. The responses will only be based on what the AI can see in the screenshot, however, users can also prompt Gemini to access the Internet to find more information. One of its use cases could be asking for a short summary of a news article.
To ask a follow-up question, users can just tap the microphone icon or the keyboard icon at the bottom of the floating window. Notably, the captured screenshot is not saved on the device. Additionally, long screenshots are not supported with this feature.
The ‘Ask about this video’ feature only works for a YouTube video where captions have been added. In our testing, we found that no other video platform, despite the status of captions, was compatible with the feature. However, on a YouTube video, using the feature allows users to ask questions about the video. The answers are generated based on the captions, and the video itself is not analysed.