Google’s new Voice Access update uses AI to detect in-app icons

Tidio Live Chat Software - Add Tidio live chat software to your website in minutes. Contact visitors and turn them into happy customers. Enhance their experience and boost your sales. Get it for Free

WP Rocket - WordPress Caching Plugin

Must Read

Google launched an update for Voice Access so that it can automatically detect icons using only the pixel values displayed on the screen, regardless of whether icons have been given suitable accessibility labels or not. With this update, the users will have more control over Android devices using voice commands.

Apparently, Voice Access needs on-screen user interface (UI) elements to have reliable accessibility labels to function properly. These elements are provided to the operating system’s accessibility services via the accessibility tree. However, the usability of Voice Access sometimes reduces because, in many apps, adequate labels aren’t always available for UI elements.

Also Read

Therefore, to address the issue, Google came up with a way where a machine learning model automatically detects icons on the screen based on UI screenshots. In this way, Voice Access can determine whether elements like images and icons have accessibility labels, or the labels are provided to Android’s accessibility services.

The updated Google Voice Access introduces a vision-based object detection model called IconNet. This model can automatically detect 31 types of icons on the screen in a manner that is agnostic to the underlying structure of the app being used, launched as part of the latest version of Voice Access.

Elegant Themes - The most popular WordPress theme in the world and the ultimate WordPress Page Builder. Get a 30-day money-back guarantee. Get it for Free

Furthermore, this model is optimized to run on-device for mobile environments, with a compact size and fast inference time to enable a seamless user experience. According to a Google blog post, the tech giant is currently working on improving the model and hopes that it can extend the icon detection capability to 70 types in near future.

How does Google detect icons?

The Google scientists wrote in their blog post that the problem of detecting icons on app screens is similar to classical object detection. Based on the novel CenterNet architecture, here, IconNet extracts features from input images and then predicts appropriate bounding box centers and sizes. In order to train the IconNet, they collected and labeled more than 700,000 app screenshots, streamlining the process by using heuristics, auxiliary models, and data augmentation techniques to identify rarer icons and enrich existing screenshots with infrequent icons.

Now, Voice Access can easily identify the icons when actions are performed by tapping anywhere in the region of the UI element of interest. So, whenever the users refer to icons detected by IconNet by their names, e.g. “Tap Gallery”, Voice Access can detect it faster than before.

Gilles Baechler and Srinivas Sunkara, the Google Software Engineers at Google Research said in a blog post, “We are constantly working on improving IconNet. Among other things, we are interested in increasing the range of elements supported by IconNet to include any generic UI element, such as images, text, or buttons. We also plan to extend IconNet to differentiate between similar looking icons by identifying their functionality. On the application side, we are hoping to increase the number of apps with valid content descriptions by augmenting developer tools to suggest content descriptions for different UI elements when building applications”.

Google introduced Voice Access back in 2016 and is still working on a number of projects based on the voice command coupled with AI.

iThemes WordPress Hosting

Stay updated

Subscribe to our newsletter and never miss an update on the latest tech, gaming, startup, how to guide, deals and more.



- Advertisement -
- Advertisement -


Grow Your Business

Place your brand in front of tech-savvy audience. Partner with us to build brand awareness, increase website traffic, generate qualified leads, and grow your business.

- Advertisement -

Grow Your Business

Get these business solutions, tools and services to help your business grow.

Elementor -Join 5,000,000+ Professionals Who Build Better Sites With Elementor. Build your website with 100% visual design that loads faster and speeds up the process of building them.

WP Rocket

WP Rocket - Speed up your website with the most powerful caching plugin in the world. The website speed increase means better SEO ranking, user experience, and conversation. It’s a fact that Google loves a fast site.


Kinsta - If you are looking for WordPress managed hosting, Kinsta is in the leading front. Kinsta provides WordPress hosting for a small or large business that helps take care of all your needs regarding your website with cutting-edge technology.


OptinMonster - Instantly boost leads and grow revenue with the #1 most powerful conversion optimization toolkit in the world. 700,000+ websites are using OptinMonster to turn their traffic into leads, subscribers, and sales.


- Advertisement -
- Advertisement -
India to Produce Upcoming iPhones: Trade Minister Japanese Professor Developed A Power Semiconductor made of Diamond Google Releases New Product for India’s Merchants Indian EV Startup Unveil Two AutoBalancing Electric Scooters OPPO Find X6 Pro Images Render via Weibo Sony Develops New Tech to Reduce Noise of Image Sensors Tesla’s S & Y Models Earn Best-in-Class Cars of 2022 India Approved $320m to Promote Homegrown Payments Network Twitter Roll Out TikTok-like ‘For You’ Timeline on iOS Nothing Phone Enters US with Beta Membership Program
Finally! Apple’s VR Headset Coming this Spring Harvard Scientists Develop New Cell Therapy to Kill, and Prevent Brain Cancer HTC Vive XR Elite VR/AR Headset Now Selling for $1099 Realme 10 Launched in India with Helio G99 Gaming Chipset Twitter Data Breach Revealing User’s Private Information Instagram New Feature to Create Albums With Friend Redmi Note12 Pro Plus Launched in India POCO C50 Launched in India