Machine learning is becoming increasingly integrated in a wide range of fields. Its widespread use has spread to all industries, including the world of user interfaces (UI), where it is important to predict semantic data. This application not only improves accessibility and simplifies testing, but also helps automate UI-related tasks, resulting in a more streamlined and effective application.
Currently, many models rely primarily on static screenshot datasets evaluated by humans. However, this approach is expensive and may lead to unexpected mistakes in some activities. Since it is not possible to interact with her UI elements in the live app to confirm the conclusion, the human annotator relies solely on visual cues when assessing whether her UI elements are tappable from the snapshot. is needed.
Despite the disadvantages of using datasets that only record fixed snapshots of mobile application views, they are expensive to use and maintain. However, due to their data richness, these datasets remain invaluable for training deep neural networks (DNNs).
As a result, Apple researchers collaborated with Carnegie Mellon University to develop the Never-Ending UI Learner AI system. The system can continuously interact with real mobile applications and continually improve its understanding of UI design patterns and emerging trends. Autonomously download apps from the app store for your mobile device and thoroughly explore each one to find new and challenging training scenarios.
Never Ending UI Learner has studied over 5,000 hours of devices and performed over 500,000 actions across 6,000 apps. This long-term interaction trains three different computer vision models. One to predict tappability, another to predict draggability, and a third to determine screen similarity.
The study performs a number of actions, such as taps and swipes, on components within each app’s user interface. The researchers emphasize using designed heuristics to classify his UI elements and identify characteristics such as whether buttons can be touched or images can be moved.
The data collected is used to train a model that predicts the tappability and draggability of UI elements and the similarity of the screens they appear on. The end-to-end procedure does not require any more human-labeled samples, even though the process can start from a model trained on human-labeled data.
The researchers highlighted the benefits of this method of actively exploring apps. This helps machines identify difficult situations that might be missed by typical human-labeled datasets. In some cases, the image is not always so clear that you may not notice everything you can touch on the screen. However, the crawler taps on an item and instantly monitors what happens, giving you clearer and better information.
The researchers demonstrated how a model trained on this data improved over time, reaching 86% accuracy in predicting tappability after five training sessions.
The researchers emphasized that applications focused on accessibility remediation may benefit from more frequent updates to catch subtle changes. Conversely, for tasks such as design pattern summarization and mining, longer intervals may be desirable to allow more significant UI changes to accumulate. Further research is needed to find the optimal schedule for retraining and updating.
This research highlights the potential for never-ending learning, allowing systems to adapt and advance by continuously incorporating more data. While current systems focus on modeling simple semantics such as tappability, Apple wants to apply similar principles to learn more sophisticated representations of mobile UI and interaction patterns. .
Please check paper. All credit for this study goes to the researchers of this project.Also, don’t forget to join us 31,000+ ML subreddits, 40,000+ Facebook communities, Discord channeland email newsletterWe share the latest AI research news, cool AI projects, and more.
If you like what we do, you’ll love our newsletter.
I’m also on WhatsApp. Join our AI channel on Whatsapp.
Rachit Ranjan is a consulting intern at MarktechPost. He is currently pursuing his bachelor’s degree from Indian Institute of Technology (IIT) Patna. He is actively developing a career in the fields of artificial intelligence and data science and has a passion and dedication to exploring these fields.