Sketch-Driven AI: A Game-Changer in Cancer Detection and More

Leading academics at Surrey University believe that teaching machine learning applications to selectively recognize objects within images, whilst ignoring others, could radically transform cancer diagnosis and spark significant breakthroughs in the field.

Research: What Can Human Sketches Do for Object Detection? Image Credit: Jameelaug91 / Shutterstock

Unveiling a Unique Sketch-Driven Tool

Surrey University is gearing up to exhibit its unique sketch-driven object identification tool at the annual Computer Vision, Pattern, and Recognition Conference (CVPR). This tool empowers users to sketch an object that the AI system will use as a reference to search within an image and spot a corresponding object while disregarding more generic possibilities.

Professor Yi-Zhe Song, who leads this research at the Institute for People-Centred AI at Surrey University, shared his perspective on the project, stating:

"An artist's sketch carries unique cues that can't be succinctly expressed through words, echoing the sentiment that 'a picture paints a thousand words.' While simple descriptive words assist in generating images for modern AI systems, they can't capture the user's individuality or accurately satisfy the user's specific demands," explained Professor Yi-Zhe Song.

We train an object detector using SBIR models. (a) First, we train an FG-SBIR model using existing sketch–photo pairs that generalise to unseen categories. (b) To train the object detector module, we tile multiple object-level photos from SBIR datasets [75] and use its paired sketch encoding via a pre-trained sketch encoder to align the region embedding of detected boxes. (c) Inclusion of sketches for object detection opens several avenues like detecting a specific object for query sketch (e.g., detect a “zebra” eating grass) or part of an object (e.g., “head” of “zebra”).

Broad Applications and Potential Impact

"This is exactly where our sketch-driven tool enters the picture. Artists can direct the AI system using sketches to pinpoint a particular object while ignoring others. This feature carries enormous potential in various sectors, such as healthcare, where it can assist in identifying more aggressive tumors, or in wildlife preservation initiatives by aiding the detection of rare animals," Professor Yi-Zhe Song expounded.

In their conference paper, the researchers illustrate the use of a tool to search for an image containing zebras. The tool depends exclusively on a sketch of a single zebra feeding as a guiding reference for the search. Incorporating visual cues like pose and structure, the AI tool makes decisions based on the precise instructions given by the amateur artist.

Human-AI Interaction and Future Outlook

Professor Song continued: 

The ability of AI to discern objects through individual amateur sketches represents an extraordinary leap in harnessing human creativity in the Computer Vision field. It allows humans to interact with AI from a new angle, transitioning from AI making decisions to instructing it to behave exactly as directed while preserving the crucial aspect of human intervention.

Surrey University is preparing to introduce this revolutionary research at the renowned Computer Vision, Pattern, and Recognition Conference (CVPR) 2023, a respected stage for exhibiting leading AI research on an international scale. Impressively, Surrey University has outperformed other educational bodies by having an exceptional number of papers accepted, with more than 18 papers passing the review. Additionally, one of their papers has been nominated for the highly sought-after Best Paper Award, underlining the exceptional quality and influence of their research efforts.