To increase the level of abstraction that computer vision algorithms can work with, Meta AI has developed a neural network that animates children’s drawings. Anyone can test the model.
While many models are designed to process realistic images of people, children’s drawings add variety and unpredictability, which greatly complicates the definition of the depicted objects.
At the first stage, the Meta model highlights the figures of the characters in the drawing. Object detection using existing techniques works quite well in children’s drawings, but segmentation masks are not accurate enough to be used in animation. To solve this problem, Meta used the bounding boxes obtained using an object detector and applied a number of image processing steps to obtain masks.
When extracting anthropomorphic characters from a child’s drawing, an object detection model based on the convolutional neural network Meta Mask R–CNN was used. Then the skeleton of the character is created. Changing the character’s posture is achieved by rotating parts of the skeleton around the joints.
The model can be tested by following the link.