IEEE PULSE presents

AI, Virtual Reality, and Robots Advancing Autism Diagnosis and Therapy

Feature September/October 2021
Author: Leslie Mertz

Autism spectrum disorder (ASD) is a challenge in multiple ways. Just getting diagnosed can take months of visits to doctors and specialists. After the diagnosis, children are often put on long waiting lists to begin therapy, which itself consists of frequent sessions that while helpful, are usually quite taxing for both the children and their parents. And while child-directed therapies are available, adults who are on the spectrum often find little continuing support. Recent technologies are using artificial intelligence (AI), machine learning (ML), virtual reality (VR), and other advanced methods to address all of those issues with faster and easier diagnostics, and in-home therapeutic approaches designed for all ages.

Three of the groups behind the technologies are:

  • Cognoa of Palo Alto, CA, which plans to make its diagnostic aid available before the end of the year; and is simultaneously working on therapeutics;
  • Floreo, a startup based in Washington, DC, that has paired VR with a coaching app to help children with autism;
  • A Yale University group that has developed robots that teach social skills to children, and is now working on job-skills training for adults on the spectrum.

Speeding diagnosis

ASD diagnosis typically takes about 18 months, which is far too long, according to Halim Abbas, the chief AI officer at Cognoa (Figure 1). “The delay isn’t because it’s not possible to diagnose a child sooner; it’s just because there are long waiting lines full of false positives that specialists and the final diagnostician have to see one by one, while the child that really needs the diagnosis is waiting in line.”

Figure 1. Halim Abbas, chief AI officer of Cognoa, which has developed technology it calls Canvas Dx to streamline diagnosis of ASD. Canvas Dx combines information from parents, analysts’ review of videos of the child to identify behavioral features of ASD, and additional data from the child’s primary care physician, and with the help of an ML predictive model that scans for subtle patterns representative of the phenotype of ASD, it provides an outcome: positive for autism, negative, or inconclusive. The “inconclusive” outcome suggests a need for further screening. (Photo courtesy of Cognoa.)

That delay cuts into the critical window of neuroplasticity when the child’s brain is most responsive to therapy, Abbas said. “It is established in the literature (that) you have to start early with therapy to be able to position the child for the best outcome, and reducing that delay is a huge unmet need in health care right now in the United States.”

To accomplish that, Cognoa developed a diagnostic aid, called Canvas Dx (1), to streamline the process. With Canvas Dx, parents complete a questionnaire about the child’s behavior and capabilities, and also upload two short videos of the child at home; Cognoa-trained analysts score the videos for behavioral features of autism; and primary-care physicians complete a separate questionnaire based on a short meeting with the parents and child, Abbas described. “So now we have three sets of inputs from three agents regarding the same child. Next, we have an AI algorithm that evaluates all the inputs—this is an ML predictive model that is looking for the subtle patterns that represent the phenotype of autism—and that algorithm gives a singular outcome to the physician to be used as a diagnostic aid.”

Within minutes, the Canvas Dx algorithm provides an outcome: positive for autism, negative, or inconclusive, which suggests a need for further screening.

In developing its predictive model, Cognoa spent several years reviewing and ranking data gathered by the traditional diagnostic methodologies, consulting with its own and outside autism specialists, and going through multiple rounds of successive improvements through clinical-trial validations to build a system with high predictive power for both positive and negative diagnoses and without the gender, racial, or other biases that can creep into traditional diagnoses, Abbas said.

The U.S. Food and Drug Administration (FDA) provided its final marketing authorization for Canvas Dx in June, when it reported that it had assessed the safety and effectiveness of the software and found that it “made an accurate ASD determination in 98.4% of patients with the condition and in 78.9% of patients without the condition” (2). Cognoa plans to release Canvas Dx later this year.

Figure 2. Parent of a child who is on the spectrum, Vijay Ravindran began toying with the idea of using VR to assist therapy. Now six years later, Ravindran is the CEO of a company called Floreo, which offers a VR platform and some 200 different lessons that allow the user, or “learner,” to interact with scenes and characters as a way to develop social and behavioral skills. (Photo courtesy of Floreo.)

Therapy for kids

The idea for Floreo’s VR platform, which is already available and providing autism therapy to children, had its start with a parent’s desire to benefit his child who was on the spectrum. “Seeing my son’s interactions with VR when he was six years old, really seeded the idea that VR could be a powerful new medium for therapy,” recalled company CEO Vijay Ravindran (Figure 2). He took his idea—along with his past experience as a software engineer at companies ranging from Amazon to The Washington Post—and ran with it.

Within a year of its founding in 2016, the company had created a set of prototype therapy lessons. Through a National Institutes of Health Fast-Track grant in 2017 and considerable venture capital investments (totaling $6 million as of July), Floreo today offers about 200 different VR lessons, all of which run on inexpensive, off-the-shelf VR goggles (Figure 3).

Floreo’s lessons allow the user, or “learner,” to interact with scenes and characters as a way to develop social and behavioral skills, Ravindran said. Examples include how to engage socially with peers in a school setting; cross a street safely; interact with law enforcement officers; or simply relax by peacefully watching fish in an aquarium [3], [4]. “Our lessons really span early-development skills to social and safety skills to emotional regulation,” he said. In July 2021, the company added lessons on executive-function skills, including planning, organization, flexibility, and impulse control, which could also be helpful for children with other conditions, such as attention deficit hyperactivity disorder or anxiety, he noted.

Figure 3. Enhancing the lessons further, the Floreo system is not only able to record the learner’s in-VR
activity and performance to identify areas of improvement or of deficit, but can also pair with an app so that a supervising parent or professional—whether in-person or via telehealth—can see what the learner is seeing in VR, puppeteer characters, and provide real-time coaching. (Photo courtesy of Floreo.)

VR works well as a training vehicle for several reasons, Ravindran said. For one thing, it allows control of audio and visual distractions—important for children on the spectrum—so they can focus on the lesson. The lessons are also realistic enough that the child can transfer the material learned to the real world. In the street-crossing lesson, for instance, a child can watch the cars go by, identify where the walk sign and button are, hear the sounds that go with the sign and button, and practice crossing the street safely, he described. “In addition, we can create peer-to-peer interactions and allow the child to practice social or other skills in a school or another setting.”

To enhance the experience further, the Floreo system is able to record the learner’s in-VR activity and performance to identify areas of improvement or of deficit, and pairs with an app that runs on an iPad, so a supervising parent or professional—whether in-person or via telehealth—can see what the learner is seeing in VR, puppeteer characters, and provide real-time coaching (Figure 4).

The Floreo platform is already approved for Medicaid reimbursement in several states, and others will hopefully follow, Ravindran reported. While such approvals move forward, the company has an ongoing partnership with the Frist Center for Autism and Innovation at Vanderbilt University to expand its VR lessons to include independent-living training for adults who have ASD [5].

Figure 4. Floreo is also looking into the possibility of combining an individual’s profile information with data collected during VR sessions and using it to suggest the next lesson. If it proves successful, Ravindran believes it “will lead to incredible new capabilities powered by AI and ML, [including] the opportunity to be more adaptive in the types of treatments we recommend and the type of feedback we can give.” (Photo courtesy of Floreo.)

Extending to adults

A Yale University research group has already done considerable work developing therapy for children, and is now also turning its attention to adults who are on the spectrum.

Figure 5. Brian Scassellati, Ph.D., Yale University professor of computer science, and his research group have developed fully autonomous robots to help children with ASD, and is now working on robot-delivered therapy to assist adults who are on the spectrum. (Photo courtesy of Yale University.)

The group, headed by Brian Scassellati, Ph.D., professor of computer science (Figure 5), gained notice in 2018 for developing fully autonomous robots to teach social-communication skills to children with ASD [6]. “We placed our robots in the homes for 30 days, and because every child is at a different point on the spectrum, our robots had to adapt and change based on what they needed to do,” he remarked. “That meant we were putting the systems out there and we had no idea what they were going to look like a month later. It’s probably the most frightening thing I’ve ever done as a roboticist and technologist, but as it turned out we had a really successful deployment and were able to show the kids made clinically verified improvements in social skills.” These included progress with making eye contact and initiating communication—improvements that persisted after the robots were removed from the homes (Figure 6).

Today, Scassellati’s team is part of a multi-institution group of engineers, roboticists, and researchers who are developing new job-skill coaching software aimed at adults who are on the spectrum. “If you look at what has been done in the past, 95% of the autism-related technologies focused on kids. It’s not that adults don’t need any kind of support system, but rather that there’s nothing there for them.”

Figure 6. As part of its new technology, Scassellati’s group is designing a system (shown here being tested by student Emmanuel Adeniran) to teach adults with ASD how to deal effectively and quickly with the myriad little interruptions that occur during a typical work day. (Photo by Rebecca Ramnauth.)

For their part, Scassellati and his research group are again developing robot-delivered therapy, but this time are designing it to teach adults how to deal with little interruptions during the work day (Figure 5). “While most adults can focus on a simple interruption, address it, and get back to work quickly, many adults with ASD can become derailed from what they’re doing for 5, 10, or 20 minutes, and that’s a real problem on many jobs,” he said.

The researchers have already done a small pilot trial with adults, and so far, have found that, like the children, the older audience responds very well to the robots. He believes that part of the attraction is that robots are physical entities delivering the lesson, rather than a lesson that appears on an app on a phone or other screen, Scassellati contended. “People respond to being ‘in person,’ which is why business people don’t call or Skype or Zoom to make the big deal, but rather meet in person. And with this training, they connect more with a physical robot.”

The next phase for the interruption-skills project is to look at results. “It’s not enough just to build the technology. You really have to show that it’s actually helping people. And not just that they like it, but that it’s actually providing real benefit,” he said.

Tech with results

These groups are among many that are focusing on autism support. “There are a lot of new technologies coming out,” Scassellati said “Our hope is that we are doing our job, which is to be useful to people.”

Floreo’s Ravindran commented, “This is the most gratifying thing I’ve ever worked on. The families are just incredibly grateful that you’re thinking about them, and have a sense of challenges they face.” He added, “I know. I’ve been through it, too.”


  1. H. Abbas, F. Garberson, S. Liu-Mayo, E. Glover, and D. P. Wall, “Multi-modular AI approach to streamline autism diagnosis in young children,’’ Sci. Rep., vol. 10, Mar. 2020, Art. no. 5014. Accessed: Aug. 26, 2021. [Online]. Available:
  2. U.S. Food and Drug Administration, Press Release. (Jun. 2, 2021). FDA Authorizes Marketing of Diagnostic Aid for Autism Spectrum Disorder. Accessed: Aug. 26, 2021. [Online]. Available:
  3. Floreo, Company Website. Using Virtual Reality to Impact Individuals With Autism. Accessed: Aug. 26, 2021. [Online]. Available:
  4. J. P. McCleery et al., “Safety and feasibility of an immersive virtual reality intervention program for teaching police interaction skills to adolescents and adults with autism,’’ Autism Res., vol. 13, no. 8, pp. 1418–1424, Aug. 2020. Accessed: Aug. 26, 2021. [Online]. Available:
  5. V. Ravindran, M. Osgood, V. Sazawal, R. Solórzano, and S. Turnacioglu, “Virtual reality support for joint attention using the Floreo joint attention module: Usability and feasibility pilot study,’’ JMIR Pediatrics Parenting, vol. 2, no. 2, Jul./Dec. 2019, Art. no. e14429. Accessed: Aug. 26, 2021. [Online]. Available:
  6. B. Scassellati et al., “Improving social skills in children with ASD using a long-term, in-home social robot,’’ Sci. Robot., vol. 3, no. 21, Aug. 2018, Art. no. eaat7544.

Related Articles