VR Robot Teleoperation through ROS Reality

ROS Reality creates a platform for globally distributed robotic labor, enabling seamless robotic teleoperation in Virtual Reality.

Teleoperation is still critical in robotics as there are many tasks that are outside the capabilities of autonomous systems. A world with ubiquitous robots will for the near term require ubiquitous operators. These operators, however, need not be collocated with the robot. Modern communications infrastructure allows for the control of robots from across the planet. 

Existing screen and joystick or screen and keyboard teleoperation interfaces have not changed much in the past 30 years. In the 90s and 2000s mechanical gantry systems that recorded exact human position showed the promise of recording human movement and mapping that to robotic control in complex environments, but these systems were expensive and cumbersome.

In recent years we have seen a proliferation of low cost, commercial, virtual reality and motion capture systems. ROS reality permits the use of these systems as a virtual control gantry for robotic operation. The virtual reality interface permits an intuitive interface for the user to engage a robot in complex tasks. The example mundane, yet complex, task of folding a T-shirt is the subject of this video showing ROSReality in action: https://youtu.be/VTRcfj7eeJY

With improvement to the ROSReality interface we envision a future of work where robotic labor can be controlled in VR "call centers" where operators interject themselves into robotic control tasks at points where automation is insufficient to achieve the goal. This would have a massive impact on labor markets as robotic labor could be outsourced from high cost markets to markets with lower labor costs, without the barriers of migration.

ROSReality will be a tool that permits the future of physical labor to be as global as the modern markets for intellectual labor. This will be especially important in regions with low migration and aging populations such as Japan. It also aids in AI big data development as robots can be completely virtual. We plan to use ROSreality to build learning from demonstration datasets.

We are among the first in the world to bridge ROS, the robot operating system, and the Unity Game Engine used for many Virtual Reality gaming environments. This enables an interface for users to control robots intuitively from great distances. The ability to fluidly control a robot from a distance will lead to the removal of the need for proximity for physical labor. Construction workers or fast food workers, for instance, could work from another city, state or country.

At heart we are a purely technical solution leveraging Virtual Reality, Robotics, Augmented Reality and in the future some Artificial Intelligence in Learning From Demonstration situations.

To deploy ROS reality in a STEAM gaming interface to be able to deploy virtual robot control games for large scale learning from demonstration data set development to be able to do machine learning on human driven teleoperation data. This will enable us to use AI to create smoother teleoperation interfaces, and be able to leverage the skills of the best gamers in VR.

Long term this would develop into a robotic teleoperation company. We see this being deployed as the last step in complex automation tasks, where some human in the loop is still required. Right now one barrier to automation fully taking over processes in environments like factory floors or fast food back of restaurant is the need for a human to perform a few small tasks that don't amount to full time work. This could usher in the full automation of these industries by giving human operators the chance to do those tasks across multiple locations.

We will likely reach out to factories looking for co-robotics and teleoperation solutions.

The solution is still pure research and is not implemented in trials yet.

If we develop a steam VR game for learning from demonstration that would engage a minimum of 10,000 people.

If this were to develop into a for profit venture an experienced CEO would likely be sought. We could use additional developers.

This would likely be an Uber style model where people operate robots as a part of the gig economy and companies hire them on a job by job basis.

Looking for funding to take the research and prototype to the next level. The ability to hire additional developers would be a huge help. Additional publicity from SOLVE could help raise awareness in grant applications.

ROSreality might just stay in research.