Controlling Robots with Web Apps

Fortunately, controlling robots doesn't require speaking machine language. Yeti's leveraged technology developed for real-time web functionality like chat to control robots through a simple touchscreen web interface.

Static vs. Dynamic Web Content

Back when the Internet was first created, it was intended for showing static web pages, not dynamic web applications. As a result, the protocols that underlie the web are synchronous. Browsers send a request and wait for a response. Nothing else happens until the request is answered.

Of course, people don't like waiting, so workarounds were created to provide asynchronous communication, using techniques like AJAX, polling, streaming, and long polling. These techniques let web applications run in a single page, refreshing only the parts that need to be updated rather than reloading the entire page when the user clicks a button. But the data transfer and update are still initiated by an event in the browser.

As web users have become more demanding, the limitations inherent in HTTP have led to the development of websockets, which allow full duplex communication. Unlike other techniques that workaround the standard HTTP protocols, websockets are both efficient and compatible with multiple platforms. (Because the technology isn't supported everywhere, applications usually attempt to establish a websocket connection and then fall back to another technique if it isn't supported, or first establish a long polling connection, then try establishing a websockets connection in the background and switch over if it succeeds.)

What this means is that instead of the request/response protocol, where the requester sits waiting for a response, websockets use a publish/subscribe protocol, where the subscriber is notified whenever information is published on the other end. Subscribers just have to register their interest in the topic, and then information is sent as it becomes available. This allows more dynamic interactions and a better user experience for collaboration or game playing—data can be sent to the browser without the user explicitly asking for it.

Frameworks for Developing Dynamic Web Applications

Whenever communications are involved, it's best to use a tested framework that encapsulates the handshaking protocol mechanics and abstracts away the details, allowing developers to focus on the methods and processing of the communicated data.

In previous projects we’ve experimented with autobahn.ws, which runs on top of websockets and adds support for remote procedure calls, plus the publish-and-subscribe communication pattern. Using one of these frameworks is highly recommended as using raw websockets can become overwhelming.

There are several other websocket libraries available, socket.io and xSockets, but we found their integrations to be mostly javascript based and we’re excited by the number of supported languages offered by Autobahn. Using a simple tutorial we were able to quickly get websockets to control an LED screen connected to an Arduino using Autobahn in real time.

Yeti relies on Django’s open source framework as a key part of Yeti's development toolkit so we wanted to look into how we could integrate it further. Luckily Django’s third party community has made a big effort to provide real-time functionality. This has been a challenge since it was initially designed to work on the web server gateway interface (WSGI) protocol, which is synchronous. Several packages like django-websockets and django-realtime have been created to offer support for real-time behavior in Django.

The Real-Time Web and Robotics

Now, back to robots: Using these technologies, Yeti worked with Flextronics as a partner to build a proof-of-concept robotics system, a vending machine where touchscreens determined which products the robot would retrieve or return. This was just a demo, but it could have real-world applications in businesses that process shipments and handle returns.

Those real-world applications of real time web and real time robotics are unlimited. Any device with sensors can respond to instructions sent this way. The Internet of Things already lets homeowners control their thermostats remotely via a web interface.

Maybe web-controlled robots will find an entry-level job in a fast food restaurant, like many of us. Momentum Machines' hamburger-making robot adds custom toppings just before serving; the device’s next generation will custom-grind blends of meat before broiling to order.

Right now, building robots and controlling them is complicated, but technologists are working to streamline the development and integration process. The Defense Advanced Projects Research Agency (DARPA) has made controlling robots wirelessly part of its robotics challenge. There are many possible technologies for connecting and controlling robots, using websockets or alternate means like the Windows runtime. Robots can even be controlled through speech apps. Someday we may walk into our favorite fast food restaurant and tell the robot chef exactly how we want our burger cooked.

As software engineering techniques become more refined, controlling robots will become simpler—maybe as simple as child's play. Wonder Workshop's app already lets children as young as toddlers control a simple robot using a tablet and Bluetooth connection. With some luck, we'll be able to use simple real-time controls to keep humanity safe from robotic devastation.