ai-toolkit

[ai-toolkit]

Delft Toolkit v2 status

Since my fellowship last fall, I’ve had lots of other things going on, but this summer I’ve been working on a new version of the toolkit. This post is a brief status update for this summer, in anticipation of a full release this fall.

The primary feature of the new version is a replacement for Node Canvas (which was great, but a paid Unity asset) with an open source node system called xNode -- so the entire system will be open source and free. In addition, the entire system has been rewritten to simplify the architecture and clean up the code.

Here’s a video of the new system doing voice and object recognition.

(Note that I’m about to replace the raspi local object recognition system with something that’s much faster -- more on this below).

xNode

In addition to making the toolkit node system fully open source, using xNode allows me to customize the nodes to be optimized for the features we need. For example, the system is evolving towards a hybrid of behavior tree and data-flow models. As a bonus, I’ve hired the author of xNode (Thor Brigsted) to do a lot of the customization and feature additions, which is really accelerating the toolkit development. 

Here’s an example of the new node system.

As you can see, the new Action nodes support multiple actions within a single node, which will keep the complexity of the node graph lower. The nodes also have features to repeat the sequence of actions, as well allow execution of random actions from the sequence.

New Architecture

The new version of the toolkit simplifies the architecture significantly, eliminating the node.js server on the computer, and the bluetooth connection to the Arduino. Unity now communicates directly to the robot’s Raspi directly over OSC, and the Raspi forwards appropriate commands to the Arduino via a direct serial connection.

Faster Raspi Object Recognition

Among the things I’m working on for this next version is a new on-board Raspi visual object recognition system. As seen in the above video, the old system takes about 6 seconds to process an image taken with the pi camera (using standard TensorFlow and the Inception model). The new system provides multiple models (balancing speed with accuracy and size of dataset) and a Raspi native compilation of OpenCV. Speeds now range from less than a second to under 2 seconds. Special thanks to Nik Martelaro for all the research he’s done to optimize for the Raspi, and helping out with this.

Not only will this new system be much faster, but it will provide a simple interface for the user to see how different models behave. In the “recognize” action, you will select the model with a drop down in the action. This will allow people to easily see how different models perform (or not), and learn to make decisions about what I’m calling the MVD (Minimum Viable Data) for their project. Ultimately, I hope this system can allow for user trained models as well.

Up Next

Other things I’m working on supporting:

IBM Watson APIs for services such as speech to text, text to speech, chatbots, sentiment analysis, etc.

Unity’s Machine Learning Agents that do reinforcement learning.

A range of robot configurations. Especially one oriented more towards IoT contexts than mobility, where the device can express through through movement and light in addition to voice.

[ai-toolkit]

Update 5 - Voice and Object Recognition

It’s colder, the leaves have fallen, and this was the view today as I walked to TU Delft for my last day of working on the toolkit during this fellowship. More to come in 2018!

This update video shows the robot listening for voice commands, one of which is to use its camera to perform object recognition. You can also ask “Ding” (Dutch for “thing”) the robot to dance. 

Object Recognition

As mentioned in the previous update, the object recognition is being done entirely on the Raspberry Pi, using Google’s Tensorflow software and model based on the ImageNet database of 12000 images. It takes about six seconds to do the recognition.

Voice Recognition

The voice recognition is done using the Seeed ReSpeaker 2-Mic Hat to record an audio file. That file is then sent to Google’s speech API for processing. The API sends the text it identifies back to the Raspberry Pi, which sends it to over to Unity via the NodeJS Hub using OSC.

Github

The project is now on Github in rough form, no documentation right now. I’ll work on making a proper repo in the coming months.

[ai-toolkit]

An interesting platform that has some characteristics I’m interested in:

* Personality control * A finite state machine approach to programming behavior * A mix of text, voice, and animated movement * Enough mechanical articulation to express and show what it’s attending to

Could be interesting to experiment with.

[ai-toolkit]

Update 4 - Machine Learning Object Recognition

Success integrating machine learning object recognition!

The image above was taken by the Raspberry Pi, and using Google’s TensorFlow running directly on the RasPi, it recognized that there was a laptop in the scene. Based on finding a laptop, the system triggered the robot to move.

After a couple days of wrangling the RasPi, TensorFlow, TU Delft’s restrictive WiFi, OSC, and Python, the system can now tell the RasPi to take a picture and analyze it. The results are sent back to Unity in OSC, and if the Condition node matches one of the objects in the scene, the robot will be triggered to move.

The above image shows the behavior tree sequence, where one condition is waiting for “bottle” and the other is waiting for “laptop.” The designer can set for any match, and if desired, matches from among several options (say laptop or bottle for a single action).

This photo shows how it was able to recognize a water bottle, even though it was upside down and the top is cropped out. Not sure where the “lampshade” comes from, but the white thing in the background is in fact a bunch of lampshades put together to make a chandelier. 

RapidMix Integration

Last week, I was able to replace Wekinator in the system with the RapidMix ML library. This library is used for the realtime gesture training and recognition shown in the last update. RapidMix is still in beta, and I uncovered a couple bugs, but they’ve been very supportive, and I was able to work around the issues.

This is great, since it means I don’t have to have Wekinator running, which simplifies everything since the training/recognition is now integrated into my Node.js server.

Github Coming Soon

Over the next week, I’ll be cleaning up the code and posting it to Github. Stay tuned for more details.

Voice Recognition Next

The next feature to implement is voice recognition, using the Seeed Studio ReSpeaker on the RasPi.

[ai-toolkit]

Update 3 - Sensing, Machine Learning, and Responding

The toolkit now has the ability to sense and respond to the environment, as well as recognize patterns in the environment using machine learning. Explanations after the break.

Machine Learning - Training and Classifying

The toolkit can now take sensor data from the robot and feed it to a machine learning system to train on time based gestures. Once trained, it can classify those gestures and use those recognized gestures to trigger different behaviors.

In the video, I train the system for two different gestures, tilting the robot to the left for one, and tilting it to the right for the other. The robot uses the accelerometer in the Arduino 101 to sense this motion.

In the authoring environment, these gestures are associated with two different behaviors - the first behavior spins the robot around, and the other has the robot repeating a “turn-right, move forward” sequence four times, which makes it drive in a square pattern.

Sensing and Responding

The robot also has a proximity sensor on it, and the ability to pass that sensor data directly to the authoring system. In the authoring system, you can create behaviors that occur in response to the sensor data. In the example, I’ve created a two behaviors that are triggered depending on what the robot senses in front of it.

The first behavior moves the robot towards an object that’s in front of and near the robot. The second behavior turns away when the robot gets closer to the object. This means that the robot sits still until something is nearby. Then when it senses an object in front, it travels towards it until it gets close. Then it turns away and waits until something is in front again.

[ai-toolkit]

Prototypes,  Tools, and Techniques

I took the above image walking to work today, and cool fall day here in Delft put me in an explanatory mood. So it seems like a good time to discuss how I’m approaching this work.

Working Prototypes

This project investigates what a tool for designing AI systems could be. As such, I’m making prototypes, not finished systems. My current task is to create examples and explorations. Even though what I’m making is interactive and actually works, what you see is not the design of a completed tool. My approach is very much a strategy of “sketching” in software/hardware. This makes it possible to explore the potential affordances and interactions of tools by actually using them. 

Tools for Tool Making

Much of what you see in this project is based on off-the-shelf tools that I’m customizing to make my own tool. So for example, I’m using Unity3D as my primary environment for the user “authoring” part of the tool. And within Unity, I’m using a commercial add-on asset for Unity called NodeCanvas and its companion FlowCanvas. I’m also using the the Adafruit CurieBot.

As an example, the node graph above is from my prototype, but is the NodeCanvas interface and is not my UI design. NodeCanvas provides a rich system for creating Behavior Trees (as well as Finite State Machines) in a visual authoring environment. What is my design are the custom nodes and system behind it that integrates Unity with an iPad as a marionette control, a 3D representation of the thing, and a physical robot as a real example of the thing being designed. See this update post for a video of how it’s currently working.

I’m using off-the-shelf tools like NodeCanvas because I think a visual authoring environment similar to this is needed for an AI design tool. And also because it allow me to go fast and rapidly iterate.

Inspiration from Game Design

One of the things that’s becoming clear is that designing for AI is much more than the classic Machine Learning strategies of classification and regression.  Because we have to design the entire interaction for the end-user, including the specific behaviors of smart things in many situations, we need tools that support a broad palette of techniques and strategies.

I’m finding that some of the practices from the game design world are applicable, especially since many of the aspects of “smart” things are relatively dumb (e.g. Alexa lighting up when her name is spoken), and the shortcuts that games use work well.

Game designers use the term Non-Player Character (NPC) for agents within a game that are autonomous and are not controlled by the “player,” such as the zombies, animals and helpers one encounters and interacts with. This seems analogous to the networks of smart objects that we’re faced with designing for IoT and AI collaboration (but hopefully we can avoid the undead!).

I also like that in game design, the use of the term “AI” is used fairly loosely and is more about creating the impression of intelligence than an academic definition. I think interaction designers need to do the same. This is a separate discussion that I’ll address in the future, but I think the design of AI is really a process of creating a fiction of intelligence that honestly reveals the affordances and limits of complex and otherwise opaque systems.

“The design of AI is really a process of creating a fiction of intelligence that honestly reveals the affordances and limits of complex and otherwise opaque systems.”

In designing the behavior of NPCs, game designers have used Finite State Machines, but due to their limitations, they’ve moved on to Behavior Trees which allow for hierarchical definition of conditions for the sequences of behaviors (if this, then that and that, and then if this, then that, etc.) that compose seemingly intentional, autonomous activity.

More recently, game designers are using Utility AI which is derived from Utility Theory in economics that proposes that people do what has the highest utility value for them. In a game, the Utility AI scores potential behaviors depending on a range of factors, where the behavior with the highest utility is chosen. This approach is good at creating emergent behaviors, as well as enabling more “irrational” and unpredictable behavior, which makes games, and I would argue ecologies of AI systems, more interesting and challenging. 

What’s Next

I have the basic system implemented for designing with Behavior Trees, and the next step is to begin to integrate some machine learning capability. This will allow for combining the logic of behavior trees with the pattern recognition of machine learning.

[ai-toolkit]

Update 2 - System Working

Last update we saw the robot being controlled via bluetooth. In this Update 2 video you can see the robot is integrated into a growing toolkit system that allows a designer to create a sequence of actions that form a behavior. You use the visual authoring system to set up and refine your sequence (e.g. move forward, turn LEDs on, move backward, etc.), and simulate it on-screen in 3D. Then, when you are ready to try the behaviors out in hardware, the same system controls the robot.

The toolkit allows any number of behaviors to be created, triggered by any actions. In the video, I’m showing the laptop keyboard and an iPad used to start actions, and this is a way to Wizard-of-Oz how a system might react to a person (in the video, I pretend to give the robot a voice command).

The next step will be to integrate sensors on the robot so that the system can be triggered by real-world input. This will enable the designer to create a fully working prototype. After that, I’ll begin integrating a Raspberry Pi (single board computer) in the robot, and this will give it (hardware gods willing) the ability to “see” and “hear” so it can do object recognition and voice interactions.

[ai-toolkit]

Great tool for easily trying out simple machine learning. Makes me realize that I should incorporate several pre-made examples in my tool that allow people to instantly try out different applications and approaches to AI, and then be able to modify them.

[ai-toolkit]

First Days - Robot Working!

I’ve just arrived at TU Delft, and have begun work on my project. The first task was to get a simple wireless robot working. This will be the basis for the #AI “smart thing” that a person can design the behavior and #IxD of, using the designer’s AI authoring tool I’m prototyping.

In the video, you can see that I’m controlling the robot (Adafruit’s CurieBot) via Bluetooth on the phone -- for now, I’ve implemented forward/backward/turn L-R, set color, and wiggle nose.

The next step is to get Unity3D to talk to the robot, because the prototype visual authoring system will be in the Unity environment.  I’ll enable communication of Unity to the robot using NodeJS as a go-between:

Unity <-> OSC <-> NodeJS <-> Bluetooth <-> robot

Eventually, I’ll add a Raspberry Pi to the robot to enable vision and hearing.

[ai-toolkit]

Human-Agent Collaboration: Can an Agent be a Partner? Panel Rachel K. E. Bellamy, Sean Andrist, Timothy Bickmore, Elizabeth F. Churchill, Thomas Erickson CHI '17: ACM CHI Conference on Human Factors in Computing Systems Session: 

Panel Abstract 

Human-Agent Interaction has been much studied and discussed in the last two decades. We have two starting points for this panel. First we observe that interaction is not the same as collaboration. Collaboration involves mutual goal understanding, preemptive task co-management and shared progress tracking. Second, that much of the on-going discourse around human-agent interaction implies that agents can be trusted, collaborative partners. Our position is that while virtual and embodied agents have the potential to be work partners, for this goal to be achieved we need to better understand what partnership in collaboration involves. In this panel we ask: Can this potential for trusted collaboration be realized? If so, what will it take? This panel will bring together HCI experts who work on collaboration, virtual agent design and human-robot-interaction. Panelists will engage the audience through discussion of their shared and diverging visions, and through suggestions for opportunities and challenges for the future of human-agent collaboration.

[ai-toolkit]

Abstract

User interaction with intelligent systems need not be lim- ited to interaction where pre-trained software has intelligence “baked in.” End-user training, including interactive machine learning (IML) approaches, can enable users to create and customise systems themselves. We propose that the user ex- perience of these users is worth considering. Furthermore, the user experience of system developers—people who may train and configure both learning algorithms and their user interfaces—also deserves attention. We additionally propose that IML can improve user experiences by supporting user- centred design processes, and that there is a further role for user-centred design in improving interactive and classical ma- chine learning systems. We are developing this approach and embodying it through the design of a new User Innovation Toolkit, in the context of the European Commission-funded project RAPID-MIX.

[ai-toolkit]

Image of the RAFON visual state machine authoring system