Getting started with Misty II robot and its Python SDK

Contents

Overview

There are several methods to interact with Misty II robot as indicated on its official website. These are also summarized below.

  1. Command Center : It is a Graphical User Interface (GUI) with several commands to choose from and interact with Misty II. One would simply need to connect to Misty II robot using its IP address and once it is connected to the PC, the user can send any commands that are available in the Command Center such as to move it around, or move its limbs, etc. It also shows the live data from the robot sensor and its state, when enabled/subscribed. So, this comes handy when the user either wants to study some aspects of Misty’s sensors or simply wants to have a minimal interaction with the robot. However, if a user intends to do something more advanced such as make Misty II have a conversation or anything else that is not included in the Command Center, then there are at least two other options available. Those are, using the SDKs or APIs.
  2. SDKs: Currently, Misty Robotics has three official SDKs (JavaScript SDK, .NET SDK and Python SDK). The programs called Skills, written using the SDKs to have Misty do something useful are run directly on the robot. That is, there is no external processor needed to run the skills. (Note: this does not mean that one cannot use Misty II’s REST API when using the SDKs! The API can still be used to extend the capabilities that are not part of the SDKs). For the JavaScript and .NET SDK, there is excellent documentation available on Misty II website. However, Python SDK is fairly new (released on June 1st, 2021), it does not have extensive documentation yet. This post will attempt to introduce basic examples using the Python SDK.
    1. One difference to note between the JavaScript SDK/.NET SDK and Python SDK is that the skills written in Javascript and .NET can be uploaded on the Skill Runner and can be run on the robot from here. However, the Skill Runner does not have such support for the skills written in Python. So, those have to be run from elsewhere, such as from your PC.
  3. API: Finally, Misty II can also be programmed using the REST API. The programs that make use of the API are called “applications” (per Misty Robotics terminology). These applications are run on an external device and they communicate with Misty II using the HTTP requests and Websockets. APIs can be written in any programming language. Some examples using REST API for Misty II (using JavaScript) are provided here.
    1. API Explorer: This is a GUI that allows to send HTTPS requests. However, it currently appears to be under development.

Programming Misty II using Python SDK

In order to use Python SDK for Misty II, we need to install it first and set up the environment. The steps below will walk through the process of installing the SDK, followed by some basic examples.

Installation

We will use Anaconda to manage/install Python libraries that are needed and Visual Studio Code to write Python code. If not already downloaded, go ahead and download these two by following the instructions for Anaconda and Visual Studio Code installation on the respective websites. Next, follow the steps below to download and install necessary packages for programming Misty II with Python-SDK.

Note: The following instructions were tested on Windows 10 and Ubuntu 20.04. Please note that for Ubuntu, the conda commands are slightly different. For example, to activate a conda environment in Windows, we generally type $activate ENV_NAME…. For Ubuntu, the same command is $ conda activate ENV_NAME. So, just be careful of these differences when following the steps below.

STEPS

  1. Create a new folder in any directory on your computer.
  2. Next, we install the Python-SDK for Misty II. Before we do that, it is best to create a new virtual environment first (this is always a good practice as it allows to work on multiple projects using different Python versions or packages).
    1. Create a new virtual environment with a Python version >=3.8. See Option 3 on the official Anaconda guide or follow the steps below (Figure 1). Make sure to give any name to the new environment (that is, replace ENV_NAME with some meaningful name)
    2. After the new virtual environment is created, activate it ($activate ENV_NAME). (Again, for Ubuntu, you may have to do $conda activate ENV_NAME)
    3. Then install any desired libraries in this environment. For using Misty II Python SDK, we need requests, misty-sdk, websocket-client, yapf libraries. See the specific versions needed on this page. These can be installed using pip install. Example: pip install Misty-SDK to install misty-sdk
$ cd NEW_FOLDER
$ conda create - n ENV_NAME python=3.8
$ activate ENV_NAME
$ pip install ipykernel
$ python -m ipykernel install --user --name ENV_NAME --display-name "ENV_NAME"

Figure 1: Creating a virtual conda environment

After the minimal required libraries have been installed, we can do a check by simply typing conda list. It will show all the installed libraries in this virtual environment. Below is a snapshot from my setup.

Figure 2: The list of installed Python libraries

So far, we have created a virtual environment and installed necessary Python libraries. Next we will set up Visual Studio Code to use the virtual environment that was created above. (The Visual Studio Code is used as an IDE to write and run Python programs.)

An important thing to note here is that we want to use the conda virtual environment that was created above to run Python programs in Visual Studio Code.

Assuming that Visual Studio Code is installed. Follow a tutorial in this short video to point Visual Studio Code to the virtual environment that we created.

And that is it with the setup!

Test the installation

Next, let’s start coding Misty II using Python SDK. I followed the sample in the github repository to test the installation and the setup discussed above.

Assuming that Misty II is already connected to the device running the Visual Studio Code, add the IP address for Misty in the following sample code and then run the code. The following code should move Misty II’s arms to the position specified in the command MoveArms(X,Y). If Misty II responds by moving her arms, then we have succeeded with the setup. The command below, ‘misty.MoveArms ( )’ uses REST API. Therefore, we get a response code when the request is sent to Misty II.

from mistyPy.Robot import Robot

def start_skill():
    current_response = misty.MoveArms(50, 150)
    print(current_response)
    print(current_response.status_code)
    print(current_response.json())
    print(current_response.json()["result"])

if __name__ == "__main__":
    ipAddress = "IP ADDRESS"
    misty = Robot(ipAddress)
    start_skill()
Figure 2: Example

Interacting with Misty II

Now that we have tested that the installation is good, we can try some advanced stuff such as sending API requests or getting live data from MistyII

API requests

The examples below show a POST and a GET request.

from mistyPy.Robot import Robot
from mistyPy.Events import Events
import requests

if __name__ == "__main__":
    ipAddress = "IP_ADDRESS"
    misty = Robot(ipAddress)

    #### POST Method --- Display a TEXT on Misty's screen
    textToDisplay = {"Text": "Hello World"}
    text = json.dumps(textToDisplay, separators=(',', ':'))
    url = "http://" + ipAddress + "/api/text/display"
    r1 = requests.post(url, json=textToDisplay) # Note, POST request for Misty expects a JSON payload
    print(r1.status_code)
    #print(r1.json())
from mistyPy.Robot import Robot
from mistyPy.Events import Events
import requests

if __name__ == "__main__":
    ipAddress = "IP_ADDRESS"
    misty = Robot(ipAddress)

   #### GET Method -- get a specific audio file stored on Misty
    url = "http://" + ipAddress + "/api/audio?FileName=FILENAME.wav&Base64=True"
    r5 = requests.get(url, headers={'Content-Type': 'application/json'})
    print(r5.status_code)
    print(r5.json())
Getting live data from Misty II

We can also get live data from Misty II by registering to various events such as IMU, Time of Flight, etc. See example below.

from mistyPy.Robot import Robot
from mistyPy.Events import Events
from mistyPy.EventFilters import EventFilters

# The callback function must only accept one parameter, which will be the event message data
def callback_func(data):
    print(data)
    print("Printing message only ...")
    print(data["message"])
    print("Print first key:value pair in the message .......")
    Msg = data["message"]
    element  = list(Msg.items())[0]
    print(element)
   
if __name__ == "__main__":
    try:
        # First create the robot object
        ip_address = "IP ADRESS"
        misty = Robot(ip_address)

        # Register the event, which has a minimum of 2 parameters: the user defined name of the event, and the event type
      
        misty.RegisterEvent("battery_charge", Events.BatteryCharge,callback_function=callback_func, keep_alive= False)


        # Use the KeepAlive function to keep running the main python thread until all events are closed, or the script is killed due to an exception
        misty.KeepAlive()

    except Exception as ex:
        print(ex)
    finally:
        # Unregister events if they aren't all unregistered due to an error
        misty.UnregisterAllEvents()

########## Expected Output ###################
# {'eventName': '7464431873', 'message': {'chargePercent': 0.83, 'created': '2021-07-08T07:09:01.6610701Z', 'current': -0.411, 'healthPercent': 0.63, 'isCharging': False, 'sensorId': 'charge', 'state': 'Discharging', 'temperature': 0, 'trained': True, 'voltage': 7.772}}
# Printing emssage only ...
# {'chargePercent': 0.83, 'created': '2021-07-08T07:09:01.6610701Z', 'current': -0.411, 'healthPercent': 0.63, 'isCharging': False, 'sensorId': 'charge', 'state': 'Discharging', 'temperature': 0, 'trained': True, 'voltage': 7.772}
# Print first key:value pair in the message .......
# ('chargePercent', 0.83)
# Event connection has closed for event: battery_charge

Helpful terms/concepts

  1. Application Programming Interface (API): It is a software interface that allows two applications to talk to each other. As an example, when we search for a flight using services such as Google Flights, Kayak, Expedia, etc., we are essentially using an API. These sites are all different APIs that interact with websites for different airlines individually and bring desired data (available flights) back to us. In simple terms, an API is analogous to a waiter in a restaurant. Just like a waiter is an interface who allows us to interact with the kitchen (gives us the menu, takes the order to the kitchen and delivers the food), APIs in software allow two applications to communicate with each other. See [2].
  2. REST / RESTful: Representational State Transfer defines a set of constraints on an architecture as to how the communication must happen between the server and a client. It has at least 6 constraints as explained in [10, 11]
  3. REST/RESTful API: The APIs that meet the constraints of the REST architecture are called REST or RESTful APIs. See [11]
  4. API Endpoints: These are unique URLs, where the client must point its HTTP request at, to communicate with the server. See [3]
  5. HTTP: This is a protocol that defines a set of request methods to indicate the desired action to be performed for a given resource. The RESTful APIs use HTML requests to facilitate communication between the server and a client.
    1. HHTP request in Python: Python has a requests library that is popular. See [14,15]
  6. HTTP verbs: The intent of the communication/request is contained in HTTP verbs that make up the HTTP request. In other words, HTTP verbs tell the server what to do with the data identified in the URL. Example POST, GET, DELETE, etc. See 6,7,8,9.
    1. GET: Retrieve information from the server that is identified in the URL included in the request.
      1. Syntax of a GET request for Misty II:
    2. POST: Submit an entity to a specified resource, causing a change in the state on the server side.
      1. Syntax for a POST request for Misty II:
    3. PUT: Replaces the target resource with payload identified in the request.
      1. Syntax for a PUT request for Misty II:
  7. Uniform Resource Locator (URL)/ Uniform Resource Identifier (URI): URI includes both URL and URN. See [4]
  8. Webservices: There are few different definitions of webservices. According to one of them, webservices provide a way for two applications to communicate with each other just like the APIs. The difference is that the webservices must be available over the network, whereas not all APIs are required to be available over the network. Therefore, all APIs are webservices but not all web services are APIs. See [5], [13].
  9. Websockets vs HTTP request: Websocket allow communication/data flow between two applications (server and a client). The difference between a websocket and an API is that REST API (which uses HTTP) is based on a request/response protocol (a client sends a request and the server responds to the request), whereas websocket based communication is a two way (such as publisher/subscriber method). The server is always sending data. If a client has subscribed to this data then the server pushes data to the client whenever there is a change. The HTTP request is stateless whereas websocket communication is stateful. The real time applications typically use websockets, such as getting sensor data. See [16]
  10. SDK vs API: See [17]
  11. Events: Misty has several onboard sensors and services such as object and facial recognition. These sensors and services provide useful data. The skills/applications written for Misty II make use of this data to do something meaningful. Thus, we can say that skills are driven by events. The process of getting data from various events is called ‘registering for an event‘. The methods to register for an event depend on the how the skill is written. For example, whether it is written in JavaScript SDK/.NET SDK/ websockets/REST API and so on. Read more here.
    1. There are many Event Types. See here for the full list.
    2. Events have event names and messages they send
  12. Skills vs Application: In Misty Robotics terminology, Skills are programs that are able to run on a robot using its onboard software, whereas applications are programs that are run on external devices such as PC, raspberry PI and communicate with Misty II using the APIs (REST API to be specific).

References

  1. https://docs.mistyrobotics.com/misty-ii/get-started/meet-misty/ – Misty II official documentation
  2. https://www.mulesoft.com/resources/api/what-is-an-api – What is API?
  3. https://dev.socrata.com/docs/endpoints.html – API Endpoints
  4. https://rapidapi.com/blog/url-vs-url/ – URI/URL
  5. https://rapidapi.com/blog/api-vs-web-service/ – API vs web services
  6. https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods – HTTP verbs
  7. https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html – HTTP verbs
  8. https://www.c-sharpcorner.com/UploadFile/47fc0a/working-with-http-verbs/ – HTTP verbs
  9. https://dev.socrata.com/docs/verbs.html – HTTP verbs
  10. https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm – REST explained – original thesis
  11. https://blog.ndepend.com/rest-vs-restful/ – RESTful
  12. https://nordicapis.com/what-is-the-difference-between-an-api-and-an-sdk/ – SDK vs API
  13. https://www.tutorialspoint.com/webservices/what_are_web_services.htm – what are webservices?
  14. https://www.datacamp.com/community/tutorials/making-http-requests-in-python – How to define HTTP requests in Python?
  15. https://docs.python-requests.org/en/master/ – How to define HTTP requests in Python?
  16. https://blog.stanko.io/do-you-really-need-websockets-343aed40aa9b – websockets
  17. https://squareup.com/us/en/townsquare/sdk-vs-api – SDK vs API
  18. https://www.mistyrobotics.com/blog/what-can-the-misty-ii-platform-do/ – MISTY internal architecture
  19. Using Python with Anaconda and Visual Studio Code (Windows) – Anaconda and Visual Studio
  20. https://pypi.org/project/Misty-SDK/ – Misty -SDK installation