Serverless compute for AI development in ROS environments

Robotics based in ROS has been a popular choice for AI development. The Robot Operating System (ROS) is a set of software libraries and tools for building robot applications. From drivers to state-of-the-art algorithms, and with powerful developer tools, ROS has what you need for your next robotics project. And it’s all open source. Here you will find the official documentation on ROS 2, the LTS version of ROS that you will use.

Despite the popularity of ROS2 for AI development, there are still challenges in deploying ROS applications in the cloud. The main challenge is the complexity of the ROS2 environment and the need for specialized hardware to run ROS2 applications such as the need to have a Virtual Desktop or a GPU physical machine. In this project you will be designing a serverless ROS2 environment that can be used for AI development in robotics without the need to have specialized hardware, taking advatntage of the web browser technology (Web2, WebAssembly, WebRTC, Websockets, … ) and the facilities provided by Hugging Face Hub and other cloud providers.

Management plane

Every application requires centralized management and this will be provided by Django framework. The Django framework is a leading high-level Python web framework and below we have included the architecture of JDE Robotics that has implemented a Django based management plane for a similar application.

The management plane will be responsible for:

  1. Provisioning the ROS environment in the cloud or locally via docker containers.
  2. Provisioning the JupyterHub environment for AI development in the cloud or locally via docker containers. JupyterHub will be configured with ROS and other kernels to allow developers to design models and consume them in Jupyter notebooks.

Essentially the JDE Robotics architecture uses the Django framework for everything that needs serving via the web browser but we will design an architecture that Django is used for the management of the application and the SaaS tenant and delegate the execution of the ROS2 environment in the Jupyter Hub. As such our architecture will roll in Robostack taking full advantage of the existing user base and at the same time allowing the user to run ROS2 applications in a local setup.

Your task is to outline the minimal set of changes that need to be made to the JDE Robotics architecture to make the JupyterHub environment as the execution target.

Real-time perception pipelines

In most ROS applications we need to author perception pipelines such as object detection. Latency is of parammount importance. We will use this repo to seamlessly connect GStreamer and ROS2.

GStreamer is an open source pipeline-based cross-platform extensible multimedia framework and has been adopted by major players in mission-critical use cases that need to process video and other ream-time streams. It also underpins NVIDIA’s DeepStream implementation. To know more please watch this GStreamer overview. This additional hands-on video addresses NVIDIA pipeline elements.

You will showcase both gst_bridge and gst_pipeline for a video input and Yolo v8 object detection output and you will use existing or other WebRTC facilities to stream the video output with the superposed bounding boxes to the JupyterHub environment.

Back to top