Getting Started

Create a new conda environment for robomaster

conda create -n robomaster python=3.8
pip install robomaster dora-rs==0.3.3

Create a new conda environment for idefics2. This requirements file suppose that your using cu122.

conda create -n idefics2 python=3.10
conda activate idefics2
pip install -r requirements.txt

Robomaster Jailbreak

Installation of the Robomaster S1 Hack

This guide is an updated version of the original Robomaster S1 SDK Hack Guide and is intended for use on a Windows 11 system.

Prerequisites

Before you get started, you'll need the following:

• Robomaster S1 (do not update it to the latest version, as it may block the hack).
• Robomaster App.
• Android SDK Platform-Tools. Simply unzip it and keep the path handy.
• A micro USB cable. If this guide doesn't work, there might be an issue with the cable, and you may need to replace it with one that supports data transfer.

Instructions

1. Start the Robomaster App and connect the Robomaster S1 using one of the two options provided (via router or via Wi-Fi).

2. While connected, use a micro USB cable to connect the robot to the computer's USB port. You should hear a beep sound, similar to when you connect any device. (Please note that no other Android device should be connected via USB during this process).

3. In the Lab section of the app, create a new Python application and paste the following code:

   def root_me(module):
       __import__ = rm_define.__dict__['__builtins__']['__import__']
       return __import__(module, globals(), locals(), [], 0)
   
   builtins = root_me('builtins')
   subprocess = root_me('subprocess')
   proc = subprocess.Popen('/system/bin/adb_en.sh', shell=True, executable='/system/bin/sh', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
   

4. Run the code; there should be no errors, and the console should display Execution Complete

5. Without closing the app, navigate to the folder containing the Android SDK Platform-Tools and open a terminal inside it.

6. Run the ADP command .\adb.exe devices . If everything is working correctly, you should see output similar to this:

7. Execute the upload.sh script located in the folder s1_SDK.

8. Once everything has been executed, restart the S1 by turning it off and then back on. While it's booting up, you should hear two chimes instead of the usual single chime, indicating that the hack has been successful.

Robomaster Connection

Make sure to be connected using the wifi hotspot of the robomaster which is the most stable one.

The default password for the hotpsot is: 12341234

You might need to have a second wifi card if you want to be able to run the demo with internet on.

Post-Installation test

Please try running idefics2 with:

conda activate idefics2
python tests/test_idefics2.py

Please try running robomaster with:

conda activate robomaster
python tests/test_robomaster.py

Running the demo

export HF_TOKEN=<TOKEN>
conda activate idefics2
# This requires dora==0.3.3, update with:
# cargo install dora-cli
dora up
dora start graphs/dataflow_robot_vlm.yml --attach --hot-reload

Current way to interact is by press up arrow key on laptop to record a message and send to the VLM

Running the demo without robot

export HF_TOKEN=<TOKEN>
conda activate idefics2
# This requires dora==0.3.3, update with:
# cargo install dora-cli
dora up
dora start graphs/dataflow_vlm_basic.yml --attach --hot-reload

Current way to interact is by press up arrow key on laptop to record a message and send to the VLM

Kill process in case of failure

Due to a Python GIL issue, we currently meed to kill processes manually. You can use the following command to do so:

pkill -f 'import dora;'

LICENSE

While the source of this library is licensed under Apache-2.0, the usage of the Text to Speech(TTS) SystemEngine is licensed under Mozilla Public License 2.0 and GNU Lesser General Public License (LGPL) version 3.0.

Feel free to remove the TTS SystemEngine.