Project 36：LLM-Based Interactive Robot

Do you want an intelligent interactive robot that follows your commands? It can chat with you, be your companion, understand voice commands, control hardware, read environmental data, and play music. In this class, we'll build an exclusive interactive robot on the UNIHIKER K10 using the MCP protocol, unlocking the experience of hardware control via large language models!

Project Objectives

Knowledge Objectives

1.Understand MCP protocol and service types;

2.Use LLM to control IoT devices.

Practical Objective

Flash the Chatbot Xiaozhi firmware with MCP functionality into the UNIHIKER K10 to realize functions like controlling RGB lights via dialogue, reading sensor data, and playing music.

Materials List

Preparations

Hardware

Connect the UNIHIKER K10 to the computer using a USB Cable.

Software

Open Mind+, switch to“Upload Mode”, connect the UNIHIKER K10 as shown in the figure below, and load the UNIHIKER K10 library.

Hands-on Practice

Next, based on Project 35, we will re-flash the firmware for Chatbot Xiaozhi and adjust the background configuration to build an interactive robot.

Task 1 Configure and Test the Intelligent Assistant

The content of this task is similar to the corresponding part in Project 35, except for the version of the Chatbot Xiaozhi firmware to be flashed.

Task 2 Invoke MCP Settings

Then, we'll configure MCP service for Chatbot Xiaozhi to enhance its functions.

Task 3 Build an Interactive Robot

Last, we'll use two UNIHIKER K10 to expand AI capabilities through remote control and other means with MCP service, and then make an interactive robot.

Task 1 Configure and Test the Intelligent Assistant

Code

STEP 1: Download the flashing and firmware tools as detailed in the attachment and shown below.

For more details, please refer to the link: https://www.unihiker.com/wiki/K10/Playground/XiaozhiAI/.

STEP 2:  Flash Firmware on Windows

Press and hold the BOOT button on UNIHIKER K10's back. Then, open“Flash Download Tool”, fill in the relevant information as per the picture below, and finally click“OK”to confirm.

Then, in the newly appeared interface, follow the steps below to flash the Chatbot Xiaozhi firmware:

(1) In the first blank box, click the"…"button, select the downloaded and extracted firmware .bin file, and check the corresponding arrow.

(2) Fill the starting address with 0x00.

(3) Select the correct COM port and set the baud rate to 1152000.

(4)Then press "ERASE" button to erase the current firmware in UNIHIKER K10.

(5) After the erasure is completed, press "START" to flash Chatbot Xiaozhi firmware into UNIHIKER K10.

STEP 3: Flash Firmware on Mac/GNU Linux

Open ESP LAUNCHPAD at the URL https://igrr.github.io/esp-launchpad/, press and hold the BOOT button on the back of UNIHIKER K10, connect the mainboard to your Mac or PC and then release the button, after which you can click “Connect” and select the corresponding port. The operations are as follows:

After a successful connection, enter "DIY" page, select the .bin firmware you just downloaded, set "Flash Address" to 0x00, and finally click "Program" to flash the Xiaozhi firmware into the device. The specific steps are as follows:

STEP4 : Configure Wi-Fi and the language model

After flashing the firmware successfully, press“RST”button on UNIHIKER K10's back. Then, the board will enter Wi-Fi configuration mode.

Use a computer or phone to connect to Xiaozhi's hotspot. Then, configure the Wi-Fi SSID and password so Xiaozhi can connect to the Internet.

Note: The UNIHIKER K10 only supports 2.4GHz Wi-Fi.

Once Wi-Fi is connected, the UNIHIKER K10 will restart in 3 seconds and show a 6-digit device code on the screen.

Next, go to the Xiaozhi configuration site to set up the language model and speaker. Open the website by entering“xiaozhi.me”in browser, click Console, and log in to Xiaozhi account.

After entering the console, click“Add Device”, enter the 6-digit code on the UNIHIKER K10 screen, and click“Confirm”to deploy Xiaozhi on UNIHIKER K10.

Run the Code

After successful configuration, you can talk with Chatbot Xiaozhi and listen to its responses. Before use, wake it up by saying "Hello, Xiaozhi".

Task 2 Invoke MCP Settings

Code

STEP 1: Modify the MCP settings of the role

On the website https://xiaozhi.me/, click“Configure Role”option corresponding to your predefined role.

Scroll down the page, find “Official Services” under “MCP Settings”, and then select functions as needed.

You can also configure the knowledge base by yourself. Currently, Xiaozhi only supports creating one knowledge base & one document. Details are as follows:

Among them, the knowledge base name is shown in the knowledge base list during agent configuration. The knowledge base description explains its purpose, and the AI decides whether to call it based on the description. It follows a fixed format, as shown below:

This document contains relevant content such as "xxxxx". When you mention such content, I cannot respond directly; instead, I must invoke the tool to retrieve information in each conversation and then answer based on the query results.

The following is the content of "xxxxx" (Fill in the summary of the document here)

To make it easier to understand, an example is given below:

After creating the knowledge base, upload documents. Click“View”in the knowledge base list to enter the upload page. Then click “Create Document”  in the upper-right corner to select a document. Once uploaded, parsing starts automatically. Click the adjacent button to refresh progress. When finished, the status changes to Parsing Completed. Details are as follows:

At this point, the knowledge base configuration is complete. Click "Save" at the page bottom and restart the UNIHIKER K10.

Note: Chatbot Xiaozhi's firmware differs from that for graphical programming on the UNIHIKER K10. If you want to use the UNIHIKER K10 for further programming, use Mind+ to restore it to factory settings, as shown below.

Run the Code

After the device is added, you can start using the interactive robot (you need to say "Hello, Xiao Zhi" to wake it up before use).

(1) When you issue "turn on the light" command, the RGB light turns on.

(2) When you issue "play music" command, music plays and then pauses automatically.

(3) When you request to "identify the content in the camera frame", the camera will turn on and introduce the content.

(4) When you request to "tell news/jokes" and other commands, it will output the corresponding content.

(5) You can ask questions related to the knowledge base you imported.

Task 3 Build an Interactive Robot

Control Schematic Diagram

In this task, two UNIHIKER K10 devices are used. One is Chatbot Xiaozhi AI voice terminal for user dialogue and decision-making, and the other is environmental execution terminal to detect ambient temperature and light intensity.

For simplicity, the UNIHIKER K10 with Xiaozhi firmware is defined as UNIHIKER K10 - A, and the one for program design is named UNIHIKER K10 - B. Refer to the figure below for the specific control logic.

Code

After clarifying the control logic of the two UNIHIKER K10 boards, we begin to write programs for UNIHIKER K10-B acting as the execution terminal.

After opening Mind+, click“Extensions”. In“Module”, search for“Xiaozhi”, then locate and load“Xiaozhi MCP Client”, as shown below.

Note: Select version 0.0.2 for Xiaozhi MCP Client.

Next, we'll develop relevant programs using Chatbot Xiaozhi perceiving indoor temperature and humidity and judging environmental comfort as an example. To implement this function, define an environment detection MCP tool on the execution terminal UNIHIKER K10-B. Then, Chatbot Xiaozhi firmware terminal (K10-A) can call the tool via MCP to get real-time data.

STEP 1: Initialize the WiFi and connection endpoint of UNIHIKER K10 Xiaozhi MCP Client.

The initialization of WiFi and connection endpoints requires two blocks from the “Xiaozhi MCP Client”: "Initialize WiFi SSID (ssid) Password (Passwaord)" and "Initialize MCP Connection Endpoint ". The specific locations are shown below:

The WiFi network should be the same as that of UNIHIKER K10-A and the computer. The MCP connection endpoint should match the MCP access address of UNIHIKER K10-A, which is on the Configure Role Interface of the corresponding agent on UNIHIKER K10-A, as shown in the figure below.

STEP 2: Register the tool and provide its description.

After connection success, use “Register Tool (my_tool) Description” block from“Xiaozhi MCP Client”.Meanwhile, to enable Chatbot Xiaozhi-equipped UNIHIKER K10 to learn about the functions of this tool, a tool description shall be added. The corresponding block position is as follows:

Taking the scenario where Chatbot Xiaozhi senses indoor temperature and humidity and judges environmental comfort as an example, the corresponding program is as follows.

Note: In MCP, tool names and parameters should be named clearly for the large model to understand their functions. Avoid abbreviations and provide a comment to explain the tool's function and usage scenarios.

STEP 3: Set the functions of the corresponding tools.

Next, define the specific execution function of get_environment. Use "When MCP receives tool call (my_tool)" block. When Chatbot Xiaozhi firmware terminal (UNIHIKER K10-A) needs environmental information, it sends a request to K10-B via MCP. This block indicates that the subsequent program will execute upon receiving a request.

Meanwhile, when reading the temperature and humidity data detected by the on-board sensor on the execution-terminal UNIHIKER K10 and returning it to the Chatbot Xiaozhi-enabled UNIHIKER K10, use“Return MCP result (result) =(status)”block. The specific positions of the two blocks are as follows.

The corresponding code is as follows:

STEP4: Define the Fan Control MCP Tool

Next, based on the ambient temperature and humidity readings, we'll program Chatbot Xiaozhi to control the room fan's on/off for comfortable room temperature and air circulation. To implement this, connect a DC fan to the execution terminal (UNIHIKER K10-B), as shown below.

First, set P0 connected to the fan to output mode. Then define a boolean parameter "on" related to the fan status and add a specific description for it. This requires using the block "Add Parameter To (my_tool) Name (state) Title () Type (string) Desc()”block from“Xiaozhi MCP Client”. The corresponding position is shown below.

Next, define an MCP tool named set_fan. Enter description: Controls fan on/off for cooling and ventilation to adjust indoor environment.

During interaction with users, if the user wants to control the fan (e.g., saying "It feels hot"), Chatbot Xiaozhi can activate the fan by sending the on parameter.

First, obtain the value of the on parameter sent by Xiaozhi. This requires using the block“Get MCP boolean param (state)” from“Xiaozhi MCP Client”. The corresponding block position is shown below.

If the value is True, set the PWM output of Pin P0 to 1023 to turn on the fan; if the value is False, set the output to 0 to turn off the fan.

Through the on parameter, we have built a complete connection bridge: User Voice→Xiaozhi Comprehension→Parameter Conversion→UNIHIKER Execution.The corresponding code is as follows:

STEP5: Define the Light Control MCP Tool

Next, we'll define the light control MCP tool. In this task, light needs adjustment for different scenarios like studying, resting and nighttime, not just simple on and off. Thus, we've designed a multi-functional light control tool named set_light_environment for Chatbot Xiaozhi.

With this tool, Xiaozhi can automatically adjust lighting according to user needs or environmental changes, always maintaining a cozy indoor atmosphere.

To achieve more flexible control, multiple parameters are added to this tool. The parameter specifications of the light control tool are as follows:

Next, register the light control tool and name it set_light_environment.

Enter the tool description: Adjust indoor lighting for current scenarios. Supported modes: auto-adjust brightness based on light; study-All LEDs on, warm white, medium brightness; relax-Blue lighting, all three LEDs lit; night-One pale yellow LED at low brightness; off-Turn off all lights; manual-Customize LED quantity (0, 1, 2; -1 for all), brightness (0-9) and RGB color.

The corresponding code is as follows:

After defining light control MCP tool, write its execution logic. When Xiaozhi decides to "adjust the lighting", the execution-end UNIHIKER K10 takes action. The UNIHIKER K10 running Chatbot Xiaozhi decides on light adjustment and the mode, and the execution-end UNIHIKER K10 adjusts the lights accordingly.

We use the onboard ambient light sensor of the execution-end UNIHIKER K10 to detect indoor ambient light intensity. UNIHIKER K10 with Xiaozhi controls different operating states of the execution-end device by configuring distinct values for the mode parameter.

In the code, lighting performance is determined by identifying the specific value of the mode parameter, with each mode corresponding to an independent daily living scenario.

When the mode is set to study, the light is configured as a warm white light with medium brightness for study mode;

When the mode is set to relax, the light is configured as a blue ambient light with low brightness for relaxation mode;

When the mode is set to night, the light is configured as an illuminating light with the lowest nighttime brightness for night mode.

The corresponding program is as follows:

When mode="off", turn off all lights.When mode="manual", it indicates manual mode, which allows precise control of the number, color and brightness of the LED beads according to user instructions.The corresponding code is as follows:

The complete code for reading is as follows:

Run the Code

Click the“Upload”button and wait for upload completion. Return to the Chatbot Xiaozhi role configuration interface, click "Refresh" to view the newly registered tool. Then ask Xiaozhi relevant questions to remotely obtain real- time data.

Note: For MCP-related knowledge points, refer to the Knowledge Base.

Set the smart butler role

To make Xiaozhi Robot-equipped UNIHIKER K10 better fit the role of a proactive, self-judging and responsive smart butler, we'll compile the corresponding role description.

Please copy the relevant content from file “Role Introduction.txt” to the corresponding device in Xiaozhi Console.The specific file location is as follows.

The specific paste position is as follows:

After clicking Save, restart the UNIHIKER K10-A, and you will get an intelligent chatbot with the corresponding role.

Code Review

Category	Blocks	Function
		Configure the device's wireless network parameters to establish the foundation for network communication.
		Set the MCP communication access address to build a connection channel between the device and the AI service.
		Maintain a persistent MCP connection to avoid communication interruptions.
		Register custom tools with the AI service, and define the tool names and functions clearly for the AI to recognize and invoke.
		Configure name, description, data type and detailed explanation for parameters of custom MCP tools, clarify tool parameter specifications to facilitate AI recognition and correct parameter transmission and invocation.
		Trigger the execution module; when the AI initiates a call for a specified tool, the corresponding functional program will run.
		When the AI initiates a tool call via MCP, read and extract the text string data corresponding to the state parameter.
		Parse the state parameter in MCP communication, obtain the corresponding numeric data, and use it for program logic like data calculation and numerical judgment.
		Extract the state parameter from MCP, identify and return its switch on-off/boolean state to implement switch control functions like device start and stop.
		Send the tool execution results back to the AI service to complete the instruction feedback.

Knowledge Base

Next, let's learn and summarize the hardware knowledge used in this lesson.

MCP Protocol

1.Introduction to MCP Protocol

MCP, short for Model Context Protocol, is a powerful technical solution. As a "universal adapter" in the AI field, it allows the Xiaozhi voice assistant to break through traditional function limitations and connect seamlessly with local MCP services and large language models. With MCP support, Xiaozhi is not just for casual chatting and entertainment. It can remotely control devices, perform complex calculations, process emails automatically, and search for knowledge quickly and accurately. For example, when you ask about the weather, Xiaozhi can use MCP to call the weather API and provide accurate information instantly, enhancing daily life and work efficiency.

2.MCP Functions of Chatbot Xiaozhi AI

The Xiaozhi MCP Tool, based on MCP (Model Context Protocol), is a function expansion interface for Xiaozhi AI, an open - source voice interaction robot. It enables Xiaozhi AI to call external services and tools via voice commands, expanding the robot's capabilities. Without MCP, the AI can only chat and answer questions. With MCP, it can control devices (lights, vehicles, robots), call programs (calculation, data query), and execute tasks (automated processes).

3.Types of MCP Services

MCP's core services focus on the standardized interaction between models and the external environment, mainly offering three core services: tool, resource, and prompt services. Here's an introduction to tool services:

4.Advantages of the MCP Protocol

Unlike traditional protocols for simple data transmission, the MCP Protocol's core advantage is multi-functional collaboration. It enables fast data transfer, schedules hardware computing power, parses device commands, and manages AI model operation.

For the UNIHIKER K10, MCP is the underlying core for its AI functions. The voice interaction of “Xiaohang&Xiaozhi”on UNIHIKER, like triggering voice playback by saying“Xiaozhi, read an ancient poem”, is driven by MCP. It parses users' voice commands, schedules the on-board speech recognition AI model, converts instructions into hardware actions, and completes speech synthesis and playback, without users needing to understand low-level logic.

In practice, MCP lowers the barrier to AI and hardware development. Mind+ graphical programming and MicroPython programming for UNIHIKER K10 use MCP for low-level encapsulation. Users can use MCP to invoke UNIHIKER’s functions such as screen display, data collection, and speech synthesis by dragging graphical blocks or writing basic Python code, without manually developing complex AI model inference code or hardware drivers.

In short, MCP is a built-in steward for UNIHIKER K10. It unifies communication rules among hardware, AI models and software, coordinates resources, parses commands and transfers data, which simplifies complex AI-hardware operations and makes them intuitive. It solves the problem of running AI models on small embedded devices, enabling users to access AI + hardware functions easily. It truly reflects the idea that technology should serve usage.

Challenge Task

Combining the MCP protocol functions of Xiaozhi Robot and the onboard hardware of UNIHIKER K10, design an intelligent scene interaction task: when a designated voice command is issued, Xiaozhi Robot will perform a series of operations in sequence, including turning on the RGB light with a specified color and automatically playing a piece of music.