Wonder how ancient pictographs combine with modern technology? Pose before the camera, the device will recognize, match with a pictograph, and unveil its mystery for culture-tech interaction. In this lesson, we'll build an interactive human pictograph device on UNIHIKER K10 using YOLO pose recognition for a fun cross-ancient-modern human-computer exploration.
Project Objectives
Knowledge Objectives
1.Understand human key points;
2.Use K10 network image streaming;
3.Use YOLO for human keypoints.
Practical Objective
Create an interactive human pictograph device on UNIHIKER K10. When a person poses, the UNIHIKER K10 uses the camera to recognize poses, matches the most similar pictograph to the action, shows the character image on the screen, and introduces its meaning.
Materials List
Preparations
Hardware
Connect the UNIHIKER K10 to the computer using a USB Cable.
Software
Open Mind+, switch to“Upload Mode”, connect the UNIHIKER K10 as shown in the figure below, and load the UNIHIKER K10 library.
Hands-on Practice
Next, we'll use UNIHIKER K10's human pose recognition function to capture and recognize human poses in real-time. Then, we'll display the most similar oracle bone script image and relevant introduction on the screen to finish the production of the interactive human pictograph device.
Task 1: Network Configuration & Network Camera Startup
First, we'll connect the UNIHIKER K10 to the network, enable the network camera function, and establish a visual communication channel with the computer to transmit video data.
Task 2: Visual Detection and Data Upload
Then, use UNIHIKER K10's camera to capture human actions in real-time and transmit to the computer. Also, connect UNIHIKER K10 to the MQTT platform, and the computer will upload detection results to the SIoT platform.
Task 3: Production of the Interactive Human Pictograph Device
Last, UNIHIKER K10 will remotely obtain the inference results from SIoT, and display the most similar oracle bone script image and its relevant introduction on the screen according to different human actions.
Task 1: Network Configuration & Network Camera Startup
Code
Network Configuration
First, retrieve and add the Wi-Fi module from the Module in the "Extensions". The specific steps are as follows:
Next, we can complete the core code follow the main logic:connect to the network→wait for successful connection→output IP address, as shown below:
Note: Connect UNIHIKER K10 and computer to the same Wi-Fi network.
Network Camera Startup
After connecting the UNIHIKER K10 to the network, search for and add the “K10webcam”from "Module" in extensions. The specific steps are as follows:
After loading, follow the main logic to display the UNIHIKER K10 camera and enable the network camera to transmit the feed to the LAN. Then, any LAN computer can access it at any time and use existing computer vision libraries for image recognition.
To enable the network camera, use“Webcam (open)”block from UNIHIKER K10 Webcam, as shown below.
The complete reading code is shown below:
Run the Code
Click the“Upload”button, wait for upload completion, then open the Serial Monitor. You'll see the IP address of the UNIHIKER K10, as shown below.
Open the browser and enter IP/stream to view the camera feed. For example, enter 10.119.123.130/stream (the IP in the above figure) in the browser. The corresponding screen is as follows:
Task 2: Visual Detection and Data Upload
Code
Next, connect the UNIHIKER K10 to the MQTT platform. Create a SIoT topic to store results, and use a computer for facial keypoint detection to analyze expressions based on these keypoints. Then send the detection results to SIoT for the UNIHIKER K10 to do subsequent processing.
Computer Environment Preparation
On the computer side, download the Windows version of SIoT V2 and extract the SIoT_V2_Win.zip file.
After extraction, double-click start SIoT.bat to launch the new version of SIoT. Once launched, a small black command window pops up to start the server. Don't close it while the project is running, and note your computer's IP address.
Next, open the Mind+ software to complete the MQTT connection.
First, search for and add the“MQTT”from the“Module” in Extensions. The specific steps are as follows:
Then complete the code in this sequence: Initialize parameter settings→Initiate connection→Judge connection status & give prompt. Reference code is as follows:
Click to fill in the parametersClick to fill in the parametersEnter the local IP addressocal IP addressusername：siotpassword：dfrobotEnter the local IP addressocal IP
Combine Task 1 code, omit prompt section after successful connection, and the complete reading code is as follows:
Next, to recognize and process human pose information, install the Python dependent libraries for this project. The specific steps are as follows for Mind+ v1.8（you also can use your own Python IDE）:
STEP 1: Open a new Mind+ window, select and enter“Code”under “Python”.
STEP 2: Click“Library Management”to open the library installation page. In “PIP Mode”, run the following commands in sequence to install six libraries, such as the mediapipe and ultralytics libraries.
Python Code
STEP 1: Create a Topic
In the computer's web browser, enter local IP address:8080.For example, enter 192.168.35.1:8080 in the browser, as shown below:
STEP 2: Enter the username "siot" and password "dfrobot" to log in to the SIoT IoT platform.
STEP 3: Create a new topic "abs" for data transmission.
STEP4: Write code for human pose detection. It mainly detects human key points, calculates angles, and sends detection info to the SIoT IoT platform.
First, create a new Python file named "visiondetect" in”File in the project” in Mind+'s right sidebar.
Next, use the YOLOv8 model to detect human key points, and make judgments based on the human body tilt angle (bounded by 15°and 45°) and arm angle (whether > 170°) in the image. Then, publish the commands to the abs topic on the SIoT platform. The corresponding code is in the attached file visiondetect.py. Copy it into the visiondetect.py file created in Mind+.
Replace it with the IP address of your UNIHIKER K10.Replace it with the IP address of your UNIHIKER K10.
Run the Code
Click button and wait for loading to finish. Ensure your whole body is visible in UNIHIKER K10 camera window. The program will detect your body pose. When you perform different actions in front of the camera, the action recognition results will be shown on the video screen in real-time and updated synchronously in the terminal window, as shown below.
Task 3: Production of the Interactive Human Pictograph Device
In this task, based on human pose info from the SIoT platform, we'll show the oracle bone script image for the pose and its relevant intro.
In the image above, from left to right are the characters: Da (Great), Li (Standing), Lao (Elderly), Fu (Crouching).
Subscribe to the corresponding SIoT topic in the main program of the code for Task 2, as shown below:
Next, use"When (“topic/a”)received (MQTT message)" block in MQTT to enable the smart terminal to process control blocks from "siot/abs", as shown below.
Finally, use "If ()Then ()" block to make judgments and execute corresponding functions (display recognition results and introductions of corresponding oracle bone scripts via cached images) to complete the production of the interactive device. The specific logic is as follows:
The corresponding code is as follows:
The complete code for reading is as follows:
Run the Code
Click the“Upload”button and wait for the upload to finish. Then strike different human poses in front of UNIHIKER K10 camera and observe the output on the UNIHIKER K10 screen.
Knowledge Base
Next, let's learn and summarize the hardware knowledge used in this lesson.
Human Keypoints
The human keypoints model in computer vision detects and locates specific key body parts. It extracts body structure information from pixel data for quantitative analysis of body movements and postures. By analyzing keypoint positions and relationships, it helps understand human posture and movement information.
These models serve as the foundation for:
- - Abnormal behavior detection: Analyze dynamic changes of key points in human posture to identify dangerous behaviors, then trigger an alarm.
- - Service robot interaction: Robots understand instructionsby recognizing key information such as human gestures and standing postures.
- - Character animation generation: By capturing the key point movement data of actors, quickly generate the pose frames of animated characters, reducing the cost of manual modeling.
Human Action Detection Method
Human motion detection identifies and analyzes human postures, movement trajectories, and behavioral patterns via technical means. Based on technical principles and dependent devices, it can be divided into:
(1) Computer vision-based (non-contact) detection;
(2) Sensor-based (contact/near-field) detection,;
(3) Hybrid (multimodal fusion) detection.
In daily life, human action detection has wide applications. For example:
- - Security and surveillance: Track pedestrian movement trajectories to optimize layout or police deployment.
- - Healthcare and rehabilitation: Enable disabled people to control prosthetics and wheelchairs via EMG sensors or pose estimation using gestures.
- - Industrial and production safety: Monitor whether workers follow standard operating procedures to prevent safety incidents.
Challenge Task
Based on the existing device, add recognition function to match pictographs "Zou (Walk)" and "Zuo (Sit)", and adjust angle judgment parameters for human pose detection. When corresponding poses are made, the UNIHIKER K10 screen will accurately show oracle bone script images of "Zou (Walk)" and "Zuo (Sit)" and briefly introduce their meanings.
project34.zip 
