Color Piano | Huskylens Playground with micro:bit EP02

[Color Piano]

Have you ever had a musical dream? Have you ever imagined playing the piano as gracefully as a pianist? Piano, the "King of Musical Instruments", interprets perfect sound effects and romantic feelings with fluent notes, bringing pure enjoyment to people. But for various reasons, maybe you have never studied piano or failed to own a piano.

Now with the HUSKYLENS sensor, we can make a color piano for ourselves and realize the musical dream. Let us use the colorful keys to play beautiful music.

Function Introduction:

This project uses the color recognition function of HuskyLens to recognize different color keys and play different notes, so that your "playing" is both beautiful and pleasant, with absolutely wonderful stage effects.

Materials:

Micro:bit     https://www.dfrobot.com/product-2125.html

IO Extender for micro:bit V2.0     https://www.dfrobot.com/product-1867.html

HUSKYLENS     https://www.dfrobot.com/product-1922.html

Knowledge Field:

Like other biological characteristics of the human body (fingerprint, iris, etc.), the face is inherent, unique, and not easily copied. It is the first kind of image that has been studied, and also the widely used one in the field of computer vision. This project is making use of the face recognition of HUSKYLENS.

1. What is color recognition?

What is color recognition? First of all, we must understand what color is. Color is a visual effect on light produced by our eyes, brain and our life experience. The light we see with the naked eye is caused by a very narrow wavelength range produced by electromagnetic waves. The electromagnetic waves of different wavelengths appear in different colors. Color recognition is based on the color attributes under different brightness to recognize and distinguish.

The principle of color recognition:

Color recognition is based on Lab color space, with dimension L representing brightness, a and b representing opposite dimensions of color, which is based on the CIE XYZ color space coordinates of nonlinear compression. We can regard these three parameters of Lab as the XYZ of the three-dimensional coordinate system. Compare the Lab parameters of the recognized and learned colors. When the two colors match within a certain error range, they are judged to be the same color.

Lab Coordinate System

n our usual use of color recognition, the hue and saturation in the color attributes of the same module are fixed, but the brightness will change due to the influence of the ambient brightness, so you must try to ensure that when using the HUSKYLENS color recognition function, the environment brightness during learning and recognition is consistent with that during actual work.

Application Scenario of Color Recognition:

1. Color recognition is now widely used in industries such as printing, coatings and textiles for color monitoring and calibration.

2. Help people with color weakness or visual impairment to recognize and enhance their understanding of color.

2. Demonstration of Color Recognition of HUSKYLENS

If you want a color piano to play smoothly, first you should let the HUSKYLENS sensor learn the color of the keys, and let it know the notes corresponding to each color. The color recognition function in the HUSKYLENS sensor uses the sensor's built-in algorithm to identify the ID of different colors and feed them back to the main control board by learning and recording different colors.

The HUSKYLENS sensor is set by default to learn, recognize and track only one color, but there is not only one key on the piano, so we need to set it to recognize multiple colors.

Operational Settings - Learn Multiple:

1. Dial the function button to the right or left until the word "Color Recognition" is displayed at the top of the screen.

2. Long press the function button to enter the parameter setting of the color recognition function.

3. Dial the function button until "Learn Multiple" is displayed, then short press the function button, and dial to the right to turn on the "Learn Multiple" switch, that is, progress bar turns blue and the square icon on the progress bar moves to the right. Then short press the function button to confirm this parameter.

4. Dial the function button to the left until "Save & Return" shows. And the screen prompts "Do you want to save the parameters?" Select "Yes" in default, now short-press the function button to save the parameters and return automatically.

In this way, the multiple learning function is set up.

Learning and Recognition

1. Detect Color

Point the icon "+" in the center of the HuskyLens screen to the target color block, and a white frame will appear on the screen, which selects the target color block automatically. Adjust the angle and distance of the HuskyLens to the color block so that the white frame can include the entire target color block as far as possible.

2. Color Learning

When detecting colors, press “learning button” to learn the first color, and release it to finish learning. Meanwhile, a message “Click again to continue! Click other buttons to finish” will be displayed. Please short press the "learning button" before the countdown ends if you want to learn other colors. If not, short press the "function button" before the countdown ends, or do not press any button to let the countdown ends. The color ID displayed is consistent with the order of learning, that is, ID will be displayed as “ID1”, “ID2”, “ID3”, and so on, and different colors correspond to different colors of frames.

3. Color Recognition

When encountering the same or similar color blocks, a color frame with an ID will be automatically displayed on the screen, and the size of the blue frame is the same as the size of the color blocks. Multiple colors can be recognized and tracked at the same time and different colors correspond to different colors of frames.

When multiple blocks of the same color appear, the separated color blocks cannot be recognized at the same time, and only one block can be recognized at one time.

Tips: Color recognition is greatly affected by ambient light. Sometimes HuskyLens may misidentify similar colors. Please try to keep the ambient light unchanged.

Project Practicing:

We will complete the task with two steps. First, we will learn to use the color recognition function of HuskyLens and output the recognized color ID. Then we can play their corresponding sound according to the output color ID, so that we can complete our color piano.

Task 1: Multiple Color Learning

At the very beginning, we need to allow the HuskyLens camera to recognize multiple colors, distinguish the differences in these colors, and be able to give feedback so that we can add notes later.

Task 2: Define Notes for Each Color

After being able to accurately identify each color, we can define a sound for each color, so that they can be played according to a certain rule, so that a color piano can be realized.

Task1: Multiple Color Recognition

1. Hardware Connection

HuskyLens sensor uses IIC interface, you need to pay attention to the wire sequence, and don't connect it wrongly or reversely.

2. Program Design

Here we need to let the HuskyLens sensor learn the color of each key and be able to output the color ID so that we can play the corresponding sound with the corresponding color.

STEP1: Learning and Recognition

Before designing the program, we need to let the HuskyLens sensor learn the color of each key. (Note that you need to open the multiple learning function first)

STEP2: Mind+ Software Setting

Open Mind+ (version 1.62 or above), switch to "Offline", click "Extension", click “micro:bit” under “Board”, click “HUSKYLENS AI Camera” under “Sensor”.

STEP3: Instruction Learning

Here are the instructions mainly used.

① Initialize only once between the beginning of the main program and looping executions. You can select I2C or Soft-serial, and no need to change I2C address. Please note that the “Output protocol”of your HuskyLens sensor should be set to be consistent with the program, otherwise, data cannot be read.

② You can switch to other algorithms freely, but please note that you can run only one algorithms at each time, and it takes some time to switch algorithms.

③ The main controller requests HuskyLens to store data in the “Result” once(stored in the memory variable of the main board, and once requested, it will refresh the data in the memory once), then the data can be obtained from the “Result”. The latest data can be got from the “Result”only when this module is called.

④ Check whether there is frame or arrow in the screen from the requested “Result”, including the learned(id>0) and unlearned, and returns 1 if there is one or more.

⑤ Check whether the IDx has been learned from the requested “Result”.

⑥ Check if the IDx requested from the “Result”is in the screen. The frame refers to the algorithm of the frame on screen, arrow refers to the algorithm of the arrow on screen. Select arrow when the current is only line-tracking algorithm, for others, choose frame.

⑦ Get the parameter of the IDx from the “result”. If this ID is not in screen or unlearned, return -1.

STEP4: Flowchart Analysis

2. The Sample Program

3. Operating Effect

When the ID of the corresponding color is recognized in the HUSKYLENS sensor, the corresponding number will be displayed on the micro:bit.

Task2: Define Notes for Each Color

1. Program Design

How to write a program? In Mind+, it comes with instructions for playing notes, which are divided into low, mid, and high notes, as well as various beats. We just need to add the corresponding note to the corresponding ID.

The program logic diagram is as follows:

2. Sample Program

Add the note module to the program completed in the previous step.

3. Operating Effect

Fix the HuskyLens sensor. When the key is not pressed, the color key must be outside the recognition range, and when the key is pressed, it should be in the recognition range. When we press the key, the corresponding note will be played according to the recognized color.

Project Summary:

In this lesson, we learned about the working principle of color recognition, and learned the color recognition function of the sensor by using the HuskyLens sensor.

Color recognition is a very important function in artificial intelligence visual recognition, and it has been widely used in the industry. Think about what other interesting functions can be achieved by color recognition?

Review:

1. Learned the principle of color recognition;

2. Learned the color recognition function of HuskyLens sensor and the operation method of multiple recognition.

Project Development:

After completing the color piano, we will find a problem that the number of keys is relatively small. If we want to increase the number of keys, as the color increases, there will be many keys with similar colors, which may cause misidentification. If there are more keys, the camera's recognition range will not cover them all. So, is there any way we can broaden the range of our color piano?

(Tips: We can use the AB button on the micro:bit to realize the upgrade and downgrade functions.)

Next Class: Smart Spotlights | Huskylens Playground with micro:bit EP03 >