Project 1: Numbered Musical Notation of Color

Is music related to color?

Many musicians compare music with color. The famous American musician Maryon once said: "Sound is audible color, color is visible music". The use of different timbres in music works is very similar to the use of different colors in art works. Both timbre and color can give people feelings like clear, bright, warm or dim.

Although there is no scientific theory to prove the correlation between music and color, we can make fun interactive devices between music and color. The simplest thing is to connect the seven tones in the scale with seven colors so that each color represents a scale. Let’s make a colored score, through which the wonderful music generated!

Function Description:

This project uses color recognition function of HuskyLens to recognize color blocks of different colors. By using Maqueen Plus to play different scales, your music score is not only good-looking but also pleasant to listen, which makes an absolutely wonderful audio-visual effect.

Materials Checklist:

Knowledge Extension:

In today's society, automated production has become the development trend. As the eyes of "robots", machine vision is particularly significant. As one of the most important technical directions, color recognition has undergone many generations of technological upgrading. This project is based on the color recognition function of HuskyLens sensor.

I.What is color identification?

What is color identification? First we need to know what color is.

Color is a visual effect on light generated by eyes, brain and our life experience. Light we see with our naked eyes is generated by electromagnetic waves with very narrow wavelength range. Electromagnetic waves with different wavelengths show different colors.

Color recognition is based on color attributes under different lightness to identify and distinguish.

II. The Working Principle of Color Recognition

Color recognition is based on Lab color space, with dimension L representing lightness, a and b representing dimension of opposite color value, and CIE XYZ color space coordinates based on nonlinear compression.

Comparing the Lab parameters of the recognized and learned colors, when the two colors match within a certain error range, they are identified as the same color.

In our usual use of color recognition, the hue and saturation in the color attribute of the same module are fixed, but the lightness will change under the influence of the environment light change. Therefore, when using color recognition function of HuskyLens, we must try our best to ensure that the environment light during learning and recognition is consistent with actual work.

III. Main Application Fields of Color Recognition

Industrial field: Color recognition is currently widely used in the industrial field, such as printing, paint coating and textile, for color monitoring and calibration.

Personal life: usage as an auxiliary recognition for people with color weakness or visual impairment can enhance their understanding of color.

IV. Demonstration of HuskyLens Color Recognition Function

The color recognition function of HuskyLens sensor is to use the built-in algorithm, by learning and recording different colors, the ID of different colors can be identified and fed back to the mainboard.

The default setting in HuskyLens is to learn, recognize and track only one color. But we can change the default setting to make it recognize multiple colors.

How to operate it step by step? First, take out your HuskyLens. Let's do it together.

For first-time users, please refer to WIKI website for firmware burning and language setting:https://wiki.dfrobot.com/HUSKYLENS_V1.0_SKU_SEN0305_SEN0336

STEP1 Switch On

HuskyLens has its own independent USB connector, which can be switched on by connecting the USB cable.

STEP2 Operation and Setting - Learn Multiple

① Dial the function button to the right or left until the word "Color Recognition" is displayed at the top of the screen.

② Long press the function button to enter the parameter setting of the color recognition function.

③ Dial the function button until "Learn Multiple" is displayed, then short press the function button, and dial to the right to turn on the "Learn Multiple" switch, that is, progress bar turns blue and the square icon on the progress bar moves to the right. Then short press the function button to confirm this parameter.

④ Dial the function button to the left until "Save & Return" shows. And the screen prompts "Do you want to save the parameters?" Select "Yes" in default, now short-press the function button to save the parameters and return automatically.

In this way, we finish the set-up of "Learn Multiple" function.

STEP3 Operation and Setting - Learning and Detection

(1) Color Detection

Point the icon "+" in the center of the HuskyLens screen to the target color block, and a white box will appear on the screen, which selects the target color block automatically. Adjust the angle and distance of the HuskyLens to the color block so that the white box frames the entire target color block as far as possible.

(2) Color Learning

After detecting the color, long press the "learning button" to learn the first color, and then release the "learn button" to finish learning. A message will prompt on the screen: "Click again to continue! Click other button to finish".

Please short press the "learning button" before the countdown ends if you want to learn next color. If not, short press the "function button" before the countdown ends, or do not press any button to let the countdown ends.

The color ID displayed by HuskyLens is in the same order as the learning color, that is, the ID will be marked as "ID1", "ID2", "ID3" and so on. And the frame colors corresponding to different colors are also different.

(3) Color Identification

When encountering the same or similar color blocks, a colored frame with an ID will be automatically displayed on the screen, and the size of the colored frame is same as the size of the color blocks. The frame will automatically track the color block. Different colors can be identified and tracked at the same time, and the frame colors corresponding to different colors are also different.

When multiple color blocks in the same color appear in the picture, only one color block can be identified at a time.

* Color recognition is greatly affected by environment light. Sometimes HuskyLens may misidentify similar colors. Please try to keep the environment light consistent and use under moderate lighting.

Project practice:

How is HuskyLens color recognition function used? How can we map colors to scales one by one? Let's break down the whole project into several small tasks and complete the color numbered musical notation step by step.

This project will be divided into three steps to complete the task. First, we will learn to use color recognition function of HuskyLens and output the recognized color ID. Then we can play the scales according to the output color ID.

Finally, we’ll improve the whole project.

Task 1: Get to Know Color Recognition

1. Hardware Settings

Connection Diagram: HuskyLens sensor uses IIC connector. Please pay attention to the cable sequence. Do not connect it wrongly or reversely.

Assembly Diagram: HuskyLens sensor has its own bracket structure, which can be fixed to Maqueen Plus by screws, and HuskyLens can adjust various angles.

2. Program Design

Here we take HuskyLens sensor to learn 3 colors as an example, and output color ID through serial port to facilitate real-time viewing.

* Serial port: Serial port is a way of real-time communication between computer and hardware. For example, in this task, through the serial port, the data of HuskyLens on the Maqueen Plus can be viewed in real time on the computer.

STEP 1 Learning and Recognition

Before designing the program, we need HuskyLens sensor to learn each color.

(Note: First switch to "Learn Multiple" function)

STEP 2 Mind+ Software Settings

Open Mind+ Software (Version 163 or above):

① Switch to "Upload mode";

② Click "Extensions" and load "micro:bit" under "Board";

③ Continue to click and load "Maqueen Plus Domestic Edition" under "Expansion";

④ Click and load "HuskyLens AI Camera" under "Sensor" to finish.

STEP3 Instruction Learning

Let's take a look at some of the main instructions.

①Initializing: placed between the start of the main program and loop execution, only need to be executed once. I2C or serial port can be selected, I2C address does not need to be changed.

Note: HuskyLens side needs to adjust the "output protocol" in the setting to be consistent with the program, otherwise the data cannot be red.

②Switching algorithms: other algorithms can be switched to at any time, and only one algorithm can exist at the same time.

Note: Algorithm switch takes some time.

③The main board requests HuskyLens to store the data in the "result" once (data stored in the memory of the main board, refreshed by each request), then the data can be obtained from the "result", and the latest data will be obtained in the "result" only after this module is called.

④The information near the center box in the current interface is obtained from the requested "result". The box id that has not been learned is 0, otherwise, -1 is returned.

⑤Variable: Variable refers to the amount of change, which is convenient to store the changed number.

Click "Make a Numeric Variable" to create a new variable. You can directly set the value of the variable as well as increase or decrease the value of the variable.

3 Program Example

4 Execution Result

When using a serial port, the computer and the main board need to be connected through a USB cable. In Mind+, connect the corresponding COM port.

Serial port Operation Method:

1. Serial baud rate defaults to 9600

2. Open the serial port

Run window:

When the HuskyLens sensor has learned three colors, it recognizes the ID of the corresponding color and displays the corresponding number on the serial port.1 denotes the first-learned color, 2 denotes the second -learned color, 3 denotes the third-learned color, and -1 denotes the color that has not been learned.

Task 2: Define Scale for Each Color

1. Program Design

STEP1 Operation and Setting-Learning and Recognition

Before designing the program, we need have HuskyLens to learn 7 colors. In order to avoid false recognition, 7 colors with big color difference should be selected.

The colors of ID1-ID7 are respectively set to blue, red, green, yellow, cyan, purple and black. Please learn colors in this order to correspond to the following procedures. After learning the 7 colors, HuskyLens stopped learning.

In combination with the lights on Maqueen Plus, identify a color and let the lights display the corresponding color.

See? Is the visual effect more glorious?

STEP2 Instruction Learning

Let's take a look at some of the main instructions.

①Obtaining whether IDx has learned from the requested "result".

②Obtaining whether there is a box or arrow in the current interface from the requested "result", including learned (id is greater than 0) and not learned, if there is one or more, return 1.

③Set the car lights to display different colors, and the program provides 8 colors.

④Instructions of playing notes: are divided into low, medium and high notes, and various beats.

⑤Function: A function can be understood as a program that performs a specified function.

Click "Make a Block" to create a new function.

Sometimes too many instructions in the main program will affect our understanding. Functions can simplify the main program and facilitate understanding of the whole program.

STEP 3 Flow Chart Analysis

2.Program Examples

3. Execution Result

We have learned colors in the order of blue, red, green, yellow, cyan, purple and black given above. Then execute the program, when the center of HuskyLens display screen is blue, play note 1 and the light is blue; when it is red, play note 2, and the light turns red, and so forth.

If your programs do not work properly, please check the following points:

(1) Whether they have learned colors in the order of blue, red, green, yellow, cyan and purple;

(2) Whether the mode is switched to “Stop Learning” after learning the above 7 colors; if not, a white box will always be output on the camera screen, which will affect the program execution.

Task 3: Remove Misjudged Interference of Small Color Blocks

1. Program Design

STEP1 Analysis of Problems Encountered

The program in Task 2 can basically realize the project function, but some small color blocks may become interference when there are multiple color blocks unexpectedly appeared in the camera screen. How to eliminate such interference? One solution is to judge whether the color block is the target color block by the area proportion on the camera screen. We may set a threshold value, and the color block below the threshold value will automatically be judged as interference.

STEP 2 Instruction Learning

Let's take a look at some of the main instructions.

①Obtain the information in the frame near the center box from the requested "result" in the current interface, the frame area can be obtained by multiplying "Object width" and "Object height".

STEP3 Flow Chart Analysis

2. Program Examples

Modify the program in Task 2. The function "note" remains unchanged. The main program is modified as follows.

3. Execution Result

When the center of HuskyLens displays blue, play note 1 and the light is blue; when turns red, play note 2, the light turns red, and so forth.

Project Summary:

Project Review

Understand the working principle of color recognition in this lesson, and learn the color recognition function by using HuskyLens.

Color recognition is a very important function in AI vision recognition and is widely used in industrial fields. What other interesting functions can color recognition achieve? Please think about it.

Knowledge Nodes Recap

After this project is completed, you can search for a numbered musical notation through internet, spell out the numbered musical notation with color blocks to have Maqueen Plus play music after printing it out!

However, we will definitely find a problem: the number of scales is relatively small, and the medium, low and high notes cannot be realized. If we want to add color blocks, with the increase number of colors, there will be many similar colors, which may lead to misrecognition. So is there any solution to widen the sound range?

(Tips: The AB button on micro:bit board can be used to realize the functions of raising and lowering the key.)