Summary
A client who does a lot of work with animatronics, was interested in a player piano developed by Brandon Switzer that used solenoids to drive piano keys based on a MIDI input stream. This client wanted to take that project a step further and create different animatronics interfaces that is controlled by a MIDI stream. The types of outputs included solenoids, digital outputs, PWM channels, servo motors, and analog outputs. This embedded system became a MIDI Animatronics Interface.
Hardware Configuration
The key hardware component is an ESP32. ESP32 is a 2-core, 80MHz processor that is supported by the Arduino IDE and FreeRTOS. In the player piano multi-channel PWM outputs are driven by a chain of 74HC595 shift registers controlled using SPI. Each shift resister output controlled a channel/key/note. Because we wanted to control many different types of devices, the interface was changed to I2C, which supports many peripherals. Most of the interface outputs are PWM based (this includes the digital, full on or off). After reviewing the market, the PCA9685 was selected for PWM outputs.
The PCA9685 is a 16-channel, 12-bit, PWM controller. This device has lots of capability including Fast I2C interface, output frequency from 24Hz to 1.5kHz, different output driving modes, and operates from 2V3 to 5V. There are several available Arduino ready supporting libraries. Since the device supports up to 16-channels, this became a nice system building block.
Since the product is highly configurable, a Nextion touchscreen interface was used. This was an inexpensive interface allowing the user to control the system configuration and runtime parameters using a small, 2.4″ touchscreen. Again an Arduino library was available, although a couple of source code changes were required.
The ESP32 interfaces include the I2C, MIDI serial (UART 2), and Nextion serial (UART 1). I also use the Arduino IDE serial monitor (UART 0) for string based feedback.
System Configurations
Using a MIDI stream one can create a MIDI network using the IN and THRU connections. There are a number of different configuration that can be created using nodes formed by ESP32s and connected I2C devices. Each node can have the same or different MIDI address, different starting note, and support a different number of channels. In the player piano example instead of using a single ESP32 you could use two, one for the left hand and a second for the right hand. On a single MIDI channel you can control up to 127 outputs (notes).
Implementation
The ESP32 forms the core of the MIDI node. The firmware currently supports using I2C to interface PCA9685 16-channel, 12-bit PWM and MCP4725 single channel DAC ICs. Up to 128 outputs are support. Also if necessary, a 3V3 to 5V translation interface is used for those applications that require higher output voltage.
As a basis, this firmware started with the worked published by Brandon Switzer (2019) that created a player piano using solenoids to move the keys. Modifications were made to combine the operation of three separate processors to a single ESP32. Other modification included using a different MIDI library, use of the PCA9685 for the PWM output instead of 595 shift registers, and Nextion user control interface. Also added were modes with generic parameters to provide digital control (full on/full off), PWM (piano mode without note handling), servo motor, and analog outputs.
One of the key changes was to move from SPI to I2C to support different I2C peripherals that are managed from a single piece of common firmware. New I2C device may require code changes in the future. Also moving to the PCA9685 requires less processor management of the PWM stream.
The firmware has three tasks distributed on two processor cores. Core 0 handles the MIDI input that includes MIDI read callbacks for Note On, Note Off, and Control Change (pedal control), note handling, and note scheduling.
When a note is ready for output, it is sent to a queue with its velocity. Waiting on the queue is the task managing output to the I2C device(s). This task receives the queue data and activates the note with a modified velocity dependent on the mode. For digital mode, any non-zero velocity is set to 127 (maximum MIDI velocity value). The MIDI velocity is also converted based on the output word size. In servo mode the velocity is mapped to a calibrated servo min/max (i.e., velocity 0 = 0 degrees, velocity 127 = 180 degrees).
The third task is on Core 1 and is used to interface with the Nextion touchscreen. The Nextion interface is used to select an operating mode (necessary prior to handling MIDI inputs), setup configuration parameters, and to change runtime parameters.
Operation
Before using the system, an operating mode must be selected. Without an operating mode the system does not process MIDI inputs. In this condition the operating mode selection screen is shown. The user pushes the desired mode and the operating mode screen is shown. The user then configures the parameters for this mode. Once a mode is configured, the user sets the configuration by pressing the Save button and the information is saved to EEPROM.
After the Save button is pressed, the system reboots and the operating mode screen is shown again. The system now handles MIDI input stream. The system is currently configuration to re-transmit the input MIDI stream (future user parameter to disable or filter accepted commands). Pressing Reset erases the current operating mode and parameters and returns to the operating mode selection screen.
Anyware, LLC 