Serial Buffer Questions

Hey Everyone,

I am currently building a custom midi controller. I am building it with midi input as well, so that you can run midi notes into the controller and it will just pass it along directly to the output.

So at the beginning of my code in the “loop()” area (I am using Arduino), I started by using this

if (Serial.available() > 0) {
incomingByte = Serial.read();
Serial.write(incomingByte);
}

So that works for passing midi received in the input to the midi output of the controller, but the problem is I have code afterwards which sends other midi messages when I am moving knobs on the controller. So let’s say I receive a Note On message, (144, 60, 80) for example, the arduino will receive and pass on the 144 byte but then proceed to execute other knob movement messages instead of passing the 2 data bytes that directly follow the 144. So the result is faulty output.

To correct this issues I modified the code to look like this:

If(Serial.available() > 0) {
incomingByte = Serial.read();
if (incomingByte > 127 && incomingByte < 160) {
Serial.write(incomingByte);
incomingByte1 = Serial.read();
Serial.write(incomingByte1);
incomingByte2 = Serial.read();
Serial.write(incomingByte2);
}
}

So the theory was with this code that when a midi note on or off message was received it would read the following 2 bytes (note number, and velocity) before proceeding to execute the code which reacts to knob movements and therefore doesn’t mix up strings of midi bytes. However, this code only passes on Note On messages for some reason when I play midi notes on my keyboard into it, and Note Off messages don’t get pass on. I have no idea why.

Maybe I don’t understand the serial buffer enough or midi protocol? Can someone help point out the flaw in my code or understanding here?

Thanks mutable community

I think the root of the problem is that your code doesn’t handle running status. It doesn’t parse incoming data with running status, and it doesn’t handle the following situation: you will need to re-insert status bytes if you decide to insert a control change between two noteon/off messages whose status byte is stripped because of running status.

Another potential problem I see with your code is that it mixes blocking and non-blocking I/O. At some point you are switching to blocking I/O, so the calls to Serial.read() will actually block (and for about 1ms your device will freeze). This is not a good idea. Sitting in a tight loop waiting for things to happen is a bad idea.

Finally, your code won’t work if the other synth upstream is sending clocks, CCs, etc…

You can’t escape doing things correctly…

  • Read the MIDI data one byte at a time.
  • Parse the data you get into full messages (noteOn, noteOff…).
  • As soon as you have a full message, write it to a transmission buffer. Keep track of the status byte - write it or not if necessary (If the first byte of the message matches the running status, do not write the first byte; if the first byte of the message does not match, write the first byte and update the running status).
  • Whenever you need to generate a CC, write it to the transmission buffer - with or without the status byte (same rule as above).
  • In your main loop, check the status of the transmission buffer - if it contains data, and if the serial port is available for writing, pop a byte from the transmission buffer and write it to the serial output.

If you think that parsing MIDI data is too tedious for you (and that’s indeed a bit daunting for a beginner project), ditch the MIDI input on your device and use a third party MIDI merger known to work.

Fantastic. Thank you, you’re response answers all my questions perfectly! I will definitely spend the time writing the code to parse the MIDI properly. I read more about running status after reading your response and I now understand how it is implemented in MIDI protocol.

“You can’t escape doing things correctly”.

This is certainly true.

“You can’t escape doing things correctly”.

ill make a greeting card from this

Ok, after spending a lot of time reading over midi specifications I wrote code that properly parses incoming midi in my controller. It properly handles all types of control mode and system messages. Hurray!

However, I ran into another problem. I am getting serial buffer overflow when I output computer drawn envelopes from my DAW into my midi controller with the merger software. Once I start using envelopes that change extremely fast continuously the arduino can’t keep up. I have been reading online that one solution is to just increase the serial buffer size to 256 bytes instead of the default 64. I’m not sure that will be enough though, it might only just take longer for the buffer to overflow.

Are there other better ways to truncate continuous streams of data flow at high speeds into the arduino? Or other ways to optimize the arduino for this kind of performance?

Thanks so much!

You don’t need to truncate and you don’t need to increase the serial buffer size.

MIDI is 31250 bauds ; which means about 4kbyte / s ; which means 4000 CPU cycle to process each byte. If your code is well written the bytes should not even accumulate in the serial RX buffer, and in fact, the fast majority of lightweight MIDI products do not even buffer - they process incoming bytes straight away as they are received, in an ISR.

So I would suggest first to check what causes the incoming buffer to get full. Is there a path in your code that would take more than 4000 CPU cycles? Maybe some blocking transaction with an I/O device that would freeze your device for more than 64 / 4000 = 16ms - like some shitty graphic LCD drawing code?

Hmm I’m not sure what would be slowing it down. My midi controller has 3 multiplexer chips full of potentiometers, 4 shift registers for switches and buttons. So it’s making a lot of digital.read() and analogue.read() commands every loop() cycle. Could that be slowing it down enough that it’s not reading the serial buffer fast enough?

Oh one other thing I forgot to mention. It’s converting incoming midi CC values to SysEx commands, so sometimes streams of bytes coming out of the controller are up to like 10 bytes long.

I use:

What about handling incoming MIDI inside an Interrupt Routine that breaks the UI handling and handle all the UI stuff in the Main Event loop thus with a lower priority?

Hmm thanks for your input fcd72. I will look into Interrupt Routines, I haven’t used them before since I’m a beginner. Seems like something that could streamline my code.

I’m not sure you’ll be able to create your own UART data RX routine as the arduino library already puts its own there. But it’s not a big deal. Without the right practices doing things in the ISR is all wrong altogether, and has the risk of indeed crashing your device if you don’t service it fast enough, causing reentry and ultimately a stack overflow (if you don’t understand these last words, this is the very first thing you have to focus on! understanding this!)

You really have to identify what takes so long in the main loop. Out of curiosity, how do you write the data you want to transmit?

The best way to time stuff is to toggle pins HIGH / LOW before and after bits of code and measure them on a scope (analog or digital phosphor) - you see the pulse width and its deviation.

“You really have to identify what takes so long in the main loop. Out of curiosity, how do you write the data you want to transmit?”

So here’s like a nutshell of what the code in the main loop looks like:

Section 1.Parsing/gathering incoming MIDI data and sending it to the output

  • I followed your previous advice here, and there is only 1 call to Serial.read() per cycle of the main loop. Once and only after all the data bytes of a midi message have been gathered does it execute a series of Serial.write() functions to send the data.

Section 2. Converting certain MIDI Control Change messages from the input into SysEx messages

  • So if certain Control Change messages are received from the input then they do not get passed directly to the ouput, instead they are converted into a SysEx message:

for (int count = 0; count<7; count++) {
Serial.write(paramChange[count]);
}
Serial.write(whichCC);
paramValue = formatConVal(whichCC, conCCMessages[1]);
Serial.write(paramValue);
Serial.write(247);

Section 3. Code for scanning for changes in potentiometer movement

Section 4.Code for scanning for changes in button and switch movement. This code here is executed to create arrays of data from the shift registers which the code checks afterwards to see if there are any changes in positions. If so, only then is a Serial.write code executed.

digitalWrite(latchPin, 1);
digitalWrite(latchPin,0);

for (int i=7; i>=0; i–) {
digitalWrite(clockPin, 0);
shiftdata0[i] = !digitalRead(dataPin0);
shiftdata1[i] = !digitalRead(dataPin1);
shiftdata2[i] = !digitalRead(dataPin2);
shiftdata3[i] = !digitalRead(dataPin3);
digitalWrite(clockPin, 1);
}

Could this chunk here in Section 4 be what is slowing the main loop down? It’s a lot of digitalRead and Write calls in one loop.

Those digital reads and writes takes ALOT of time.
Use direct port manipulation instead.

My synths are built like this:

systick or timer irq doing the dsp calculation and output of audio.

uart irq receiving MIDI and manipulating parameters and initializing new notes for the dsp.

mainloop is doing realtime wave calculations and knob scanning.

Ok, I have read up some more on Interrupts.

So potentially I should just have changes in the serial input attach to an ISR so that it gets read and calls a function to execute output right away, instead of having to wait for the main code loop to finish?

But I should definitely use direct port mapping at least for the button and switch code?

I don’t know what else would be slowing down my code other than constant polling of the multiplexers for the pots.

> But I should definitely use direct port mapping at least for the button and switch code?

I would disagree. They are inefficient but not that inefficient. On something that is not doing realtime audio this would be a bit overkill. Your project is doing 1 or 2% of what the Shruthi is doing in terms of CPU so it would be fine, even with the arduino abstractions…

> systick or timer irq doing the dsp calculation and output of audio. uart irq receiving MIDI and manipulating parameters and initializing new notes for the dsp.

That’s one way of doing things.

On my side, I do only basic I/O read/writes in interrupt handlers and write/read the data into buffers. Buffers are processed/examined in the main loop. For audio, they are processed in short blocks of samples to factorize some code during their processing (mostly: to have everything in registers and no memory/read write, and keep it so over several samples).

My diagnostic regarding your problem: check the arduino libraries documentation to see if your “Serial.write” calls are blocking. Back in 2009 they were. Looks like the Arduino is freezing while you do all these writes - so the input buffer fills up during that time.

Instead of doing so many blocking writes in one go, use an output buffer. So your main loop would read / write bytes on the serial output just one by one. Check if there is a way, through the arduino libraries, of polling the serial output so that you can know in advance if Serial.write will block or return immediately. That would be the very first thing I would start doing through direct I/O register manipulation rather than through the arduino libs.

On the Arduino serial.write is blocking.

So don’t use it :wink:

Awesome, thanks again so much for all your valuable insight. I will start working on modifying my code in the way you guys suggested and let you know how it goes.

Can someone confirm this assumption for me:

If I have a continuously changing midi CC signal (3 bytes (2 bytes in runtime status)) running at the maximum speed it can send (31250 bauds) and it’s being translated real-time into a SysEx message of 9 bytes going to the output, then serial output buffer would definitely overflow. No matter how streamlined the code was. Because for every 2 bytes received the serial output has to send 9. So in order for me to have a MIDI to SysEx converter function, it would need to truncate the continuous signal somehow in order to avoid overflow.

Is this correct? (Thanks)