You should not need to have massive latency and I have not had this problem myself doing the same thing with Yarns.
Basic questions would be for me 1) where are you running the midi cable from (soundcard? USB port?). 2) If it is USB to midi cable, is cable good quality? 3) Are you running a small enough buffer size in Ableton preferences? I am not a technical expert at all and it is all trial and error for me but I found upgrading my USB to midi cable to a better model really helped performance. Likewise, when I have worked with running midi from soundcard, upgrading my soundcard to a Thunderbolt one virtually eliminated latency straight away.