What's the difference between the headphone and speaker modes?

Headphone mode produces binaural audio intended to be presented directly to the listeners ears via headphones. The speaker modes process the binaural audio with a crosstalk canceller which cancels the crosstalk from the left speaker to the right ear and vice-versa.

Unfortunately, there isn’t any way to create binaural audio which is fully stereo compatible and can be listened to over both headphones or loudspeakers.

Without crosstalk cancellation, binaural audio over loudspeakers sounds like stereo, but lacks the reality of headphone presentation due to the crosstalk which ruins the spatial cues. Conversely, if you listen to crosstalk-cancelled audio over headphones, it sounds very unnatural because the crosstalk cancellation tends to drive the speakers out of phase at low frequencies.