So far I have only been able to reproduce this on devices running Android 8.0 and older, is this the same for you?
By changing the DSP buffer size to 4096 x 2 I was able to solve the distortion on one of the devices, but the other still had a small amount of distortion. This is likely due to the low latency mode not being supported on that device. The only solution in that case is to have a switch in the app to make the latency really high/extra large for bad devices.