Yes. Just as on a Macintosh, you will have to install each version in a different directory. (The default install directory for FMOD Studio includes the installed version’s version number, so this should happen automatically.)
In addition, the .fspro filename extension will be associated with the version that was most recently installed. Because of this, if you want to open an FMOD Studio project in a version of FMOD Studio that you currently have installed, but which is not the most recently installed version on your machine, you will have to open that version of FMOD Studio manually, and then open the FMOD Studio project by selecting “File > Open…”
Many of our users work on multiple game projects at a time, each using a different version of FMOD Studio. As such, having multiple versions installed is not uncommon.