Is it possible to open a custom .bank file from the game using FMOD Studio?
No.
Bank files aren’t compressed archives that can be “unzipped” to get back the project that was used to create them. They’re a propetary format for dynamic audio content that can be used by the FMOD Engine to play audio as part of a game, and building them is a one-way process.
If you want to edit the contents of a .bank file, you must open the original FMOD Studio project in FMOD Studio, edit the project, and then build a new version of the bank.
If you do not have the original FMOD Studio project, you should contact the developers of the game and ask them. If they agree to give you permission to work with their content, they should be able to furnish you with a copy of the FMOD Studio project; if they do not give you permission, you could not legally work with that content in any case.