InstallShield uses a tool to extract COM information (spying on the registry) called IsRegSpy.exe. It is trying to register COM libraries and watching what happens in the registry to capture all the keys and values written during registration.
After adding some our own internal libraries I noticed that the IDE startet to hang when extracting COM information from the components in the setup project. After inspecting what was happening using ProcExplorer from Microsoft (http://www.sysinternals.com) I saw that the IDE spawned IsRegSpy.exe but IsRegSpy.exe never seemed to return.
After digging back and forth for a while, I tried to extract the COM information from within the InstallShield IDE to avoid getting this error each time we built the project. The same thing happened, the IDE launched IsRegSpy.exe and stopped responding.
I tried killing IsRegSpy.exe from within ProcExplorer, and noticed that InstallShield managed to add the correct COM information to the IDE! It also turned off the "Extract COM at build" setting for the component! Doing this for all COM components in the Setup project I managed to get past the problem of hanging the IDE when building.
EDIT: The last paragraph was obviously written in a very optimistic tone.. Some of the information was saved, some was not. My final solution was to export the registry hives before and after registering the dlls and then diffing the output...