Are you disabling/stopping the ICA dial-in service at any point? If you are, a reboot is required when you re-enable it.

Another possibility is that the modem is micsonfigured or the cable damaged. The modem should raise hardware signals when in use, and drop them on a hang-up/reset. If it doesn't the port will appear to be busy, and the last program to use ut may hang waiting for the hardware signals to drop. This program will probably be the ICA listener.

The software may also be waiting for a hardware "pulse", which is a signal that the device is cleared. From very vague memory it is a DTR drop or toggle.

Can you try a different modem & cable? Preferably a different model of modem.

Also check for any extra settings - hardware flow-control and connectivity are the most relevant.

When it next happens, try disconnecting the modem cable at the server end and see if the port clears.

FWIW We've supported ICA dial-in across 12 devices on two different server platforms for the last 4 years. The modems are old USR Sportsters, and we don't have any problems.

Finally, check the Citrix support site for modem issues - make sure your modem is supported.