While the section Troubleshooting lists problems by symptom, this section explains what will happen if interrupts are set incorrectly. This section helps you understand what caused the symptom, what other symptoms might be due to the same problem, and what to do about it.
The symptoms depend on whether or not you have a modern serial port with FIFO buffers or an obsolete serial port without FIFO buffers. It's important to understand the symptoms for the obsolete ones also since sometimes modern ports seem to behave that way.
For the obsolete serial ports, only one character gets thru every several seconds. This is so slow that it seems that nothing is working (especially if the character that gets thru is invisible (such a space or newline). For the modern ports with FIFO buffers you will likely see bursts of up to 16 characters every several seconds.
If you have a modem on the port and dial a number, it seemingly doesn't connect since the CONNECT message doesn't make it thru. But after a long wait it finally does connect and you finally do see part of a login message (or the like).
If you use minicom, a common test to see if things are working is to type the simplest "AT" command and see if the modem responds. Typing just at<enter> should normally (if interrupts are OK) result in an "OK" response from the modem. With bad interrupts you type at<enter> and may see nothing. But then after 10 seconds or so you see the cursor drop down one line. What is going on is that the FIFO is behaving like it can only hold one byte. The "at" you typed caused it to overrun and both letters were lost. But the final <enter> got thru since you waited for it and you "see" this invisible character by noting that the cursor jumped down one line. If you were to type a single letter and then wait about 10 seconds, you should see it echo back to the screen. This is fine if your typing speed is less that one word per minute :-)
If you don't understand what an interrupt does see Interrupts. If a serial port has one IRQ set in the hardware but a different one set in the device driver, the device driver will not receive any interrupts sent by the serial port. Since the serial port uses interrupts to tell its driver when it needs service (fetching bytes from it's 16-byte receive buffer or putting another 16-bytes in its transmit buffer) one might expect that the serial port would not work at all.
But it still may work anyway --sort of. Why? Well, besides the interrupt method of servicing the port there's a slow polling method that doesn't need interrupts. The way it works is that every so often the device driver checks the serial port to see if it needs anything such as if it has some bytes that need fetching from its receive buffer. If interrupts don't work, the serial driver falls back to this polling method. But this polling method was not intended to be used a substitute for interrupts. It's so slow that it's not practical to use and may cause buffer overruns. Its purpose may have been to get things going again if just one interrupt is lost or fails to do the right thing. It's also useful in showing you that interrupts have failed.
For the 16-byte transmit buffer, 16 bytes will be transmitted and then it will wait until the next polling takes place (several seconds later) before the next 16 bytes is sent out. Thus transmission is very slow and in small chunks. Receiving is slow too since bytes that are received by the receive buffer are likely to remain there for several seconds until it is polled.
This explains why it takes so long before you see what you typed. When you type say AT to a modem, the AT goes out the serial port to the modem. The modem then echos the AT back thru the serial port to the screen. Thus the AT characters have to pass twice thru the serial port. Normally this happens so fast that AT seems to appear on the screen at the same time you hit the keys on the keyboard. With polling delays thru the serial port, you don't see what you typed until many seconds later.
What about overruns of the 16-byte receive buffer? This will happen with an external modem since the modem just sends to the serial port at high speed which is likely to overrun the 16-byte buffer. But for an internal modem, the serial port is on the same card and it's likely to check that this receive buffer has room for more bytes before putting received bytes into it. In this case there will be no overrun of this receive buffer, but text will just appear on your screen in 16-byte chunks at intervals of several seconds.
Even with an external modem you might not get overruns. If just a few characters (under 16) are sent you don't get overruns since the buffer likely has room for them. But attempts to send a larger number of bytes from your modem to your screen may result in overruns. However, more than 16 (with no gaps) can get thru OK if the timing is right. For example, if 32 bytes were received (and no more bytes followed), the polling might just happen after the first 16 bytes had been received. Then there would be space for the next 16 bytes so that 32 bytes gets thru OK. Similar conditions might pass between 16 to 31 bytes thru OK. But it's also likely that only an occasional 16-byte chunk will get thru and huge gaps of missing data will be lost.
If you have an obsolete serial port with only a 1-byte buffer (or it's been incorrectly set to work like a 1-byte buffer) then the situation will be much worse than described above and only one character will occasionally make it thru the port. Every character received causes an overrun (and is lost) except for the last character received. This character is likely to be just a line-feed since this is often the last character to be transmitted in a burst of characters sent to your screen. Thus you may type AT<return> to the modem but never see AT on the screen. All you see several seconds later is that the cursor drops down one line (a line feed). This has happened to me with a 16-byte FIFO buffer that was behaving like a 1-byte buffer.
When a communication program starts up, it expects interrupts to be working. It's not geared to using this slow polling-like mode of operation. Thus all sorts of mistakes may be made such as setting up the serial port and/or modem incorrectly. It may fail to realize when a connection has been made. If a script is being used for login, it may fail (caused by timeout) due to the polling delays.
When two devices have the same IRQ number it's called sharing interrupts. Under some conditions this sharing works out OK. Starting with kernel version 2.2, serial ports may, in some cases, share interrupts with other serial ports. Devices on the PCI bus may share the same IRQ interrupt with other devices on the PCI bus. In other cases where there is potential for conflict, there should be no problem if no two devices with the same IRQ are ever "in use" at the same time. More precisely, "in use" really means "open" (in programmer jargon). In cases other than the exceptions mentioned above (unless special software and hardware permit sharing), sharing is not allowed and conflicts arise if sharing is attempted.
Even if two processes with conflicting IRQs run at the same time, one of the devices will likely have its interrupts sent to its device driver and may work OK. The other device will not have its interrupts sent to the correct driver and will likely behave just like a process with mis-set interrupts. See Mis-set Interrupts for more details.
If you are getting a very slow response as described above, then one test is to change the IRQ to 0 (uses polling instead of interrupts) and see if the problem goes away. If so, then there was likely something wrong with the interrupt and you need to fix it. Check /proc/interrupts to see if it is currently in use by another process. If it's in use by another serial port you could try "top" (type f and then enable the TTY display) or "ps -e" to find out which serial ports are in use. If you suspect that setserial has a wrong IRQ then see What is the current IO address and IRQ of my Serial Port ?