Dear ma and pa,
Your question brings to light the point that in this new age of technology, we need to develop a "code of ethics" or at least adopt some universally accepted etiquette relative to these situations.
You are correct to assume many people become angry when told they are being put on hold or "left" for another call. You're also correct to assume the anger is born out of feeling unimportant, compared to the next caller. No one wants to be in a competition (with another person) for your attention.
In the situation you are outlining, the key is the issue of power. "I'm going to put you on hold" or "I need to call you back because I need to take this call" are both responses which take away mutuality of power. In both cases, the other person may feel uncomfortable/angry/inadequate because you are not proposing a choice. In so doing, you are not communicating to them, their value to you. In addition, what may add insult to injury, is the fact that this communication to the first caller must occur in a very short amount of time.
In order to deliver a thoughtful response, we need to determine our general values relative to "interrupting calls" in advance. At the same time, it's important to remember our priorities must be fluid when it comes to phone calls.
I know people who do not have the "call waiting" option, believing no call is worth interrupting a previous call. That's one choice.
You have determined there are certain "interrupting calls" which should take precedent, due to the high priority of the call, the limited availability of the second caller, or the timeliness of the subject matter.
Others may have their own ideas about which "interrupting calls" take precedent. Some people for instance, allow calls from their children to trump all other calls.
Suppose you receive an "interrupting call" that you immediately decide you want to take.Your first consideration still lies with your primary (first) caller. To demonstrate that consideration, you do not take away the mutuality of power. Instead, you offer them the right to determine what will happen next. Say, "I have a very important call I need to take. Would you prefer I put you on hold or would you like me to call you right back?" (If it applies, you may include, "Would you prefer to be on hold knowing the call may take awhile?") The key is allowing the first caller as many options as possible.