I've actually gotten reasonably good at saying that one (not perfect, but reasonably good.

The real problem comes when I have to say to X, "Please find out from Y and Z which of the things they have previously asked me to do they will now get someone else to do so that I have time to give to this thing for you." Because then X, Y and Z all get together and usually decide that I surely must have time to do everything they all need. And then I have to tell them all that no, someone has to yield something before I can add a new something from anybody to the list, and they all freak out and insist that no one can possibly help them but me, which is not true, because there are four other people in the company who, like me, are junior researchers whose job is at least in part to help out the seniors (although it is true that one of those people has pretty much been monopolised by the company's fearless leader to do all of his stuff, so she's rarely available).

At least it gives me some assurance of job security. ;-)