Are you able to identify the caller with Caller ID before you pick up? If so, you could simply not answer the call and let it go to voicemail. Then call back when it's convenient to you.
There is no need to be constantly available. Same applies with emails, too - it's not necessarily a good idea to reply too promptly, even if it's possible. People start to get used to it.