Have you ever heard the statement, "I am too busy"? As you know, many times that statement is a way of saying no. A person or organization has asked someone to do something that they do not want to do. They may feel unqualified to do what is asked of them. It may also be that they do not understand what is asked of them. But, a lot of the time, it is because in their mind they have more important things to do.
Yes, many times we may need to say no to people because we need our priorities to be right. We see this illustrated in the book of Nehemiah. He and the people of God were busy building the wall when the enemies of God asked him to come and meet with them. I like the answer given by Nehemiah 6:3. "And I sent messengers unto them saying, I am doing a great work, so that I cannot come down: why should the work cease, whilst I leave it, and come down to you?"
It is easier to say no to all things, if we are busy. What we must ask is: are our priorities right in our lives, and are our priorities right with the will of God? "But seek ye first the kingdom of God, and His righteousness; and all these things shall be added unto you.