Does the deceased hear the supplications or recitations of those visiting his/her grave? And does making constant supplications for the deceased who didn`t adhere to performing the prescribed prayers ameliorate his/her affliction?

Praise be to Allah the Lord of the Worlds.

In principle, the dead hear the living since it has been rigorously authenticated that the Prophet (PBUH) ordered the bodies of the idolaters slain in the Battle of Badr to be buried. He then approached them and called them one by one and said: "Have you found what your Lord promised to be true for we have found what our Lord promised us to be true." Umar asked him: "O messenger of God! Why do you speak to lifeless bodies?" The Prophet (PBUH) replied: "By the One Who has sent me with the truth! You do not hear my words better than they do except that they cannot respond." [Agreed upon].

However, not all the deceased are in the same situation because Allah The Almighty Knows: "Before them is a Partition till the Day they are raised up." [Al-Mu`minun/100]. Moreover, supplication reaches the deceased. And Allah Knows Best..

What is the ruling on working for orphans' institution?

I am an author, and I have books stored with me. Is it permissible to pay my zakat in the form of books containing beneficial knowledge for poor students, such as chemistry and mathematics, or religious books that spread Islamic sciences?

Zakat on wealth must be paid in cash from the wealth itself. Therefore, you should pay your zakat in monetary form so that the poor can purchase what they need. And Allah Knows Best.

Is it permissible for someone who has forgotten to mention the supplication pertaining to answering the call of nature to say it while in the toilet?

It is disliked for a Muslim to mention Allah while in the toilet, but he/she may recall the supplication in his/her head without uttering it; that is in case he/she had forgotten mentioning it before entering the toilet.