Is it permissible for a woman to shake hands with her brother-in-law?

It is forbidden for the woman to shake hands with her brother-in-law because he isn`t a Mahram (unmarriageable) of hers.

Is it permissible for a person who had vowed to give a certain amount of money to another, but didn`t find the latter to donate that money to the mosque?

In principle, the vowing person should abide by his vow as much as possible. Therefore, if he couldn`t find the person that he had made the vow for, then the vow itself is countless and nothing is due on its maker. However, if the latter happens to find the former later on then, he has to give him that money.

Is it disliked to clip one`s nails at night, and to burn them afterwards?

It is recommended to bury hair and nails that have been removed and to conceal them in the ground. As for burning nails, we have not found any specific prohibition against it, but adhering to what is supported by evidence—burying the nails—is preferable. And Allah Knows Best.

Is it permissible for the big brother to force his sisters to wear Hijjab (Islamic wear) if their father didn`t do his part in this regard ?

All Perfect Praise is due to Allah, The Lord of The Worlds

The father`s attention should be drawn gently and politely, and the sisters should also be reminded with good words since Allah, The Almighty, Says (What means): "O ye who believe! Save yourselves and your families from a Fire whose fuel is Men and Stones." [Al-Tahreem/6]. And Allah Knows Best.