Is it permissible for the big brother to force his sisters to wear Hijjab (Islamic wear) if their father didn`t do his part in this regard ?

All Perfect Praise is due to Allah, The Lord of The Worlds

The father`s attention should be drawn gently and politely, and the sisters should also be reminded with good words since Allah, The Almighty, Says (What means): "O ye who believe! Save yourselves and your families from a Fire whose fuel is Men and Stones." [Al-Tahreem/6]. And Allah Knows Best.

When is Istikharah (guidance prayer) Du`a offered, before Tasleem (saying asalamu alikum warahmatu allah wa barakatuh to end the prayer), or after it?

It is offered after Tasleem.

What should a praying person who doubts having offered one, or two prostrations in the third Rakah (unit of prayer) do?

Whoever has doubts about their prayer should consider what is less i.e. the above person should offer another Sajda (prostration) and complete his/her prayer, then offer Sujood As-Sahw (prostration of forgetfulness). And Allah Knows Best.

Is it permissible to offer mandatory prayer while sitting?

All perfect praise be to Allah, The Lord of The Worlds, and may His Peace and Blessings be upon our Prophet Muhammad and upon all of his family and companions.                                                                                                                                                                                                                            Standing up is a pillar in the mandatory prayer and the prayer is not valid without it unless one cannot do it. As for Nafila (supererogatory prayer), one can perform it in the state of sitting even if he/she can stand, but his/her reward is half the reward of the person who prays while standing if he/sh has no legal excuse for such an act. And Allah Knows Best.