Is prayer nullified by having thoughts while performing it?

Prayer isn`t nullified by having thoughts while performing it, provided that the praying person offers all of its pillars and meets all of its conditions. However, the reward decreases significantly in the absence of reverence since a praying person is rewarded according to the attentiveness of his/her heart during prayer.

Is it permissible for the big brother to force his sisters to wear Hijjab (Islamic wear) if their father didn`t do his part in this regard ?

All Perfect Praise is due to Allah, The Lord of The Worlds

The father`s attention should be drawn gently and politely, and the sisters should also be reminded with good words since Allah, The Almighty, Says (What means): "O ye who believe! Save yourselves and your families from a Fire whose fuel is Men and Stones." [Al-Tahreem/6]. And Allah Knows Best.

Is it obligatory for a wife who has been abandoned for many years to observe the waiting period (iddah) after her husband passes away?

If her husband passes away, the wife is required to observe the waiting period (Iddah), even if he had abandoned her, because she is his wife and inherits from him, so she must observe the iddah after his death. And Allah Knows Best.

What should a praying person who doubts having offered one, or two prostrations in the third Rakah (unit of prayer) do?

Whoever has doubts about their prayer should consider what is less i.e. the above person should offer another Sajda (prostration) and complete his/her prayer, then offer Sujood As-Sahw (prostration of forgetfulness). And Allah Knows Best.