What is the ruling on not fulfilling a vow?

A vow is a commitment towards Allah, The Almighty, and so, it should be observed as much as possible, and it is impermissible to take it lightly. Allah, The Exalted, Says –in commending the believers (What means): "(Because) they perform the vow and fear a day whereof the evil is wide-spreading." [Al-Insan/7].

Is it permissible for a woman who has become pure before the end of forty days of confinement to perform different acts of worship? and is it lawful for her husband to have sex with her?

Once postpartum bleeding ceases and she is certain that it won`t happen again, she becomes pure, and so she should perform Ghusl (ritual bath), pray, and fast. She can also have sex with her husband even before the end of (40) days because the minimum of postpartum bleeding is a moment, and the maximum is (60) days. However, the postpartum bleeding for the majority of women is (40) days, but this does not apply to rest of them.

Is it permissible for a praying person to make supplication in each Sajdah (a prostration) of obligatory prayer?

It is permissible for a praying person, whether praying individually, or in congregation, to make supplication during Sujood (prostration) in obligatory, or voluntary prayers. However, it is disliked for the Imam to do so since he is required to spare the prayer performers any sort of hardship, unless he is leading a certain group who don`t mind him extending the prayer.

Is it permissible for the big brother to force his sisters to wear Hijjab (Islamic wear) if their father didn`t do his part in this regard ?

All Perfect Praise is due to Allah, The Lord of The Worlds

The father`s attention should be drawn gently and politely, and the sisters should also be reminded with good words since Allah, The Almighty, Says (What means): "O ye who believe! Save yourselves and your families from a Fire whose fuel is Men and Stones." [Al-Tahreem/6]. And Allah Knows Best.