Is it permissible for a woman who has become pure before the end of forty days of confinement to perform different acts of worship? and is it lawful for her husband to have sex with her?

Once postpartum bleeding ceases and she is certain that it won`t happen again, she becomes pure, and so she should perform Ghusl (ritual bath), pray, and fast. She can also have sex with her husband even before the end of (40) days because the minimum of postpartum bleeding is a moment, and the maximum is (60) days. However, the postpartum bleeding for the majority of women is (40) days, but this does not apply to rest of them.

What is the expiation for breaking fast due to being on a journey, or being sick, or being in a state of menstruation?

No expiation is due on the aforesaid categories, but they are obliged to make up for the missed fast. However, if any of them failed to do so while being able to, and the next Ramadhaan has come, then making up for those days is incumbent on him/her , and paying the ransom as well. And Allah Knows Best.

Does the clipping of nails nullify ablution?

No, it does not, and it is preferable to wash the hands afterwards.

Is it permissible for the big brother to force his sisters to wear Hijjab (Islamic wear) if their father didn`t do his part in this regard ?

All Perfect Praise is due to Allah, The Lord of The Worlds

The father`s attention should be drawn gently and politely, and the sisters should also be reminded with good words since Allah, The Almighty, Says (What means): "O ye who believe! Save yourselves and your families from a Fire whose fuel is Men and Stones." [Al-Tahreem/6]. And Allah Knows Best.