All praise is due to Allah, Lord of The Worlds. Peace and blessings be upon Prophet Mohammad, and upon all his family and companions.
Whoever utters an oath like phrase without the intention of swearing is not sinful from a Sharia point of view since it does not mean to swear but it is something which is uttered according to their habit. Allah, The Almighty, says: “God will not take you to task for a slip in your oaths, but He will take you to task for that to which you have pledged oaths” {Al-Mai`dah/89}.
Al-imam Al-Nawawi (God bless his soul) reported that the Prophet (PBUH) said: ” If one has to take an oath, he should swear by Allah”. This Hadeeth addresses the one who pledges himself by the oath which involves the glorification of the sworn by through comparing them to Allah, The Almighty. But, the person swears with the intention of pledging himself to that oath is sinful since swearing should be by Allah alone for the Prophet (PBUH) says: ”Whoever has to take an oath should swear by Allah or remain silent." (i.e. He should not swear by other than Allah). [Agreed upon] Thus, neither his oath is valid nor liable for Kafara [an expiation].
In the book, [Hashiat Aljaml Ala Sharh Alminhaj,(5/288)]. An oath is not pledged if sworn by a creature like saying “I swear by the Prophet or by the Kabah”, but it is disliked as mentioned in the Hadith: ”Verily! Allah Forbids you to swear by your fathers. If one has to take an oath, he should swear by Allah or otherwise keep quiet.” One should avoid such an act as it could lead to underestimating the dignity of the Prophet (PBUH). And Allah knows best.