All perfect praise be to Allah, The Lord of The Worlds; and may his blessings and peace be upon our Prophet Mohammad and upon all his family and companions.
There is no harm in discontinuing Tawaf or Sa`y to perform prayer. If someone starts the Tawaf or Sa`y and then the Iqaama (call for prayer) is called before he completes it, then he should discontinue his Tawaf or Sa`y and pray with the congregation. When he finishes the prayer, he continues his Tawaf or Sa`y from where he left off because performing all the circuits of Tawaf successively isn`t an obligation.
Al-Khateeb Ash-Shirbini (May Allah bless him) said: "If it was called for prayer while he was performing Tawaf, then he should break it and perform the prayer because prayer has a certain time span; whereas, he can continue Tawaf from where he left off." He added: "If what hindered him from continuing his Tawaf has ended, then he should continue his Tawaf from where he left off. For example, if his ablution was broken while he was performing Tawaf, then he should re-make ablution and continue Tawaf from where he left off because succession isn`t a condition in Tawaf and ablution, for both of them are acts of devotion that may be broken by something." {Moghni Al-Mohtajj}.
Accordingly, your (questioner) breaking Tawaf to perform prayer, then continuing Tawaf from where you left off is valid, even if you weren`t able to continue your prayer due to crowdedness. And Allah knows best