Don’t worry about your lack of vocabulary. The words you know are enough to speak well!
“I can’t remember the right word when I am talking in English!” This by far is the most common answer I have received to my question “What is the main problem you face when speaking in English?”
Now, for an average person who does not spend his days and nights thinking about why someone finds it difficult to speak confidently in English, this may sound like a fully legitimate explanation as to why the perosn does not feel confident in using English.