I am invisible

When I speak English to people in the office, I suddenly turn invisible to my non-English speaking colleagues. I was just having a conversation with Raju and, in mid-sentence, I was interrupted no less than 5 times by various people who did not even look at me when they began their conversation with him, and continued until they’d finished whatever it was they had to say.