No, it's not rude. People who think it's rude are simply mad because they can't listen in on a conversation that doesn't involve them.

When my wife was alive, she and her relatives had plenty of conversations in their native language while I was in the same room with them.

I didn't get upset because I had no need to know what they were saying.

Unless you are supposed to be participating in their conversation, there is no need for you to understand what two people are saying to each other.

I beg to differ, but to each their own. If someone is in MY house, and in my presence, I do think I should be involved in the conversation. It has nothing to do with fear of what they are saying or even wanting to know. It is just flat out rude to do that in others domains. I would not do it in their home and I expect the same courtesy.