Literally it’s more like “non-speakers”, though, isn’t it? Nie + mowić = Not + to speak.
So, maybe in contemporary Polish the word has been polished to mean “mute”, but could be that they were “those damn non-speakers [of our Polish] across that river-thing!”



At least Wiktionary completely agrees with you!