Open source baseband only won't be enough, there are plenty of CPUs/MCUs capable of running backdoors/malware spread on any modern board (heck, even battery controller often runs on a full fledged MCU). Until we have an open hardware + software, we will never be safe from government and other well-funded crooks.
Let not the perfect be the enemy of the good though.
Every time I hear someone asking for some improvement in openness and security, someone jumps in to point out how something else will circumvent it. Fine -- but how is anything going to get done if we sit paralysed a proliferation of worries?