The virus survives in water droplets or aerosol which fall down faster than smoke.
You can easily smell cigar smoke at 10 meters but you are ver unlikely to get infected at the same distance, current estimates put it to 4 meters in hospital settings.
Also,the amount of virus you receive matters, and distance decreases it.
You are right about viral load being a factor. You are not right about only macro droplets being a problem. Micro dropplets, that are suspended in air and do not "fall to the ground" soon carry enough of the virus to be infectious in accumulation. This is why indoor areas are problematic regardless of the 6 foot spacing, as the air just gets saturated over time unless you have a continual strong crossdraft.
The smoke analogy is not perfect, as it's composition is less moist, but it is closer to the truth than the '6 foot' model that was itself already a compromise between the now accepted as disproved macro fluid projectile model and economic concerns.
There is not much difference between smelling cigars and smelling my neighbours smoking weed, is there?
It never occurred to me, that being able to smell this might indicate danger.
Not OP, but I had the same theory
but with cigarettes. According to Google results, cigarette smoke particles are 0.25 micron at the largest, Sars-CoV-2 virus is 0.12 micron, so about half. They're both so light that gravity probably has little effect.
So, sharing a room with someone sick probably gives you a high chance of encountering the virus, but outdoors one hopes the concentrations would be so low that our bodies can fight it off.
And if you're wearing a mask and can still smell cigarette smoke, then surely it means that mask would let the virus particles through as well.
And in fact, this is how you test the respirator fit in professional setting. You don your respirator, and then somebody sprays a funny-smelling substance. If you can smell it, it means it doesn't have a proper seal on your face.