At speeds up to 100 km/h (62 mph) I would say you can drop the windblocker, only raising the rear window. Longer journeys at higher speeds make it more comfortable with the (very easy) windblocker.
But hey, when I drive topless it's because I enjoy life, no stress at high speeds