Falling, no matter how you cut it, is a bad thing. Falling from a couple hundred or a few thousand feet in the air is really bad. Whether it's due to a midair collision, or just because the passenger is leaning over the side to get a better view, at least a few people fall out of balloons each year [source: Facworld].
Falling primarily occurs in ballooning, but it can also be an issue in paragliding and hang gliding. If it's not abundantly clear at this point, air sports are incredibly dangerous on their own, so putting yourself into needless danger, like hanging on the outside of a balloon or not being properly secured, is a bad idea.