The main downsides of fixed gear bikes are that these types of bicycles are not adapted to all riding conditions. If you are riding somewhere with lots of hills, you might find yourself missing your gears are freewheel pretty quickly!
It's also harder on the knees. Up to a point it can probably help you build stronger legs and knees, but you could still get part of that benefit by learning good pedaling technique on a regular bike and avoid the extra wear & tear.
Also, something important to remember: If you're braking with the rear wheel, you're not getting maximum braking power because during deceleration, weight is transferred to the front wheel. This is another reason why it's important to have brakes on a fixie.
But the best way to find out of a fixed gear bicycle is right for you is to try one. Just make sure you do it in a safe environment; you don't want to go riding in traffic the first time you try a fixie...
Planet Green Video: G Word's Electric Bike Conversion