If you are a perfectionist like me, you probably notice when a picture is just a bit askew. It's like the person was standing on a hill to take a picture or one of his legs is shorter than the other. Whatever the reason, it just doesn't look right. So when you are trying to hold your camera straight, make sure it's straight.
However, don't feel like you always have to have your camera in either the horizontal or vertical position. Sometimes tilting your camera off of the axis can make your image much more interesting - just make sure it looks like you did it on purpose.
Check out these examples:
