In UK (and probably, mostly universal
) :
If you are standing on public ground (e.g.the high street) :
You can photograph any building you like. It's not the taking of the photograph but the usage of the photograph which is under restriction. Put simply, you may not make money from a photograph of any privately owned building without the permission of the owner. Beware, such buildings can include museums, churches, and other buildings which you might think are public buildings! Think of it this way, just about everything in this world is owned by someone.
If you are standing on private property :
Get permission to take photographs. It is just simpler to do this, to avoid any questions or arguments later. No hassle, just ask the nearest official if it's ok or not. Most tourist-related places will display a notice in this regard in any event. I never understand the fuss people make about asking permission, honestly what's the hassle in asking; it's courtesy as well as saving a whole lot of hassle later.