Every 1 armor means the damage it takes to kill you is extended by 1% of your hitpoints. At 100 armor, it takes 100% more damage to kill you, ie it requires double damage, ie you're taking half-damage. At 300 armor you're taking 25% damage, etc.
Magic resistance works in the exact same fashion, except it does so against spells only. There is no overlap between armor and magic resistance at all. (Ex. Having 500 Armor would have no impact on spells, having 500 Magic Resistance would have no impact on physical attacks)
Q: What order is armor penetration/MR penetration applied in?
From top to bottom
Armor/MR Reduction
Flat/Linear Armor/MR Penetration
% Based Armor/MR Penetration
Q: Can you reduce armor or magic resistance below 0?
Yes, although the only things that can reduce below 0 is Armor/MR Reduction. Both percentage and flat penetration do not have an effect after they get to 0 armor/MR.