Hmm... A better way to think of it might be along the same vein as potions. We have buttons for buff potions, health potions, and mana potions. All three are used in certain circumstances, but not all the time.
A misused potion sets you back in terms of ingredients (Magic Mirror in terms of time). If it's during a boss, a misused health potion could only heal a fraction of its max and leave you with potion sickness (Magic Mirror would remove you completely and in most cases despawn the boss).
Because of this you don't want potions in your Hotbar so you don't use them by mistake; plus it makes more sense to have weapons or building materials available. Yet you still want ease of access to them when it comes to the times you need them. A quick health potion when in need can give you the health necessary to survive (Magic Mirror in a time of need can be crucial to saving your life, especially on mediumcore/hardcore). Those few seconds it takes to use a potion (or mirror) without a button can be the difference between survival and death.
Neither potions nor the Magic Mirror are items that get used constantly, but they're items people always carry with them because they do in fact need them. A button for them seems applicable.
So far a couple of people have stated why they think it's not a necessity to have. It's not a necessity, more of a convenience factor. I would like to ask this. Is there any reason why a Magic Mirror button would be a bad thing? In what way would it hurt the game?