Both hammer and axe have very limited uses, but it wouldn't make much sense for the pickaxe-axe hybrids to let them change and destroy blocks.
Which means you would have another tool carrying around at the time you have hybrid tools.
Destroying walls happens when holding the button, but changing tiles only at button presses, so I think it should be good if the button is held before focussing on a block.