I'm pretty sure they only increase the swing speed, but not the hit speed. (Hit being each click you hear when mining) as mining speed is a different attribute to use time.
Terraria Community Forums
The Storm
J Bame
"Speed-modifying prefixes affect both mining speed and use time, potentially leading to the misconception that "melee speed" bonuses from armor and accessories improve mining speed. In truth, melee speed bonuses affect only use time, which is useful for combat but has absolutely no influence on mining speed."