From what I know, Bags of Potential are dropped by enemies at random. It's kind of rare, but it shouldn't be impossible to get.
Thorium ore can also be found at random below cavern level. Most commonly found near lava, from my experiences.
