Igneous rocks tend to be quite hard, because most of them form from molten material cooling, so they are usually really well glued together with very little space or weakness in them (that’s not to say that they don’t get broken or stressed later). I work in NUIG and most of the campus has a rock called migmatite under it – this is a form of granite and it is particularly hard. When they work on the campus here it always takes much longer to dig or drill foundations in this rock.
Diamonds are the hardest mineral and they are made of carbon. They form really deep beneath the ground (up to 100km deep) where they are subject to extreme pressure and heat. The exact same carbon forms graphite if you find it nearer the surface (where temperatures and pressures are a lot lower). Graphite and diamonds are the same thing (chemically), they just formed in different places.