A mould grows profusely on a moist slice of bread because it gets the moisture and nutrients both from it.
But, in a dry slice of bread, the moisture is not available and it only gets the nutrients from it, that is why they don't grow that profusely.