The Milford Housing Development Corporation has withdrawn from the Dutchman’s Harvest workforce housing project in Lewes, citing significant challenges such as rising costs and interest rates.