Growth in Milton has town leaders considering a new impact fee that would charge residential developers $13,127 for each unit of new construction.