It was 1875 when the Milton Library Association was organized. The first physical library was one bookcase at Welch's Store on Union Street. The library would later move to a building on Front Street, then to Palmer Building on Federal Street.
In September of 1978 taxpayer funds from the federal government were used by Sussex County to purchase the Conwell Building on Union Street, and in July 1980 under Sussex County management, the new library opened.
The demand for its services would grow as the 20th century neared its end so in 1998 an adjoining property was purchased leading to a library expansion.
In addition to books the library provides cds, dvds, access to computers and hosts meetings and events for a number of organizations.
the marker on the outside wall was erected in 2004.
