Milford was once home to numerous granite quarries, which produced a stone that was used, among other things, to make the pillars for the Treasury building in Washington, D.C. - pillars that can still be seen on the American $10 bill. Its nickname remains "The Granite Town," although only one small quarry is in operation as of 2007.
Milford was a stop on the underground railroad for escaped slaves. It was also the home of Harriet E. Wilson, who published the semi-autobiographical novel Our Nig: Or, Sketches in the Life of a Free Black in 1859, making it the first novel by an African-American published in the country.






