One of the most impressive mineral specimens unearthed in New York City is the famous Kunz garnet (also called the sewer garnet or the subway garnet) discovered in 1885. It's a massive 9 pound, 10 ounce almandine garnet, seven inches in diameter, dug up on West 35th Street near Broadway.
Today the area looks like this:
Anyway, Kunz presented the garnet to the New York Mineralogical Club which later donated it to the American Museum of Natural History in Manhattan. I believe it's on display in the New York City Minerals collection but I haven't been there in a few years.
Hundreds of spectacular mineral specimens have been discovered in the bedrock of Manhattan (primarily in the 19th century when a lot of the subway, skyscraper, and water tunnel excavations were going on). The Kunz garnet was found in the Manhattan Formation (also called the Manhattan Schist), a rock unit which underlies much of the island and formed from the metamorphism of sedimentary rocks by a mountain-building event some 440 million years ago (Taconic Orogeny).
I'll write more about the Manhattan Formation another day.