Along Route 209 near Kingston, between Route 28 and Sawkill Road are a series of outcrops along both sides of the road. Cliffs and hills of shale interbedded with thin layers of sandstone. These rocks belong to what's known as the Mount Marion Formation (after a town up the road a bit) which is part of what's called the Hamilton Group (a regionally extensive series of rock formations).
If you know where to look along these outcrops, being careful not to get run over by all the speeding cars on Route 209, it's relatively easy to find weathered out coral fossils like the one below.
This is actually a fairly large specimen, most are a bit smaller (some of the smaller ones actually preserve better details as well). This belongs to a group of corals (phylum cnidaria, class anthozoa) that are called the rugose corals (many people call them horn corals because of their shape). Rugose refers to their wrinkled appearance (this one's been weathered smooth).
This particular species lived during the Devonian Period of geologic time (359-416 million years ago). A little searching can also turn up shelled fossils called brachiopods and even a marine snail (gastropod) or two. Below is a reconstruction of a Devonian seafloor with some rugose corals (among other things).
This is basically what Kingston looked like during the Devonian Period. A shallow, warm, subtropical sea floor. The evidence is right there laying in shale chips in ditches along the side of the highway!