Spelunker is a perfectly reputable term coming from the Greek σπήλαιον and Latin spelunca (meaning cave or grotto). Speleology is the scientific study of caves and those interested in the field typically belong to the NSS - National Speleological Society.
The term spelunker was routinely used to describe cave explorers until a few decades ago when it began to take on negative connotations and came to applied sarcastically to unprepared and inexperienced yahoos who decide to explore a cave. They typically are wearing sneakers, cotton clothing (an invitation to hypothermia), and carry a single flashlight. If the cave has a pit, they'll slide down a piece of clothsline with no thought of how they're going to climb back out. Sometimes a group enters a cave because they think it's a neat place to party. They often need to be rescued from a cave (or worse yet, their bodies need to be recovered).
So, if you ever run into someone who is serious about caving, don't call them a spelunker - odds are they're driving around with a popular bumper sticker in the caver community - "Cavers rescue spelunkers!"
My family of "spelunkers" underground a few years ago (not as bad
as it looks, it's Ape Cave, a lava tube near Mt St Helens in Washington
State and a relatively safe and tame underground environment).