Look at the image below. On the left is a nice amethyst crystal. No one would deny that this is crystalline. On the right is a piece of kaolinite - a soft, powdery clay mineral. Is this crystalline? Turns out it is indeed crystalline.
Crystalline does not mean it only occurs as a crystal, crystalline means that the atoms which link together to form the mineral do so in an orderly, repeating, fashion. This mineral quartz and silica glass, for example, both can be denoted by the formula SiO2 - the difference is that in quartz, the silicon and oxygen atoms are linked up in a hexagonal lattice. In glass, they're linked together, but not in a regular repeating fashion. Glass is therefore not crystalline.
Back to the lump of kaolinite clay (Al2Si2O5(OH)4) above. It actually has a crystalline structure as shown below (it's a bit complicated and difficult to see in 2-D).
The final characteristic of minerals, having a well-defined chemical composition, sound simple but also has some complications we can mention. Most common minerals have simple chemical formulas - gold (Au), halite (NaCl), quartz (SiO2), calcite (CaCO3), hematite (Fe2O3), and orthoclase feldspar (KAlSi3O8) are examples.
Other minerals, however, have formulas that are a bit more odd looking. One fairly common mineral, called hornblende, has a formula that may be written as:
It's not an exact formula. It can have varying amounts of different elements. Is that really a well-defined chemical formula? I guess so since hornblende is a mineral.
So there it is. To a geologist, a mineral is "any naturally occurring inorganic solid that possesses an orderly crystalline structure and a well-defined chemical composition."
One oddity of this definition, by the way, is that ice should be considered a mineral. It's naturally-occurring, inorganic, solid (certainly tonight here in the Hudson Valley since it's 11° F out right now), has an orderly crystalline structure, and a well-defined chemical compositon (H2O).