I can't image sleeping directly on the ground, which is pretty much what you're doing if you camp without a sleep pad of some sort. It doesn't matter much if it's an air mattress or a closed-cell foam pad, anything helps. Without one there's nothing to keep the heat from your body from being lost through transfer to the ground.
I remember camping once as a kid without sleeping pad. I remember tossing and turning most of the night and shifting from side-to-side to keep warm. I also remember shoving some extra cloths under my body to get my torso off the ground. It wasn't a great night of sleep, but I was okay.
Looking back I suspect I was able to "get away" with not having any ground insulation because I was using a big, thick heavy 20 degree synthetic sleeping bag. My suspicion is that down or modern synthetic bags, which have much thinner insulation, would not work at all without a ground pad.
Sleeping pads help with comfort, but they're critical for maintaining body temperature during the night.