Do you line dry? Line drying will make any fabric, new or old, diaper or clothing, feel very stiff and crunchy. Dryer drying will soften them up instantly or you can play around with part line drying and part dryer drying. Most find that drying about halfway in the dryer and then finishing up on the line offers softness almost equivalent to 100% in the dryer.
If you already dry in the dryer then I'd suspect hard water mineral build-up and/or detergent build-up (hard water can cause both). You can try stripping your diapers (which just means putting them through one or more washes with just water). If you have hard water I'd add some water softener (like Calgon) and/or vinegar. You can even try a soaking in lots of vinegar (like 3-5 cups or more in a washer full of water) to get things started, the acidity in the vinegar will help break-down the build-up.
All that said, it's highly unlikely they'll irritate your little one's skin. It doesn't feel comfy to us moms and we worry, but honestly the vast majority of babies don't even notice! You could also always try using a fleece liner, that way you'd have a layer of softness right next to his skin. Fleece stays soft even when line dried plus it provides great stay-dry benefits.