A house with a dirt crawl space is dysfunctional, and a dysfunctional house is not worth building - at any price. Think about it this way, what you would and would not do in a basement, the same rules apply to a crawl space. We would never create a dirt floor basement - so why would we have dirt floor crawl spaces?
Your house is one building. You can't have one part of the building that is sick and another part this is healthy. The crawl space air is bad. It's damp. It's cold if it's winter. It's full of mold spores. And it smells. Whatever is in your crawl space air is in your house and affecting you, whether you spend any time in the crawl space or not.