This is the reason you want a Subaru. I live in a rural town that has only 2 paved roads, the rest are all gravel. My 2010 Forester has survived wonderfully for the last 10 years on these roads. No repairs yet other than brakes. I expect my Ascent to be just as good.
I had a buddy who took his Forest off-roading all the time and was near the limits of it's capabilities. Basically he beat the heck out of it. He did need new shocks at 90k miles but other than that it worked perfectly!
I've had my Ascent on several dirt roads and it handles it very well. I'd agree on the 18" wheels (even if you want a fancier trim, you can still buy 18" rims/tires aftermarket or possibly get the dealer to swap).
I've got a couple rattles already, and the trim piece worked loose off of the lower part of the driver's front door in four months and 4000 miles. I am mostly on pretty smooth roads, although I drive a mile-and-a-half on rutted, washboard roads about four or five times a month to our other house. If you keep your speeds reasonable you'll be okay. With my 18" winter wheels & tires (considerably softer tread and sidewall, in addition to the larger sidewall) it took the impacts much better than with 20" stock tires. I had mentioned in other posts from day one that the suspension sounded like it's about to break every time it hits a good-sized bump, but it does absorb the impacts quite well and I haven't had any suspension issues.