My experience, as someone who isn't a gnarly-experienced 4x4 guy, but has driven both of those places in my F250 4x4, FWC rig:
If it's not wet and muddy there's no need for 4x4 to get to Toroweap. The last tiny bit to get to the campground is over a few spots of bedrock, but all kinds of people/rigs go in there.
When I drove the White Rim trail this past April I was in 4-low in a few steep spots -- both steep up and steep down. Even with those steep spots and a lot of places driving on bedrock, it's not technical. It's possible that people have done it in any kind of rig -- even sedans (either as a stunt or as stupid) -- but I'm glad I had 4x4 for a few spots.