I think weight index is only part of the equation. My Toyota came with P rated tires that only made it about 20,000 miles, mostly on Forest Service roads and no serious off roading. One road in particular was full of sharp rocks that were just too much for the P rated tire. Went to E rated all terrain tires with no regrets. Only lost about 1 mpg on fuel mileage and would never consider P rated for any sort of off road use, even just light duty tooling around on Forest Service roads.
I went through 2 sets of Toyo Open Country AT 2 and got almost 70,000 miles on each set. But they were kind of noisy and it only got worse as the tires wore down. Currently have about 50,000 miles on a set of Hankook Dynapro AT2 and have been happy with them so far.
So it is not just me - I am replacing the original tires at less than 20,000. and it sounds like my use is similar to yours - well, even my driveway is rather rough.
I think the Toyos have more aggressive tread than I need and you are not the first to mention the noise. After years of big, noisy tires on another truck I have been spoiled by my "passenger" tires.
Thank you for your response.