Yeah. It sucks. I have a roof rack (25 lbs?) with Yakima rocket box (another 25 lb?) and sometimes carry a 14' Old Town Guide canoe (50, 60# ?) as well. Bench pressing it up isn't too tough (start with the back first,then the front,) but putting it down without smashing yourself in the top of the head can be a rather um... "abrupt" procedure. It's a bit difficult to make sure the sides are all stuffed in properly from the inside, so you have to go around the outside and do it a little bit at a time. So definitely doable, but a total PITA.
Just getting the canoe on and off without dropping it and then strapping down is a bit of a task as well.