Rolling his eyes at Kellyanne Conway is hardly the thing I would identify as a deal breaker, OP. He maintained composure with the dangerous frauds of the Trump administration as long as almost anyone.
But yes, I do loathe him for straddling the line of celebrity and journalist, and not by accident.
Years ago, while still closeted, he was given a network daytime talk show, and the show was me-me-me. It was called Anderson and it could have been called All About Anderson, except that it didn't address the elephant in the room of his sexuality. He interviewed his friends, his mother, his fans. He talked in disturbing detail about the moments involved in his brother's bizarre suicide, he probed celebrity guests including his octogenarian mother about their sex lives but never commented on his own romantic life, and he somehow thought that showing off how boring he is would be compelling to viewers: he owns dozens of the same black shirt and wears the same shirt and jeans every day; he doesn't enjoy food and so eats the same foods every day. Who care? (No one. The show was canceled.)
After he came out, he played up the gayness quotient for attention. He hitched himself to Kathy Griffin and created that weird New Year's Eve show for CNN. I have no problem with it except that the show was all about his nervous giggling while Kathy Griffin joked and Don Lemon got drunk. Funny-ish and fun, but why does CNN, a news network, air it?
Then he latched onto Andy Cohen, who at this point is pretty universally reviled and known to be nasty and drug addicted behind the scenes. And since he spent years making his personal life a soap opera for all to scrutinize, his befriending of and then abandonment of Kathy Griffin, including condemning her own air after going silent on her because she made a controversial political joke, I think it's fair to judge him for being a bad friend who chose Andy Cohen over everyone else.
He's revealed he is a starfucker, engaging in public adoration with the likes of Kelly Ripa and her husband, et al.—the Andy Cohen entourage—and he posts Narcissus-like photos of himself lying back with his biceps in the frame over and over and over so famous and nonfamous friends can tell him how dreamy he is on Instagram.
Finally, there's his reporting on 60 Minutes. Nearly everyone's stories are serious and investigative on that show. Cooper's recent segments have included profiling the luxurious world of Monaco's royal family, another profiling the GoPro camera. There's nothing inherently wrong with his reporting, but it is damned superficial and inconsequential compared with everything else on 60 Minutes.
He does strike me as an heir on par with Meghan McCain, JFK, Jr. and others who have been handed multimedia platforms and paid heaps of money to just do and say whatever they want when what they want to say is nothing anyone else would ever be hired to put out into the world.