He told the News of the World: "Simon and Dannii have both had more work done since last year. I'm the only judge who's had nothing done, not even my teeth. They are still my own."
But then remembering stunning Cheryl Cole is now on the panel, he added: "She doesn't need it. Not yet anyway, she's too young.
"But the other two have definitely had a few touch-ups since last year. Just look at them."
Simon, 48, bristled at Louis' sideswipe and then turned the tables by offering to pay for him to have surgery.
He said angrily: "Louis should be looking in the mirror and realising what he needs doing. I've actually offered to pay for his teeth, one at a time."
Louis, 56, caused a storm last year when he told the News of the World Simon had work on his famous pecs.
He said: "Simon's definitely had a boob job. They all have new tits."
The boobs of Xtra Factor presenter Holly Willoughby have also hit the headlines. She has refused to stop wearing plunging necklines, despite criticism during Dancing on Ice that her cleavage distracted viewers.
Holly, 27, said: "I'm not at all worried about wearing plunging dresses at all. I do have curves and a bottom so I have to make sure I dress well. I always wear what I think are beautiful and glamorous dresses."
She added: "I dress well for my shape and that's my secret."
And Holly blamed other women for kicking up about her curves.
She said: "I think women focus on it so much themselves, they are the worst culprits when it comes to being critical of other women.
"The majority of guys I know don't want that skinny thing. It's not them creating the fuss, it's women.
"We are our own worst enemy I think.
"Sometimes, it's too easy to see a blonde girl on TV and think she's a bit of a bimbo."