The Prime Minister made the untrue allegation in the Commons that Sir Keir had failed to prosecute the notorious sex offender while he was director of public prosecutions.
He later dialled back, admitting Sir Keir had not been directly involved.
Mr Kwarteng was asked whether he would have used the same words as the PM, and told Trevor Phillips On Sunday on Sky News: “I think it’s entirely legitimate… it depends what the context was.”
“In that context, I think it was perfectly reasonable to mention the fact that Sir Keir had apologised. Sir Keir himself apologised on behalf of the organisation that he led about the fact that they failed to prosecute Jimmy Savile.
“So the fact that he apologised suggests that he does at some level bear some responsibility.”
Mr Kwarteng said he was not saying Sir Keir had “personal blame”.
Mr Johnson accused the Labour leader of having “used his time prosecuting journalists and failing to prosecute Jimmy Savile” on Monday as he came under pressure in the Commons over alleged lockdown breaches in No 10.
The remark sparked a furure with former home secretary Amber Rudd accusing the Prime Minister of a “Trumpian” attempt to link the Labour leader to Savile.
Amid a welter of criticism, Mr Johnson sought to “clarify” his comments on Thursday.
Speaking to broadcasters in Blackpool, the Prime Minister said: “I want to be very clear about this because a lot of people have got very hot under the collar, and I understand why.
“Let’s be absolutely clear, I’m talking not about the Leader of the Opposition’s personal record when he was DPP and I totally understand that he had nothing to do personally with those decisions. I was making a point about his responsibility for the organisation as a whole.”
But the controversy only intensified as Mr Johnson’s long-standing policy chief Munira Mirza dramatically the same day over her boss’s comments.
In a resignation letter, she said: “I believe it was wrong for you to imply this week that Keir Starmer was personally responsible for allowing Jimmy Savile to escape justice.
“There was no fair or reasonable basis for that assertion. This was not the usual cut and thrust of politics.”
Chancellor Rishi Sunak also distanced himself from the Prime Minister’s attacks, saying he would not have made the comments.
Health Secretary Sajid Javid later also distanced himself from the controversial attack.
Mr Javid said the Labour leader had done a “good job” when he was director of public prosecutions and deserved “absolute respect” for his work in the post.