In a report Thursday, the Center for Responsible Lending, a nonpartisan research firm, called on lenders to modify mortgage terms for subprime borrowers, saying the financial damage caused by subprime lending will fester without modifications.

The organization pointed out that 1.5 million families have lost their homes and another 2 million families are facing that fate.

“Because the foreclosure crisis is at the root of this recession, the continuing flood of foreclosures stymies any chance of real economic recovery,” says Center for Responsible Lending president Michael Calhoun. “Any economic stimulus would be a Band-aid solution unless we stop the hemorrhaging in our housing market.”

In its report, the group urges the Treasury to require recipients of funds from the governments’s Troubled Assets Relief Program to adopt streamlined modification programs and also guarantee sustainably modified mortgages against default.

The Center also says Congress should act fast to lift the ban on judicial loan modifications, which would prevent hundreds of thousands of foreclosures without requiring any taxpayer funding.