Loans backed by investment portfolios have become a booming business for Wall Street brokerages. Now the bill is coming due—for both the banks and their clients.