Investment banks and brokers find themselves reeling by the prospect that what they considered routine for decades could open them up to criminal charges.