Leon G. Cooperman plans to return money to outside investors and convert his firm, Omega Advisors, into a family office by year’s end.