Mutual funds that mimic hedge fund strategies — the so-called liquid alternatives sector — were among the hottest investments just a few years ago. Despite lagging returns and setbacks at several noteworthy funds, retail investors until recently have stuck with them even as they have pulled billions of dollars out of other funds.