Squeamish from the start about pursuing profits in one of the darker corners of finance, established firms this year slowed their already halting efforts to make a business out of Bitcoin mania.