The City of Boston's retirement board on Wednesday voted unanimously to end its relationship with money manager Kenneth Fisher, whose firm has lost almost $1 billion in assets after allegations he made disparaging remarks about women last week.