Several venture capital professionals, and founding members of leading VC firms, are setting up their own firms as investors look to back smaller funds, hoping their nimbler approach will deliver bigger returns.