Some of the world's biggest private equity firms are growing increasingly nervous that the sector is heading for a downturn after years of stellar returns fueled by low interest rates and strong demand for alternative investments.