One would think that offering potentially big rewards would inspire workers to create better work. A study found a more complicated picture.