Currently the "Display Field" is set globally, but sometimes you want the display field to be set at the view level.
For example, imagine a project management app. When you're selecting members to remove from your project, you want to see the Email Address as the Display Field because it's the most user-friendly identifier. However, this forces you to use email addresses as the Display Field throughout the entire app, which means any User Connection picker on any public pages will expose the email address field.
I guess this feature was requested already twice.