It's unexpected for the users.

Options:

• 1 clear button that removes locations only.
• Clear locations, clear all (2 buttons).
• Clear dropdown with options.
• 1 clear button -> dialog with options and Cancel Delete buttons.