Normally refresh is called by right clicking on an artifact (e.g. a folder) and choosing refresh, not from the drop down menu on the top right of the panel.

It's very counterintuitive having it up there, it should be available on all folders views (and I would argue not in that menu at all)