There should be a global shortcut to open the directory on the most recent list.
At my workplace i have to use a Mac, and there is a tool called DefaulFolder X, which has this function.
Basically you press Ctrl+Alt+Down arrow, and it jumps through the recent directory list, one by one.
By “jumps through” i mean opening it in the currently active window.
(Ctrl+Alt+Up goes forward in history.)
It would be a really quick way to browse through your folder history, and it eliminates the constant Middleclick->Recents->Choose Folder procedure.
Usually when you are working on something, the 2 most recent directories is as far as you need to go.