It does not work to remove the selection of someone else's fortified wall.
Windows 10 64 bit, 8 Gm RAM, Intel(R) Core(TM) I7-4500 CPU1.80 GHz 2.40GHz GPU: AMD Radeon R7 M265, Intel(R) HD Graphics Family. Version game fresh.
Watch the video. I can choose my wall. And to make a choice. But I can not remove the choice from someone else's wall.
I am forced to use neutral ground. This is not convenient and will become a problem if the no land is not (or not noticed) - because the workers are digging someone else's wall for a very long time, and in most cases, this is a useless occupation (and there are workers to do).
I can not reproduce this, neither on the map save you attached nor on another map I tried.
Seems I can always remove my selection with one click without issues.
Might it be that you had an AI player playing for yourself when you recorded the video and that AI player was messing with your selection?