Those who build walls are their own prisoners.

A quote from The Dispossessed. Also:

Like all walls it was ambiguous, two-faced. What was inside it and what was outside it depend upon which side of it you were on. (Ursula K. Le Guin, The Dispossessed)

Only once you build a wall to keep others on their side do you now have to worry about them getting over to your side.

Johann Hari in Lost Connections on inflated egos being like walls:

Our egos protect us. They guard us. They are necessary. But when they grow too big, they cut us off from the possibility of connection. Taking them down, then, isn’t something to be done casually. To people who feel safe only behind walls, dismantling their walls won’t feel like a jail break; it will feel like an invasion.