Libertarians are very nice(in theory) but they often forget that their ideology only works with the equally minded. Many, maybe most, people love to be led and thus are not able to be "free". These are exactly what the powerful use against the "freer souls".
Being a Libertarian IMO is more wishful thinking.
Wanting things to change so drastically.
But it will never happen.
You are correct where in order for it to work we would all need to be relatively the same in mindset. Or have simple rules that all agree upon.
That's the problem. It's hard to get people to agree on anything, the more people involved the harder it becomes.
That's why a true Libertarian society would only work on a small scale.
Too many minds.