Use a stack to track open brackets as you scan left to right.
When you see an open bracket, push it. When you see a close bracket, the top of the stack must be its matching open bracket. If it matches, pop. If not, the string is invalid.
At the end, the stack must be empty. Any remaining open brackets mean unmatched opens.
The stack naturally handles nesting because inner brackets get pushed after outer ones and must close first.