I want to ignore square brackets when using javascript regex

I want to ignore square brackets when using javascript regex

I am using javascript regex to do some data validation and specify the
characters that i want to accept (I want to accept any alphanumeric
characters, spaces and the following !&,'\- and maybe a few more that I'll
add later if needed). My code is:
var value = userInput;
var pattern = /[^A-z0-9 "!&,'\-]/;
if(patt.test(value) == true) then do something
It works fine and excludes the letters that I don't want the user to enter
except the square bracket and the caret symbols. From all the javascript
regex tutorials that i have read they are special characters - the
brackets meaning any character between them and the caret in this instance
meaning any character not in between the square brackets. I have searched
here and on google for an explanation as to why these characters are also
accepted but can't find an explanation.
So can anyone help, why does my input accept the square brackets and the
caret?