There may be several reasons you might want to quit your job: Human relationships, salaries, career design, etc.
Whatever reason you have, the fact you don’t get along with your boss or colleagues is not trivial. You share the same time and space with them every day. Therefore, if you are uncomfortable with the current work environment, it is reasonable that you want to make a big decision to change your job.
But as you may have already noticed, this is just an “ad hoc solution.” You will likely have to work with someone whom you are uncomfortable with. Instead, it is unlikely that you will be able to work surrounded by only those you like.
The phrase “don’t fit well” may include something like harassment that you really have to avoid. Therefore, I do not deny changing jobs for that reason. But at the same time, when considering a career change, you’ll need to think about how you can behave to minimize stress among those who don’t fit.