Fiction shouldn’t be hyper-realistic. It’s fiction, and you’re the storyteller. Fiction isn’t meant to mimic life, but to analyze it or somehow represent it. Writers use fiction to make sense of events or characters, to explore possibilities, to reshape memory, to make a statement, or simply to have fun. Readers use fiction to escape life or to relax or to go on an adventure in their living rooms. If they wanted to experience life, they can just go out and live it, right? Fiction =/= real life, and it never should. It should say something more than real life does.
When people think of mannerisms, some things that come to mind are affected speech and nervous gestures. Other possibilities include dialect/accent or diction (someone who is college-educated might use different vocabulary or sentence structure than someone who is life-educated).
These can be used effectively to paint a character’s personality, but they can also “generify” it and bog down the writing. Often, these sorts of mannerisms are empty, wasted words on the page. These are the worthless “um”s and “uh”s that riddle natural speech, but shouldn’t have any place in fiction. These sorts of pauses and meaningless phrases slow down the story, turn the characters bland, and just generally don’t serve any purpose. Besides, writers need to trust that readers are smart enough to use their imaginations to fill in these pauses and nervous hand wavings or hair brushings.
As for dialect, again, authors have used it well, but use it too heavily or too often and you risk irritating the reader, who then has to sit there puzzling out each line of dialogue. This slows or even stops the reader, and yanks them right out of your story. If you’re writing dialect, best to have it flow as naturally as possible, and allow the reader to imagine the accent themselves.
Bottomline, if it doesn’t serve the story or develop the character, or it slows the story down, get rid of it. Mannerisms or style of dialogue should add flavor, a little bit of spice to the cooking pot. It shouldn’t obliterate the rest of your ingredients.