I look at it this way: if they have an impact on the game, then they have an impact on fantasy. It could be minimal or it could decide the outcome of the game. No harm in arming yourself with some extra knowledge. In my nearly 30 years of watching baseball, I can say this for sure: you don’t hear the names of good umpires very often. Think about it. Everyone knows Angel Hernandez’ name, and it isn’t because he’s good. You tend to hear the bad umps’ names way more often. Food for thought.
Edit: BTW, this would all be rendered moot with robot umps.