The general concept is sending the affected enemy run in horror, unable to perform any actions, away from the caster or the fearsome object. If feared enemies are immobilized, cornered or otherwise cannot flee any further, they just stand idle, effectively being paralyzed. Fear only prevents use of attacks and active skills; all other effects will work as normal, and some skills can be used while Feared.
Fear has no effect on bosses in both games.
In Diablo II, Fear has no effect on players and super unique monsters.
Fear is a more common effect in Diablo III, and it can affect players as well. Monsters can even have a Nightmarish affix that gives a chance on attack to send the enemy fleeing in terror away from the caster. A chance to Fear on hit can roll on items, mostly weapons and Helms, as a secondary stat.
CC resistance reduces the duration of Fear. Note that players, while Feared, can still use some skills and even drink potions.
Bosses are immune to Fear, but all other enemies are affected by it normally.