A fundamental question in the emotional memory literature is why emotion enhances memory in some conditions but disrupts memory in other conditions. For example, separate studies have shown that emotional stimuli tend to be better remembered in long-term episodic memory (EM), whereas emotional distracters tend to impair working memory (WM) maintenance. The first goal of this study was to directly compare the neural correlates of EM enhancement (EME) and WM impairing (WMI) effects, and the second goal was to explore individual differences in these mechanisms. During event-related functional magnetic resonance imaging (fMRI), participants maintained faces in WM while being distracted by emotional or neutral pictures presented during the delay period. EM for the distracting pictures was tested after scanning and was used to identify successful encoding activity for the picture distracters. The first goal yielded two findings: (1) emotional pictures that disrupted face WM but enhanced subsequent EM were associated with increased amygdala (AMY) and hippocampal activity (ventral system) coupled with reduced dorsolateral PFC (dlPFC) activity (dorsal system); (2) trials in which emotion enhanced EM without disrupting WM were associated with increased ventrolateral PFC activity. The ventral-dorsal switch can explain EME and WMI, while the ventrolateral PFC effect suggests a coping mechanism. The second goal yielded two additional findings: (3) participants who were more susceptible to WMI showed greater amygdala increases and PFC reductions; (4) AMY activity increased and dlPFC activity decreased with measures of attentional impulsivity. Taken together, these results clarify the mechanisms linking the enhancing and impairing effects of emotion on memory, and provide insights into the role of individual differences in the impact of emotional distraction.