What processes or mechanisms mediate interpersonal matching of facial expressions remains a debated issue. As theoretical approaches to underlying processes (i.e., automatic motor mimicry, communicative intent, and emotional appraisal) make different predictions about whether facial responses to others' facial expressions are influenced by perceived gaze behavior, we examined the impact of gaze direction and dynamic facial expressions on observers' autonomic and rapid facial reactions (RFRs). We recorded facial electromyography activity over 4 muscle regions (Corrugator Supercilli, Zygomaticus Major, Lateral Frontalis, and Depressor Anguli Oris), skin conductance response and heart rate changes in participants passively exposed to virtual characters displaying approach-oriented (anger and happiness), and avoidance-oriented (fear and sadness) emotion expressions with gaze either directed at or averted from the observer. Consistent with appraisal theories, RFRs were potentiated by mutual eye contact when participants viewed happy and angry expressions, while RFRs occurred only to fear expressions with averted gaze. RFRs to sad expressions were not affected by gaze direction. The interaction between emotional expressions and gaze direction was moderated by participants' gender. The pattern of autonomic reactivity was consistent with the view that salient social stimuli increase physiological arousal and attentional resources, with gaze direction, nature of emotion, and gender having moderating effects. These results suggest the critical role of self-relevance appraisal of senders' contextual perceptual cues and individual characteristics to account for interpersonal matching of facial displays.