Objective Deficits in social cognition are common and significant in people with temporal lobe epilepsy (TLE), but the functional and structural underpinnings remain unclear. The present study investigated how the side of seizure focus impacts face-processing networks in temporal lobe epilepsy. Methods We used functional magnetic resonance imaging (fMRI) of a face-processing paradigm to identify face-responsive regions in 24 individuals with unilateral temporal lobe epilepsy (left = 15; right = 9) and 19 healthy controls. fMRI signals of face-responsive regions ipsilateral and contralateral to the side of seizure onset were delineated in TLE and compared to the healthy controls with right and left sides combined. Diffusion tensor images were acquired to investigate structural connectivity between face regions that differed in fMRI signals between the two groups. Results In TLE, activation of the cortical face-processing networks varied according to side of seizure onset. In temporal lobe epilepsy, the laterality of amygdala activation was shifted to the side contralateral to the seizure focus, whereas controls showed no significant asymmetry. Furthermore, compared to controls, patients with TLE showed decreased activation of the occipital face-responsive region on the ipsilateral side and an increased activity of the anterior temporal lobe in the side contralateral to the seizure focus. Probabilistic tractography revealed that the occipital face area and anterior temporal lobe are connected via the inferior longitudinal fasciculus, which in individuals with TLE showed reduced integrity. Significance Taken together, these findings suggest that brain function and white matter integrity of networks subserving face processing are impaired on the side of seizure onset, accompanied by altered responses on the side contralateral to the seizure.