Childhood trauma has a negative impact on the developing brain and increases the risk for almost all psychiatric disorders including bipolar disorder. White matter abnormalities may play a role in the persistently increased risk for bipolar disorder following childhood trauma. We therefore examined the influence of childhood abuse and neglect on white matter integrity using diffusion tensor imaging (DTI), quantified as fractional anisotropy (FA), in patients with bipolar I disorder (N = 251) and healthy controls (N = 163). Bipolar patients experienced more childhood abuse (30.6% vs 8.0%; p< 0.001) and childhood neglect (36.3% vs 22.7%; p = 0.003) than controls. Childhood abuse had different effects on whole brain FA in patients with bipolar disorder compared to healthy individuals (F[1,410] = 3.060; p = 0.006). Specifically, whereas patients with bipolar disorder with childhood abuse had lower FA in widespread regions of the brain relative to patients without childhood abuse (t = 2.28; p = 0.024), no differences were found between healthy individuals with and without abuse (t=-0.18; p = 0.986). Differences in mean FA significantly mediated the association between childhood abuse and bipolar disorder. In contrast, childhood neglect was not significantly associated with FA in patients with bipolar disorder nor in healthy controls. Together, these results show that childhood abuse but not neglect is associated with lower integrity of white matter microstructure across the brain in patients with bipolar I disorder but not in healthy individuals. Therefore, white matter integrity might be involved the relationship between childhood abuse and bipolar disorder, even though the directionality cannot be proven due to the cross-sectional design of our study.