Several diffusion tensor imaging (DTI) studies in attention deficit hyperactivity disorder (ADHD) have shown a delay in brain white matter (WM) development. Because these studies were mainly conducted in children and adolescents, these WM abnormalities have been assumed, but not proven to progress into adulthood. To provide further insight in the natural history of WM maturation delay in ADHD, we here investigated the modulating effect of age on WM in children and adults. 120 stimulant-treatment naive male ADHD children (10–12 years of age) and adults (23–40 years of age) with ADHD (according to DSM-IV; all subtypes) were included, along with 23 age and gender matched controls. Fractional anisotropy (FA) values were compared throughout the WM by means of tract-based spatial statistics (TBSS) and in specific regions of interest (ROIs). On both TBSS and ROI analyses, we found that stimulant-treatment naive ADHD children did not differ in FA values from control children, whereas adult ADHD subjects had reduced FA values when compared to adult controls in several regions. Significant age × group interactions for whole brain FA (p = 0.015), as well as the anterior thalamic radiation (p = 0.015) suggest that ADHD affects the brain WM age-dependently. In contrast to prior studies conducted in medicated ADHD children, we did not find WM alterations in stimulant treatment naïve children, only treatment-naïve adults. Thus, our findings suggest that the reported developmental delay in WM might appear after childhood, and that previously reported differences between ADHD children and normal developing peers could have been attributed to prior ADHD medications, and/or other factors that affect WM development, such as age and gender.