Background: Direct-acting oral anticoagulant use for stroke prevention in atrial fibrillation is limited by bleeding concerns. Asundexian, a novel, oral small molecule activated coagulation factor XIa (FXIa) inhibitor, might reduce thrombosis with minimal effect on haemostasis. We aimed to determine the optimal dose of asundexian and to compare the incidence of bleeding with that of apixaban in patients with atrial fibrillation. Methods: In this randomised, double-blind, phase 2 dose-finding study, we compared asundexian 20 mg or 50 mg once daily with apixaban 5 mg twice daily in patients aged 45 years or older with atrial fibrillation, a CHA2DS2-VASc score of at least 2 if male or at least 3 if female, and increased bleeding risk. The study was conducted at 93 sites in 14 countries, including 12 European countries, Canada, and Japan. Participants were randomly assigned (1:1:1) to a treatment group using an interactive web response system, with randomisation stratified by whether patients were receiving a direct-acting oral anticoagulant before the study start. Masking was achieved using a double-dummy design, with participants receiving both the assigned treatment and a placebo that resembled the non-assigned treatment. The primary endpoint was the composite of major or clinically relevant non-major bleeding according to International Society on Thrombosis and Haemostasis criteria, assessed in all patients who took at least one dose of study medication. This trial is registered with ClinicalTrials.gov, NCT04218266, and EudraCT, 2019-002365-35. Findings: Between Jan 30, 2020, and June 21, 2021, 862 patients were enrolled. 755 patients were randomly assigned to treatment. Two patients (assigned to asundexian 20 mg) never took any study medication, resulting in 753 patients being included in the analysis (249 received asundexian 20 mg, 254 received asundexian 50 g, and 250 received apixaban). The mean age of participants was 73·7 years (SD 8·3), 309 (41%) were women, 216 (29%) had chronic kidney disease, and mean CHA2DS2-VASc score was 3·9 (1·3). Asundexian 20 mg resulted in 81% inhibition of FXIa activity at trough concentrations and 90% inhibition at peak concentrations; asundexian 50 mg resulted in 92% inhibition at trough concentrations and 94% inhibition at peak concentrations. Ratios of incidence proportions for the primary endpoint were 0·50 (90% CI 0·14–1·68) for asundexian 20 mg (three events), 0·16 (0·01–0·99) for asundexian 50 mg (one event), and 0·33 (0·09–0·97) for pooled asundexian (four events) versus apixaban (six events). The rate of any adverse event occurring was similar in the three treatment groups: 118 (47%) with asundexian 20 mg, 120 (47%) with asundexian 50 mg, and 122 (49%) with apixaban. Interpretation: The FXIa inhibitor asundexian at doses of 20 mg and 50 mg once daily resulted in lower rates of bleeding compared with standard dosing of apixaban, with near-complete in-vivo FXIa inhibition, in patients with atrial fibrillation. Funding: Bayer.