Depletion of high-quality natural sand deposits and sustainability concerns are popularizing manufactured sand use in cementitious composites. Meanwhile, palm fruit bunch fiber (PFBF) improves the properties of cementitious composites, but it is unclear how PFBF interacts with different fine aggregates to affect mortar strength. This study investigated the impact of PFBF on the compressive strength of cement mortars containing manufactured sand (granite quarry dust) and natural sands (river and pit). The aggregates were used with Portland cement to fabricate mortar cubes, which were tested after 28 days. The control mortars (0% PFBF) of quarry dust, river sand, and pit sand recorded strength of 24.2 MPa, 21.5 MPa, and 10.4 MPa, respectively. At the optimum fiber content, the strength of the quarry dust and pit sand mortars increased marginally to 24.7 MPa and 12.2 MPa, respectively. However, river sand mortar strength considerably increased to 26.1 MPa. Interestingly, the quarry dust and pit sand mortars generally experienced strength loss before reaching their peak at 2.0% and 2.5% fiber content, respectively. In comparison, river sand mortar consistently gained strength before peaking at 2.5% PFBF. Hence, pre-optimum fiber contents could enhance river sand mortar strength but hinder quarry dust and pit sand mortar strengths. By standardizing the PFBF-reinforced mortar strengths against the control strengths, PFBF enhanced pit sand mortar strength the most, followed by river sand mortar, but it mainly reduced quarry dust mortar strength. Mortar design must, therefore, optimize PFBF dosage considering the unique characteristics of each sand type.