They like to be kept on the dry side as too much water will inhibit fruit set.
Check the flowers and help pollination. If the flower parts are perfect then maybe too much water and vegetative growth caused flower drop. Ants also like to eat part of the flowers so apply a band with a sticky substance to the trunk.
To produce big fruit, the key is to pollinate them very well. With hand pollination, adequate irrigation, right fertilization and fruit thinning, with the right variety, you can grow bigger fruits.