Artificial intelligence (AI) and computer vision (CV) advancements have paved the way for more efficient agricultural activities such as predicting and estimating fruit yield. Durian, a fruit native to tropical regions, necessitated using high-tech solutions to keep up with its rising global demand. This work aimed to apply the image analysis technique using deep learning to identify and estimate the number of durian fruits using image recognition. A new dataset was specifically constructed in this work, consisting of 500 images split for training and testing the object detection model. Various pre-trained object detection models such as YOLOv3, YOLOv4, YOLOv3 tiny, and YOLOv4 tiny are used for performance comparison on the newly constructed dataset. The best model is then chosen as the inference model for the drone-captured video dataset, assisted by the DeepSORT algorithm as the counting mechanism. Our investigations showed that the YOLOv4 model significantly performs best among all four state-of-art detection networks where it computes the highest mean average precision (mAP) performance with 96.02% accuracy on the constructed dataset. This work enables more efficient and precise durian cultivation with less labour and higher-quality yields.