Aiming at the problems of low detection accuracy, difficult feature extraction and large amount of model calculation of small target defects in transmission lines under UAV inspection scenarios, a transmission line defect detection algorithm RBI-YOLO based on YOLOv8 n is proposed. The RFCAConv (Receptive Field Coordinate Attention Convolution) module integrating receptive field and coordinate attention is introduced to optimize the backbone network, strengthen defect feature extraction and suppress background interference. The neck network is reconstructed, and the P2 shallow prediction branch is added to make full use of the high-resolution fine-grained features. At the same time, the BiFPN (Bidirectional Feature Pyramid Network) is used to optimize the aggregation path of cross-scale features, so as to improve the fusion and perception ability of the model to the shallow features of small targets. The Inner-MPDIoU (Inner Multi-Patch Distance Intersection over Union) loss function is designed to optimize the regression accuracy and stability of dense small target bounding box. The experimental results on the transmission line defect dataset show that compared with the baseline model YOLOv8n, the accuracy, recall, mAP50, and mAP50-95 of RBI-YOLO are increased by 1.3, 5.2, 3.9, and 4.2 percentage points, respectively. At the same time, the number of model parameters is reduced by 23.9 %, and the volume is reduced by 14.4 %, which can be effectively applied to the transmission line defect detection task.