It is still a challenge for fruit robot to automatic detecting young green apples in a complex grove environment due to color similarity with the background and varying illumination conditions. The purpose of this study was developing a robust method to detect young green apples in the tree canopy from low-cost color images acquired with diverse fruit sizes and under varying light circumstances. Adaptive green and blue chromatic aberration map was designed and combined with the iterative threshold segmentation algorithm to detect the region of interest contains potential apple fruits pixels. Then every potential fruit was identified by using an improved circular Hough transformation after morphological operation and blob analysis of the ITS outs which kept as many potential apple fruits pixels as possible. Finally, a kernel support vector machine classifier optimized by using grid search algorithm was built and combined with histogram of oriented gradients feature descriptor to distinguish and remove false fruit objects. The experimental result shows that the proposed method has better detection performance for young green apples.