Object recognition is one of the hottest issues in the field of vision system for harvesting robot. How efficiently and accurately to remove the background and get the object in image is the key research. The attention mechanisms of human visual system (HVS) can be segmented an image into the region of interesting (ROI) which is considered important and the background which is less important, and recognized the object from ROI using the local information. In this paper, an algorithm based on the characteristic of HVS is proposed. In algorithm, the image was partitioned into many blocks of equal size. ROI was got through calculating the factor of weight of each sub-block image, and the object was extracted by segmenting the ROI. Experiment results show that the algorithm can be recognized the object efficiently and accurately. A new method for vision system of harvesting robot is provided.