The quick and high accuracy detection of point targets is a difficult but important technique in infrared surveillance. In this work, we proposed a patch image based low-rank and sparse matrices recovery (LSMR) method to detect moving point target in infrared marine surveillance. After analyzing the relationship between detection speed and image size, we wisely use frame difference and local threshold to efficiently exclude majority non-target patches before LSMR, which greatly accelerated the detection speed. Then integrated with the powerful classification ability of LSMR, the method proposed in this paper finally gains a high accuracy in point target detection in a marine environment. The experiment results show that the proposed method could effectively enhance the system’s signal-to-clutter ratio gain and background suppression factor while keeping a high detection speed compared to other similar algorithms.