The human brain is composed of multiple modular subsystems, with a unique way interacting among each other. These subsystems have their own unique characteristics and interact to support cognitive functions such as memory, attention and cognitive control. Nowadays, deep learning methods based on the above-mentioned functions accompanied with knowledge are widely used to design more dynamic, robust and powerful systems. We first review and summarize the progresses of cognition-based deep neural networks, and how cognitive mechanisms can be applied to more brain-like neural networks. Then we propose a general framework for the design of cognition-based deep learning system. Although great efforts have been made in this field, cognition-based deep learning is still in its early age. We put forward the potential directions towards this field, such as associative memory in deep learning, interpretable network with cognitive mechanisms, and deep reinforcement learning based on cognitive science.