Support Vector Machine (SVM) cannot process imbalanced problem and matrix patterns. Thus, Fuzzy SVM (FSVM) is proposed to process imbalanced problem while Support Matrix Machine (SMM) is proposed to process matrix patterns. FSVM applies a fuzzy membership to each training pattern such that different patterns can make different contributions to the learning machine. However, how to evaluate fuzzy membership becomes the key point to FSVM. Although SMM can process matrix patterns, it still has no ability to process imbalanced problem. This paper adopts SMM as the basic and proposes an entropy-based support matrix machine for imbalanced data sets, i.e., ESMM. The contributions of ESMM are: (1) proposing an entropy-based fuzzy membership evaluation approach which enhances importance of certainty patterns, (2) guaranteeing importance of positive patterns and getting a more flexible decision surface. Experiments on real-world imbalanced data sets and matrix patterns validate the effectiveness of ESMM.