Emotion recognition is an essential function to realize human-machine interaction devices. Physiological signals which can be collected easily and continuously by wearable sensors are good inputs for emotion analysis. How to effectively process physiological signals, extract critical features, and choose machine learning model for emotion classification has been a big challenge. In this paper, an entropy-based processing scheme for emotion recognition framework is proposed, which includes entropy domain feature extraction and prediction by XGBoost classifier. We experiment on AMIGOS database and the experimental results show that the proposed scheme for multi-modal analysis outperforms conventional processing approaches. It achieves approximately 80% and 68% accuracy of prediction for two affect dimensions, valence and arousal. For one modality case, we found that galvanic skin response (GSR) channel is the most potential modality for prediction, which leads to best performances.