Smartwatches, which contain an accelerometer and gyroscope, have recently been used to implement gait/activity-based biometrics. However, many research questions have not been addressed in the prior work such as the training and test data was collected in the same day from a limited dataset, using unrealistic activities (e.g., punch) and/or the authors did not carry out any particular study to identify the most discriminative features. This paper aims to highlight the impact of these factors on the biometric performance. The acceleration and gyroscope data of the gait and game activity was captured from 60 users over multiple days, which resulted in a totally of 24 h of the user’s movement. Segment-based approach was used to divide the time-series acceleration and gyroscope data. When the cross-day evaluation was applied, the best obtained EER was 0.69%, and 4.54% for the walking and game activities respectively. The EERs were significantly reduced into 0.05% and 2.35% for the above activities by introducing the majority voting schema. These results were obtained by utilizing a novel feature selection process in which the system minimizing the number of features and maximizing the discriminative information. The results have shown that smartwatch-based activity recognition has significant potential to recognize individuals in a continuous and user friendly approach.