Irony detection is a difficult task because the intended meaning of a sentence differs from the literal meaning or sentiment of that sentence. Most existing work on this subject has focused on irony detection in the English language. Since no public dataset is available for this task in the Bengali domain, we have created a Bengali irony detection dataset that contains a total of 1500 labeled Bengali tweets. This paper presents the description of the Bengali irony detection dataset developed by us and reports some results obtained on our Bengali irony dataset using several widely used machine learning algorithms such as Naïve Bayes, Support Vector Machine, K-Nearest Neighbor and Random Forest.