To preserve competitive advantage in a more and more digitalized environment, today’s organizations seek to assess their level of digital maturity. Given this particular practical relevance, a plethora of digital maturity models, designed to asses a company’s digital status quo, has emerged over the past few years. Largely developed and published by practitioners, the academic value of these models remains obviously unclear. To shed light on their value in a broader sense, in this paper we critically evaluate 17 existing digital maturity models – identified through a systematic literature search (2011–2019) – with regard to their validity of measurement. We base our evaluation on established academic criteria, such as generalizability or theory-based interpretation, that we apply in a qualitative content analysis to these models. Our analysis shows that most of the identified models do not conform to the established evaluation criteria. Based on these insights, we derive a detailed research agenda and suggest respective research questions and strategies.