An early characterization of drug-induced cardiotoxicity may be possible by combining comprehensive in vitro proarrhythmia assay and deep learning techniques. We aimed to develop a method to automatically detect irregular beating rhythm of field potentials recorded from human pluripotent stem cells (hPSC) derived cardiomyocytes (hPSC-CM) by multi-electrode array (MEA) system. We included field potentials from 380 experiments, which were labeled as normal or arrhythmic by electrophysiology experts. Convolutional and recurrent neural networks (CNN and RNN) were employed for automatic classification of field potential recordings. A preparation phase was initially applied to split 60-s long recordings into a series of 5-s windows. Subsequently, the classification phase comprising of two main steps was designed and applied. The first step included the classification of 5-s windows by using a designated CNN. While, the results of 5-s window assessments were used as the input sequence to an RNN that aggregates these results in the second step. The output was then compared to electrophysiologist-level arrhythmia detection, resulting in 0.83 accuracy, 0.93 sensitivity, 0.70 specificity, and 0.80 precision. In summary, this paper introduces a novel method for automated analysis of “irregularity” in an in vitro model of cardiotoxicity experiments. Thus, our method may overcome the drawbacks of using predesigned features that restricts the classification performance to the comprehensiveness and the quality of the designed features. Furthermore, automated analysis may facilitate the quality control experiments through the procedure of drug development with respect to cardiotoxicity and avoid late drug attrition from market.