An intelligence-based electrocardiogram (ECG) signal classification algorithm is very effective in monitoring Cardiac arrhythmias and helps the specialist make a decision and start a safe treatment routine for patients. Preventing wrong decisions about this category of patients that may even lead to death is one of the biggest challenges in the field of medicine, which requires very detailed analyses of the cardiac signals recorded by the patients. The diagnosis accuracy of cardiovascular diseases (CVDs) is largely dependent on the type and space of the extracted features and the ability of the learned classifier. This paper proposes an efficient 34-layer ResNet deep network to classify three types of CVDs based on the features extracted from the time–frequency domain as a scalogram. Also, the introduced ResNet-34 model is combined with the transfer learning technique and the results improvement has been shown. The presented algorithms are compared to other deep networks such as two different structures of a convolutional neural network (CNN) and a recurrent neural network (RNN). Also, these results are evaluated with a dictionary learning-based sparse Non-Negative Matrix Factorization (SNMF) classifier. The results show that the model presented based on ResNet-34 has better performance in different types of evaluation criteria such as accuracy, sensitivity, and robustness that has a positive role in clinical practices.