In recent years, deep neural networks have developed into important computing models for deep learning, whose robustness is essential for their deployment in safety-critical areas. Therefore, the way to train robust neural networks is a popular issue that has attracted the attention of academia and industry. In this paper, three mainstream classes of training methods for robust neural networks are introduced, i.e., the method based on data enhancement, that based on adversarial training, and the Lipschitz robust training method. Meanwhile, their core ideas, representative work, and the application scope are introduced. Then, the advantages and disadvantages of the training methods in recent years are compared, and in-depth analysis and comparison are carried out when they correspond to key elements in neural network training. The robustness evaluation metrics of neural networks obtained by these training methods are introduced and compared. Finally, hotspots and challenges of robust neural network training are analyzed, and the possible future directions and some suggestions are briefly summarized.