Top Buy Now Summary References
Home
>
Section
>
Chapter

SPSA for layer-wise training of deep networks

Wulff, B.; Schuecker, J.; Bauckhage, C.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11141 LNCS: 564-573

2018

ISSN/ISBN: 0302-9743
DOI: 10.1007/978-3-030-01424-7_55
Accession: 104607094

Full-Text Article emailed within 0-6 h
Payments are secure & encrypted
Powered by Stripe
Powered by PayPal

Summary
Concerned with neural learning without backpropagation, we investigate variants of the simultaneous perturbation stochastic approximation (SPSA) algorithm. Experimental results suggest that these allow for the successful training of deep feed-forward neural networks using forward passes only. In particular, we find that SPSA-based algorithms which update network parameters in a layer-wise manner are superior to variants which update all weights simultaneously.

References

Liu, X.; Li, S.; Gao, K.; Wang, B. 2024: A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks Advances in Neural Information Processing Systems 37
Chen, L.; Ling, Z.; Liu, L.; Dai, L. 2014: Voice Conversion Using Deep Neural Networks with Layer-Wise Generative Training IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(12): 1859-1872
Hong, D.; Qi, Y.; Wang, Y. 2025: LaSNN: Layer-wise ANN-to-SNN distillation for effective and efficient training in deep spiking neural networks Neurocomputing 656: 131351
Shin, Y. 2022: Effects of depth, width, and initialization: A convergence analysis of layer-wise training for deep linear neural networks Analysis and Applications 20(1): 73-119
Hanggara, F.S.; Anam, K. 2020: Sequence-based protein-protein interaction prediction using greedy layer-wise training of deep neural networks AIP Conference Proceedings 2278: 020050
Wu, C.; Masoomi, A.; Gretton, A.; Dy, J.G. 2022: Deep Layer-wise Networks Have Closed-Form Weights Proceedings of Machine Learning Research 151: 188-225
Ishii, M.; Sato, A. 2018: Layer-Wise Weight Decay for Deep Neural Networks Lecture Notes in Computer Science 10749 LNCS: 276-289
Lacey, G.; Taylor, G.W.; Areibi, S. 2018: Stochastic layer-wise precision in deep neural networks 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 2: 663-672
Pina, O.; Vilaplana, V. 2025: Layer-Wise Training of Graph Neural Networks with Self-Supervised Learning IEEE Transactions on Neural Networks and Learning Systems 36(10): 18437-18447
You, Y.; Chen, T.; Wang, Z.; Shen, Y. 2020: L2-GCN: Layer-wise and learned efficient training of graph convolutional networks Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition: 2124-2132
Shen, Y.; Lu, L.; Zhu, T.; Wang, X.; Clifton, L.; Chen, Z.; Clarke, R.; Clifton, D.A. 2024: AutoNet-Generated Deep Layer-Wise Convex Networks for ECG Classification IEEE Transactions on Pattern Analysis and Machine Intelligence 46(10): 6542-6558
Kubota, S.; Hayashi, H.; Hayase, T.; Uchida, S. 2021: Layer-wise interpretation of deep neural networks using identity initialization Proceedings - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing 2021-June: 3945-3949
Xu, K.; Shen, X.; Yao, T.; Tian, X.; Mei, T. 2018: Greedy layer-wise training of long short term memory networks 2018 IEEE International Conference on Multimedia and Expo Workshops, ICMEW 2018: 8551584
Chen, S.; Zhao, Q. 2019: Shallowing Deep Networks: Layer-Wise Pruning Based on Feature Representations IEEE Transactions on Pattern Analysis and Machine Intelligence 41(12): 3048-3056
Yu, D.; Xiong, W.; Droppo, J.; Stolcke, A.; Ye, G.; Li, J.; Zweig, G. 2016: Deep convolutional neural networks with layer-wise context expansion and attention Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 08-12-September-2016: 17-21
Ahmed Zaid, D.; Djamaa, B.; Benatia, M.A. 2025: Efficient and dynamic layer-wise structured N:M pruning of deep neural networks Neurocomputing 653: 131090
Adesuyi, T.A.; Kim, B.-m. 2019: A layer-wise Perturbation based Privacy Preserving Deep Neural Networks 1st International Conference on Artificial Intelligence in Information and Communication, ICAIIC 2019: 389-394
Chen, S.; Wang, W.; Pan, S.J. 2019: Deep Neural Network Quantization via Layer-Wise Optimization Using Limited Training Data Proceedings of the AAAI Conference on Artificial Intelligence 33: 3329-3336
Sharma, A.K.; Ki Kim, K. 2024: Enhanced Image Classification through Layer-Wise Feature Concatenation in Deep Neural Networks Proceedings - International SoC Design Conference 2024, ISOCC 2024: 165-166
Dong, X.; Chen, S.; Pan, S.J. 2017: Learning to prune deep neural networks via layer-wise optimal brain surgeon Advances in Neural Information Processing Systems 2017-December: 4858-4868