No Access

Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems

Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090 Wien, Austria

E-mail Address: andres.felipe.lerma.pineda@univie.ac.at

and

Faculty of Mathematics, and Research Network Data Science @ Uni Vienna, University of Vienna, Kolingasse 14-16, 1090 Wien, Austria

E-mail Address: philipp.petersen@univie.ac.at

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0219530522400097Cited by:5 (Source: Crossref)

This article is part of the issue:

Special Issue on 20th Anniversary of Analysis and Applications — Part II
Editors: Steve Smale, Roderick Wong and Ding-Xuan Zhou

Abstract

We study the problem of reconstructing solutions of inverse problems when only noisy measurements are available. We assume that the problem can be modeled with an infinite-dimensional forward operator that is not continuously invertible. Then, we restrict this forward operator to finite-dimensional spaces so that the inverse is Lipschitz continuous. For the inverse operator, we demonstrate that there exists a neural network which is a robust-to-noise approximation of the operator. In addition, we show that these neural networks can be learned from appropriately perturbed training data. We demonstrate the admissibility of this approach to a wide range of inverse problems of practical interest. Numerical examples are given that support the theoretical findings.

Keywords:

AMSC: 35R30, 41A25, 68T05

References

1. G. S. Alberti and M. Santacesaria , Infinite-dimensional inverse problems with finite measurements, Arch. Ration. Mech. Anal. 243(1) (2022) 1–31. Crossref, Web of Science, Google Scholar
2. G. S. Alberti, M. Santacesaria and S. Sciutto, Continuous generative neural networks, preprint (2022), arXiv:2205.14627. Google Scholar
3. D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng and G. Chen , Deep speech 2: End-to-end speech recognition in english and mandarin, in Int. Conf. Machine Learning (PMLR, 2016), pp. 173–182. Google Scholar
4. M. Anthony and P. L. Bartlett , Neural Network Learning: Theoretical Foundations (Cambridge University Press, Cambridge, 1999). Crossref, Google Scholar
5. V. Antun, F. Renna, C. Poon, B. Adcock and A. C. Hansen, On instabilities of deep learning in image reconstruction-does AI come at a cost? preprint (2019), arXiv:1902.05300. Google Scholar
6. K. Armanious, C. Jiang, S. Abdulatif, T. Küstner, S. Gatidis and B. Yang , Unsupervised medical image translation using cycle-MedGAN, in 2019 27th Eur. Signal Processing Conf. (EUSIPCO) (IEEE, 2019), pp. 1–5. Crossref, Google Scholar
7. S. Arridge, P. Maass, O. Öktem and C. B. Schönlieb , Solving inverse problems using data-driven models, Acta Numer. 28 (2019) 1–174. Crossref, Web of Science, Google Scholar
8. V. Bacchelli and S. Vessella , Lipschitz stability for a stationary 2d inverse problem with unknown polygonal boundary, Inverse Probl. 22(5) (2006) 1627. Crossref, Web of Science, Google Scholar
9. D. Bahdanau, K. Cho and Y. Bengio, Neural machine translation by jointly learning to align and translate, preprint (2014), arXiv:1409.0473. Google Scholar
10. J. Berner, P. Grohs, G. Kutyniok and P. Petersen, The modern mathematics of deep learning, preprint (2021), arXiv:2105.04026. Google Scholar
11. P. J. Bickel and B. Li , Local polynomial regression on unknown manifolds, in Complex Datasets and Inverse Problems, Vol. 54 (Institute of Mathematical Statistics, 2007), pp. 177–187. Crossref, Google Scholar
12. H. Boche, A. Fono and G. Kutyniok, Limitations of deep learning for inverse problems on digital hardware, preprint (2022), arXiv:2202.13490. Google Scholar
13. S. Boucheron, G. Lugosi and P. Massart , Concentration Inequalities: A Nonasymptotic Theory of Independence (Oxford University Press, 2013). Crossref, Google Scholar
14. S. Boyd, N. Parikh, E. Chu, B. Peleato and J. Eckstein , Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends Mach. Learn. 3(1) (2011) 1–122. Crossref, Google Scholar
15. S. L. Brunton and J. N. Kutz , Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control (Cambridge University Press, 2022). Crossref, Google Scholar
16. M. Chen, H. Jiang, W. Liao and T. Zhao, Nonparametric regression on low-dimensional manifolds using deep relu networks, preprint (2019), arXiv:1908.01842. Google Scholar
17. C. K. Chui and H. Mhaskar , Deep nets for local manifold learning, Front. Appl. Math. Stat. 4 (2018) 12. Crossref, Google Scholar
18. A. Cloninger and T. Klock , A deep network construction that adapts to intrinsic dimensionality beyond the domain, Neural Netw. 141 (2021) 404–419. Crossref, Web of Science, Google Scholar
19. M. J. Colbrook, V. Antun and A. C. Hansen , The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and smale’s 18th problem, Proc. Natl. Acad. Sci. USA 119(12) (2022) e2107151119. Crossref, Google Scholar
20. K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian , Image denoising by sparse 3-D transform-domain collaborative filtering, IEEE Trans. Image Process. 16(8) (2007) 2080–2095. Crossref, Web of Science, Google Scholar
21. D. Elbrächter, P. Grohs, A. Jentzen and C. Schwab, DNN expression rate analysis of high-dimensional PDEs: Application to option pricing, preprint (2018), arXiv:1809.07669. Google Scholar
22. H. W. Engl, M. Hanke and A. Neubauer , Regularization of Inverse Problems, Vol. 375 (Springer Science & Business Media, 1996). Crossref, Google Scholar
23. M. Genzel, J. Macdonald and M. März , Solving inverse problems with deep neural networks-robustness included, IEEE Trans. Pattern Anal. Mach. Intell. 45(1) (2022) 1119–1134. Crossref, Web of Science, Google Scholar
24. I. Goodfellow, Y. Bengio and A. Courville , Deep Learning (MIT Press, 2016), http://www.deeplearningbook.org. Google Scholar
25. N. M. Gottschling, V. Antun, B. Adcock and A. C. Hansen, The troublesome kernel: Why deep learning for inverse problems is typically unstable, preprint (2020), arXiv:2001.01258. Google Scholar
26. F. A. Graybill and G. Marsaglia , Idempotent matrices and quadratic forms in the general linear hypothesis, Ann. Math. Stat. 28(3) (1957) 678–686. Crossref, Google Scholar
27. K. Gregor and Y. LeCun , Learning fast approximations of sparse coding, in Proc. 27th Int. Conf. Machine Learning (Omnipress, 2010), pp. 399–406. Google Scholar
28. P. Grohs and F. Voigtlaender, Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces, preprint (2021), arXiv:2104.02746. Google Scholar
29. I. Gühring, M. Raslan and G. Kutyniok, Expressivity of deep neural networks, preprint (2020), arXiv:2007.04759. Google Scholar
30. J. Han, A. Jentzen and W. E , Solving high-dimensional partial differential equations using deep learning, Proc. Natl. Acad. Sci. USA 115(34) (2018) 8505–8510. Crossref, Web of Science, Google Scholar
31. J. He, L. Li, J. Xu and C. Zheng, ReLU deep neural networks and linear finite elements, preprint (2018), arXiv:1807.03973. Google Scholar
32. K. He, X. Zhang, S. Ren and J. Sun , Deep residual learning for image recognition, in Proc. IEEE Conf. Computer Vision and Pattern Recognition (IEEE, 2016), pp. 770–778. Crossref, Google Scholar
33. M. Hein, J.-Y. Audibert and U. V. Luxburg , From graphs to manifolds–weak and strong pointwise consistency of graph Laplacians, in Int. Conf. Computational Learning Theory (Springer, 2005), pp. 470–485. Google Scholar
34. J. Hu, L. Shen and G. Sun , Squeeze-and-excitation networks, in 2018 IEEE/CVF Conf. Computer Vision and Pattern Recognition (IEEE, 2018), pp. 7132–7141. Crossref, Google Scholar
35. Y. Huang, T. Würfl, K. Breininger, L. Liu, G. Lauritsch and A. Maier , Some investigations on robustness of deep learning in limited angle tomography, in Int. Conf. Medical Image Computing and Computer-Assisted Intervention (Springer, 2018), pp. 145–153. Crossref, Google Scholar
36. B. Kaltenbacher, A. Neubauer and O. Scherzer , Iterative Regularization Methods for Nonlinear Ill-Posed Problems (de Gruyter, 2008). Crossref, Google Scholar
37. A. Kirsch , An Introduction to the Mathematical Theory of Inverse Problems, Vol. 120 (Springer, 2011). Crossref, Google Scholar
38. M. Kohler, A. Krzyzak and S. Langer, Estimation of a function of low local dimensionality by deep neural networks, preprint (2019), arXiv:1908.11140. Google Scholar
39. M. Kontak and V. Michel , A greedy algorithm for nonlinear inverse problems with an application to nonlinear inverse gravimetry, GEM Int. J. Geomath. 9(2) (2018) 167–198. Crossref, Web of Science, Google Scholar
40. A. Krizhevsky, I. Sutskever and G. E. Hinton , Imagenet classification with deep convolutional neural networks, in Adv. Neural Inf. Process. Syst., Vol. 25 (Curran Associates, 2012), pp. 1097–1105. Google Scholar
41. Y. LeCun, Y. Bengio and G. Hinton , Deep learning, Nature 521(7553) (2015) 436–444. Crossref, Web of Science, Google Scholar
42. J. M. Lee , Smooth manifolds, in Introduction to Smooth Manifolds (Springer, 2013), pp. 1–31. Crossref, Google Scholar
43. Q. Li, T. Lin and Z. Shen , Deep learning via dynamical systems: An approximation perspective, J. Eur. Math. Soc. (2022). https://doi.org/10.4171/JEMS/1221. Crossref, Web of Science, Google Scholar
44. T. T. Marinov and A. S. Vatsala , Inverse problem for coefficient identification in the Euler–Bernoulli equation, Comput. Math. Appl. 56(2) (2008) 400–410. Crossref, Web of Science, Google Scholar
45. W. McCulloch and W. Pitts , A logical calculus of ideas immanent in nervous activity, Bull. Math. Biophys. 5 (1943) 115–133. Crossref, Google Scholar
46. E. J. McShane , Extension of range of functions, Bull. Am. Math. Soc. 40(12) (1934) 837–842. Crossref, Google Scholar
47. H. N. Mhaskar , Eignets for function approximation on manifolds, Appl. Comput. Harmon. Anal. 29(1) (2010) 63–87. Crossref, Web of Science, Google Scholar
48. R. K. Miller , Volterra integral equations in a Banach space, Funkcial. Ekvac. 18(2) (1975) 163–193. Google Scholar
49. J. R. Munkres , Topology, Vol. 2 (Prentice Hall, Upper Saddle River, 2000). Google Scholar
50. R. Nakada and M. Imaizumi , Adaptive approximation and generalization of deep neural network with intrinsic dimensionality, J. Mach. Learn. Res. 21(174) (2020) 1–38. Google Scholar
51. F. Natterer , Regularisierung schlecht gestellter Probleme durch Projektionsverfahren, Numer. Math. 28(3) (1977) 329–341. Crossref, Web of Science, Google Scholar
52. F. Natterer , The Mathematics of Computerized Tomography (SIAM, 2001). Crossref, Google Scholar
53. F. Natterer and F. Wübbeling , Mathematical Methods in Image Reconstruction (SIAM, 2001). Crossref, Google Scholar
54. S. Niwas and D. C. Singhal , Aquifer transmissivity of porous media from resistivity data, J. Hydrol. 82(1–2) (1985) 143–153. Crossref, Web of Science, Google Scholar
55. G. Ongie, A. Jalal, C. A Metzler, R. G. Baraniuk, A. G. Dimakis and R. Willett , Deep learning techniques for inverse problems in imaging, IEEE J. Sel. Areas Inf. Theory 1(1) (2020) 39–56. Crossref, Google Scholar
56. J. A. A. Opschoor, P. C. Petersen and Ch. Schwab , Deep ReLU networks and high-order finite element methods, Anal. Appl. 18(05) (2020) 715–770. Link, Web of Science, Google Scholar
57. P. C. Petersen , Neural network theory, University of Vienna (2020). Google Scholar
58. P. C. Petersen and F. Voigtlaender , Optimal approximation of piecewise smooth functions using deep ReLU neural networks, Neural Netw. 180 (2018) 296–330. Crossref, Web of Science, Google Scholar
59. R. Plato and G. Vainikko , On the regularization of projection methods for solving ill-posed problems, Numer. Math. 57(1) (1990) 63–79. Crossref, Web of Science, Google Scholar
60. M. Raissi, P. Perdikaris and G. E. Karniadakis , Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys. 378 (2019) 686–707. Crossref, Web of Science, Google Scholar
61. M. Razzaghi and Y. Ordokhani , Solution of nonlinear Volterra-Hammerstein integral equations via rationalized Haar functions, Math. Probl. Eng. 7(2) (2001) 205–219. Crossref, Web of Science, Google Scholar
62. L. Rondi , A remark on a paper by Alessandrini and Vessella, Adv. Appl. Math. 36(1) (2006) 67–69. Crossref, Web of Science, Google Scholar
63. J. Schmidt-Hieber, Deep ReLU network approximation of functions on a manifold, preprint (2019), arXiv:1908.00695. Google Scholar
64. B. Sepehrian and M. Razzaghi , Solution of nonlinear Volterra-Hammerstein integral equations via single-term Walsh series method, Math. Probl. Eng. 2005(5) (2005) 547–554. Crossref, Web of Science, Google Scholar
65. U. Shaham, A. Cloninger and R. R. Coifman , Provable approximation properties for deep neural networks, Appl. Comput. Harmon. Anal. 44(3) (2018) 537–557. Crossref, Web of Science, Google Scholar
66. A. Singer , From graph to manifold Laplacian: The convergence rate, Appl. Comput. Harmon. Anal. 21(1) (2006) 128–134. Crossref, Web of Science, Google Scholar
67. J. Sirignano and K. Spiliopoulos , Dgm: A deep learning algorithm for solving partial differential equations, J. Comput. Phys. 375 (2018) 1339–1364. Crossref, Web of Science, Google Scholar
68. J. Sun, H. Li and Z. Xu , Deep admm-net for compressive sensing MRI, in Adv. Neural Inf. Process. Syst., Vol. 29 (Curran Associates, 2016), pp. 10–18. Google Scholar
69. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin , Attention is all you need, in Adv. Neural Inf. Process. Syst., Vol. 30 (Curran Associates, 2017), pp. 5998–6008. Google Scholar
70. S. V. Venkatakrishnan, C. A. Bouman and B. Wohlberg , Plug-and-play priors for model based reconstruction, in 2013 IEEE Global Conf. Signal and Information Processing (IEEE, 2013), pp. 945–948. Crossref, Google Scholar
71. D. Yarotsky , Error bounds for approximations with deep ReLU networks, Neural Netw. 94 (2017) 103–114. Crossref, Web of Science, Google Scholar
72. W. W.-G. Yeh , Review of parameter identification procedures in groundwater hydrology: The inverse problem, Water Resour. Res. 22(2) (1986) 95–108. Crossref, Web of Science, Google Scholar
73. B. Zhu, J. Z Liu, S. F Cauley, B. R. Rosen and M. S. Rosen , Image reconstruction by domain-transform manifold learning, Nature 555(7697) (2018) 487–492. Crossref, Web of Science, Google Scholar
74. J. Y. Zhu, T. Park, P. Isola and A. A. Efros , Unpaired image-to-image translation using cycle-consistent adversarial networks, in Proc. IEEE Int. Conf. Computer Vision (IEEE, 2017), pp. 2223–2232. Crossref, Google Scholar