Article Contents
Yue Wu, Jun-Wei Liu, Chen-Zhuo Zhu, Zhuang-Fei Bai, Qi-Guang Miao, Wen-Ping Ma, Mao-Guo Gong. Computational Intelligence in Remote Sensing Image Registration: A survey. International Journal of Automation and Computing. doi: 10.1007/s11633-020-1248-x
Cite as: Yue Wu, Jun-Wei Liu, Chen-Zhuo Zhu, Zhuang-Fei Bai, Qi-Guang Miao, Wen-Ping Ma, Mao-Guo Gong. Computational Intelligence in Remote Sensing Image Registration: A survey. International Journal of Automation and Computing. doi: 10.1007/s11633-020-1248-x

Computational Intelligence in Remote Sensing Image Registration: A survey

Author Biography:
  • Yue Wu received the B. Eng. circuits and systems Ph. D. degree in control science and engineering from Xidian University, China in 2011 and 2016, respectively. Since 2016, he has been a teacher with Xidian University. He is currently an associate professor with Xidian University. He has authored or co-authored more than 40 papers in refereed journals and proceedings. His research interests include computer vision and computational intelligence. E-mail: ywu@xidian.edu.cnORCID iD: 0000-0002-3459-5079

    Jun-Wei Liu is a master student in Key Laboratory of Big Data and Intelligent Vision, School of Computer Science and Technology, Xidian University, China. His research interests include evolutionary computing and image registration. E-mail: jwliu_2@stu.xidian.edu.cn

    Chen-Zhuo Zhu is a master student in Key Laboratory of Big Data and Intelligent Vision, School of Computer Science and Technology, Xidian University, China. His research interests include deep learning and image registration. E-mail: chenzhuozhu@xidian.edu.cn

    Zhuang-Fei Bai is a master student in Key Laboratory of Big Data and Intelligent Vision, School of Computer Science and Technology, Xidian University, China. His research interests include computer vision and remote sensing image understanding. E-mail: zfbai@stu.xidian.edu.cn

    Qi-Guang Miao received the M. Eng. degree in computer science from Xidian University, China in 1996, received the Ph. D. degree in computer science from Xidian University, China in 2005. He is currently a professor with the School of Computer Science and Technology, Xidian University, China. His research interests include intelligent image processing and multiscale geometric representations for images. E-mail: qgmiao@xidian.edu.cn

    Wen-Ping Ma received the B. Sc. degree in computer science and technology and the Ph. D. degree in pattern recognition and intelligent systems from Xidian University, China in 2003 and 2008, respectively. Since 2006, she has been with the Key Laboratory of Intelligent Perception and Image Understanding of the Ministry of Education, Xidian University, where she is currently an associate professor. She has published more than 30 SCI papers in international academic journals, including the IEEE Transactions on Evolutionary Computation, IEEE Transactions on Image Processing, Information Sciences, Pattern Recognition, Applied Soft Computing, Knowledge-based Systems, Physica A − Statistical Mechanics and its Applications, and IEEE Geoscience and Remote Sensing Letters. She is a member of the Chinese Institute of Electronics and the China Computer Federation. Her research interests include natural computing and intelligent image processing. E-mail: wpma@mail.xidian.edu.cn

    Mao-Guo Gong received the B. Sc. degree in electronic engineering and the Ph. D. degree in electronic science and technology from Xidian University, China in 2003 and 2009, respectively. Since 2006, he has been a teacher with Xidian University. In 2008 and 2010, he was promoted as an associate professor and as a full professor, respectively, both with exceptive admission. He is an executive committee member of the Chinese Association for Artificial Intelligence and a senior member of the Chinese Computer Federation. He received the prestigious National Program for the support of Top-Notch Young Professionals from the Central Organization Department of China, the Excellent Young Scientist Foundation from the National Natural Science Foundation of China, and the New Century Excellent Talent in University from the Ministry of Education of China. He is currently the vice-chair of the IEEE Computational Intelligence Society Task Force on Memetic Computing. His research interests include computational intelligence with applications to optimization, learning, data mining, and image understanding. E-mail: gong@ieee.org (Corresponding author)ORCID iD: 0000-0002-0415-8556

  • Received: 2020-06-15
  • Accepted: 2020-08-14
  • Published Online: 2020-12-29
Fund Project:  Recommended by Associate Editor De Xu
  • In recent years, computational intelligence has been widely used in many fields and achieved remarkable performance. Evolutionary computing and deep learning are important branches of computational intelligence. Many methods based on evolutionary computation and deep learning have achieved good performance in remote sensing image registration. This paper introduces the application of computational intelligence in remote sensing image registration from the two directions of evolutionary computing and deep learning. In the part of remote sensing image registration based on evolutionary calculation, the principles of evolutionary algorithms and swarm intelligence algorithms are elaborated and their application in remote sensing image registration is discussed. The application of deep learning in remote sensing image registration is also discussed. At the same time, the development status and future of remote sensing image registration are summarized and their prospects are examined.
  • 加载中
  • [1] H. Ghassemian.  A review of remote sensing image fusion methods[J]. Information Fusion, 2016, 32(): 75-89. doi: 10.1016/j.inffus.2016.03.003
    [2] P. Z. Zhang, M. G. Gong, L. Z. Su, J. J. Liu, Z. Z. Li.  Change detection based on deep feature representation and mapping transformation for multi-spatial-resolution remote sensing images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 116(): 24-41. doi: 10.1016/j.isprsjprs.2016.02.013
    [3] J. H. Cui.  Remote sensing image feature recognition and monitoring of ecological vegetation restoration in football field[J]. Ekoloji, 2019, 28(108): 1571-1575.
    [4] Y. M. Zhen, Z. Sun, J. B. Li, Y. Peng. An airborne remote sensing image mosaic algorithm based on feature points. In Proceedings of the 6th International Conference on Instrumentation & Measurement, Computer, Communication and Control, IEEE, Harbin, China, pp.202−205, 2016. DOI: 10.1109/IMCCC.2016.145.
    [5] S. H. Wang, J. D. Sun, P. Phillips, G. H. Zhao, Y. D. Zhang.  Polarimetric synthetic aperture radar image segmentation by convolutional neural network using graphical processing units[J]. Journal of Real-time Image Processing, 2018, 15(3): 631-642. doi: 10.1007/s11554-017-0717-0
    [6] Y. D. Zhang, L. N. Wu.  Crop classification by forward neural network with adaptive chaotic particle swarm optimization[J]. Sensors, 2011, 11(5): 4721-4743. doi: 10.3390/s110504721
    [7] N. Alborzi, F. Poorahangaryan, H. Beheshti.  Spectral-spatial classification of hyperspectral images using signal subspace identification and edge-preserving filter[J]. International Journal of Automation and Computing, 2020, 17(2): 222-232. doi: 10.1007/s11633-019-1188-5
    [8] J. Du, C. X. Jiang, Q. Guo, M. Guizani, Y. Ren.  Cooperative earth observation through complex space information networks[J]. IEEE Wireless Communications, 2016, 23(2): 136-144. doi: 10.1109/MWC.2016.7462495
    [9] J. Y. Ma, Y. Ma, C. Li.  Infrared and visible image fusion methods and applications: A survey[J]. Information Fusion, 2019, 45(): 153-178. doi: 10.1016/j.inffus.2018.02.004
    [10] B. Zitova, J. Flusser.  Image registration methods: A survey[J]. Image and Vision Computing, 2003, 21(11): 977-1000. doi: 10.1016/S0262-8856(03)00137-9
    [11] G. Haskins, U. Kruger, P. K. Yan.  Deep learning in medical image registration: A survey[J]. Machine Vision and Applications, 2020, 31(1–2): 8-. doi: 10.1007/s00138-020-01060-x
    [12] J. Le Moigne. Introduction to remote sensing image registration. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, IEEE, Fort Worth, USA, pp.2565–2568, 2017. DOI: 10.1109/IGARSS.2017.8127519.
    [13] S. R. J. Ramson, K. L. Raju, S. Vishnu, T. Anagnostopoulos.  Nature inspired optimization techniques for image processing − a short review[J]. Nature Inspired Optimization Techniques for Image Processing Applications, Hemanth J., Balas V. E., Eds., Cham, Germany: Springer, 2019, (): 113-145. doi: 10.1007/978-3-319-96002-9_5
    [14] H. Liu, G. F. Xiao.  Remote sensing image registration based on improved kaze and brief descriptor[J]. International Journal of Automation and Computing, 2020, 17(4): 588-598. doi: 10.1007/s11633-019-1218-3
    [15] L. Ma, Y. Liu, X. L. Zhang, Y. X. Ye, G. F. Yin, B. A. Johnson.  Deep learning in remote sensing applications: A meta-analysis and review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152(): 166-177. doi: 10.1016/j.isprsjprs.2019.04.015
    [16] H. Liu, G. F. Xiao, Y. L. Tan, C. J. Ouyang.  Multi-source remote sensing image registration based on contourlet transform and multiple feature fusion[J]. International Journal of Automation and Computing, 2019, 16(5): 575-588. doi: 10.1007/s11633-018-1163-6
    [17] Y. Wu, W. P. Ma, Q. X. Su, S. D. Liu, Y. H. Ge.  Remote sensing image registration based on local structural information and global constraint[J]. Journal of Applied Remote Sensing, 2019, 13(1): 016518-. doi: 10.1117/1.JRS.13.016518
    [18] X. P. Liu, S. L. Chen, L. Zhuo, J. Li, K. N. Huang.  Multi-sensor image registration by combining local self-similarity matching and mutual information[J]. Frontiers of Earth Science, 2018, 12(4): 779-790. doi: 10.1007/s11707-018-0717-9
    [19] X. L. Ma, X. D. Li, Q. F. Zhang, K. Tang, Z. P. Liang, W. X. Xie, Z. X. Zhu.  A survey on cooperative co-evolutionary algorithms[J]. IEEE Transactions on Evolutionary Computation, 2019, 23(3): 421-441. doi: 10.1109/TEVC.2018.2868770
    [20] J. W. Zhang, L. N. Xing. A survey of multiobjective evolutionary algorithms. In Proceedings of IEEE International Conference on Computational Science and Engineering and IEEE International Conference on Embedded and Ubiquitous Computing, IEEE, Guangzhou, China, pp.93–100, 2017. DOI: 10.1109/CSE-EUC.2017.27.
    [21] K. De Jong. Evolutionary computation: A unified approach. In Proceedings of Genetic and Evolutionary Computation Conference Companion, ACM, Denver, USA, pp.185–199, 2016. DOI: 10.1145/2908961.2926973.
    [22] K. R. Opara, J. Arabas.  Differential evolution: A survey of theoretical analyses[J]. Swarm and Evolutionary Computation, 2019, 44(): 546-558. doi: 10.1016/j.swevo.2018.06.010
    [23] Z. N. He, G. G. Yen, J. C. Lv.  Evolutionary multiobjective optimization with robustness enhancement[J]. IEEE Transactions on Evolutionary Computation, 2020, 24(3): 494-507. doi: 10.1109/TEVC.2019.2933444
    [24] Y. F. Zhong, A. L. Ma, Y. Soon Ong, Z. X. Zhu, L. P. Zhang.  Computational intelligence in optical remote sensing image processing[J]. Applied Soft Computing, 2018, 64(): 75-93. doi: 10.1016/j.asoc.2017.11.045
    [25] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis.  Deep learning for computer vision: A brief review[J]. Computational Intelligence and Neuroscience, 2018, 2018(): 7068349-. doi: 10.1155/2018/7068349
    [26] B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, I. Iusgum. End-to-end unsupervised deformable image registration with a convolutional neural network. In Proceedings of International Workshop on Deep Learning in Medical lmage Analysis, International Workshop on Maltimodal Learning for Clinical Decision Support, Springer, Quebec city, Canada, pp.204–212, 2017. p.204–212, 2017. DOI: 10.1007/978-3-319-67558-9_24.
    [27] Z. Q. Zhao, P. Zheng, S. T. Xu, X. D. Wu.  Object detection with deep learning: A review[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(11): 3212-3232. doi: 10.1109/TNNLS.2018.2876865
    [28] M. G. Gong, H. L. Yang, P. Z. Zhang.  Feature learning and change feature classification based on deep learning for ternary change detection in SAR images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 129(): 212-225. doi: 10.1016/j.isprsjprs.2017.05.001
    [29] Y. Liu, X. Chen, Z. F. Wang, Z. J. Wang, R. K. Ward, X. S. Wang.  Deep learning for pixel-level image fusion: Recent advances and future prospects[J]. Information Fusion, 2018, 42(): 158-173. doi: 10.1016/j.inffus.2017.10.007
    [30] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5MB model size. https://arxiv.org/abs/1602.07360, 2016.
    [31] K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. [Online], Available: https://arxiv.org/abs/1409.1556, 2014.
    [32] C. Szegedy, W. Liu, Y. Q. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich. Going deeper with convolutions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Boston, USA, pp.1–9, 2015. DOI: 10.1109/CVPR.2015.7298594.
    [33] A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th Information Processing Systems, ACM, Lake Tahoe, USA, pp.1097–1105, 2012.
    [34] J. X. Gu, Z. H. Wang, J. Kuen, L. Y. Ma, A. Shahroudy, B. Shuai, T. Liu, X. X. Wang, G. Wang, J. F. Cai, T. Chen.  Recent advances in convolutional neural networks[J]. Pattern Recognition, 2018, 77(): 354-377. doi: 10.1016/j.patcog.2017.10.013
    [35] P. Murugan. Feed forward and backward run in deep convolution neural network. https://arxiv.org/abs/1711.03278, 2017.
    [36] A. Giusti, D. C. Cirecsan, J. Masci, L. M. Gambardella, J. Schmidhuber. Fast image scanning with deep max-pooling convolutional neural networks. In Proceedings of IEEE International Conference on Image Processing, IEEE, Melbourne, Australia, pp.4034–4038, 2013. DOI: 10.1109/ICIP.2013.6738831.
    [37] W. Ma, J. Lu. An equivalence of fully connected layer and convolutional layer. [Online], Available: https://arxiv.org/abs/1712.01252, 2017.
    [38] F. M. Ye, Y. F. Su, H. Xiao, X. Q. Zhao, W. D. Min.  Remote sensing image registration using convolutional neural network features[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(2): 232-236. doi: 10.1109/LGRS.2017.2781741
    [39] R. Iqbal, F. Doctor, B. More, S. Mahmud, U. Yousuf.  Big data analytics: Computational intelligence techniques and application areas[J]. Technological Forecasting and Social Change, 2020, 153(): 119253-. doi: 10.1016/j.techfore.2018.03.024
    [40] Y. S. Chen.  Performance identification in large-scale class data from advanced facets of computational intelligence and soft computing techniques[J]. International Journal of High Performance Computing and Networking, 2019, 13(3): 283-293. doi: 10.1504/IJHPCN.2019.098569
    [41] A. E. Eiben, M. Schoenauer.  Evolutionary computing[J]. Information Processing Letters, 2002, 82(1): 1-6. doi: 10.1016/S0020-0190(02)00204-1
    [42] H. Al-Sahaf, M. J. Zhang, A. Al-Sahaf, M. Johnston.  Keypoints detection and feature extraction: A dynamic genetic programming approach for evolving rotation-invariant texture image descriptors[J]. IEEE Transactions on Evolutionary Computation, 2017, 21(6): 825-844. doi: 10.1109/TEVC.2017.2685639
    [43] W. A. Albukhanajer, J. A. Briffa, Y. C. Jin.  Evolutionary multiobjective image feature extraction in the presence of noise[J]. IEEE Transactions on Cybernetics, 2015, 45(9): 1757-1768. doi: 10.1109/TCYB.2014.2360074
    [44] J. Pramanik, S. Dalai, D. Rana.  Image registration using discrete wavelet transform and particle swarm optimization[J]. International Journal of Computer Science and Information Technologies, 2015, 6(): 1521-1525.
    [45] Y. Tian, H. D. Ma.  Image registration based on improved ant colony algorithm[J]. Advanced Materials Research, 2013, 765--767(): 683-686. doi: 10.4028/www.scientific.net/AMR.765-767.683
    [46] J. Santamaria, S. Damas, J. M. Garcia-Torres, O. Cordon.  Self-adaptive evolutionary image registration using differential evolution and artificial immune systems[J]. Pattern Recognition Letters, 2012, 33(16): 2065-2070. doi: 10.1016/j.patrec.2012.07.002
    [47] V. T. Ingole, C. N. Deshmukh, A. Joshi, D. Shete. Medical image registration using genetic algorithm. In Proceedings of the 2nd International Conference on Emerging Trends in Engineering & Technology, IEEE, Nagpur, India, pp.63–66, 2009. DOI: 10.1109/ICETET.2009.143.
    [48] X. Wang, M. Adjouadi. Automatic registration of FDG_CT and FLT_CT images integrating genetic algorithm, powell method and wavelet decomposition. In Proceedings of IEEE Signal Processing in Medicine and Biology Symposium, IEEE, Philadelphia, USA, 2015. DOI: 10.1109/SPMB.2015.7405475.
    [49] L. Schwab, M. Schmitt, R. Wanka. Multimodal medical image registration using particle swarm optimization with influence of the data′s initial orientation. In Proceedings of IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology, IEEE, Niagara Falls, Canada, 2015. DOI: 10.1109/CIBCB.2015.7300314.
    [50] M. Abdel-Basset, A. E. Fakhry, I. El-Henawy, T. Qiu, A. K. Sangaiah.  Feature and intensity based medical image registration using particle swarm optimization[J]. Journal of Medical Systems, 2017, 41(12): 197-. doi: 10.1007/s10916-017-0846-9
    [51] K. K. Delibasis, P. A. Asvestas, G. K. Matsopoulos.  Automatic point correspondence using an artificial immune system optimization technique for medical image registration[J]. Computerized Medical Imaging and Graphics, 2011, 35(1): 31-41. doi: 10.1016/j.compmedimag.2010.09.002
    [52] J. Inglada, F. Adragna. Automatic multi-sensor image registration by edge matching using genetic algorithms. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, Scanning the Present and Resolving the Future, IEEE, Sydney, Australia, pp.2313–2315, 2001. DOI: 10.1109/IGARSS.2001.977986.
    [53] S. K. Makrogiannis, N. G. Bourbakis, S. Borek. A stochastic optimization scheme for automatic registration of aerial images. In Proceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence, IEEE, Boca Raton, USA, pp.328–336, 2004. DOI: 10.1109/ICTAI.2004.18.
    [54] Q. Zhang, G. J. Wen, C. X. Zhang, Z. R. Lin, Z. M. Shang, H. M. Wang. Image registration with position and similarity constraints based on genetic algorithm. In Proceedings of the 10th International Conference on Natural Computation, IEEE, Xiamen, China, pp.568–572, 2014. DOI: 10.1109/ICNC.2014.6975897.
    [55] Z. J. Gou, H. B. Ma. An automatic registration based on genetic algorithm for multi-source remote sensing. In Proceedings of the 2nd International Conference on Control, Automation and Robotics, IEEE, Hong Kong, China, pp.318–323, 2016. DOI: 10.1109/ICCAR.2016.7486748.
    [56] Y. Y. Li, Q. J. Liu, L. H. Jing, S. Liu, F. X. Miao. A genetic-optimized multi-angle normalized cross correlation SIFT for automatic remote sensing registration. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, IEEE, Beijing, China, pp.2586–2589, 2016. DOI: 10.1109/IGARSS.2016.7729668.
    [57] S. Yavari, M. J. V. Zoej, B. Salehi.  An automatic optimum number of well-distributed ground control lines selection procedure based on genetic algorithm[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 139(): 46-56. doi: 10.1016/j.isprsjprs.2018.03.002
    [58] I. De Falco, A. Della Cioppa, D. Maisto, E. Tarantino. Differential evolution for the registration of remotely sensed images. Soft Computing in Industrial Applications: Recent Trends, A. Saad, K. Dahal, M. Sarfraz, R. Roy, Eds., Berlin, Heidelberg: Springer, pp.40–49, 2007. DOI: 10.1007/978-3-540-70706-6_4.
    [59] I. De Falco, A. Della Cioppa, D. Maisto, E. Tarantino.  Differential evolution as a viable tool for satellite image registration[J]. Applied Soft Computing, 2008, 8(4): 1453-1462. doi: 10.1016/j.asoc.2007.10.013
    [60] I. De Falco, D. Maisto, U. Scafuri, E. Tarantino, A. Della Cioppa. Distributed differential evolution for the registration of remotely sensed images. In Proceedings of the 15th EUROMICRO International Conference on Parallel, Distributed and Network-based Processing, IEEE, Naples, Italy, pp.358–362, 2007. DOI: 10.1109/PDP.2007.36.
    [61] I. De Falco, A. Della Cioppa, D. Maisto, U. Scafuri, E. Tarantino. Distributed differential evolution for the registration of satellite and multimodal medical imagery. Evolutionary Image Analysis and Signal Processing, S. Cagnoni, Ed., Berlin, Heidelberg: Springer, pp.153–169, 2009. DOI: 10.1007/978-3-642-01636-3_9.
    [62] Z. B. Hu, W. Y. Gong, Z. H. Cai.  Multi-resolution remote sensing image registration using differential evolution with adaptive strategy selection[J]. Optical Engineering, 2012, 51(10): 101707-. doi: 10.1117/1.OE.51.10.101707
    [63] W. P. Ma, X. F. Fan, Y. Wu, L. C. Jiao.  An orthogonal learning differential evolution algorithm for remote sensing image registration[J]. Mathematical Problems in Engineering, 2014, 2014(): 305980-. doi: 10.1155/2014/305980
    [64] Y. Lu, Z. W. Liao, W. F. Chen. An automatic registration framework using quantum particle swarm optimization for remote sensing images. In Proceedings of International Conference on Wavelet Analysis and Pattern Recognition, IEEE, Beijing, China, pp.484–488, 2007. DOI: 10.1109/ICWAPR.2007.4420718.
    [65] Y. Zhang, Y. Guo, Y. F. Gu, W. Z. Zhong. Particle swarm optimization with powell′s direction set method for remote sensing image registration. In Proceedings of the 5th International Conference on Natural Computation, IEEE, Tianjin, China, pp.388–392, 2009. DOI: 10.1109/ICNC.2009.402.
    [66] R. An, C. Y. Chen, H. L. Wang. An improved particle swarm optimization algorithm for image matching. In Proceedings of International Forum on Computer Science-Technology and Applications, IEEE, Chongqing, China, pp.7–10, 2009. DOI: 10.1109/IFCSTA.2009.8.
    [67] R. Gharbia, S. A. Ahmed, A. ella Hassanien. Remote sensing image registration based on particle swarm optimization and mutual information. In Proceedings of the 2nd International Conference INDIA, Vol. 2, pp.399–408, 2015. DOI: 10.1007/978-81-322-2247-7_41.
    [68] S. Yavari, M. J. Valadan Zoej, M. R. Sahebi, M. Mokhtarzade.  Accuracy improvement of high resolution satellite image georeferencing using an optimized line-based rational function model[J]. International Journal of Remote Sensing, 2018, 39(6): 1655-1670. doi: 10.1080/01431161.2017.1410294
    [69] Y. Wu, Q. G. Miao, W. P. Ma, M. G. Gong, S. F. Wang.  PSOSAC: Particle swarm optimization sample consensus algorithm for remote sensing image registration[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(2): 242-246. doi: 10.1109/LGRS.2017.2783879
    [70] Y. Wu, W. P. Ma, Q. G. Miao, S. F. Wang.  Multimodal continuous ant colony optimization for multisensor remote sensing image registration with local search[J]. Swarm and Evolutionary Computation, 2019, 47(): 89-95. doi: 10.1016/j.swevo.2017.07.004
    [71] M. Črepinšek, S. H. Liu, M. Mernik.  Exploration and exploitation in evolutionary algorithms: A survey[J]. ACM Computing Surveys, 2013, 45(3): 35-. doi: 10.1145/2480741.2480752
    [72] J. H. Holland. Adaption in Natural and Artificial Systems. Michigan, USA : The University of Michigan Press, 1975.
    [73] N. Saini.  Review of selection methods in genetic algorithms[J]. International Journal of Engineering and Computer Science, 2017, 6(12): 22261-22263.
    [74] P. Sharma, H. Sharma, J. C. Bansal.  Effective competency based differential evolution algorithm[J]. Journal of Statistics and Management Systems, 2019, 22(7): 1223-1238. doi: 10.1080/09720510.2019.1609560
    [75] V. Kachitvichyanukul.  Comparison of three evolutionary algorithms: GA, PSO, and DE[J]. Industrial Engineering and Management Systems, 2012, 11(3): 215-223. doi: 10.7232/iems.2012.11.3.215
    [76] S. Das, S. S. Mullick, P. N. Suganthan.  Recent advances in differential evolution − an updated survey[J]. Swarm and Evolutionary Computation, 2016, 27(): 1-30. doi: 10.1016/j.swevo.2016.01.004
    [77] P. K. Yadav, N. L. Prajapati.  An overview of genetic algorithm and modeling[J]. International Journal of Scientific and Research Publications, 2012, 2(9): 1-4.
    [78] R. D. Al-Dabbagh, F. Neri, N. Idris, M. S. Baba.  Algorithmic design issues in adaptive differential evolution schemes: Review and taxonomy[J]. Swarm and Evolutionary Computation, 2018, 43(): 284-311. doi: 10.1016/j.swevo.2018.03.008
    [79] R. Storn, K. Price.  Differential evolution − a simple and efficient heuristic for global optimization over continuous spaces[J]. Journal of Global Optimization, 1997, 11(4): 341-359. doi: 10.1023/A:1008202821328
    [80] R. S. Parpinelli, H. S. Lopes.  New inspirations in swarm intelligence: A survey[J]. International Journal of Bio-Inspired Computation, 2011, 3(1): 1-16. doi: 10.1504/IJBIC.2011.038700
    [81] C. Kolias, G. Kambourakis, M. Maragoudakis.  Swarm intelligence in intrusion detection: A survey[J]. Computers & Security, 2011, 30(8): 625-642. doi: 10.1016/j.cose.2011.08.009
    [82] I. Fister Jr, X. S. Yang, I. Fister, J. Brest, D. Fister. A brief review of nature-inspired algorithms for optimization. https://arxiv.org/abs/1307.4186, 2013.
    [83] J. kennedy, R. Eberhart. Particle swarm optimization. In Proceedings of IEEE International conference on Neural Netuorks, Perth, Australia, pp.1942−1948, 1995.
    [84] Q. H. Bai.  Analysis of particle swarm optimization algorithm[J]. Computer and Information Science, 2010, 3(1): 180-184.
    [85] D. P. Rini, S. M. Shamsuddin, S. S. Yuhaniz.  Particle swarm optimization: Technique, system and challenges[J]. International Journal of Computer Applications, 2011, 14(1): 19-27. doi: 10.5120/1810-2331
    [86] M. Dorigo, V. Maniezzo, A. Colorni.  Ant system: optimization by a colony of cooperating agents[J]. IEEE Transactions on Systems, Man, and Cybernetics − Part B, 1996, 26(1): 29-41.
    [87] W. Peng, R. F. Tong, G. P. Qian, J. X. Dong. A constrained ant colony algorithm for image registration. In Proceedings of International Conference on Intelligent Computing, Computational Intelligence and Bioinformatics, Springer, Kunming, China, pp.1–11, 2006. DOI: 10.1007/11816102_1.
    [88] F. Yang, H. L. Zhang.  Multiresolution 3D image registration using hybrid ant colony algorithm and Powell′s method[J]. Journal of Electronics & Information Technology, 2007, 29(3): 622-625.
    [89] H. Rezaei, M. Shakeri, S. Azadi, K. Jaferzade. Multimodality image registration utilizing ant colony algorithm. In Proceedings of the 2nd International Conference on Machine Vision, IEEE, Dubai, Emirates, pp.49–53, 2009. DOI: 10.1109/ICMV.2009.21.
    [90] G. S. Raghtate, S. S. Salankar. Modified fuzzy c means with optimized ant colony algorithm for image segmentation. In Proceedings of International Conference on Computational Intelligence and Communication Networks, IEEE, Jabalpur, India, pp.1283–1288, 2015. DOI: 10.1109/CICN.2015.246.
    [91] B. Khorram, M. Yazdi.  A new optimized thresholding method using ant colony algorithm for MR brain image segmentation[J]. Journal of Digital Imaging, 2019, 32(1): 162-174. doi: 10.1007/s10278-018-0111-x
    [92] G. H. Zou.  Ant colony clustering algorithm and improved Markov random fusion algorithm in image segmentation of brain images[J]. International Journal Bioautomation, 2016, 20(4): 505-514.
    [93] G. Rellier, X. Descombes, J. Zerubia.  Local registration and deformation of a road cartographic database on a spot satellite image[J]. Pattern Recognition, 2002, 35(10): 2213-2221. doi: 10.1016/S0031-3203(01)00180-7
    [94] X. G. Du, J. W. Dang. Y. P. Wang, X. G. Liu, S. Li. An algorithm multi-resolution medical image registration based on firefly algorithm and Powell. In Proceedings of the 3rd International Conference on Intelligent System Design and Engineering Applications, IEEE, Hong Kong, China, pp.274–277, 2013. DOI: 10.1109/ISDEA.2012.68.
    [95] J. Zhang, J. L. Hu. A novel registration method based on coevolutionary strategy. In Proceedings of IEEE Congress on Evolutionary Computation, IEEE, Vancouver, Canada, pp.2375–2380, 2016. DOI: 10.1109/CEC.2016.7744082.
    [96] X. S. Yang, X. S. He.  Bat algorithm: Literature review and applications[J]. International Journal of Bio-inspired Computation, 2013, 5(3): 141-149. doi: 10.1504/IJBIC.2013.055093
    [97] Y. H. Shi. Brain storm optimization algorithm. In Proceedings of the 2nd International Conference in Swarm Intelligence, Springer, Chongqing, China, pp.303–309, 2011. DOI: 10.1007/978-3-642-21515-5_36.
    [98] D. G. Lowe.  Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110. doi: 10.1023/B:VISI.0000029664.99615.94
    [99] E. Shelhamer, J. Long, T. Darrell.  Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 39(4): 640-651. doi: 10.1109/TPAMI.2016.2572683
    [100] L. C. Chen, Y. K. Zhu, G. Papandreou, F. Schroff, H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp.833–851, 2018.
    [101] B. Singh, M. Najibi, L. S. Davis. SNIPER: Efficient multi-scale training. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, ACM, Montreal, Canada, pp.9310–9320, 2018.
    [102] K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp.770–778, 2016. DOI: 10.1109/CVPR.2016.90.
    [103] J. Zabalza, J. C. Ren, J. B. Zheng, H. M. Zhao, C. M. Qing, Z. J. Yang, P. J. Du, S. Marshall.  Novel segmented stacked autoencoder for effective dimensionality reduction and feature extraction in hyperspectral imaging[J]. Neurocomputing, 2016, 185(): 1-10. doi: 10.1016/j.neucom.2015.11.044
    [104] Y. Q. Chen, L. C. Jiao, Y. Y. Li, J. Zhao.  Multilayer projective dictionary pair learning and sparse autoencoder for polSAR image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(12): 6683-6694. doi: 10.1109/TGRS.2017.2727067
    [105] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, ACM, Montreal, Canada, pp.2672–2680, 2014.
    [106] Y. Zhan, D. Hu, Y. T. Wang, X. C. Yu.  Semisupervised hyperspectral image classification based on generative adversarial networks[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(2): 212-216. doi: 10.1109/LGRS.2017.2780890
    [107] N. Dalal, B. Triggs. Histograms of oriented gradients for human detection. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, San Diego, USA, pp.886–893, 2005. DOI: 10.1109/CVPR.2005.177.
    [108] R. Alshehhi, P. R. Marpu, W. Lee Woon, M. Dalla Mura. Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks. ISPRS Journal of Photogrammetry and Remote Sensing, vol.130, pp.139–149, 2017. DOI: 10.1016/j.isprsjprs.2017.05.002.
    [109] W. P. Ma, J. Zhang, Y. Wu, L. C. Jiao, H. Zhu, W. Zhao.  A novel two-step registration method for remote sensing images based on deep and local features[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(7): 4834-4843. doi: 10.1109/TGRS.2019.2893310
    [110] W. P. Ma, Z. L. Wen, Y. Wu, L. C. Jiao, M. G. Gong, Y. F. Zheng, L. Liu.  Remote sensing image registration with modified SIFT and enhanced feature matching[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(1): 3-7. doi: 10.1109/LGRS.2016.2600858
    [111] Y. Y. Dong, W. L. Jiao, T. F. Long, L. F. Liu, G. J. He, C. J. Gong, Y. T. Guo.  Local deep descriptor for remote sensing image feature matching[J]. Remote Sensing, 2019, 11(4): 430-. doi: 10.3390/rs11040430
    [112] A. Sedaghat, M. Mokhtarzade, H. Ebadi.  Uniform robust scale-invariant feature matching for optical remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 49(11): 4516-4527. doi: 10.1109/TGRS.2011.2144607
    [113] A. Sedaghat, H. Ebadi.  Remote sensing image matching based on adaptive binning SIFT descriptor[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(10): 5283-5293. doi: 10.1109/TGRS.2015.2420659
    [114] J. T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller. Striving for simplicity: The all convolutional net. https://arxiv.org/abs/1412.6806, 2014.
    [115] X. F. Han, T. Leung, Y. Q. Jia, R. Sukthankar, A. C. Berg. MatchNet: Unifying feature and metric learning for patch-based matching. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Boston, USA, pp.3279–3286, 2015. DOI: 10.1109/CVPR.2015.7298948.
    [116] Z. Yang, T. Dan, Y. Yang. Multi-temperal remote sensing image registration using deep convolutional features. IEEE Access, vol. 6, pp. 38544–38555, 2018.
    [117] Y. Yang, S. H. Ong, K. W. C. Foong.  A robust global and local mixture distance based non-rigid point set registration[J]. Pattern Recognition, 2015, 48(1): 156-173. doi: 10.1016/j.patcog.2014.06.017
    [118] S. Zhang, Y. Yang, K. Yang, Y. Luo, S. H. Ong. Point set registration with global-local correspondence and transformation estimation. In Proceedings of International Conference on Computer Vision, IEEE, Venice, Italy, pp.2669–2677, 2017. DOI: 10.1109/ICCV.2017.291.
    [119] S. Wang, D. Quan, X. F. Liang, M. D. Ning, Y. H. Guo, L. C. Jiao.  A deep learning framework for remote sensing image registration[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 145(): 148-164. doi: 10.1016/j.isprsjprs.2017.12.012
    [120] D. Quan, S. Wang, X. F. Liang, R. J. Wang, S. Fang, B. Hou, L. C. Jiao. Deep generative matching network for optical and SAR image registration. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, IEEE, Valencia, Spain, pp.6215–6218, 2018. DOI: 10.1109/IGARSS.2018.8518653.
    [121] D. G. Kim, W. J. Nam, S. W. Lee. A robust matching network for gradually estimating geometric transformation on remote sensing imagery. In IEEE International Conference on Systems, Man and Cybernetics, IEEE, Bari, Italy, pp.3889–3894, 2019. DOI: 10.1109/SMC.2019.8913881.
    [122] M. Vakalopoulou, S. Christodoulidis, M. Sahasrabudhe, S. Mougiakakou, N. Paragios. Image registration of satellite imagery with deep convolutional neural networks. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, IEEE, Yokohama, Japan, pp.4939–4942, 2019. DOI: 10.1109/IGARSS.2019.8898220.
    [123] J. H. Park, W. J. Nam, S. W. Lee.  A two-stream symmetric network with bidirectional ensemble for aerial image matching[J]. Remote Sensing, 2020, 12(3): 465-. doi: 10.3390/rs12030465
    [124] J. Bromley, I. Guyon, Y. LeCun, E. Sackinger, R. Shah. Signature verification using a “Siamese” time delay neural network. In Proceedings of the 6th International Conference on Neural Information Processing Systems, Denver, USA, pp. 737–744, 1993.
    [125] L. H. Hughes, M. Schmitt, L. C. Mou, Y. Y. Wang, X. X. Zhu.  Identifying corresponding patches in SAR and optical images with a pseudo-Siamese CNN[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(5): 784-788. doi: 10.1109/LGRS.2018.2799232
    [126] H. Zhang, W. P. Ni, W. D. Yan, D. L. Xiang, J. Z. Wu, X. L. Yang, H. Bian.  Registration of multimodal remote sensing image based on deep fully convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(8): 3028-3042. doi: 10.1109/JSTARS.2019.2916560
    [127] A. Mishchuk, D. Mishkin, F. Radenovic, J. Matas. Working hard to know your neighbor′s margins: Local descriptor learning loss. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc., Long Beach, USA, pp.4826–4837, 2017.
    [128] C. Harris, M. Stephens. A combined corner and edge detector. In Proceedings of the 4th Alvey Vision Conference, Alvey Vision Club, Manchester, UK, pp.147–151, 1988.
    [129] N. Merkle, W. J. Luo, S. Auer, R. Muller, R. Urtasun.  Exploiting deep matching and sar data for the geo-localization accuracy improvement of optical satellite images[J]. Remote Sensing, 2017, 9(6): 586-. doi: 10.3390/rs9060586
    [130] W. J. Luo, A. G. Schwing, R. Urtasun. Efficient deep learning for stereo matching. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp.5695–5703, 2016. DOI: 10.1109/CVPR.2016.614.
    [131] H. Q. He, M. Chen, T. Chen, D. J. Li.  Matching of remote sensing images with complex background variations via Siamese convolutional neural network[J]. Remote Sensing, 2018, 10(2): 355-. doi: 10.3390/rs10020355
    [132] M. A. Fischler, R. C. Bolles.  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981, 24(6): 381-395. doi: 10.1145/358669.358692
    [133] H. Bay, T. Tuytelaars, L. Van Gool. SURF: Speeded up robust features. In Proceedings of the 9th European Conference on Computer Vision, Springer, Graz, Austria, pp.404–417, 2006. DOI: 10.1007/11744023_32.
    [134] J. M. Morel, G. S. Yu.  ASIFT: A new framework for fully affine invariant image comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2): 438-469. doi: 10.1137/080732730
  • 加载中
  • [1] Wei Jia, Jian Gao, Wei Xia, Yang Zhao, Hai Min, Jing-Ting Lu. A Performance Evaluation of Classic Convolutional Neural Networks for 2D and 3D Palmprint and Palm Vein Recognition . International Journal of Automation and Computing,  doi: 10.1007/s11633-020-1257-9
    [2] Farid Menacer, Abdelmalek Kadr, Zohir Dibi. Modeling of a Smart Nano Force Sensor Using Finite Elements and Neural Networks . International Journal of Automation and Computing,  doi: 10.1007/s11633-018-1155-6
    [3] Punyanuch Borwarnginn, Worapan Kusakunniran, Sarattha Karnjanapreechakorn, Kittikhun Thongkanchorn. Knowing Your Dog Breed: Identifying a Dog Breed with Deep Learning . International Journal of Automation and Computing,  doi: 10.1007/s11633-020-1261-0
    [4] Fu-Qiang Liu, Zong-Yi Wang. Automatic “Ground Truth” Annotation and Industrial Workpiece Dataset Generation for Deep Learning . International Journal of Automation and Computing,  doi: 10.1007/s11633-020-1221-8
    [5] Bin Hu, Jiacun Wang. Deep Learning Based Hand Gesture Recognition and UAV Flight Controls . International Journal of Automation and Computing,  doi: 10.1007/s11633-019-1194-7
    [6] Kittinun Aukkapinyo, Suchakree Sawangwong, Parintorn Pooyoi, Worapan Kusakunniran. Localization and Classification of Rice-grain Images Using Region Proposals-based Convolutional Neural Network . International Journal of Automation and Computing,  doi: 10.1007/s11633-019-1207-6
    [7] Viet-Anh Le, Hai-Xuan Le, Linh Nguyen, Minh-Xuan Phan. An Efficient Adaptive Hierarchical Sliding Mode Control Strategy Using Neural Networks for 3D Overhead Cranes . International Journal of Automation and Computing,  doi: 10.1007/s11633-019-1174-y
    [8] Xiang Zhang, Qiang Yang. Transfer Hierarchical Attention Network for Generative Dialog System . International Journal of Automation and Computing,  doi: 10.1007/s11633-019-1200-0
    [9] Viet Khanh Ha, Jin-Chang Ren, Xin-Ying Xu, Sophia Zhao, Gang Xie, Valentin Masero, Amir Hussain. Deep Learning Based Single Image Super-resolution: A Survey . International Journal of Automation and Computing,  doi: 10.1007/s11633-019-1183-x
    [10] Zhen-Jie Yao, Jie Bi, Yi-Xin Chen. Applying Deep Learning to Individual and Community Health Monitoring Data: A Survey . International Journal of Automation and Computing,  doi: 10.1007/s11633-018-1136-9
    [11] Bo Zhao, Jiashi Feng, Xiao Wu, Shuicheng Yan. A Survey on Deep Learning-based Fine-grained Object Classification and Semantic Segmentation . International Journal of Automation and Computing,  doi: 10.1007/s11633-017-1053-3
    [12] Guo-Bing Zhou, Jianxin Wu, Chen-Lin Zhang, Zhi-Hua Zhou. Minimal Gated Unit for Recurrent Neural Networks . International Journal of Automation and Computing,  doi: 10.1007/s11633-016-1006-2
    [13] Yu Liu,  Rong-Hu Chi,  Zhong-Sheng Hou. Neural Network State Learning Based Adaptive Terminal ILC for Tracking Iteration-varying Target Points . International Journal of Automation and Computing,  doi: 10.1007/s11633-015-0891-0
    [14] Bao-Chang Xu,  Ying-Ying Zhang. An Improved Gravitational Search Algorithm for Dynamic Neural Network Identification . International Journal of Automation and Computing,  doi: 10.1007/s11633-014-0810-9
    [15] An Optimization Algorithm Employing Multiple Metamodels and Optimizers . International Journal of Automation and Computing,  doi: 10.1007/s11633-013-0716-y
    [16] Yan Sun,  Bao-Tong Cui. Dissipativity Analysis of Neural Networks with Time-varying Delays . International Journal of Automation and Computing,  doi: 10.1007/s11633-008-0290-x
    [17] S. C. Chiam,  K. C. Tan,  A. Al Mamum. Evolutionary Multi-objective Portfolio Optimization in Practical Context . International Journal of Automation and Computing,  doi: 10.1007/s11633-008-0067-2
    [18] Francisco Flórez-Revuelta, José Manuel Casado-Díaz, Lucas Martínez-Bernabeu. An Evolutionary Approach to the Delineation of Functional Areas Based on Travel-to-work Flows . International Journal of Automation and Computing,  doi: 10.1007/s11633-008-0010-6
    [19] Computational Intelligence Determines Effective Rationality . International Journal of Automation and Computing,  doi: 10.1007/s11633-008-0063-6
    [20] Wei-Hua Gui,  Chun-Hua Yang,  Jing Teng. Intelligent Fault Diagnosis in Lead-zinc Smelting Process . International Journal of Automation and Computing,  doi: 10.1007/s11633-007-0135-z
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Figures (3)  / Tables (5)

Metrics

Abstract Views (37) PDF downloads (7) Citations (0)

Computational Intelligence in Remote Sensing Image Registration: A survey

Fund Project:  Recommended by Associate Editor De Xu

Abstract: In recent years, computational intelligence has been widely used in many fields and achieved remarkable performance. Evolutionary computing and deep learning are important branches of computational intelligence. Many methods based on evolutionary computation and deep learning have achieved good performance in remote sensing image registration. This paper introduces the application of computational intelligence in remote sensing image registration from the two directions of evolutionary computing and deep learning. In the part of remote sensing image registration based on evolutionary calculation, the principles of evolutionary algorithms and swarm intelligence algorithms are elaborated and their application in remote sensing image registration is discussed. The application of deep learning in remote sensing image registration is also discussed. At the same time, the development status and future of remote sensing image registration are summarized and their prospects are examined.

Yue Wu, Jun-Wei Liu, Chen-Zhuo Zhu, Zhuang-Fei Bai, Qi-Guang Miao, Wen-Ping Ma, Mao-Guo Gong. Computational Intelligence in Remote Sensing Image Registration: A survey. International Journal of Automation and Computing. doi: 10.1007/s11633-020-1248-x
Citation: Yue Wu, Jun-Wei Liu, Chen-Zhuo Zhu, Zhuang-Fei Bai, Qi-Guang Miao, Wen-Ping Ma, Mao-Guo Gong. Computational Intelligence in Remote Sensing Image Registration: A survey. International Journal of Automation and Computing. doi: 10.1007/s11633-020-1248-x
    • Remote sensing image registration has important applications in remote sensing image processing, and is a basic problem in many remote sensing information extraction and processing technologies. In the practical application of remote sensing images, registration is an indispensable step in the process of image fusion[1], change detection[2], feature recognition[3], image mosaics[4], image segmentation[5] and image classification[6,7]. Many later theories and applications are carried out on the assumption that the registration problem has been solved. The accuracy of remote sensing image registration will directly affect the accuracy and application effects of the final application results. In recent years, with the continuous improvement of human ability in observing the earth[8], the registration of remote sensing images has made great progress[9]. The purpose of image registration is trying to match two or more images which are about the same object but taken in different situations, such as with different time, different viewpoints and different sensors[10,11]. Compared with natural image registration, remote sensing image registration is more challenging because remote sensing images have more complex features[12]. The procedure of remote sensing image registration can be divided into five steps[10], which are preprocessing, feature selection, feature correspondence, determining transformation functions, and resampling. In this paper, we mainly discuss the applications of computational intelligence in remote sensing image registration, which mainly includes evolutionary algorithms[13,14] and deep learning[15,16].

      Evolutionary algorithms have been widely used in remote sensing image registration, and have achieved great performance[17,18]. An evolutionary algorithm is a heuristic global optimization probability search algorithm, of which the principle is to simulate the evolution of the survival of the fittest in the ecosystem[19,20]. Compared with traditional optimization algorithms, evolutionary algorithms have the following advantages:

      1) Evolutionary algorithms do not require the strict definitions of mathematical models to solve problems[21]. When the actual application requirements are abstracted as an optimization problem, the evolutionary algorithm could make full use of the corresponding fitness function to find the approximate optimal solution without relying on other information, and the actual application requirements could be satisfied.

      2) In the optimization process, the traditional optimization strategy is to start from an initial value and then iterate step by step to find the optimal solution, but the evolutionary algorithm starts from a population that contains many points, and finds the optimal or suboptimal solution through the evolution iteration of the population[22].

      3) The robustness and embedded parallelism of the evolutionary algorithm makes the algorithm very effective in finding the global optimal solution, and it is not easy to fall into the dilemma of the local optimal solution, and realize the global optimization in the probability sense[23].

      Because the task of remote sensing image registration could be regarded as an optimization problem by establishing a reasonable model and evolutionary algorithms have great performance in optimization problems, evolutionary algorithm has important applications in solving the task of remote sensing image registration[24].

      Due to the powerful learning ability of deep learning, it has been widely used in computer vision tasks[25], such as image registration[26], object detection[27], change detection[28] and image fusion[29]. Especially with the introduction of a series of outstanding feature extraction networks, such as AlexNet[30], VGGNet[31], and GoogleNet[32], remote sensing image registration based on deep learning has achieved satisfactory performance. The convolutional neural networks (CNNs) simulate the connection pattern between neurons in animal visual cortical tissue, which are a deep learning structure of a multilayer feed-forward artificial neural network[33]. CNNs are generally composed of multiple convolutional layers, pooling layers and fully connected layers connected to each other[34]. The convolution layer uses various convolution kernels to perform convolution operations on the input to extract various features[35]. The pooling layer performs dimensionality reduction on the input through pooling operations, thereby reducing the number of network parameters[36]. The fully connected layer is usually the last part of CNNs and is a traditional multilayer perceptron network[37]. Each neuron is connected to each neuron of the previous layer. The powerful feature extraction and representation capabilities of CNNs can overcome the instability of low-level features and improve the reliability of registration[38].

      Compared with natural images, remote sensing images could be obtained in various ways, the image content is difficult to understand, and their features are not obvious. The method of manually designed features has a narrow scope of application in remote sensing images. It is impossible to predict uncontrollable changes in remote sensing images, and it is difficult to extract discriminative features. For the traditional method, there is no feedback of information between feature extraction and feature matching. Deep neural networks have powerful learning capabilities, which can extract higher-dimensional features for registration. Deep learning further improves the accuracy and robustness of remote sensing image registration[38].

      Computational intelligence is intelligent[39], parallel[40] and robust[24], and does not depend on the characteristics of the problem itself. The computational intelligence-based algorithms are basically optimized for solving problems in a group collaboration manner, which is very suitable for large-scale parallel processing. Algorithms based on computational intelligence have good fault tolerance and are insensitive to initial conditions. It can find optimal solutions under different conditions.

      The remainder of this paper is organized as follows. In Table 1, we show the abbreviations that appear in the text. The remote sensing image registration based on the evolutionary algorithm is described in detail in Section 2. The remote sensing image registration based on deep learning is reported in Section 3, and Section 4 draws the conclusions.

      Abbreviation Full name
      PSO Particle swarm optimization
      Gbest Optimal position found by the group
      Pbest Optimal position found by the particles
      SAR Synthetic aperture radar
      CNN Convolutional neural networks
      AE Autoencoder
      GANs Generative adversarial networks
      DNN Deep neural networks
      HOG Histogram of oriented gradient
      RANSAC Random sample consensus
      SIFT Scale invariant feature transform
      SURF Speeded up robust features
      ASIFT Affine-SIFT
      VGGNet Visual geometry group network

      Table 1.  Abbreviation table

    • Evolutionary computing[41] is a kind of adaptive artificial intelligence technology that can simulate the evolution processes and mechanisms of biology to solve problems. There are many methods based on evolutionary computation that have been surveyed in the field of natural image registration[4246] and medical image registration[47-51]. Although not all the algorithms are effective in remote sensing image registration, it has been proved that some algorithms can achieve better performance in remote sensing image registration. According to the frequency of various algorithms used in remote sensing image registration, this paper mainly divides them into two parts: remote sensing image registration based on evolutionary algorithms[5263] and remote sensing image registration based on swarm intelligence algorithms[64-70].

    • Evolutionary algorithms[71] are inspired by biological evolution and genetic theory. An evolutionary algorithm is a general problem-solving algorithm based on natural selection and population genetics. It uses the selection mechanism of survival of the fittest to find the most suitable solution. Among many evolutionary algorithms, genetic algorithms[72,73] are more widely used in remote sensing image registration than genetic programming, evolution strategies, and evolutionary planning because of their relatively mature development. At the same time, as a new and efficient heuristic parallel search technology, the differential evolution algorithm[74-76] has the advantages of fast convergence, fewer control parameters and robust optimization results, so its application in remote sensing image registration has also been studied. Next, we give a brief overview of genetic algorithms and differential evolution algorithms, and discuss the applications of genetic algorithms[52-57] and differential evolution algorithms[58-63] in remote sensing image registration. Table 2 shows a summary of the application of evolutionary algorithms in remote sensing image registration.

      Author Year Method improvement Advantages
      Inglada and Adragna[52] 2001 Using genetic algorithm to find the optimal set of control points. Avoid local minima.
      Makrogiannis et al.[53] 2004 Combining coarse registration with genetic algorithm and fine registration with scale optical flow method. Achieve higher accuracy registration while avoiding excessive calculation.
      De Falco et al.[58] 2007 Using differential evolution for remote sensing image registration. Innovative introduction of differential evolution algorithm to avoid local optimization.
      De Falco et al.[61] 2009 Applying distributed differential evolution to remote sensing image registration. Reduce the risk of population particle stagnation.
      Hu et al.[62] 2012 An adaptive differential evolution strategy is proposed, which combines probability matching and multi-resolution techniques to register remote sensing images. Probability matching technology is used in DE to independently choose the most suitable strategy when solving problems.
      Ma et al.[63] 2014 Orthogonal learning strategy applied to remote sensing image registration. Use fewer experimental samples to find the best combination between levels of factors.
      Zhang et al.[54] 2014 Applying genetic algorithm to the combination of coarse matching with grid matching and fine matching with triangle mapping. Reduce the amount of calculation and reduce the error of manual decision.
      Gou and Ma[55] 2016 Construct a graph by delaunay triangulation, and solve the graph similarity by genetic algorithm to find the best matching point. In the case of large gray scale differences, the accuracy is still high.
      Li et al.[56] 2016 The genetic algorithm mainly corrects the image rotation to preprocess the image. More matching control points can be obtained to improve the accuracy of registration.
      Yavari et al.[57] 2018 Use genetic algorithm to get the best quantity and best combination of well-distributed ground control information. Avoid the influence of some manual operations on the experiment in the registration process and improve the accuracy of registration.

      Table 2.  Application research of evolutionary algorithm

    • A genetic algorithm was first proposed by Holland, a professor of the University of Michigan[72]. It is a computational model simulating the natural selection and genetic mechanisms of Darwinian biological evolution, and a method of searching the optimal solution by simulating the natural evolution[77]. Genetic algorithms are mainly used in feature matching[52,54,55,57] and parameter estimation of transformation models[53,56] in remote sensing image registration. We combine the basic flow of genetic algorithms and take [55,57] for example. The basic flow chart of the algorithms for applying genetic algorithm to the feature matching step in the image registration process is shown in Fig. 1. First of all, we use the existing methods to roughly match the information of the reference image and the sensed image. Secondly, we encode the information after the coarse matching to construct the initial population. And then, the population was randomly selected according to the fitness function value, and the population position was updated by crossover and mutation population coding. The fitness function is calculated to update the optimal solution and judge the convergence condition. If the convergence condition is satisfied, the optimal solution is output, otherwise the crossover and mutation operation is performed again. Next, the transformation model is estimated, and finally the registration image is obtained.

      Figure 1.  Remote sensing image registration-genetic algorithm flow chart

      When using genetic algorithms to solve specific problems, the coding rules of chromosomes, the selection of fitness function, the design of genetic operators and the determination of algorithm parameters need to be analyzed for specific problems. This is also the difficulty of using genetic algorithms to solve problems. Therefore, the rational application of genetic algorithms is the key to solving the problem.

    • Genetic algorithms are mainly used in feature matching and parameter estimation of transformation models in remote sensing image registration. Next, we will discuss these two parts. In [52,54,55,57], a genetic algorithm is applied to feature matching in image registration. The genetic algorithms are applied to the parameter estimation of transformation models in image registration. Next, we discuss them separately[53-56].

      Using the genetic algorithm for matching. Inglada and Adragna[52] proposed automatic multisensor image registration by edge matching using genetic algorithms. A genetic algorithm is used to find the optimal set of control points by minimizing the distance on the basis of the proposed minimization measurement method. This method does not require too high extraction accuracy to achieve high registration accuracy, and at the same time, the use of genetic algorithms avoids the problem of local optimal solutions. In order to register aerial images as satellite images or map images, Zhang et al.[54] proposed an image registration with position and similarity constraints based on a genetic algorithm. The genetic algorithm is mainly to minimize the mean square error of the two grids and minimize the total deformation of the grid points. Compared with similar methods in the past, the main advantage of this method is to avoid errors in the manual operation and manual decision-making process. Gou and Ma[55] presented a method based on delaunay triangulation and a genetic algorithm for multi-source remote sensing registration. The genetic algorithm mainly solves the optimal solution of graph similarity, so as to obtain the optimal set of control points for solving the transformation model. A prominent point of this method is to use the genetic algorithm to eliminate mismatches and find the best set of corresponding points, so as to improve the accuracy of registration. Yavari et al.[57] proposed a method for selecting the best distribution and the best amount of ground control information based on genetic algorithms to improve the accuracy of registration. Compared with the past methods, this method is a significant achievement in reducing the errors caused by the extraction process and achieving sub-pixel precision registration.

      Using genetic algorithms to estimate the transformation parameters. Makrogiannis et al.[53] proposed an affine mapping scheme based on random optimization in order to solve the distortion caused by changes in time, viewpoint and terrain in remote sensing images. The genetic algorithm is used to search the searching space defined by the affine transformation parameters in the coarse registration stage. In order to achieve sub-pixel registration and avoid the high cost of genetic algorithms, the registration scheme is completed by a hierarchical optical flow approach. The combination of coarse registration and fine registration reduces the computational complexity while ensuring registration accuracy. Li et al.[56] proposed a multi-angle normalized cross-correlation method based on genetic optimization. This method can determine the maximum value of the correlation coefficient between two images through the genetic algorithm, so as to detect the angle offset between the target image and the reference image, and complete the rotation offset correction. The advantage of this newly added preprocessing operation is that more matching points can be obtained, thereby improving the accuracy of registration. The authors used a genetic algorithm to preprocess the registration image, which provided a new way for us to solve the problem of image registration.

    • A differential evolution algorithm is a heuristic random search algorithm based on population difference. The algorithm is proposed to solve Chebyshev polynomials[22,78,79]. A differential evolution algorithm is successfully applied to remote sensing image registration due to its simple principle, few controlled parameters and strong robustness. According to the existing references, a differential evolution algorithm is only used for parameter estimation of a transformation model.

      Combined with the basic flow of the differential evolution algorithm, the flow chart of the algorithm applied to remote sensing image registration is shown Fig. 2. Firstly, the reference image and the image to be registered are input, and the parameters of the transformation model are randomly initialized in the search space of the transformation parameters. Then, according to the fitness function, the operations of mutation, crossover and selection are carried out until the output conditions are met. Finally, according to the optimal transformation parameters, the image to be registered is matched with the reference image to obtain the final image registration result.

      Figure 2.  Remote sensing image registration-differential evolution algorithm flow chart

    • De Falco et al.[58,59] proposed a differential evolution for the registration of remotely sensed images. The authors innovatively introduce differential evolution into remote sensing image registration to effectively avoid the problem of local optimization. In order to increase the chance of finding the optimal value, De Falco et al.[60,61] applied the distributed difference algorithm to remote sensing image registration. This algorithm reduces the risk of subpopulation stagnation of the differential evolution algorithm. The distributed difference algorithm is used to solve the global optimal parameters of the affine transformation to maximize mutual information. Hu et al.[62] proposed a remote sensing image registration algorithm based on the differential evolution algorithm and adaptive strategy selection. The method is mainly used to solve the traditional differential evolution algorithm used for remote sensing image registration when the calculation is large and the convergence speed is slow. Compared with traditional differential evolution algorithms, this algorithm can achieve sub-pixel accuracy registration. Ma et al.[63] proposed an algorithm based on orthogonal learning differential evolution for remote sensing image registration. The salient point of this method is that orthogonal learning strategies are used to optimize the crossover process in differential evolution, so that people use fewer experimental samples to find the best combination between various factor levels. This method has faster convergence speed and more robust result than the previous meta-heuristic algorithms in remote sensing images.

      Some problems in remote sensing image registration can be solved by using evolutionary algorithms. Through reasonable chromosome coding and an adaptive cross mutation strategy, the convergence speed of the algorithm can be significantly improved under the premise of ensuring the diversity of solution sets. The application of evolutionary algorithm in remote sensing image registration provides more possibilities for remote sensing image registration. How to use the improved evolutionary algorithm to solve the optimization problem of remote sensing image registration parameters is still an important research question in the future.

    • Swarm intelligence algorithms[80-82] are widely used in remote sensing image registration, and particle swarm optimization algorithms play an important role because of their outstanding performance. Next, the principles and applications of particle swarm optimization algorithms are introduced[64-69]. In addition to particle swarm optimization algorithms, ant colony algorithms have been applied to remote sensing image registrations[70], so their applications will be briefly introduced. According to the existing references, particle swarm algorithms and ant colony algorithms are all applied to the parameter estimation of transformation models. Table 3 shows a summary of the applications of swarm intelligence algorithms in remote sensing image registration.

      Author Year Method improvement Advantages
      Lu et al.[64] 2007 Combination of coarse registration using particle swarm and fine registration using Harris detector. Large displacement between remote sensing images can be processed quickly.
      Zhang et al.[65] 2009 Combine the PSO algorithm with the Powell direction set algorithm. Avoid local optimization and realize accurate registration.
      An et al.[66] 2009 Innovative application of improved particle swarm optimization algorithm in remote sensing image registration. Control the amount of calculation and ensure the diversity of population.
      Gharbia et al.[67] 2015 Combination of mutual information and thin plate spline method. This method has better performance for large-scale variation, rotation and intensity variation.
      Yavari et al.[68] 2017 Remote sensing image registration using line-based rational function module with particle swarm optimization. Improve registration accuracy and reduce system errors and the amount of control information required.
      Wu et al.[69] 2018 Relative to RANSAC, the model transformation parameters are directly sampled, rather than randomly selected for trial matching. Significantly improved registration accuracy compared to the existing methods.
      Wu et al.[70] 2019 Combination of ant colony algorithm and local search algorithm. Avoiding local optimization and reducing computation.

      Table 3.  Research on the application of particle swarm optimization

    • PSO algorithm was proposed by Kennedy and Eberhart in 1995[83]. Particle swarm optimization algorithms are a kind of swarm optimization algorithms that simulate the foraging activity of birds[83-85]. Particle swarm optimization is used to find the optimal transformation model parameters in remote sensing image registration[64-69]. Combined with the basic process of particle swarm optimization algorithms, the basic process applied to remote sensing image registration is shown in Fig. 3. First, randomly sample and generate particle swarms within the transform parameter range and initialize them. Second, the optimal position found by the group (Gbest) and the optimal position found by the particles (Pbest) are calculated. Finally, the fitness value of each particle is calculated, and if the stopping criterion is not met, the velocity of the particle is updated by (1), the position of the particle is updated by (2), and the Gbest of the particle swarm and the Pbest of the particle are updated. Recycling updates the fitness value of each particle. If the stopping criterion is met, the optimal model conversion parameters are output. The classical velocity and position updating rules are as follows:

      Figure 3.  Remote sensing image registration-particle swarm optimization algorithm flow chart

      $ \begin{split} V_i(t+1) = { w}V_i(t)+c_1r_1(P{ best_i}-X_i(t)) +\\ \quad c_2r_2(G{ best_i}-X_i(t)) \end{split} $

      (1)

      $ \begin{array}{l} X_i(t+1) = X_i(t)+V_i(t) \end{array} $

      (2)

      where ${{X}} _i$(t) and ${{V}}_i$(t) are the position and velocity of the i-th particle on iteration t, c$ _1 $ and c$ _2 $ are the two learning rate weights, r$ _1 $ and r$ _2 $ are the two random parameters uniformly distributed in [0, 1], w is the inertia weight.

    • In swarm intelligence algorithms, the particle swarm optimization algorithm has been widely used in remote sensing image registration because of its fast convergence speed, simplicity of the algorithm itself, and few adjustment parameters. Lu et al.[64] adopted a registration framework from coarse to fine. In the coarse matching stage, the particle swarm optimization algorithm is used to search for the best rigid parameters to obtain the globally optimal mutual information. In the stage of fine registration, the Harris detector is used for fine registration of the coarse registration results. This method has a strong automatic registration ability and can deal with large displacements between remote sensing images quickly and steadily. Zhang et al.[65] proposed to combine the PSO algorithm with the Powell direction set algorithm, in order to avoid the problem of falling into local maxima during the registration process of multisensor remote sensing images. PSO is used to compute a good initial guess for following the Powell's direction set method (PDSM). At the same time, the combination of the PSO algorithm and the Powell algorithm also effectively avoids the calculation of the objective function gradient and reduces the computational complexity. An et al.[66] proposed an improved particle swarm algorithm based on mutual information for registration. The algorithm abandons the traditional situation of fixed population size, and uses an adaptive variable population size to optimize the function. Therefore, the algorithm cannot only control the amount of calculation, but also ensure the diversity of the population and improve the search ability of the particles. But compared with the traditional exhaustion based on mutual information, the accuracy is greatly reduced. Gharbia et al.[67] proposed the mutual information and thin plate spline method to register remote sensing images. After the thin plate spline provides the geometric representation of the corresponding landmark's relative position, the particle swarm optimization algorithm is mainly used to improve the corresponding relationship between landmarks and realize the maximum mutual information function. Compared with the previous methods, this method has better performance for large-scale variation, rotation and intensity variation. Yavari et al.[68] proposed an optimized rational function model based on linear features for high-resolution satellite image registration. The rational function model is suitable for registration due to its versatility and independence from the sensor, but the combination of over-parameterization in the rational function and the incorrect number and order of terms in the rational function is the key problem to be solved. Particle swarm optimization algorithm is used to find the optimal combination of unrelated items due to its excellent global search ability. Wu et al.[69] proposed a particle swarm optimization sample consensus algorithm. The salient point of this algorithm is to abandon the traditional random sample consensus method of tentative matching, but to directly use the particle swarm algorithm to sample the transformation model. The algorithm has remarkable performance in dealing with low accuracy and more matches.

      Particle swarm optimization has the characteristic that individuals can exchange information with each other. Particle swarm optimization has the advantage of fast convergence speed, but it is easy to fall into local search and lose the diversity of the particle swarm, and the local search ability of the particle swarm algorithm is weak. Therefore, it will be a good research direction in the future to increase the population diversity with the particle swarm algorithm to enhance its search ability, and to combine the particle swarm algorithm with some local search strategies to enhance its local search ability.

    • The ant colony algorithm was first proposed by Italian researchers Dorigo and Maniezzo in the 1990s, which was inspired by ants' foraging behavior[86]. The ant colony algorithm has been widely used in natural image registration[87-89] and image detection[90-92], but in remote sensing image registration, it is still in the exploratory stage. Wu et al.[70] have made remarkable achievements in their research.

      In order to avoid falling into local optimums when solving complex multimodal problems, Wu et al.[70] proposed a multimodal continuous ant colony optimization algorithm for multisensor remote sensing image registration. The introduction of the ant colony algorithm mainly avoids falling into the local optimal situation and improves the global search ability. At the same time, in order to reduce the computational complexity, local search is introduced to significantly improve the search efficiency rate. Compared with the traditional methods, the proposed method is more effective and robust.

    • Other evolutionary calculation methods are also used in image registration, e.g., the simulated annealing algorithm[93] for remote sensing image registration. However, due to the limitations of the simulated annealing algorithm, there is no application of the simulated annealing algorithm in remote sensing image registration in recent years. There are more evolutionary calculation methods for medical image registration, such as the artificial immune system[51], firefly algorithm[94], coevolutionary algorithm[95], etc. We also hope that these algorithms can also be applied to remote sensing image registration. Of course, there are still some popular methods of evolutionary computation that have not been found in image registration, such as the bat algorithm[96], brain storm optimization algorithm[97], etc. Whether these methods can be applied to remote sensing image registration is also a problem worthy of discussion.

    • The representative of feature-based image registration method is SIFT[98], a representative of feature-based image registration methods, has good performance in natural image processing tasks, because the generated feature descriptors have rotation, scaling, and translation invariance. However, because the imaging mechanism of remote sensing images is complex, its appearance is determined by the radiation charateristics, the geometric charateristics of the objects, and sensor's sending or receiving configuration. Therefore, the SIFT originally designed to process natural images may not continue to maintain good performance in remote sensing images task. In recent years, with the rapid development of deep learning and its outstanding performance in the field of computer vision, such as semantic segmentation[99,100], object detection[101], etc., this reflects its superiority compared to traditional methods. The representative framework is the convolutional neural network (CNN)[33]. The deeper the layer of the neural network is, the more information contained in the final extracted features is. Deep learning is a learning algorithm based on neural networks. The deep learning model consists of many network layers that convert input into specific outputs. The layer between the input layer and the output layer is usually called the "hidden layer". Neural networks that contain multiple hidden layers are called "deep neural networks". At present, deep learning is gradually developing. The CNN, GAN and AE are the most popular branches. CNN is the most widely used deep neural network framework. It is suitable for processing image data with neatly arranged pixels. Specifically, it is mainly composed of a convolutional layer, a pooling layer, and a fully connected layer. Every time the input image passes through a convolutional layer, it will convolute with a set of convolution kernels, each generates a new feature map. Compared with traditional multilayer perceptrons, CNN usually uses permutation invariant functions (e.g., maximum pooling, average pooling) to aggregate local information, currently popular frameworks in CNN include VGG network[31], residual network[102], etc. AE is a distributed data set representation used to learn compression. AE can achieve data compression and dimensionality reduction through a hidden layer. It is usually used for feature level processing. In the field of remote sensing image processing, AE is mainly used for feature representation[103,104]. The purpose of transfer learning is to improve the learning performance of target learners in the target domain by transferring knowledge contained in different but related domains, or to minimize the number of examples that need to be labeled in the target domain. Transfer learning is mainly used to solve the problems of fewer datasets in remote sensing image registration tasks. GANs[105] are a very popular category of unsupervised models in deep learning. They include a generative network and a discriminant network, and the two networks compete with each other. The generative network learns to map from the latent space to the specific data distribution of interest, and the discriminant network is used to distinguish real data and generated data. The goal of the entire network training is to deceive the discriminating network by generating generated data that looks real and has a real data distribution, prompting it to make a judgment that the generated data is the same as the real data. GAN has been successfully used in image processing tasks[106]. This section summarizes the application of deep learning in remote sensing image registration tasks. According to the difference of the loss function and the format of the network output, it can be divided into methods based on representation learning and metric learning.

    • The essence of representation learning is to transform raw data into a form that can be used by neural networks, i.e., to convert raw data into corresponding high-level features. Table 4 summarizes the methods of deep learning applied to representation learning.

      Author Year Method improvement Advantages
      Ye et al.[38] 2018 Integrating the depth features extracted by CNN and the local features, and fusing the obtained features into the PSO-SIFT algorithm. The accuracy is higher than PSO-SIFT algorithm.
      Wang et al.[119] 2018 An end-to-end registration deep network is proposed, which mainly includes two stages: 1) learning of mapping function, 2) image registration. Applying transfer learning to reduce training costs, results are more accurate than traditional methods.
      Quan et al.[120] 2018 GAN was used to automatically create more training data without manually standardizing data, and they used a dual-channel deep network as a matching network. The introduction of GAN as a data set generator makes the registration result more accurate.
      Ma et al.[109] 2019 Using CNN to obtain approximate spatial relationship, and then apply the matching strategy considering spatial relationship to the local feature-based method. Since the spatial relationship is considered, the matching result is accurate and robust.
      Dong et al.[111] 2019 A DescNet network model is designed to extract the depth features of the image, and replaced the maxpooling operati-on by increasing the stride size of the convolution filter. The number of correct matching points it obtains is higher than others′, and is robust to large gray difference.
      Kim et al.[121] 2019 An end-to-end deep neural network for estimating the transformation matrix between aerial images is proposed, which in turn estimates the rotation and affine transformation parameters between the images. Orderly estimated network hierarchy reduces estimation errors and its accuracy is higher than A2net and CNNGeo.
      Yang et al.[116] 2019 The VGG16 network was selected as the feature extractor, and chosen to gradually increase the inliers for registration. Improved accuracy of results.
      Park et al.[123] 2019 A three-input and bidirectional architecture is proposed to estimate the transformation matrix between the source image and the target image. Compared to traditional methods such as SIFT and SURF, it has a higher PCK.

      Table 4.  Analysis of deep learning method for representation learning

    • Due to the characteristics of remote sensing images, the hand-craft features such as SIFT, HOG[107] used in natural images are not suitable for the requirements of remote sensing image registration tasks, but neural networks can obtain more robust and high-level features. The CNN can automatically acquire more expressive features than statistical methods through learning[108]. Therefore, many literatures choose to directly transfer the picture to the neural network, obtain the image representation features through the network, and then use the obtained features in combination with traditional methods for registration tasks. Compared with traditional hand-craft features, it can effectively improve the accuracy of the registration task. Ma et al.[109] proposed a new two-step registration method which is based on deep and local features regions, this method includes a matching method based on CNN features and a spatial relationship remote sensing image registration method based on a point matching strategy. They selected the VGG16[31] network model as the depth feature extractor, in order to eliminate the loss of location information which was caused by maximum pooling, they proposed the location adjustment strategy, and used a local feature-based method matching strategy to obtain the final correspondences. This method not only has a good effect on SAR map registration, but also can be applied to multimodal images. Ye et al.[38] chose VGG16 as the depth feature extractor, they merged the obtained depth feature and SIFT feature into a feature vector, and then integrated it into the PSO-SIFT[110] algorithm for remote sensing image registration. Compared with PSO-SIFT, this method improves the accuracy of registration. Dong et al.[111] proposed a network (DescNet) which can effectively process images with large gray difference. Compared with other networks, this network can obtain more corresponding feature points, this can solve the problem that the number of matching points is not large enough to cause deviations or even errors in the estimated transformation model[112,113]. They adopted strategy of aggressively mining the hardest negatives to solve the problem of imbalance in the number of training samples, and because the maxpooling operation reduces the performance of the final output feature descriptor[114], they replaced the maxpooling operation by increasing the stride size of the convolution filter. Experimental results prove that Descnet has a higher inlier ratio than MatchNet[115]. Yang et al.[116] used VGG16 as the feature extractor, they didn't use the fixed outliers, but chose to gradually increase the inliers for registration. In the initial stage of registration, the most reliable feature points were used to quickly determine the rough matching, then the registration details were optimized by gradually increasing the number of feature points, and the wrong matching points were constrained according to the geometric local information. The final experimental results were significantly improved compared to GLMDTPS (global and local mixture distance and thin plate spline)[117], GLCATE (global-local correspondence and transformation estimation)[118].

    • In the image registration task, the accuracy of matching features has a great impact on the entire registration task. Wang et al.[119] proposed a novel network structure to select the matching points, they used a traditional local feature extraction method to extract feature points, constructed the corresponding local regions around these features, and used DNN networks to match these patch pairs. The entire network framework is divided into two stages: mapping function learning and image registration. The mapping function learning phase mainly learns the mapping function by using unregistered images and their affine transformed images, and the registration phase uses the above-mentioned trained DNN to predict matching labels of patch pairs from the perceived image and the reference image. If the label is 1, the center point of the matched patch pair is used as the initial keypoint of image registration. They used local constraints and global constraints to weaken the matching error, and self-learning was introduced to solve the small data problem of remote sensing images used for training. The results show this method has high accuracy. Quan et al.[120] used GAN to automatically create more training data without manually standardizing data, this method can increase the accuracy of registration. They used a dual-channel deep network as a matching network. The two channels have the same structure but do not share parameters. This kind of network is helpful to extract the features of different models. They used CNN as a feature extractor to preserve the spatial information of the image. They used multiple constraints (correlation constraints, geometric constraints) when removing mismatches. Experimental results show that this method had a good performance in processing multimodal image registration tasks.

    • Kim et al.[121] proposed an end-to-end transformation parameter estimation network to gradually estimate the rotation and affine transformation parameters between aerial image pairs and each parameter estimation stage includes: feature extraction, feature matching, gradual masking, and transformation estimation. In order to accurately estimate the transformation parameters, the gradual masking method is used to reduce the influence of irrelevant feature points. Experimental results show that this method has a high accuracy. Vakalopoulou et al.[122] proposed a CNN-based network for estimating the rigid and deformable parameters between high-resolution satellite image pairs, and then used a 2D space converter layer to align the source image with the target image. Park et al.[123] proposed an end-to-end scalable network with a two-stream architecture and a bidirectional training architecture for estimating the transformation parameters for aerial image pairs. They added a set of internally enhanced target image data, this can make the trained model robust to environmental changes. The bidirectional training architecture can make the result more accurate. This network can achieve better estimation results, but still have some weakness in detailed matching.

    • Metric learning refers to the use of neural networks to directly obtain similarity measures between the input pictures. Remote sensing image matching is the most important link in the process of remote sensing image registration. It often uses metric learning to convert image matching problems into similarity measurement problems between images. Most literatures used Siamese convolutional networks[124] and their variants for metric learning. Table 5 summarizes the methods of deep learning method applied to metric learning.

      Author Year Method improvement Advantages
      Merkle et al.[129] 2017 Based on the Siamese architecture, a network is proposed to match multisensor images of different sizes, and the dot product is used to calculate the similarity. The network model is suitable for multisensor and multi-resolution images.
      Hughes et al.[125] 2018 A pseudo-Siamese convolution network is designed to match SAR and optical remote sensing images. The network model can accurately predict the matching image blocks.
      He et al.[131] 2018 It proposed a matching network for complex background factors, and used Gaussian pyramid coupling quadtree to optimize the matching point search space, used RANSAC and the whole-to-local quadratic polynomial constraints to remove the wrong matching points. It has a high matching accuracy and can effectively solve the problem of complex background factors which will affect the matching results.
      Zhang et al.[126] 2019 Designing a matching neural network based on the Siamese network architecture for matching SAR and optical remote sensing images, and using the similarity score to remove the wrong matching points. It has accurate matching results and a simple strategy for removing mismatched points.

      Table 5.  Analysis of deep learning method for metric learning

      Hughes et al.[125] proposed a pseudo-Siamese network to solve the matching problem of optical remote sensing images and synthetic aperture radar images. The fully connected layer is used to fuse two features learned by the network branch. There is no parameter sharing between the two network streams, and only feature information was fused in the final decision stage. The final network output the 0, 1 as a match prediction for input image pairs. Zhang et al.[126] adopted the Siamese full convolutional network structure to match image pairs captured by multiple sensors. The structure of the network branch is similar to HardNet[127] and in order to ensure the spatial accuracy, they set the stride value as 1 on each convolutional layer. The parameters were shared between the two branches to solve the problem of few training data sets, the two network branches were combined through convolution operations. They used Harris' corner detector[128] to select local candidate image patches, then these patches are fed into the network, the response of the final score map indicates the similarity between the master patch and the slave sub-patch at each position of the large search patch, the point with the highest similarity can be regarded as a matched point. This model is simple, highly accurate, and suitable for processing matching images captured by multiple sensors. Merkle et al.[129] used the Siamese structure to solve the problem of matching optical-SAR remote sensing images, the feature vector corresponding to the optical image and the feature matrix corresponding to the SAR image are obtained through CNN. Then the dot product layer[130] calculates the similarity between the feature vector and each element in the feature matrix, and finally obtains a score map. The point with the highest value in the search space is used to obtain matching pairs. The experimental results show that this method has a good effect on the image matching which is obtained by multiple sensors. He et al.[131] designed a new framework for matching remote sensing images with complex backgrounds based on the Siamese network, which includes three steps: S-Harris corner detection, network training and patch matching. In this method, the grid S-Harris algorithm was used to determine the coordinates of the point matches, and a search strategy based on Gaussian pyramid coupling quadtree was used to narrow down the search space and compare multiscale conjugate pairs, the Siamese network was used to find similarities for the patch pairs and obtain the initial matches, finally the whole to local quadratic polynomial constraints and the random sample consensus algorithm[132] were used to remove false matches. This method can effectively avoid the impact of complex background changes on the matching results, but the accuracy of the matching model is affected by the unknown spatial resolution of the image.

      Due to the unique characteristics of remote sensing images, such as few data sets and complex imaging mechanisms, traditional methods (such as SIFT, SURF[133], ASIFT[134]) have difficulty meeting the requirements of current remote sensing image registration tasks, deep learning has great advantages over traditional machine learning methods. However, there are still some problems in how to use the deep learning method in remote sensing image registration. Therefore, this is still a valuable research direction in the future.

    • After decades of development, remote sensing image registration has been widely used in image fusion, change feature detection and other fields. Although many results have been achieved, it still faces many problems, such as registration of multi-source remote sensing images, image nonlinear registration, real-time registration of remote sensing images, etc. This article reviews the application of computational intelligence in remote sensing image registration, which is mainly divided into two sections: evolutionary algorithms and neural networks. In recent years, a large number of improved algorithms based on evolutionary algorithms and some novel network models have been proposed, and applied to remote sensing image registration. These methods have achieved great success, and many algorithms have been verified to be effective. With the continuous development of computational intelligence, there will be more novel remote sensing image registration methods.

    • This work was supported by National Natural Science Foundation of China (Nos. 61702392 and 61772393) and Key Research and Development Program of Shaanxi Province (Nos. 2018ZDXM-GY-045 and 2019JQ-189).

    • This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

      The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

      To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reference (134)

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return