TECHNICAL MECHANICS
ISSN (Print): 1561-9184, ISSN (Online): 2616-6380

HOME    ABOUT THIS JOURNAL    EDITORIAL BOARD    GUIDE FOR AUTHORS    CURRENT    JOURNAL ISSUES    CONTACTS

UDC 004.89+629.7

Technical mechanics, 2019, 4, 29 - 43

INTELLIGENT CONTROL OF SPACECRAFT ATTITUDE USING REINFORCEMENT LEANING

DOI: https://doi.org/10.15407/itm2019.04.029

Khoroshylov S. V., Redka M. O.

Khoroshylov S. V.
Institute of Technical Mechanics of the National Academy of Sciences of Ukraine and the State Space Agency of Ukraine

Redka M. O.
Institute of Technical Mechanics of the National Academy of Sciences of Ukraine and the State Space Agency of Ukraine

      The aim of this paper is to develop an effective algorithm for intelligent control of spacecraft based on reinforcement learning (RL) methods.
      In the development and analysis of the algorithm, methods of theoretical mechanics, automatic control and stability theories, machine learning, and computer simulation were used. To increase the RL efficiency, a statistical model of spacecraft dynamics based on the concept of Gaussian processes was used. On the one hand, such a model allows one to use a priori information about the plant and is sufficiently flexible, and on the other hand, it characterizes uncertainty in the dynamics in the form of confidence intervals and can be refined during the spacecraft operation. In this case, the problem of control/state space analysis reduces to obtaining such measurements that narrow the confidence intervals. The familiar quadratic criterion, which allows one to take into account both the accuracy requirements and the control cost, was used as the reinforcement signal. An RL-based search for control actions was made using a control law iterative algorithm. To implement the regulator and evaluate the cost function, neural network approximators were used. Spacecraft motion stability guarantees were obtained using the Lyapunov function method with account for the uncertainty in the spacecraft dynamics. The cost function was chosen as a candidate Lyapunov function, To simplify the stability test on the basis of this methodology, the dynamics of the plant was assumed to be Lipschitz continuous, which made it possible to use the Lagrange multiplier method for searching for control actions with account for the constraints formulated using the upper uncertainty bound and Lipschitz dynamics constants.
      The efficiency of the proposed algorithm is illustrated by computer simulation results. The approach makes it possible to develop control systems that can improve their performance as data are accumulated during the operation of a specific object, thus allowing one to reduce the requirements for its elements (sensors, actuators), do without special test equipment, and reduce the development time and cost.
     

reinforcement leaning, intelligent control system, spacecraft, stability, dynamic model

1. Besekersky V. A., Popov E. P. Theory of Automatic Control Systems. Saint Petersburg: Professiya, 2003. 768 pp. (in Russian).

2. Leitman G. Introduction to the Optimal Control Theory. Moscow: Nauka, 1968. 192 pp. (in Russian).

3. Zhou K., Doyle J.C., Glover K. Robust and Optimal Control. NJ: Prentice-Hall, 1996. 596 pp.

4. Alpatov A., Khoroshylov S., Bombardelli C. Relative control of an ion beam shepherd satellite using the impulse compensation thruster, Acta Astronautica. 2018. V. 151. Pp. 543-554. https://doi.org/10.1016/j.actaastro.2018.06.056

5. Astrom K. J., Wittenmark B. Adaptive Control. MA: Addison-Wesley, 1995. 580 pp.

6. Khoroshilov S. V. Space-based solar power station attitude control using an extended state observer. Teh. Meh. 2011. No. 3. Pp.117-125. (in Russian).

7. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction. MIT Press, 1998. 338 pp.

8. Gullapalli V. Skillful control under uncertainty via direct reinforcement learning. Reinforcement Learning and Robotics. 1995. V. 15(4). Pp. 237-246. https://doi.org/10.1016/0921-8890(95)00006-2

9. Kober J., Bagnell J. A., Peters J. Reinforcement learning in robotics: A survey. International Journal of Robotic Research. 2013. V. 32(11). Pp. 1238-1274. https://doi.org/10.1177/0278364913495721

10. Theodorou E., Buchli J., Schaal S. Reinforcement learning of motor skills in high dimensions. In: International Conference on Robotics and Automation (ICRA), 2010. Pp. 2397-2403. https://doi.org/10.1109/ROBOT.2010.5509336

11. Endo G., Morimoto J., Matsubara T., Nakanishi J., Cheng G. Learning CPG-based biped locomotion with a policy gradient method: Application to a humanoid robot. International Journal of Robotic Research. 2008. V. 27(2). Pp. 213-228. https://doi.org/10.1177/0278364907084980

12. Ng A. Y., Kim H. J., Jordan M. I., Sastry S. Inverted autonomous helicopter flight via reinforcement learning. In: International Symposium on Experimental Robotics, 2004. Pp. 363-372. https://doi.org/10.1007/11552246_35

13. Juang J.-N. Applied System Identification. NJ: Prentice Hall, Upper Saddle River, 1994. 394 pp.

14. Seeger M. Gaussian processes for machine learning. International Journal of Neural Systems. 2004. V. 14 (2). Pp. 69-104. https://doi.org/10.1142/S0129065704001899

15. Berkenkamp F., Turchetta M., Schoellig A.P., Krause A. Safe Model-based reinforcement learning with stability guarantees. 31st Conference on Neural Information Processing Systems, 2017. Pp. 908-919.

DOI: https://doi.org/10.15407/itm2019.04.029

Copyright (©) 2019 Khoroshylov S. V., Redka M. O.

Copyright © 2014-2019 Technical mechanics