ISSN (Print): 1561-9184, ISSN (Online): 2616-6380

Home > Journal Issues > No 4 (2020) Technical mechanics > 5

UDC 629.5

Technical mechanics, 2020, 4, 43 - 54

Relative control of an underactuated spacecraft using reinforcement learning


Khoroshylov S. V., Redka M. O.


Khoroshylov S. V.
Institute of Technical Mechanics of the National Academy of Sciences of Ukraine and the State Space Agency of Ukraine

Redka M. O.
Institute of Technical Mechanics of the National Academy of Sciences of Ukraine and the State Space Agency of Ukraine


      The aim of the article is to approximate optimal relative control of an underactuated spacecraft using reinforcement learning and to study the influence of various factors on the quality of such a solution.
      In the course of this study, methods of theoretical mechanics, control theory, stability theory, machine learning, and computer modeling were used.
      The problem of in-plane spacecraft relative control using only control actions applied tangentially to the orbit is considered. This approach makes it possible to reduce the propellant consumption of reactive actuators and to simplify the architecture of the control system. However, in some cases, methods of the classical control theory do not allow one to obtain acceptable results. In this regard, the possibility of solving this problem by reinforcement learning methods has been investigated, which allows designers to find control algorithms close to optimal ones as a result of interactions of the control system with the plant using a reinforcement signal characterizing the quality of control actions.
      The well-known quadratic criterion is used as a reinforcement signal, which makes it possible to take into account both the accuracy requirements and the control costs. A search for control actions based on reinforcement learning is made using the policy iteration algorithm. This algorithm is implemented using the actorcritic architecture. Various representations of the actor for control law implementation and the critic for obtaining value function estimates using neural network approximators are considered. It is shown that the optimal control approximation accuracy depends on a number of features, namely, an appropriate structure of the approximators, the neural network parameter updating method, and the learning algorithm parameters.
      The investigated approach makes it possible to solve the considered class of control problems for controllers of different structures. Moreover, the approach allows the control system to refine its control algorithms during the spacecraft operation.
      Pdf (English)


relative control, underactuated spacecraft, reinforcement learning, policy iteration, actorcritic

      FULL TEXT:

Pdf (English)


1. MacIsaac D. Docking at the International Space Station. Phys. Teach. 2014. V. 52. No. 126.

2. Campbell M., Fullmer R. R., Hall C. D. The ION-F formation flying experiments. Advances in the Astronautical Sciences. 2000. V. 105. Pp. 135-149.

3. Smith G. W., DeRocher W. L. Jr. Orbital servicing and remotely manned systems. Mechanism and Machine Theory. 1977. V. 12. Pp. 65-76.

4. Alpatov A. P., Khoroshylov S. V., Maslova A. I. Contactless De-Orbiting of Space Debris by the Ion Beam. Dynamics and Control. - Kyiv: Akademperiodyka, 2019. 170 pp.

5. Vassar R. H., Sherwood R. B. Formationkeeping for a pair of satellites in a circular orbit. Journal of Guidance, Control, and Dynamics. 1985. V. 8(2). Pp. 235-242.

6. Redding D. C., Adams N. J., Kubiak E. T. Linear quadratic stationkeeping for the STS orbiter. Charles Stark Draper Laboratory, Cambridge, MA, Kept. CSDL-R-1879, June 1986.

7. Dwidar H. R., Owis A. H. Relative motion of formation flying with elliptical reference orbit. International Journal of Advanced Research in Artificial Intelligence. 2013. V. 2(6). Pp. 79-86.

8. Peng H., Zhao J., Wu Z., Zhong W. Optimal periodic controller for formation flying on libration point orbits. Acta Astronautica 2011. V. 69. Pp. 537-550.

9. Starin R. S., Yedavalli R. K., Sparks A. G. Spacecraft formation flying maneuvers using linear-quadratic regulation with no radial axis inputs. AIAA Paper. August 2001. Pp. 2001-4029.

10. Kumara K. D., Bang H. C., Tahk M. J. Satellite formation flying using along-track thrust. Acta Astronautica. 2007. V. 61(7-8). Pp. 553-564.

11. Alpatov A., Khoroshylov S., Bombardelli C., Relative control of an ion beam shepherd satellite using the impulse compensation thruster. Acta Astronautica. 2018. V. 151. Pp. 543-554.

12. Khoroshylov S. Relative control of an ion beam shepherd satellite in eccentric orbits. Acta Astronautica. 2020. V. 176. Pp. 89-98.

13. Haarnoja T., Zhou A., Abbeel P., Levine S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. 2018. arXiv preprint arXiv:1801.01290.

14. Jaderberg M., Mnih V., Czarnecki W M., Schaul T., Leibo J. Z., Silver D., Kavukcuoglu K. Reinforcement learning with unsupervised auxiliary tasks. 2016. arXiv preprint arXiv:1611.05397.

15. Khadka S., Tumer K. Evolution-guided policy gradient in reinforcement learning. Advances in Neural Information Processing Systems. 2018. Pp. 1196-1208.

16. Nair A., McGrew B., Andrychowicz M., Zaremba W., Abbeel P. Overcoming exploration in reinforcement learning with demonstrations. 2018 IEEE International Conference on Robotics and Automation (ICRA). 2018. Pp. 6292-6299.

17. Kober J., Bagnell J. A., Peters J. Reinforcement learning in robotics: A survey // International Journal of Robotic Research. 2013. V. 32(11). Pp. 1238-1274.

18. Khoroshylov S. V., Redka M. O. Intelligent control of spacecraft attitude using reinforcement leaning. Teh. Meh. 2019. No. 4. Pp. 29 - 43. (in Ukrainian).

19. Yamanaka K., Ankersen F. New state transition matrix for relative motion on an arbitrary elliptical orbit. Journal of Guidance, Control, and Dynamics. 2002. V. 25 (1). Pp. 60-66.

20. Lewis F. L., Vrabie D., Syrmos V. L. Optimal Control, 3rd Edition. New York: John Wiley & Sons, Inc., 2012.

21. Sutton R. S., Barto A. G. Reinforcement learning: an introduction. MIT Press, 1998. 338 pp.

22. Glorot X., Bengio Y. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Atificial Intelligence and Statistics. 2010. Pp. 249256.

Copyright () 2020 Khoroshylov S. V., Redka M. O.

Copyright 2014-2020 Technical mechanics

Guide for Authors ==================== Open Access Policy
Open Access Policy ==================== REGULATIONS
on the ethics of publications
REGULATIONS on the ethics of publications ====================