Neural Control Systems: An introduction Federico Nesti, - PDF document

3/24/2020 Neural Control Systems: An introduction Federico Nesti, f.nesti@santannapisa.it Federico Nesti, f.nesti@santannapisa.it Credits: Brian Douglas 1

3/24/2020 Layout • Introduction to Dynamical Systems o What is a dynamical system? Features and properties o Classic Control • OpenAI Gym Environments • Tensorflow Recap • Classic Neural Control o System Identification o Reference Model Control Layout • Introduction to Dynamical Systems o What is a dynamical system? Features and properties o Classic Control • OpenAI Gym Environments • Tensorflow Recap • Classic Neural Control o System Identification o Reference Model Control 2

3/24/2020 Dynamical Systems Dynamical Systems Initial Condition Control ∫ Input System State 3

3/24/2020 Dynamical Systems Dynamical Systems 4

3/24/2020 Dynamical Systems Dynamical Systems 5

3/24/2020 Dynamical Systems Control Systems A Control System is intended to regulate the system state in a specified manner through the action of a Controller (and usually a specified manner through the action of a Controller (and usually a Feedback signal , that makes it a Closed Loop Control System). Control input State Reference Controller error - State feedback 6

3/24/2020 Simple Control Systems Design Flow 1) Choose equilibrium where to control the system (e.g., upright) 2) Linearize the system on that equilibrium, check stability 3) Design gain of proportional controller Equilibria Null Velocity 7

3/24/2020 Equilibria Linearization Stability of equilibria is critical: down position is clearly stable , upright is UNSTABLE ! How to state this? UNSTABLE ! How to state this? LINEARIZATION . Linearizing around a certain equilibrium point makes the «residual» system linear and much simpler to analyze (linear algebra tools). 8

3/24/2020 Linearization Linearization 9

3/24/2020 Eigenvalues Eigenvalues 10

3/24/2020 Eigenvalues Design Gain How to control the plant in the unstable equilibrium? ilib i ? Add state feedback . Control input K - 11

3/24/2020 Design Gain Design Gain 12

3/24/2020 Key Takeaways • Control Systems objective • State Space formalism • Reference, feedback, error signal • No matter what systems represents in reality! Control only cares about system dynamics . • Control design can be hard with simple plants . Control design can be hard with simple plants . How to control complex (and ill-modeled) systems? Layout • Introduction to Dynamical Systems o What is a dynamical system? Features and properties o Classic Control • Gym Environments • Tensorflow Recap • Classic Neural Control o System Identification o Reference Model Control 13

3/24/2020 OpenAI Gym Environments • Lots of pre-configured environments i t • Open Source • Easily configurable custom environments https://gym.openai.com/ OpenAI Gym Environments Simple gym installation: $ pip install gym Custom gym installation (needs Mujoco dependency): $ git clone http://github.com/openai/gym $ cd gym gym$ pip install –e ‘.[all]’ --user 14

3/24/2020 OpenAI Gym Environments Custom environment installation: • Create folder gym/envs/custom C f ld / / • Create gym/envs/custom/ foo_env.py and gym/envs/custom/ __init__.py • In the file gym/envs/__init__.py add line register(id=‘foo-v0’, entry_point=‘gym.envs.custom:Foo’,) • Reinstall gym Reinstall gym OpenAI Gym Environments Initialization : import gym env = gym.make(env_name) Spaces : env.observation_space Discrete or Box (i.e., continuous) env.action_space Methods : Methods : - state = env.reset() - env.render() - new_state, reward, done, info = env.step(action) 15

3/24/2020 OpenAI Gym Environments import gym env = gym.make(‘CartPole-v0’) s = env.reset() for i in range(N): env.render() s_, r, d, _ = env.step(env.action_space.sample()) MuJoCo Environments Some environments are free and open-source: Atari , box2D , classic control , algorithms , toy text . The environments in MuJoCo and Robotics need an external dependency on a physics simulator, called MuJoCo . 16

3/24/2020 MuJoCo Environments To install MuJoCo: • G t Goto https://www.roboti.us/license.html and obtain 30-days-trial license (student, personal) htt // b ti /li ht l d bt i 30 d t i l li ( t d t l) • Register PC as explained in the instructions • Download mujoco2.0, extract in /home/.mujoco and move license here • Add line “ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/(username)/.mujoco/mujoco200 /bin ”, then execute source ~/.bashrc MuJoCo Environments To install MuJoCo: • $ git clone http://github.com/openai/mujoco-py.git • $ cd mujoco-py $ cd mujoco py • mujoco-py$ sudo apt-get update • mujoco-py$ sudo apt-get install patchelf • mujoco-py$ sudo apt-get install python3 python-dev python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev libglew1.5 libglew- dev python-pip libgl1-mesa-dev +libgl1-mesa-glx libosmesa6-dev python3-numpy python3-scipy libglfw3-dev • mujoco-py$ pip install –r requirements.txt • mujoco-py$ pip install –r requirements.dev.txt • mujoco-py$ sudo python3 setup.py develop j $ d th 3 t d l • (Copy export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so in ~/.bashrc ) • $ source ~/.bashrc • Reinstall gym 17

3/24/2020 Layout • Introduction to Dynamical Systems o What is a dynamical system? Features and properties o Classic Control • Gym Environments • Tensorflow Recap • Classic Neural Control o System Identification o Reference Model Control Tensorflow 1.x Recap Walkthrough Jupyter Notebook. Basic concepts: • Computational graphs • Sessions (sess.run) • Variables and placeholders • Building models • Optimizers • Training process Training process • MNIST example • Tensorboard 18

3/24/2020 Layout • Introduction to Dynamical Systems o What is a dynamical system? Features and properties o Classic Control • Gym Environments • Tensorflow Recap • Classic Neural Control o System Identification o Reference Model Control Classic Neural Control 19

3/24/2020 System Identification System - + NN NN Learning System Identification Output Input 20

3/24/2020 Generation of input-output data Random amplitude and duration input generation Next state computation and noise addition Training of NN 21

3/24/2020 System Identification Predicted Actual Reference Model Control Reference Model - Learning + NN System controller - + model NN Learning 22

3/24/2020 Reference Model Control This is why this net is a RECURRENT net! Controller output p p depends on previous output. Reference Model Control 23

3/24/2020 Reference Model Control The architecture can be unrolled to clarify the temporal dependency. NN NN Model Model Model Model LOSS LOSS controller controller Reference Model Control Backpropagation through time NN NN Model Model Model Model LOSS LOSS controller controller 24

3/24/2020 Reference Model Control Target Reference Angle [rad] Actual Time Steps Classic Neural Control • This is how to identify an unknown system and how to control it! control it! • First applications of neural networks in control Next: “intelligent” control • Reinforcement Learning • Genetic Algorithms 25