What happens when episode length is less than n_steps parameter in PPO

[Zymoo]

Hello!

In my research with PPO I have a number of samples equal to 365, which is basically just a feature vector for every day in one year. When training the model on this data I used to keep the default parameter n_steps equal to 2048 and just set the number of total_timesteps to 100 000.

However, I do not quite understand what happens when the agent finishes these 365 steps? Does it keep restarting the data until it reaches 2048 steps overall and then updates its policy?

The results I got seemed a bit better when using 2048 steps versus setting them to 365, in both cases keeping the total_timesteps parameter the same. Could the reason for this might be that updating policy more rarely (every 2048) makes it more stable than having updates every 365 steps?

I would greatly appreciate any tip regarding these parameters and especially explanation of reaching 2048 steps with episode of 365!

Here is the sample code of parameter setting:

  # case I
  model = PPO('MlpPolicy', env, verbose=1, tensorboard_log=logdir)
  model.learn(total_timesteps=100000)

  # case II
  model = PPO('MlpPolicy', env, n_steps=365, verbose=1, tensorboard_log=logdir)
  model.learn(total_timesteps=100000)

[Miffyli]

Yup and yup, your both guesses are right! It will continue to reset the environment ("the data") every 365 steps, appending all the 2048 steps over multiple episodes into one large batch, and then doing an update on them. And yup, this (larger n_step) works better most of the time because of stabler updates. As a rule of thumb, the longer rollouts (n_steps) and the more parallel environments, the better.

[Zymoo]

Oh I see, thank you so much! If I also may ask - when using multiple parallel environments (n_envs=x) for training I would afterwards want to test the best performing agent on another environment. Do I have to do anything more than just changing the environment from parallel to alone one as I do in the code below? I understand that the model would use the optimal learned strategy in such case?

  # create_env is my internal function for creating custom environment compliant with gym interface
  env = make_vec_env(lambda: create_env(data_train, feed, renderer_feed, window, reward_window), n_envs=4)
  model = PPO('MlpPolicy', env, verbose=1, tensorboard_log=logdir)
  model.learn(total_timesteps=1000000)
  
  env =  create_env(data_test, feed, renderer_feed, window, reward_window)
  obs = env.reset()
  while True:
    action, _state = model.predict(obs, deterministic=True)
    obs, reward, done, info = env.step(action)
    if done:
      score = info["net_worth"]
      print(score)
      break

[Miffyli]

Yup that works! You could also use evaluate_policy function (see documentation), but for more precise control your step-loop works fine :).

You may close the issue if you have no further questions.

[Zymoo]

I will, thank you mate! Helped me a lot!