[Question] HER+SAC different results to SB2

[Ludilu]

Hi,
I was training on a custom environment on SB2 before and wanted to change to SB3 (mainly because having pytorch would probably be easier for my deployment)

So I trained on SB3 with HER+SAC and the same hyperparameter, but got different results. Is this to be expected due to a different SAC implementation, or what else could be the reason?

SB2 code

env = gym.make('armflex-v4')
eval_env = HERGoalEnvWrapper(env)
eval_callback = EvalCallback(eval_env, best_model_save_path=path,
                                log_path=path, eval_freq=20000,
                                deterministic=True, render=False, n_eval_episodes=15)
model_class = SAC
goal_selection_strategy = 'future'
model = HER('MlpPolicy', env, model_class, n_sampled_goal=4, goal_selection_strategy=goal_selection_strategy, verbose=1, 
    policy_kwargs=dict(layers=[512, 512]), buffer_size=1000000, batch_size=256, gamma=0.99, random_exploration=0.0, 
    ent_coef='auto', gradient_steps=1)
        
model.learn(total_timesteps=TIMESTEPS, callback=eval_callback, log_interval=1)

SB3 code

    env = make_vec_env(env_name, n_envs=1)
    env = ObsDictWrapper(env)
    eval_callback = EvalCallback(eval_env, best_model_save_path=path,
                            log_path=path, eval_freq=20000,
                            deterministic=True, render=False, n_eval_episodes=15)
    model_class = SAC
    goal_selection_strategy = 'future'
    model = HER('MlpPolicy', env, model_class, n_sampled_goal=4, online_sampling=False, 
        goal_selection_strategy=goal_selection_strategy, verbose=1, policy_kwargs=dict(net_arch=[512, 512]), 
        buffer_size=1000000, batch_size=256, gamma=0.99, ent_coef='auto', gradient_steps=1, max_episode_length=1000)
    
    model.learn(total_timesteps=TIMESTEPS, callback=eval_callback, log_interval=1)

this should also be the mean over 100 episodes

[araffin]

Hello,

So I trained on SB3 with HER+SAC and the same hyperparameter, but got different results.

the results you are showing is only for one random seed on the training reward.
Do you have the same difference when using multiple seeds and comparing the evaluation reward (using evalcallback)?
I recommend you to use the rl zoo for that ;)

https://github.com/DLR-RM/rl-baselines3-zoo

[Ludilu]

Thx for your fast reply.
So far I have a few runs on SB2 that all have a very similar shape (I hope the seed is different on each run if I don't specify it). On SB3 I have only two runs for now (training takes a while...) and both of them have a similar shape as well.

[araffin]

So far I have a few runs on SB2 that all have a very similar shape (I hope the seed is different on each run if I don't specify it). On SB3 I have only two runs for now (training takes a while...) and both of them have a similar shape as well.

SB2 and SB3 should give the same results (given enough seeds, and better to specify it, 2 seeds is usually too few to draw any conclusion).
Anyway, I would recommend you to use recommended hyperparameters from the zoo (notably online_sampling=True, larger batch size, larger network and higher learning rate), you should be able to obtain at least the same performance and probably better. And use the zoo for managing experiments ;)

(online sampling is not possible with SB2)

I did benchmark SB2 vs SB3 HER here: #120
and I could get similar results.

[Ludilu]

Ok, I will setup zoo and see what results I can achieve with more seeds and the recommended hyperparameters.
Thanks for your help!
Great project btw., keep up the good work!

[Ludilu]

Unfortunately I have a problem with online_sampling=True. I always get the following error at exactly episode = (biffersize/max_episode_length)

Traceback (most recent call last):
  File "train_her.py", line 158, in <module>
    model.learn(total_timesteps=TIMESTEPS, callback=eval_callback, log_interval=1)
  File "/home/ludilu/anaconda3/envs/sb3/lib/python3.8/site-packages/stable_baselines3/her/her.py", line 214, in learn
    self.train(batch_size=self.batch_size, gradient_steps=gradient_steps)
  File "/home/ludilu/anaconda3/envs/sb3/lib/python3.8/site-packages/stable_baselines3/sac/sac.py", line 191, in train
    replay_data = self.replay_buffer.sample(batch_size, env=self._vec_normalize_env)
  File "/home/ludilu/anaconda3/envs/sb3/lib/python3.8/site-packages/stable_baselines3/her/her_replay_buffer.py", line 141, in sample
    return self._sample_transitions(batch_size, maybe_vec_env=env, online_sampling=True)
  File "/home/ludilu/anaconda3/envs/sb3/lib/python3.8/site-packages/stable_baselines3/her/her_replay_buffer.py", line 263, in _sample_transitions
    [
  File "/home/ludilu/anaconda3/envs/sb3/lib/python3.8/site-packages/stable_baselines3/her/her_replay_buffer.py", line 264, in <listcomp>
    self.info_buffer[episode_idx][transition_idx]
IndexError: deque index out of range

[araffin]

Unfortunately I have a problem with online_sampling=True. I always get the following error at exactly episode = (biffersize/max_episode_length)

Do you have the latest version of Stable-Baselines3?
See #234

[Ludilu]

Thanks, the newer version indeed fixed the issue. Must have installed it just a few days before it was released.
I was already able to achieve higher results so I guess this question can be closes.
Thanks again for your fast support.