INFO:root:Training run: nmmo_20231024_235253 (/content/drive/MyDrive/nmmo/runs/nmmo_20231024_235253)
INFO:root:Training args: Namespace(attend_task='none', attentional_decode=True, bptt_horizon=8, checkpoint_interval=30, clip_coef=0.1, death_fog_tick=None, device='cuda', early_stop_agent_num=8, encode_task=True, eval_batch_size=32768, eval_mode=False, eval_num_policies=2, eval_num_rounds=1, eval_num_steps=1000000, explore_bonus_weight=0.01, extra_encoders=True, heal_bonus_weight=0.03, hidden_size=256, input_size=256, learner_weight=1.0, local_mode=False, map_size=128, maps_path='maps/train/', max_episode_length=1024, max_opponent_policies=0, meander_bonus_weight=0.02, num_agents=128, num_buffers=2, num_cores=None, num_envs=6, num_lstm_layers=0, num_maps=128, num_npcs=256, policy_store_dir=None, ppo_learning_rate=0.00015, ppo_training_batch_size=128, ppo_update_epochs=3, resilient_population=0.2, rollout_batch_size=32768, run_name='nmmo_20231024_235253', runs_dir='/content/drive/MyDrive/nmmo/runs', seed=1, spawn_immunity=20, sqrt_achievement_rewards=False, task_size=4096, tasks_path='reinforcement_learning/curriculum_with_embedding.pkl', track='rl', train_num_steps=10000000, use_serial_vecenv=False, wandb_entity=None, wandb_project=None)
INFO:root:Using policy store from /content/drive/MyDrive/nmmo/runs/nmmo_20231024_235253/policy_store
Allocated 93.70 MB to environments. Only accurate for Serial backend.
PolicyPool sample_weights: [128]
Allocated to storage - Pytorch: 0.00 GB, System: 3.41 GB
INFO:root:PolicyPool: Updated policies: dict_keys(['learner'])
Allocated during evaluation - Pytorch: 0.01 GB, System: 0.76 GB
Epoch: 0 - 32K steps - 0:00:38 Elapsed
Steps Per Second: Env=1905, Inference=5271
Train=756
Allocated during training - Pytorch: 0.07 GB, System: 3.61 GB
INFO:root:Saving policy to /content/drive/MyDrive/nmmo/runs/nmmo_20231024_235253/policy_store/nmmo_20231024_235253.000001
INFO:root:PolicyPool: Updated policies: dict_keys(['learner'])
Allocated during evaluation - Pytorch: 0.00 GB, System: 0.04 GB
Epoch: 1 - 65K steps - 0:02:23 Elapsed
Steps Per Second: Env=847, Inference=7591
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/pufferlib/utils.py", line 223, in wrapper
result = func(*args, **kwargs)
File "/content/drive/MyDrive/nmmo/baselines/reinforcement_learning/clean_pufferl.py", line 620, in train
self.optimizer.step()
File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 373, in wrapper
out = func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 76, in _use_grad
ret = func(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/optim/adam.py", line 163, in step
adam(
File "/usr/local/lib/python3.10/dist-packages/torch/optim/adam.py", line 311, in adam
func(params,
File "/usr/local/lib/python3.10/dist-packages/torch/optim/adam.py", line 552, in _multi_tensor_adam
bias_correction2 = [1 - beta2 ** _get_value(step) for step in device_state_steps]
File "/usr/local/lib/python3.10/dist-packages/torch/optim/adam.py", line 552, in <listcomp>
bias_correction2 = [1 - beta2 ** _get_value(step) for step in device_state_steps]
File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 84, in _get_value
def _get_value(x):
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/content/drive/MyDrive/nmmo/baselines/train.py", line 135, in <module>
reinforcement_learning_track(trainer, args)
File "/content/drive/MyDrive/nmmo/baselines/train.py", line 68, in reinforcement_learning_track
trainer.train(
File "/usr/local/lib/python3.10/dist-packages/pufferlib/utils.py", line 223, in wrapper
result = func(*args, **kwargs)
KeyboardInterrupt
Process Process-12:
Process Process-9:
Process Process-10:
Process Process-8:
Process Process-7:
Process Process-5:
Exception ignored in atexit callback: <function _exit_function at 0x7989b0c63c70>
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/util.py", line 357, in _exit_function
Process Process-4:
Process Process-2:
Process Process-11:
Process Process-3:
Process Process-6:
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
KeyboardInterrupt
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
Traceback (most recent call last):
KeyboardInterrupt
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
KeyboardInterrupt
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
KeyboardInterrupt
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
KeyboardInterrupt
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
KeyboardInterrupt
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
KeyboardInterrupt
KeyboardInterrupt
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
KeyboardInterrupt
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
KeyboardInterrupt
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
KeyboardInterrupt
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/local/lib/python3.10/dist-packages/pufferlib/vectorization.py", line 349, in _worker_process
request, args, kwargs = request_queue.get()
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
KeyboardInterrupt
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
_run_finalizers()
File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
thread.join()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
^C
The colab link has an additional section called “Generate a replay for a specific task”, which shows how to generate a replay of all agents doing the same task you specified. https://colab.research.google.com/drive/1AZt_eEGTEZrnX3iJC7jHw7lbBTNaoMoj#scrollTo=xtvJbCFFU53Z