Generating train split: 99%|█████████▉| 17177/17335 [04:34<00:02, 62.17 examples/s]E0706 01:15:57.180520 83 coordination_service.cc:1239] Shutdown barrier in coordination service has failed:
DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']
This suggests that the workers are out of sync. Either at least one worker is too fast in its execution / crashed early or too slow / hanging. Check the logs for an earlier error to identify the root cause.
E0706 01:15:57.180639 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180652 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180655 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180658 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180663 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180666 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180669 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180675 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180679 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180681 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180684 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180687 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180690 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180693 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180695 83 coordination_service.cc:797] INTERNAL: Shutdown barrier has been passed with status: 'DEADLINE_EXCEEDED: Barrier timed out. Barrier_id: Shutdown::5745716602613285649. Timed out task names:
/job:jax_worker/replica:0/task:1
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:7
/job:jax_worker/replica:0/task:14
/job:jax_worker/replica:0/task:8
/job:jax_worker/replica:0/task:13
/job:jax_worker/replica:0/task:15
/job:jax_worker/replica:0/task:9
/job:jax_worker/replica:0/task:3
/job:jax_worker/replica:0/task:4
/job:jax_worker/replica:0/task:10
/job:jax_worker/replica:0/task:11
/job:jax_worker/replica:0/task:5
/job:jax_worker/replica:0/task:12
/job:jax_worker/replica:0/task:6
[type.googleapis.com/tensorflow.CoordinationServiceError='']', but this task is not at the barrier yet. [type.googleapis.com/tensorflow.CoordinationServiceError='']
E0706 01:15:57.180780 83 coordination_service.cc:482] Stopping coordination service as shutdown barrier timed out. Check the task logs for an earlier error.
Generating train split: 100%|██████████| 17335/17335 [04:36<00:00, 62.64 examples/s]
wandb: Currently logged in as: <REDACTED>. Use `wandb login --relogin` to force relogin
wandb: Appending key for api.wandb.ai to your netrc file: /root/.netrc
wandb: WARNING `resume` will be ignored since W&B syncing is set to `offline`. Starting a new run with run id 5quqcxcb.
wandb: Tracking run with wandb version 0.17.0
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
E0706 01:16:31.111081 93 coordination_service_agent.cc:761] Coordination agent is set to ERROR: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.
Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat:
:{"created":"@1720228591.110962687","description":"Error received from peer ipv4:10.130.0.124:8476","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError='']
2024-07-06 01:16:31.111155: E external/xla/xla/pjrt/distributed/client.cc:94] Coordination service agent in error status: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.
Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat:
:{"created":"@1720228591.110962687","description":"Error received from peer ipv4:10.130.0.124:8476","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError='']
F0706 01:16:31.111196 93 client.h:81] Terminating process because the coordinator detected missing heartbeats. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. Status: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.
Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat:
:{"created":"@1720228591.110962687","description":"Error received from peer ipv4:10.130.0.124:8476","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError='']