2024-01-16 13:44:14,506 INFO StreamThr :50740 [internal.py:wandb_internal():86] W&B internal server running at pid: 50740, started at: 2024-01-16 13:44:14.505286
2024-01-16 13:44:14,507 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: status
2024-01-16 13:44:14,511 INFO WriterThread:50740 [datastore.py:open_for_write():87] open: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/run-n5zyqwu1.wandb
2024-01-16 13:44:14,512 DEBUG SenderThread:50740 [sender.py:send():382] send: header
2024-01-16 13:44:14,544 DEBUG SenderThread:50740 [sender.py:send():382] send: run
2024-01-16 13:44:14,548 INFO SenderThread:50740 [sender.py:_maybe_setup_resume():763] checking resume status for None/ReAct_Jax/n5zyqwu1
2024-01-16 13:44:15,314 INFO SenderThread:50740 [dir_watcher.py:__init__():211] watching files in: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files
2024-01-16 13:44:15,315 INFO SenderThread:50740 [sender.py:_start_run_threads():1136] run started: n5zyqwu1 with start time 1705412654.505257
2024-01-16 13:44:15,331 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: check_version
2024-01-16 13:44:15,331 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: check_version
2024-01-16 13:44:15,554 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: run_start
2024-01-16 13:44:15,565 DEBUG HandlerThread:50740 [system_info.py:__init__():27] System info init
2024-01-16 13:44:15,565 DEBUG HandlerThread:50740 [system_info.py:__init__():42] System info init done
2024-01-16 13:44:15,565 INFO HandlerThread:50740 [system_monitor.py:start():194] Starting system monitor
2024-01-16 13:44:15,565 INFO SystemMonitor:50740 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-01-16 13:44:15,566 INFO HandlerThread:50740 [system_monitor.py:probe():214] Collecting system info
2024-01-16 13:44:15,566 INFO SystemMonitor:50740 [interfaces.py:start():190] Started cpu monitoring
2024-01-16 13:44:15,567 DEBUG HandlerThread:50740 [system_info.py:probe():151] Probing system
2024-01-16 13:44:15,567 INFO SystemMonitor:50740 [interfaces.py:start():190] Started disk monitoring
2024-01-16 13:44:15,570 INFO SystemMonitor:50740 [interfaces.py:start():190] Started gpuapple monitoring
2024-01-16 13:44:15,573 INFO SystemMonitor:50740 [interfaces.py:start():190] Started memory monitoring
2024-01-16 13:44:15,574 INFO SystemMonitor:50740 [interfaces.py:start():190] Started network monitoring
2024-01-16 13:44:15,578 DEBUG HandlerThread:50740 [system_info.py:_probe_git():136] Probing git
2024-01-16 13:44:15,602 DEBUG HandlerThread:50740 [system_info.py:_probe_git():144] Probing git done
2024-01-16 13:44:15,602 DEBUG HandlerThread:50740 [system_info.py:probe():199] Probing system done
2024-01-16 13:44:15,602 DEBUG HandlerThread:50740 [system_monitor.py:probe():223] {'os': 'macOS-14.2.1-arm64-arm-64bit', 'python': '3.10.12', 'heartbeatAt': '2024-01-16T13:44:15.567486', 'startedAt': '2024-01-16T13:44:14.490216', 'docker': None, 'cuda': None, 'args': ('--save_dir', '/Users/neel/Documents/research/ReAct_Jax/ReAct/outputs/', '--lr', '1e-3', '--num_blocks', '2', '--epochs', '3', '--warmup_steps', '100', '--log_interval', '200', '--seqlen', '192', '--n_heads', '1', '--width', '64', '--batch_size', '128', '--weight_decay', '3e-3', '--drop_rate', '0.01', '--bf16', '--max_iters', '5', '--wandb'), 'state': 'running', 'program': '/Users/neel/Documents/research/ReAct_Jax/train_model.py', 'codePathLocal': 'train_model.py', 'codePath': 'train_model.py', 'git': {'remote': 'https://github.com/neel04/ReAct_Jax.git', 'commit': '88543a302cb8c420a7bbad10041ed0e371cc22cd'}, 'email': '[email protected]', 'root': '/Users/neel/Documents/research/ReAct_Jax', 'host': 'Neels-MacBook-Air.local', 'username': 'neel', 'executable': '/Users/neel/miniconda3/envs/react_jax/bin/python3', 'cpu_count': 8, 'cpu_count_logical': 8, 'disk': {'/': {'total': 460.4317207336426, 'used': 9.428455352783203}}, 'gpuapple': {'type': 'arm', 'vendor': 'Apple'}, 'memory': {'total': 24.0}}
2024-01-16 13:44:15,602 INFO HandlerThread:50740 [system_monitor.py:probe():224] Finished collecting system info
2024-01-16 13:44:15,602 INFO HandlerThread:50740 [system_monitor.py:probe():227] Publishing system info
2024-01-16 13:44:15,602 DEBUG HandlerThread:50740 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment
2024-01-16 13:44:16,321 INFO Thread-12 :50740 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/conda-environment.yaml
2024-01-16 13:44:20,972 DEBUG HandlerThread:50740 [system_info.py:_save_conda():220] Saving conda packages done
2024-01-16 13:44:20,972 DEBUG HandlerThread:50740 [system_info.py:_save_code():45] Saving code
2024-01-16 13:44:20,984 DEBUG HandlerThread:50740 [system_info.py:_save_code():66] Saving code done
2024-01-16 13:44:20,984 DEBUG HandlerThread:50740 [system_info.py:_save_patches():83] Saving git patches
2024-01-16 13:44:21,069 DEBUG HandlerThread:50740 [system_info.py:_save_patches():125] Saving git patches done
2024-01-16 13:44:21,071 INFO HandlerThread:50740 [system_monitor.py:probe():229] Finished publishing system info
2024-01-16 13:44:21,073 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: status_report
2024-01-16 13:44:21,073 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: keepalive
2024-01-16 13:44:21,074 DEBUG SenderThread:50740 [sender.py:send():382] send: files
2024-01-16 13:44:21,074 INFO SenderThread:50740 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now
2024-01-16 13:44:21,074 INFO SenderThread:50740 [sender.py:_save_file():1403] saving file code/train_model.py with policy now
2024-01-16 13:44:21,075 INFO SenderThread:50740 [sender.py:_save_file():1403] saving file diff.patch with policy now
2024-01-16 13:44:21,085 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: python_packages
2024-01-16 13:44:21,086 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: stop_status
2024-01-16 13:44:21,088 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: internal_messages
2024-01-16 13:44:21,088 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: python_packages
2024-01-16 13:44:21,091 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: stop_status
2024-01-16 13:44:21,346 INFO Thread-12 :50740 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/conda-environment.yaml
2024-01-16 13:44:21,346 INFO Thread-12 :50740 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/code/train_model.py
2024-01-16 13:44:21,346 INFO Thread-12 :50740 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/requirements.txt
2024-01-16 13:44:21,346 INFO Thread-12 :50740 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/diff.patch
2024-01-16 13:44:21,346 INFO Thread-12 :50740 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/wandb-metadata.json
2024-01-16 13:44:21,346 INFO Thread-12 :50740 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/code
2024-01-16 13:44:21,425 DEBUG SenderThread:50740 [sender.py:send():382] send: telemetry
2024-01-16 13:44:21,485 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: log_artifact
2024-01-16 13:44:21,485 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: log_artifact
2024-01-16 13:44:21,726 INFO wandb-upload_0:50740 [upload_job.py:push():131] Uploaded file /var/folders/6l/0sqsbv_d56dfzqkhyxn_6pdc0000gn/T/tmpuqu4hsy4wandb/038rwugh-wandb-metadata.json
2024-01-16 13:44:21,800 INFO wandb-upload_2:50740 [upload_job.py:push():131] Uploaded file /var/folders/6l/0sqsbv_d56dfzqkhyxn_6pdc0000gn/T/tmpuqu4hsy4wandb/2wekqvac-diff.patch
2024-01-16 13:44:21,826 INFO wandb-upload_1:50740 [upload_job.py:push():131] Uploaded file /var/folders/6l/0sqsbv_d56dfzqkhyxn_6pdc0000gn/T/tmpuqu4hsy4wandb/x0wbm1r9-code/train_model.py
2024-01-16 13:44:22,220 INFO SenderThread:50740 [sender.py:send_request_log_artifact():1470] logged artifact source-ReAct_Jax-train_model.py - {'id': 'QXJ0aWZhY3Q6NjkyMzU4NjQ4', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjk2MzgzMDU1', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NjkyMzQ5MDk3', 'versionIndex': 779}}}
2024-01-16 13:44:22,352 INFO Thread-12 :50740 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log
2024-01-16 13:44:22,760 INFO wandb-upload_3:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpm2qtgv5c
2024-01-16 13:44:22,760 INFO wandb-upload_4:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpn4coql7t
2024-01-16 13:44:22,760 INFO wandb-upload_0:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpib_ch1wp
2024-01-16 13:44:22,760 INFO wandb-upload_2:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpsywsbb4g
2024-01-16 13:44:22,760 INFO wandb-upload_5:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp3ho8mwvu
2024-01-16 13:44:22,760 INFO wandb-upload_1:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp7tff6mqs
2024-01-16 13:44:22,760 INFO wandb-upload_6:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpsdedy6hk
2024-01-16 13:44:22,760 INFO wandb-upload_7:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp9dvm9woh
2024-01-16 13:44:22,760 INFO wandb-upload_8:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp1_tf0yke
2024-01-16 13:44:22,760 INFO wandb-upload_9:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpjtmkx43g
2024-01-16 13:44:22,760 INFO wandb-upload_10:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpp6oyz83a
2024-01-16 13:44:22,760 INFO wandb-upload_11:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpob5ugug9
2024-01-16 13:44:22,760 INFO wandb-upload_12:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpjlrffhb8
2024-01-16 13:44:22,760 INFO wandb-upload_13:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpy614d5bc
2024-01-16 13:44:22,760 INFO wandb-upload_14:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp1je_yjfh
2024-01-16 13:44:22,760 INFO wandb-upload_15:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmps7o8nb68
2024-01-16 13:44:22,760 INFO wandb-upload_16:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpfcyru9zw
2024-01-16 13:44:22,760 INFO wandb-upload_17:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp8kz0ef7o
2024-01-16 13:44:22,760 INFO wandb-upload_18:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpwn26uecs
2024-01-16 13:44:22,760 INFO wandb-upload_19:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpagvc_xh6
2024-01-16 13:44:22,760 INFO wandb-upload_20:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpoyxti6id
2024-01-16 13:44:22,760 INFO wandb-upload_21:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp77n9guol
2024-01-16 13:44:22,760 INFO wandb-upload_22:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpump20818
2024-01-16 13:44:22,760 INFO wandb-upload_23:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpgo3263rz
2024-01-16 13:44:22,760 INFO wandb-upload_24:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmplghn1pao
2024-01-16 13:44:22,760 INFO wandb-upload_25:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp_xotfvdg
2024-01-16 13:44:22,760 INFO wandb-upload_27:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpv7elk7mv
2024-01-16 13:44:22,760 INFO wandb-upload_26:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp43i6yffx
2024-01-16 13:44:22,760 INFO wandb-upload_28:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpseyu_9_o
2024-01-16 13:44:22,760 INFO wandb-upload_29:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpvoslg0bm
2024-01-16 13:44:22,760 INFO wandb-upload_30:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpix9nquxf
2024-01-16 13:44:22,760 INFO wandb-upload_31:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpj7bdcbut
2024-01-16 13:44:22,760 INFO wandb-upload_32:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmppcrp9icw
2024-01-16 13:44:22,760 INFO wandb-upload_33:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp8ebyay3x
2024-01-16 13:44:22,760 INFO wandb-upload_34:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp2j6z1o60
2024-01-16 13:44:22,760 INFO wandb-upload_35:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp6nv3l7hn
2024-01-16 13:44:22,760 INFO wandb-upload_36:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp6fanmuem
2024-01-16 13:44:22,760 INFO wandb-upload_37:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpx6pgrh7u
2024-01-16 13:44:22,760 INFO wandb-upload_38:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpmqxx6oxo
2024-01-16 13:44:22,760 INFO wandb-upload_39:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp5soa99zu
2024-01-16 13:44:22,760 INFO wandb-upload_40:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpywbclbtf
2024-01-16 13:44:22,760 INFO wandb-upload_41:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp453er4jv
2024-01-16 13:44:22,760 INFO wandb-upload_42:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpcmpsn22h
2024-01-16 13:44:22,760 INFO wandb-upload_43:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp70fuiuiw
2024-01-16 13:44:22,760 INFO wandb-upload_44:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp2qq7pk0j
2024-01-16 13:44:22,760 INFO wandb-upload_45:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpv6adahqf
2024-01-16 13:44:22,760 INFO wandb-upload_46:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmprgeaelwl
2024-01-16 13:44:22,760 INFO wandb-upload_47:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpflka1lg3
2024-01-16 13:44:22,760 INFO wandb-upload_48:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp9fsfa3x4
2024-01-16 13:44:22,760 INFO wandb-upload_49:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp00j_5ida
2024-01-16 13:44:22,760 INFO wandb-upload_50:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpltqtkh7a
2024-01-16 13:44:22,760 INFO wandb-upload_51:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpdeyvxwxe
2024-01-16 13:44:22,760 INFO wandb-upload_52:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmp0ym7_5f0
2024-01-16 13:44:22,760 INFO wandb-upload_53:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpvvahk7ug
2024-01-16 13:44:22,760 INFO wandb-upload_54:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpfx9ej6wf
2024-01-16 13:44:22,760 INFO wandb-upload_55:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpppqd_g1o
2024-01-16 13:44:22,760 INFO wandb-upload_56:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpy38tiar9
2024-01-16 13:44:22,760 INFO wandb-upload_57:50740 [upload_job.py:push():86] Skipped uploading /Users/neel/Library/Application Support/wandb/artifacts/staging/tmpf78hc0me
2024-01-16 13:44:24,362 INFO Thread-12 :50740 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log
2024-01-16 13:44:25,005 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: status_report
2024-01-16 13:44:26,373 INFO Thread-12 :50740 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log
2024-01-16 13:44:28,384 INFO Thread-12 :50740 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log
2024-01-16 13:44:30,394 INFO Thread-12 :50740 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log
2024-01-16 13:44:30,730 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: status_report
2024-01-16 13:44:31,466 DEBUG SenderThread:50740 [sender.py:send():382] send: exit
2024-01-16 13:44:31,467 INFO SenderThread:50740 [sender.py:send_exit():589] handling exit code: 0
2024-01-16 13:44:31,467 INFO SenderThread:50740 [sender.py:send_exit():591] handling runtime: 15
2024-01-16 13:44:31,468 INFO SenderThread:50740 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-01-16 13:44:31,468 INFO SenderThread:50740 [sender.py:send_exit():597] send defer
2024-01-16 13:44:31,468 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:31,468 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 0
2024-01-16 13:44:31,469 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:31,469 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 0
2024-01-16 13:44:31,469 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 1
2024-01-16 13:44:31,469 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:31,469 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 1
2024-01-16 13:44:31,469 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:31,469 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 1
2024-01-16 13:44:31,469 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 2
2024-01-16 13:44:31,469 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:31,469 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 2
2024-01-16 13:44:31,469 INFO HandlerThread:50740 [system_monitor.py:finish():203] Stopping system monitor
2024-01-16 13:44:31,470 INFO HandlerThread:50740 [interfaces.py:finish():202] Joined cpu monitor
2024-01-16 13:44:31,470 DEBUG SystemMonitor:50740 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-01-16 13:44:31,470 DEBUG SystemMonitor:50740 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-01-16 13:44:31,470 DEBUG SystemMonitor:50740 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-01-16 13:44:31,471 INFO HandlerThread:50740 [interfaces.py:finish():202] Joined disk monitor
2024-01-16 13:44:31,471 INFO HandlerThread:50740 [interfaces.py:finish():202] Joined gpuapple monitor
2024-01-16 13:44:31,471 INFO HandlerThread:50740 [interfaces.py:finish():202] Joined memory monitor
2024-01-16 13:44:31,471 INFO HandlerThread:50740 [interfaces.py:finish():202] Joined network monitor
2024-01-16 13:44:31,472 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:31,472 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 2
2024-01-16 13:44:31,472 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 3
2024-01-16 13:44:31,472 DEBUG SenderThread:50740 [sender.py:send():382] send: telemetry
2024-01-16 13:44:31,472 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:31,472 DEBUG SenderThread:50740 [sender.py:send():382] send: stats
2024-01-16 13:44:31,472 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 3
2024-01-16 13:44:31,473 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:31,473 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 3
2024-01-16 13:44:31,473 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 4
2024-01-16 13:44:31,473 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:31,473 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 4
2024-01-16 13:44:31,473 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:31,473 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 4
2024-01-16 13:44:31,473 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 5
2024-01-16 13:44:31,473 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:31,473 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 5
2024-01-16 13:44:31,474 DEBUG SenderThread:50740 [sender.py:send():382] send: summary
2024-01-16 13:44:31,474 INFO SenderThread:50740 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-01-16 13:44:31,474 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:31,474 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 5
2024-01-16 13:44:31,474 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 6
2024-01-16 13:44:31,474 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:31,474 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 6
2024-01-16 13:44:31,474 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:31,474 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 6
2024-01-16 13:44:31,478 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: status_report
2024-01-16 13:44:31,670 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 7
2024-01-16 13:44:31,671 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:31,671 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 7
2024-01-16 13:44:31,671 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:31,671 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 7
2024-01-16 13:44:32,405 INFO Thread-12 :50740 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log
2024-01-16 13:44:32,406 INFO Thread-12 :50740 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/config.yaml
2024-01-16 13:44:32,406 INFO Thread-12 :50740 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/wandb-summary.json
2024-01-16 13:44:32,467 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: poll_exit
2024-01-16 13:44:34,033 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 8
2024-01-16 13:44:34,034 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: poll_exit
2024-01-16 13:44:34,034 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:34,035 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 8
2024-01-16 13:44:34,035 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:34,035 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 8
2024-01-16 13:44:34,035 INFO SenderThread:50740 [job_builder.py:build():296] Attempting to build job artifact
2024-01-16 13:44:34,036 INFO SenderThread:50740 [job_builder.py:_get_source_type():426] is repo sourced job
2024-01-16 13:44:34,039 INFO SenderThread:50740 [job_builder.py:build():402] adding wandb-job metadata file
2024-01-16 13:44:34,044 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 9
2024-01-16 13:44:34,045 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:34,045 DEBUG SenderThread:50740 [sender.py:send():382] send: artifact
2024-01-16 13:44:34,045 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 9
2024-01-16 13:44:34,413 INFO Thread-12 :50740 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log
2024-01-16 13:44:34,475 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: poll_exit
2024-01-16 13:44:34,643 INFO SenderThread:50740 [sender.py:send_artifact():1494] sent artifact job-https___github.com_neel04_ReAct_Jax.git_train_model.py - {'id': 'QXJ0aWZhY3Q6NjkyMzQ5MjE5', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjk2Mzg2MTkz', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NjkyMzQ5MjE5', 'versionIndex': 284}}}
2024-01-16 13:44:34,643 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:34,643 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 9
2024-01-16 13:44:34,643 INFO SenderThread:50740 [dir_watcher.py:finish():358] shutting down directory watcher
2024-01-16 13:44:35,418 INFO SenderThread:50740 [dir_watcher.py:finish():388] scan: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files
2024-01-16 13:44:35,419 INFO SenderThread:50740 [dir_watcher.py:finish():402] scan save: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/diff.patch diff.patch
2024-01-16 13:44:35,419 INFO SenderThread:50740 [dir_watcher.py:finish():402] scan save: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/requirements.txt requirements.txt
2024-01-16 13:44:35,419 INFO SenderThread:50740 [dir_watcher.py:finish():402] scan save: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log output.log
2024-01-16 13:44:35,420 INFO SenderThread:50740 [dir_watcher.py:finish():402] scan save: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/config.yaml config.yaml
2024-01-16 13:44:35,425 INFO SenderThread:50740 [dir_watcher.py:finish():402] scan save: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/wandb-summary.json wandb-summary.json
2024-01-16 13:44:35,427 INFO SenderThread:50740 [dir_watcher.py:finish():402] scan save: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/conda-environment.yaml conda-environment.yaml
2024-01-16 13:44:35,428 INFO SenderThread:50740 [dir_watcher.py:finish():402] scan save: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/wandb-metadata.json wandb-metadata.json
2024-01-16 13:44:35,432 INFO SenderThread:50740 [dir_watcher.py:finish():402] scan save: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/code/train_model.py code/train_model.py
2024-01-16 13:44:35,432 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 10
2024-01-16 13:44:35,433 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: poll_exit
2024-01-16 13:44:35,433 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:35,433 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 10
2024-01-16 13:44:35,434 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:35,434 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 10
2024-01-16 13:44:35,434 INFO SenderThread:50740 [file_pusher.py:finish():175] shutting down file pusher
2024-01-16 13:44:35,481 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: poll_exit
2024-01-16 13:44:35,481 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: poll_exit
2024-01-16 13:44:35,811 INFO wandb-upload_6:50740 [upload_job.py:push():131] Uploaded file /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/wandb-summary.json
2024-01-16 13:44:35,821 INFO wandb-upload_3:50740 [upload_job.py:push():131] Uploaded file /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/requirements.txt
2024-01-16 13:44:35,821 INFO wandb-upload_5:50740 [upload_job.py:push():131] Uploaded file /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/config.yaml
2024-01-16 13:44:35,900 INFO wandb-upload_0:50740 [upload_job.py:push():131] Uploaded file /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/output.log
2024-01-16 13:44:36,005 INFO wandb-upload_8:50740 [upload_job.py:push():131] Uploaded file /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/files/conda-environment.yaml
2024-01-16 13:44:36,211 INFO Thread-11 (_thread_body):50740 [sender.py:transition_state():617] send defer: 11
2024-01-16 13:44:36,212 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:36,212 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 11
2024-01-16 13:44:36,213 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:36,213 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 11
2024-01-16 13:44:36,214 INFO SenderThread:50740 [file_pusher.py:join():181] waiting for file pusher
2024-01-16 13:44:36,214 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 12
2024-01-16 13:44:36,214 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:36,214 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 12
2024-01-16 13:44:36,215 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:36,215 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 12
2024-01-16 13:44:36,215 INFO SenderThread:50740 [file_stream.py:finish():595] file stream finish called
2024-01-16 13:44:36,379 INFO SenderThread:50740 [file_stream.py:finish():599] file stream finish is done
2024-01-16 13:44:36,379 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 13
2024-01-16 13:44:36,380 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:36,380 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 13
2024-01-16 13:44:36,380 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:36,380 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 13
2024-01-16 13:44:36,381 INFO SenderThread:50740 [sender.py:transition_state():617] send defer: 14
2024-01-16 13:44:36,382 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: defer
2024-01-16 13:44:36,382 DEBUG SenderThread:50740 [sender.py:send():382] send: final
2024-01-16 13:44:36,382 INFO HandlerThread:50740 [handler.py:handle_request_defer():172] handle defer: 14
2024-01-16 13:44:36,382 DEBUG SenderThread:50740 [sender.py:send():382] send: footer
2024-01-16 13:44:36,383 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: defer
2024-01-16 13:44:36,383 INFO SenderThread:50740 [sender.py:send_request_defer():613] handle sender defer: 14
2024-01-16 13:44:36,384 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: poll_exit
2024-01-16 13:44:36,384 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: poll_exit
2024-01-16 13:44:36,384 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: poll_exit
2024-01-16 13:44:36,385 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: server_info
2024-01-16 13:44:36,386 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: poll_exit
2024-01-16 13:44:36,386 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: get_summary
2024-01-16 13:44:36,386 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: server_info
2024-01-16 13:44:36,387 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: sampled_history
2024-01-16 13:44:36,389 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: internal_messages
2024-01-16 13:44:36,390 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: job_info
2024-01-16 13:44:36,534 DEBUG SenderThread:50740 [sender.py:send_request():409] send_request: job_info
2024-01-16 13:44:36,535 INFO MainThread:50740 [wandb_run.py:_footer_history_summary_info():3857] rendering history
2024-01-16 13:44:36,535 INFO MainThread:50740 [wandb_run.py:_footer_history_summary_info():3889] rendering summary
2024-01-16 13:44:36,535 INFO MainThread:50740 [wandb_run.py:_footer_sync_info():3816] logging synced files
2024-01-16 13:44:36,536 DEBUG HandlerThread:50740 [handler.py:handle_request():146] handle_request: shutdown
2024-01-16 13:44:36,537 INFO HandlerThread:50740 [handler.py:finish():869] shutting down handler
2024-01-16 13:44:37,395 INFO WriterThread:50740 [datastore.py:close():296] close: /Users/neel/Documents/research/ReAct_Jax/wandb/run-20240116_134414-n5zyqwu1/run-n5zyqwu1.wandb
2024-01-16 13:44:37,540 INFO SenderThread:50740 [sender.py:finish():1572] shutting down sender
2024-01-16 13:44:37,540 INFO SenderThread:50740 [file_pusher.py:finish():175] shutting down file pusher
2024-01-16 13:44:37,540 INFO SenderThread:50740 [file_pusher.py:join():181] waiting for file pusher