Finally able to get more work on system that errored Tasks.

However, while the BOINC Manager indicates they are running on different GPUs, they are in fact running on the same GPU (which explains why so many errored out even though many showed as being on the GTX 1070 8GB VRAM GPU). It doesn't explain why that one Task was able to run OK after all the others did error out on the GPU that is now running Tasks OK....
And the Remaining (estimated) time is 3hr 20min, not the actual 26min or so processing time.

Re-instated application limit of 1 Task.
Exited BOINC and restarted, Task restarted from beginning (still no checkpointing),

Looking at progress indicator- appears to be based on time done and initial Remaining (estimated) time, not actual processing done.


This time around Monitoring VRAM on the RTX 2060 (6GB VRAM), and it's got up to 6097MB Maximum in use (so far).

---

Grant
Darwin NT

Something odd going on with this system.

Some Tasks are erroring out, others are Validating with odd Runtimes (and Credit).

eg

        Status            Run time  CPU time    Credit
Completed and validated     208.42      0.27    185.19
Completed and validated     390.55      0.69     85.19
Completed and validated      50.46      0.00  3,590.43
Error while computing        29.26      0.00       ---
Error while computing        33.19      0.02       ---
Completed and validated     150.12      0.00    185.19

The two Tasks that error out have these messages

<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
The access code is invalid.
 (0xc) - exit code 12 (0xc)</message>
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
  Z = torch.cross(Xn,Yn)

</stderr_txt>
]]>

and

<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
The access code is invalid.
 (0xc) - exit code 12 (0xc)</message>
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
  Z = torch.cross(Xn,Yn)
Traceback (most recent call last):
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aapredict.py", line 742, in <module>
    pred.predict(out_name+f'_{n}', 
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aapredict.py", line 550, in predict
    logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned,                msa_prev, pair_prev, state_prev = self.model(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaRoseTTAFoldModel.py", line 358, in forward
    msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaTrack_module.py", line 1106, in forward
    msa, pair, xyz, state, alpha, symmsub = self.main_block[i_m](msa, pair,
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaTrack_module.py", line 929, in forward
    xyz, state, alpha = self.str2str(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchampautocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaTrack_module.py", line 476, in forward
    neighbor = get_seqsep_protein_sm(idx, bond_feats, dist_matrix, rotation_mask)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil_module.py", line 106, in get_seqsep_protein_sm
    res_dist, atom_dist = get_res_atom_dist(idx, bond_feats, dist_matrix, sm_mask)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil_module.py", line 148, in get_res_atom_dist
    res_dist_inter[:,sm_mask] = res_dist_prot[:,closest_prot_res]
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


</stderr_txt>
]]

Yet some of those that Validate have this error message

<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
  Z = torch.cross(Xn,Yn)
Traceback (most recent call last):
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aapredict.py", line 742, in <module>
    pred.predict(out_name+f'_{n}', 
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aapredict.py", line 550, in predict
    logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned,                msa_prev, pair_prev, state_prev = self.model(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaRoseTTAFoldModel.py", line 358, in forward
    msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaTrack_module.py", line 1144, in forward
    dchiraldxyz, = calc_chiral_grads(xyz.detach(),chirals)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchutils_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaloss.py", line 1364, in calc_chiral_grads
    if l.item() == 0.0:
RuntimeError: CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

13:46:10 (13616): called boinc_finish(0)

</stderr_txt>
]]>

or even this one

<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
  Z = torch.cross(Xn,Yn)
Traceback (most recent call last):
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aapredict.py", line 742, in <module>
    pred.predict(out_name+f'_{n}', 
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aapredict.py", line 550, in predict
    logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned,                msa_prev, pair_prev, state_prev = self.model(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaRoseTTAFoldModel.py", line 358, in forward
    msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaTrack_module.py", line 1106, in forward
    msa, pair, xyz, state, alpha, symmsub = self.main_block[i_m](msa, pair,
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaTrack_module.py", line 929, in forward
    xyz, state, alpha = self.str2str(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchampautocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaTrack_module.py", line 503, in forward
    shift = self.se3(G, node.reshape(B*L, -1, 1), l1_feats, edge_feats)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aaSE3_network.py", line 96, in forward
    return self.se3(G, node_features, edge_features)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aa/SE3Transformerse3_transformermodeltransformer.py", line 185, in forward
    node_feats = self.graph_modules(node_feats, edge_feats, graph=graph, basis=basis)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aa/SE3Transformerse3_transformermodeltransformer.py", line 47, in forward
    input = module(input, *args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aa/SE3Transformerse3_transformermodellayersattention.py", line 162, in forward
    fused_key_value = self.to_key_value(node_features, edge_features, graph, basis)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagestorchnnmodulesmodule.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aa/SE3Transformerse3_transformermodellayersconvolution.py", line 319, in forward
    src, dst = graph.edges()
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagesdglview.py", line 179, in __call__
    return self._graph.all_edges(*args, **kwargs)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagesdglheterograph.py", line 3589, in all_edges
    src, dst, eid = self._graph.edges(self.get_etype_id(etype), order)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagesdglheterograph_index.py", line 696, in edges
    edge_array = _CAPI_DGLHeteroEdges(self, int(etype), order)
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagesdgl_ffi_ctypesfunction.py", line 212, in __call__
    check_call(
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libsite-packagesdgl_ffibase.py", line 70, in check_call
    raise DGLError(py_str(_LIB.DGLGetLastError()))
dgl._ffi.base.DGLError: [13:47:33] C:UsersAdministratorDGL_scriptsreleasewin-64dglsrcruntimecudacuda_device_api.cc:117: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: unknown error
13:47:34 (4532): called boinc_finish(0)

</stderr_txt>
]]>

---

Grant
Darwin NT

And speaking of odd things- every so often (roughly 1 in 10 Tasks) the amount of Credit awarded varies, a lot.
The usual amount is 185.19, but i've seen amounts as high as 3,300 on one of my Tasks. The anomalous amounts tend to be between 1,000 to 2,000.

The variation occurs much less often than with the previous application, although with the previous application the highs weren't nearly as high, but there were also lots of lows, some down in single digits.





The other odd thing- there have been no errors with the current v0.03 (nvidia_alpha) application on my RTX 2060 Super (8GB VRAM) card.
But on my RTX 2060 (6GB VRAM) card there has been 1 error (so far) out of 60 Tasks processed (59 of which completed and Validated).


RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_i_pred_225_16905_2_1

<core_client_version>8.0.2</core_client_version>
<![CDATA[
<message>
The access code is invalid.
 (0xc) - exit code 12 (0xc)</message>
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
  Z = torch.cross(Xn,Yn)
Traceback (most recent call last):
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aapredict.py", line 733, in <module>
    with zipfile.ZipFile(args.z) as z:
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libzipfile.py", line 1268, in __init__
    self._RealGetContents()
  File "C:ProgramDataBOINCprojectsralph.bakerlab.orgev1libzipfile.py", line 1335, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

</stderr_txt>
]]>

The Task itself is one that was released along with the previous version ( v0.02 (nvidia_alpha)) application in case there is a difference in Tasks released for the different applications.

---

Grant
Darwin NT

It's been a week, and still no updated application.
I had been hoping there'd be a new one by now...

---

Grant
Darwin NT

It's been a week, and still no updated application.
I had been hoping there'd be a new one by now...

You are a "rookie" here.
So you don't know that the timing of Ralph is very randomic
And admins VERY rarely interact with the forum

Well, that recent batch of work didn't last long with their significantly shorter runtimes.

Hopefully they got the info they needed and the next version isn't far off- check pointing still needs sorting out. As does the progress indicator- it's running purely on processing time and not the actual amount of work done, so a much closer Remaining Estimated (time) is needed.
Stick with the initial high estimate for the first few Tasks sent out, but as work is returned and validated, base the Remaining (estimated) time on those completed & Validated Task processing times.

---

Grant
Darwin NT

One thing i am noticing with this latest batch of work, is the low VRAM usage.

What is odd is that the video card with the larger amount of VRAM is using less than the one with the smaller amount of VRAM.
Only half a dozen Tasks completed so far per system, however the system with 8GB VRAM, the Max VRAM used so far is 4003MB. The card with 6GB of VRAM, the Max VRAM used so far is 4503MB (although the system with the smaller VRAM video card is also the one i use daily- so multiple programme windows are open), the other is a spare.

If someone with a 12GB or 16GB VRAM card could leave GPU-Z running in the background i'd be interested to see what their maximum VRAM usage is.

---

Grant
Darwin NT

All NVIDIA tasks currently failing with a Validation Error and the following output

<core_client_version>8.0.4</core_client_version>
<![CDATA[
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
Z = torch.cross(Xn,Yn)
20:46:10 (20988): called boinc_finish(0)

</stderr_txt>
]]>

---

In October 1969 I took an oath to support and defend the Constitution of the United States against all enemies, foreign and domestic;
There was no expiration date.

Rosetta and Ralph go to great lengths to compute credit for work done by volunteers. It is TOO BAD that volunteers get NO CREDIT for their 1000's of seconds of machine time when the failure is caused by the researcher submitting the broken work.

The same bug has been reported here for some time.

I have a RTX 4090 with 24gb. Windows task manager says that it is using 90%-100% GPU 3D, 7.2GB/24GB dedicated. I think I captured the end of processing on GPU-Z log file. If not, I can capture whatever you need ... with some guidance.

I have a SCREEN CAPTURE of the TASK MANAGER while running the WU.

I think it ended at:
2024-09-02 23:34:56 , 2715.0 , 1281.5 , 61.5 , 70.2 , 74.0 , 48 , 1499 , 43 , 1380 , 7749 , 1 , 0 , 0 , 0 , 90.8 , 56.8 , 24.2 , 34.1 , 12.1 , 9.8 , 12.0 , 81.0 , 12.0 , 20.2 , 4 , 1.0500 , 95.0 , 13569 ,
2024-09-02 23:34:57 , 2520.0 , 1312.7 , 60.3 , 69.4 , 72.0 , 48 , 1500 , 43 , 1380 , 1401 , 1 , 0 , 0 , 0 , 73.5 , 39.3 , 24.4 , 34.3 , 12.1 , 9.9 , 12.0 , 63.6 , 12.1 , 16.3 , 16 , 0.9650 , 95.0 , 12406 ,

let me know what you need.


Date , GPU Clock [MHz] , Memory Clock [MHz] , GPU Temperature [°C] , Hot Spot [°C] , Memory Temperature [°C] , Fan 1 Speed (%) [%] , Fan 1 Speed (RPM) [RPM] , Fan 2 Speed (%) [%] , Fan 2 Speed (RPM) [RPM] , Memory Used [MB] , GPU Load [%] , Memory Controller Load [%] , Video Engine Load [%] , Bus Interface Load [%] , Board Power Draw [W] , GPU Chip Power Draw [W] , MVDDC Power Draw [W] , PWR_SRC Power Draw [W] , PWR_SRC Voltage [V] , PCIe Slot Power [W] , PCIe Slot Voltage [V] , 16-Pin Power [W] , 16-Pin Voltage [V] , Power Consumption (%) [% TDP] , PerfCap Reason [] , GPU Voltage [V] , CPU Temperature [°C] , System Memory Used [MB] ,
2024-09-02 23:30:42 , 2700.0 , 1281.5 , 66.9 , 74.7 , 76.0 , 47 , 1488 , 42 , 1371 , 7730 , 59 , 45 , 0 , 0 , 263.1 , 190.1 , 62.7 , 73.0 , 12.0 , 10.3 , 12.0 , 252.8 , 12.0 , 58.5 , 4 , 1.0500 , 93.0 , 13929 ,
2024-09-02 23:30:43 , 2700.0 , 1281.5 , 71.9 , 81.6 , 78.0 , 47 , 1488 , 42 , 1371 , 7727 , 70 , 61 , 0 , 0 , 313.8 , 230.8 , 72.5 , 83.0 , 12.0 , 10.6 , 12.0 , 303.2 , 12.0 , 69.7 , 4 , 1.0500 , 95.0 , 13944 ,
2024-09-02 23:30:44 , 2700.0 , 1281.5 , 70.7 , 80.2 , 78.0 , 47 , 1487 , 42 , 1370 , 7730 , 69 , 58 , 0 , 0 , 252.0 , 182.8 , 58.9 , 69.1 , 12.0 , 10.3 , 12.0 , 241.7 , 12.0 , 56.0 , 4 , 1.0500 , 94.0 , 13947 ,
2024-09-02 23:30:45 , 2700.0 , 1281.5 , 71.7 , 81.8 , 76.0 , 47 , 1487 , 42 , 1371 , 7727 , 78 , 65 , 0 , 0 , 278.2 , 203.7 , 64.0 , 74.5 , 12.0 , 10.5 , 12.0 , 267.8 , 12.0 , 61.8 , 4 , 1.0500 , 95.0 , 13957 ,
2024-09-02 23:30:46 , 2700.0 , 1281.5 , 68.0 , 76.3 , 78.0 , 47 , 1487 , 42 , 1372 , 7727 , 67 , 58 , 0 , 0 , 294.9 , 215.3 , 69.2 , 79.7 , 12.0 , 10.4 , 12.0 , 284.5 , 12.0 , 65.5 , 4 , 1.0500 , 95.0 , 13940 ,
2024-09-02 23:30:47 , 2700.0 , 1281.5 , 72.2 , 82.6 , 78.0 , 47 , 1486 , 42 , 1372 , 7727 , 63 , 54 , 0 , 0 , 292.3 , 215.2 , 66.8 , 77.1 , 12.0 , 10.3 , 12.0 , 282.0 , 12.0 , 65.0 , 4 , 1.0500 , 95.0 , 13941 ,
2024-09-02 23:30:48 , 2700.0 , 1281.5 , 70.6 , 80.1 , 78.0 , 47 , 1486 , 42 , 1372 , 7727 , 66 , 52 , 0 , 1 , 300.8 , 221.6 , 68.9 , 79.2 , 12.0 , 10.3 , 12.0 , 290.5 , 12.0 , 66.8 , 4 , 1.0500 , 95.0 , 13862 ,
2024-09-02 23:30:49 , 2700.0 , 1281.5 , 69.6 , 78.6 , 78.0 , 47 , 1486 , 42 , 1372 , 7733 , 71 , 58 , 0 , 0 , 271.3 , 196.8 , 64.2 , 74.5 , 12.0 , 10.2 , 12.0 , 261.1 , 12.0 , 60.3 , 4 , 1.0500 , 95.0 , 13854 ,
2024-09-02 23:30:50 , 2700.0 , 1281.5 , 69.0 , 77.6 , 78.0 , 47 , 1487 , 42 , 1374 , 7731 , 69 , 58 , 0 , 0 , 286.7 , 210.3 , 66.2 , 76.4 , 12.0 , 10.2 , 12.0 , 276.5 , 12.0 , 63.7 , 4 , 1.0500 , 95.0 , 13852 ,
2024-09-02 23:30:51 , 2700.0 , 1281.5 , 68.8 , 77.5 , 78.0 , 47 , 1488 , 42 , 1373 , 7729 , 62 , 48 , 0 , 1 , 304.5 , 223.6 , 70.6 , 81.0 , 12.0 , 10.4 , 12.0 , 294.1 , 12.0 , 67.7 , 4 , 1.0500 , 95.0 , 13855 ,
2024-09-02 23:30:52 , 2700.0 , 1281.5 , 71.5 , 81.2 , 76.0 , 47 , 1490 , 42 , 1372 , 7729 , 67 , 57 , 0 , 0 , 308.8 , 226.5 , 71.9 , 82.3 , 12.0 , 10.4 , 12.0 , 298.4 , 12.0 , 68.6 , 4 , 1.0500 , 95.0 , 13843 ,
2024-09-02 23:30:53 , 2700.0 , 1281.5 , 70.0 , 79.7 , 78.0 , 47 , 1487 , 42 , 1373 , 7735 , 66 , 56 , 0 , 0 , 318.9 , 235.6 , 72.9 , 83.3 , 12.0 , 10.4 , 12.0 , 308.5 , 12.0 , 70.9 , 4 , 1.0500 , 94.0 , 13838 ,
2024-09-02 23:30:54 , 2700.0 , 1281.5 , 61.9 , 70.6 , 74.0 , 47 , 1488 , 42 , 1372 , 7729 , 0 , 0 , 0 , 0 , 89.5 , 55.4 , 24.4 , 34.1 , 12.1 , 9.7 , 12.0 , 79.8 , 12.0 , 19.9 , 4 , 1.0500 , 95.0 , 13787 ,
2024-09-02 23:30:55 , 2700.0 , 1281.5 , 70.8 , 80.9 , 76.0 , 47 , 1488 , 42 , 1373 , 7729 , 64 , 55 , 0 , 0 , 296.7 , 215.7 , 70.7 , 81.1 , 12.0 , 10.3 , 12.0 , 286.4 , 12.0 , 65.9 , 4 , 1.0500 , 95.0 , 13731 ,
2024-09-02 23:30:56 , 2700.0 , 1281.5 , 67.6 , 75.8 , 78.0 , 47 , 1489 , 42 , 1372 , 7719 , 70 , 56 , 0 , 0 , 278.4 , 201.1 , 67.0 , 77.3 , 12.0 , 10.3 , 12.0 , 268.1 , 12.0 , 61.9 , 4 , 1.0500 , 95.0 , 13726 ,
2024-09-02 23:30:57 , 2700.0 , 1281.5 , 64.9 , 71.8 , 76.0 , 47 , 1486 , 42 , 1374 , 7720 , 53 , 43 , 0 , 0 , 144.9 , 97.9 , 37.2 , 47.0 , 12.0 , 9.8 , 12.0 , 135.1 , 12.0 , 32.2 , 4 , 1.0500 , 95.0 , 13720 ,
2024-09-02 23:30:58 , 2700.0 , 1281.5 , 67.7 , 75.9 , 78.0 , 47 , 1486 , 42 , 1373 , 7728 , 67 , 57 , 0 , 0 , 267.5 , 193.2 , 64.0 , 74.2 , 12.0 , 10.2 , 12.0 , 257.3 , 12.0 , 59.4 , 4 , 1.0500 , 95.0 , 13789 ,
2024-09-02 23:30:59 , 2700.0 , 1281.5 , 69.3 , 78.0 , 78.0 , 47 , 1489 , 42 , 1374 , 7752 , 66 , 56 , 0 , 0 , 304.1 , 221.2 , 72.6 , 82.9 , 12.0 , 10.3 , 12.0 , 293.8 , 12.0 , 67.6 , 4 , 1.0500 , 95.0 , 13803 ,
2024-09-02 23:31:00 , 2700.0 , 1281.5 , 64.3 , 71.7 , 76.0 , 47 , 1489 , 42 , 1375 , 7740 , 50 , 38 , 0 , 0 , 145.1 , 98.2 , 37.0 , 46.8 , 12.0 , 9.8 , 12.0 , 135.3 , 12.0 , 32.2 , 4 , 1.0500 , 95.0 , 13806 ,
2024-09-02 23:31:01 , 2700.0 , 1281.5 , 68.0 , 76.4 , 76.0 , 47 , 1488 , 42 , 1375 , 7735 , 71 , 61 , 0 , 0 , 253.1 , 182.4 , 60.5 , 70.7 , 12.0 , 10.2 , 12.0 , 242.9 , 12.0 , 56.2 , 4 , 1.0500 , 95.0 , 13831 ,
2024-09-02 23:31:02 , 2700.0 , 1281.5 , 70.8 , 80.3 , 78.0 , 47 , 1487 , 42 , 1374 , 7735 , 70 , 59 , 0 , 0 , 294.6 , 214.2 , 70.0 , 80.4 , 12.0 , 10.4 , 12.0 , 284.2 , 12.0 , 65.5 , 4 , 1.0500 , 95.0 , 13830 ,
2024-09-02 23:31:03 , 2700.0 , 1281.5 , 69.0 , 77.7 , 76.0 , 47 , 1488 , 42 , 1374 , 7735 , 33 , 19 , 0 , 0 , 178.0 , 122.4 , 45.7 , 55.6 , 12.0 , 9.9 , 12.0 , 168.1 , 12.0 , 39.5 , 4 , 1.0500 , 95.0 , 13825 ,
2024-09-02 23:31:04 , 2700.0 , 1281.5 , 67.8 , 75.8 , 78.0 , 47 , 1489 , 42 , 1373 , 7735 , 70 , 61 , 0 , 0 , 275.2 , 198.8 , 66.1 , 76.3 , 12.0 , 10.2 , 12.0 , 264.9 , 12.0 , 61.1 , 4 , 1.0500 , 95.0 , 13817 ,
2024-09-02 23:31:05 , 2700.0 , 1281.5 , 69.7 , 78.8 , 78.0 , 47 , 1489 , 42 , 1374 , 7741 , 69 , 59 , 0 , 0 , 272.4 , 198.2 , 64.0 , 74.2 , 12.0 , 10.3 , 12.0 , 262.2 , 12.0 , 60.5 , 4 , 1.0500 , 95.0 , 13815 ,
2024-09-02 23:31:06 , 2700.0 , 1281.5 , 69.8 , 78.7 , 76.0 , 47 , 1487 , 42 , 1373 , 7735 , 71 , 59 , 0 , 0 , 302.7 , 219.7 , 72.4 , 82.9 , 12.0 , 10.5 , 12.0 , 292.2 , 12.0 , 67.3 , 4 , 1.0500 , 95.0 , 13813 ,
2024-09-02 23:31:07 , 2700.0 , 1281.5 , 69.3 , 78.0 , 78.0 , 47 , 1487 , 42 , 1372 , 7735 , 69 , 59 , 0 , 0 , 251.6 , 181.5 , 59.9 , 70.2 , 12.0 , 10.2 , 12.0 , 241.4 , 12.0 , 55.9 , 4 , 1.0500 , 92.0 , 13805 ,
2024-09-02 23:31:08 , 2700.0 , 1281.5 , 69.4 , 79.0 , 78.0 , 47 , 1489 , 42 , 1373 , 7741 , 66 , 57 , 0 , 0 , 306.2 , 223.8 , 72.1 , 82.5 , 12.0 , 10.4 , 12.0 , 295.9 , 12.0 , 68.0 , 4 , 1.0500 , 94.0 , 13800 ,
2024-09-02 23:31:09 , 2700.0 , 1281.5 , 69.1 , 77.9 , 78.0 , 47 , 1491 , 42 , 1374 , 7735 , 68 , 57 , 0 , 0 , 306.3 , 223.1 , 72.8 , 83.2 , 12.0 , 10.5 , 12.0 , 295.9 , 12.0 , 68.1 , 4 , 1.0500 , 95.0 , 13802 ,
2024-09-02 23:31:10 , 2700.0 , 1281.5 , 69.7 , 78.8 , 78.0 , 47 , 1488 , 42 , 1373 , 7735 , 67 , 53 , 0 , 0 , 304.7 , 222.4 , 71.9 , 82.3 , 12.0 , 10.3 , 12.0 , 294.4 , 12.0 , 67.7 , 4 , 1.0500 , 95.0 , 13797 ,
2024-09-02 23:31:11 , 2700.0 , 1281.5 , 66.4 , 73.8 , 76.0 , 47 , 1488 , 42 , 1373 , 7735 , 68 , 58 , 0 , 0 , 133.2 , 88.9 , 34.4 , 44.3 , 12.1 , 9.9 , 12.0 , 123.3 , 12.1 , 29.6 , 4 , 1.0500 , 95.0 , 13794 ,
2024-09-02 23:31:12 , 2700.0 , 1281.5 , 70.4 , 79.6 , 78.0 , 47 , 1489 , 42 , 1373 , 7735 , 66 , 56 , 0 , 0 , 315.1 , 230.0 , 74.7 , 85.2 , 12.0 , 10.5 , 12.0 , 304.7 , 12.0 , 70.0 , 4 , 1.0500 , 94.0 , 13794 ,
2024-09-02 23:31:13 , 2700.0 , 1281.5 , 69.9 , 78.7 , 78.0 , 47 , 1489 , 42 , 1373 , 7735 , 68 , 59 , 0 , 0 , 297.4 , 216.6 , 70.4 , 80.8 , 12.0 , 10.4 , 12.0 , 287.0 , 12.0 , 66.1 , 4 , 1.0500 , 95.0 , 13794 ,
2024-09-02 23:31:14 , 2700.0 , 1281.5 , 64.9 , 71.8 , 76.0 , 47 , 1489 , 42 , 1373 , 7735 , 33 , 18 , 0 , 0 , 145.8 , 98.5 , 37.3 , 47.3 , 12.0 , 9.9 , 12.0 , 135.8 , 12.0 , 32.4 , 4 , 1.0500 , 95.0 , 13792 ,
2024-09-02 23:31:15 , 2700.0 , 1281.5 , 68.2 , 76.5 , 78.0 , 47 , 1491 , 42 , 1374 , 7735 , 72 , 61 , 0 , 0 , 264.2 , 191.2 , 62.7 , 73.0 , 12.0 , 10.3 , 12.0 , 253.9 , 12.0 , 58.7 , 4 , 1.0500 , 95.0 , 13791 ,
2024-09-02 23:31:16 , 2700.0 , 1281.5 , 69.5 , 78.5 , 78.0 , 47 , 1489 , 42 , 1373 , 7735 , 69 , 59 , 0 , 0 , 309.0 , 225.8 , 72.7 , 83.1 , 12.0 , 10.5 , 12.0 , 298.5 , 12.0 , 68.7 , 4 , 1.0500 , 94.0 , 13786 ,
2024-09-02 23:31:17 , 2700.0 , 1281.5 , 62.7 , 70.9 , 76.0 , 47 , 1488 , 42 , 1374 , 7735 , 31 , 19 , 0 , 0 , 100.4 , 64.2 , 26.3 , 36.2 , 12.1 , 9.9 , 12.0 , 90.5 , 12.0 , 22.3 , 4 , 1.0500 , 95.0 , 13820 ,
2024-09-02 23:31:18 , 2715.0 , 1281.5 , 67.9 , 76.4 , 74.0 , 47 , 1488 , 42 , 1373 , 7731 , 1 , 0 , 0 , 0 , 155.7 , 105.9 , 39.7 , 49.8 , 12.0 , 10.1 , 12.0 , 145.6 , 12.0 , 34.6 , 4 , 1.0500 , 95.0 , 13689 ,
2024-09-02 23:31:19 , 2700.0 , 1281.5 , 67.1 , 75.2 , 76.0 , 47 , 1488 , 42 , 1373 , 7731 , 70 , 60 , 0 , 0 , 267.5 , 193.6 , 63.6 , 73.9 , 12.0 , 10.4 , 12.0 , 257.1 , 12.0 , 59.5 , 4 , 1.0500 , 95.0 , 13687 ,
2024-09-02 23:31:20 , 2700.0 , 1281.5 , 70.2 , 79.3 , 76.0 , 47 , 1489 , 42 , 1373 , 7731 , 72 , 57 , 0 , 0 , 298.7 , 217.9 , 70.4 , 80.8 , 12.0 , 10.5 , 12.0 , 288.2 , 12.0 , 66.4 , 4 , 1.0500 , 95.0 , 13685 ,
2024-09-02 23:31:21 , 2700.0 , 1281.5 , 69.5 , 78.8 , 76.0 , 47 , 1489 , 42 , 1375 , 7731 , 34 , 19 , 0 , 0 , 253.7 , 184.9 , 58.5 , 68.8 , 12.0 , 10.3 , 12.0 , 243.4 , 12.0 , 56.4 , 4 , 1.0500 , 94.0 , 13753 ,
2024-09-02 23:31:22 , 2700.0 , 1281.5 , 69.5 , 78.5 , 76.0 , 47 , 1490 , 42 , 1375 , 7731 , 72 , 59 , 0 , 0 , 250.7 , 182.5 , 58.0 , 68.2 , 12.0 , 10.2 , 12.0 , 240.4 , 12.0 , 55.7 , 4 , 1.0500 , 95.0 , 13749 ,
2024-09-02 23:31:23 , 2700.0 , 1281.5 , 69.5 , 79.3 , 78.0 , 47 , 1490 , 42 , 1374 , 7737 , 62 , 53 , 0 , 0 , 305.6 , 224.4 , 70.8 , 81.2 , 12.0 , 10.4 , 12.0 , 295.2 , 12.0 , 67.9 , 4 , 1.0500 , 95.0 , 13797 ,
2024-09-02 23:31:24 , 2700.0 , 1281.5 , 69.7 , 78.7 , 76.0 , 47 , 1490 , 42 , 1374 , 7731 , 24 , 13 , 0 , 1 , 323.9 , 238.1 , 75.2 , 85.8 , 12.0 , 10.6 , 12.0 , 313.3 , 12.0 , 72.0 , 4 , 1.0500 , 95.0 , 13838 ,
2024-09-02 23:31:25 , 2700.0 , 1281.5 , 68.1 , 76.4 , 76.0 , 47 , 1488 , 42 , 1374 , 7731 , 73 , 62 , 0 , 0 , 278.7 , 203.6 , 64.7 , 75.1 , 12.0 , 10.3 , 12.0 , 268.3 , 12.0 , 61.9 , 4 , 1.0500 , 95.0 , 13830 ,
2024-09-02 23:31:26 , 2700.0 , 1281.5 , 71.8 , 81.9 , 78.0 , 47 , 1489 , 42 , 1374 , 7731 , 70 , 59 , 0 , 0 , 318.3 , 233.7 , 74.1 , 84.6 , 12.0 , 10.5 , 12.0 , 307.8 , 12.0 , 70.7 , 4 , 1.0500 , 95.0 , 13826 ,
2024-09-02 23:31:27 , 2700.0 , 1281.5 , 70.5 , 80.2 , 76.0 , 47 , 1487 , 42 , 1374 , 7752 , 66 , 46 , 0 , 2 , 310.6 , 227.6 , 72.5 , 83.0 , 12.0 , 10.5 , 12.0 , 300.1 , 12.0 , 69.0 , 4 , 1.0500 , 94.0 , 13818 ,
2024-09-02 23:31:28 , 2700.0 , 1281.5 , 69.4 , 78.5 , 78.0 , 47 , 1489 , 42 , 1374 , 7752 , 71 , 61 , 0 , 0 , 309.7 , 227.9 , 71.5 , 81.8 , 12.0 , 10.4 , 12.0 , 299.3 , 12.0 , 68.8 , 4 , 1.0500 , 95.0 , 13829 ,
2024-09-02 23:31:29 , 2700.0 , 1281.5 , 69.6 , 78.8 , 78.0 , 47 , 1487 , 42 , 1373 , 7753 , 71 , 61 , 0 , 0 , 282.7 , 207.3 , 65.0 , 75.4 , 12.0 , 10.3 , 12.0 , 272.3 , 12.0 , 62.8 , 4 , 1.0500 , 95.0 , 13822 ,
2024-09-02 23:31:30 , 2700.0 , 1281.5 , 68.5 , 77.0 , 78.0 , 47 , 1489 , 42 , 1373 , 7747 , 74 , 63 , 0 , 0 , 285.9 , 209.3 , 66.3 , 76.6 , 12.0 , 10.3 , 12.0 , 275.6 , 12.0 , 63.5 , 4 , 1.0500 , 95.0 , 13815 ,
2024-09-02 23:31:31 , 2700.0 , 1281.5 , 68.2 , 77.0 , 78.0 , 48 , 1490 , 42 , 1374 , 7747 , 72 , 62 , 0 , 0 , 258.9 , 188.0 , 60.7 , 70.9 , 12.0 , 10.1 , 12.0 , 248.8 , 12.0 , 57.5 , 4 , 1.0500 , 95.0 , 13815 ,
2024-09-02 23:31:32 , 2700.0 , 1281.5 , 71.2 , 82.3 , 78.0 , 48 , 1490 , 42 , 1375 , 7747 , 66 , 56 , 0 , 0 , 288.1 , 210.3 , 67.5 , 77.7 , 12.0 , 10.3 , 12.0 , 277.8 , 12.0 , 64.0 , 4 , 1.0500 , 95.0 , 13814 ,
2024-09-02 23:31:33 , 2700.0 , 1281.5 , 68.9 , 77.7 , 76.0 , 48 , 1490 , 42 , 1374 , 7746 , 58 , 50 , 0 , 0 , 286.1 , 209.5 , 66.3 , 76.6 , 12.0 , 10.3 , 12.0 , 275.8 , 12.0 , 63.6 , 4 , 1.0500 , 95.0 , 13811 ,
2024-09-02 23:31:34 , 2700.0 , 1281.5 , 71.1 , 80.5 , 78.0 , 48 , 1490 , 42 , 1374 , 7742 , 70 , 59 , 0 , 0 , 291.0 , 212.5 , 68.1 , 78.5 , 12.0 , 10.4 , 12.0 , 280.6 , 12.0 , 64.7 , 4 , 1.0500 , 95.0 , 13807 ,
2024-09-02 23:31:35 , 2700.0 , 1281.5 , 65.5 , 72.7 , 76.0 , 48 , 1490 , 42 , 1377 , 7738 , 50 , 37 , 0 , 0 , 148.9 , 101.5 , 37.5 , 47.4 , 12.0 , 9.8 , 12.0 , 139.0 , 12.0 , 33.1 , 4 , 1.0500 , 95.0 , 13804 ,
2024-09-02 23:31:36 , 2700.0 , 1281.5 , 68.9 , 78.1 , 76.0 , 48 , 1489 , 42 , 1377 , 7738 , 72 , 61 , 0 , 0 , 251.7 , 182.5 , 59.0 , 69.2 , 12.0 , 10.2 , 12.0 , 241.4 , 12.0 , 55.9 , 4 , 1.0500 , 95.0 , 13800 ,
2024-09-02 23:31:37 , 2700.0 , 1281.5 , 69.8 , 79.3 , 78.0 , 48 , 1491 , 42 , 1375 , 7738 , 73 , 62 , 0 , 0 , 272.2 , 198.9 , 63.0 , 73.3 , 12.0 , 10.3 , 12.0 , 262.0 , 12.0 , 60.5 , 4 , 1.0500 , 95.0 , 13801 ,
2024-09-02 23:31:38 , 2700.0 , 1281.5 , 65.4 , 72.5 , 76.0 , 48 , 1491 , 42 , 1374 , 7738 , 56 , 45 , 0 , 0 , 148.0 , 100.9 , 37.2 , 47.1 , 12.0 , 9.9 , 12.0 , 138.1 , 12.0 , 32.9 , 4 , 1.0500 , 95.0 , 13800 ,
2024-09-02 23:31:39 , 2700.0 , 1281.5 , 69.7 , 78.8 , 78.0 , 48 , 1492 , 42 , 1374 , 7742 , 71 , 60 , 0 , 0 , 273.2 , 199.8 , 63.3 , 73.5 , 12.0 , 10.2 , 12.0 , 263.1 , 12.0 , 60.7 , 4 , 1.0500 , 95.0 , 13795 ,
2024-09-02 23:31:40 , 2700.0 , 1281.5 , 70.2 , 79.7 , 78.0 , 48 , 1491 , 42 , 1376 , 7746 , 67 , 58 , 0 , 0 , 328.0 , 241.4 , 76.2 , 86.6 , 12.0 , 10.4 , 12.0 , 317.6 , 12.0 , 72.9 , 4 , 1.0500 , 95.0 , 13794 ,
2024-09-02 23:31:41 , 2700.0 , 1281.5 , 64.2 , 71.6 , 76.0 , 48 , 1492 , 42 , 1377 , 7738 , 25 , 14 , 0 , 0 , 144.7 , 97.7 , 37.2 , 46.9 , 12.0 , 9.8 , 12.0 , 134.9 , 12.0 , 32.1 , 4 , 1.0500 , 95.0 , 13793 ,
2024-09-02 23:31:42 , 2715.0 , 1281.5 , 61.5 , 70.1 , 72.0 , 48 , 1492 , 42 , 1376 , 7738 , 0 , 0 , 0 , 0 , 89.3 , 55.3 , 24.4 , 34.1 , 12.1 , 9.7 , 12.0 , 79.7 , 12.1 , 19.9 , 4 , 1.0500 , 94.0 , 13690 ,
2024-09-02 23:31:43 , 2700.0 , 1281.5 , 70.5 , 80.1 , 76.0 , 48 , 1490 , 42 , 1375 , 7746 , 67 , 57 , 0 , 0 , 281.5 , 206.8 , 64.6 , 74.8 , 12.0 , 10.2 , 12.0 , 271.4 , 12.0 , 62.6 , 4 , 1.0500 , 95.0 , 13691 ,
2024-09-02 23:31:44 , 2700.0 , 1281.5 , 70.6 , 80.1 , 78.0 , 48 , 1490 , 42 , 1376 , 7737 , 69 , 56 , 0 , 0 , 309.5 , 227.3 , 71.9 , 82.3 , 12.0 , 10.4 , 12.0 , 299.2 , 12.0 , 68.8 , 4 , 1.0500 , 94.0 , 13688 ,
2024-09-02 23:31:45 , 2700.0 , 1281.5 , 64.1 , 71.0 , 74.0 , 48 , 1489 , 42 , 1375 , 7737 , 27 , 13 , 0 , 0 , 144.4 , 97.4 , 37.2 , 47.0 , 12.0 , 9.8 , 12.0 , 134.6 , 12.0 , 32.1 , 4 , 1.0500 , 95.0 , 13756 ,
2024-09-02 23:31:46 , 2700.0 , 1281.5 , 69.9 , 79.6 , 78.0 , 48 , 1489 , 42 , 1374 , 7743 , 71 , 61 , 0 , 0 , 282.6 , 207.1 , 65.1 , 75.5 , 12.0 , 10.4 , 12.0 , 272.3 , 12.0 , 62.8 , 4 , 1.0500 , 93.0 , 13754 ,
2024-09-02 23:31:47 , 2700.0 , 1281.5 , 69.8 , 79.3 , 78.0 , 48 , 1489 , 42 , 1374 , 7743 , 75 , 60 , 0 , 0 , 288.0 , 211.1 , 66.6 , 76.9 , 12.0 , 10.3 , 12.0 , 277.7 , 12.0 , 64.0 , 4 , 1.0500 , 95.0 , 13754 ,
2024-09-02 23:31:48 , 2700.0 , 1281.5 , 69.0 , 78.3 , 76.0 , 48 , 1491 , 42 , 1374 , 7737 , 32 , 20 , 0 , 0 , 145.7 , 99.5 , 36.3 , 46.2 , 12.0 , 10.0 , 12.0 , 135.7 , 12.0 , 32.4 , 4 , 1.0500 , 95.0 , 13778 ,
2024-09-02 23:31:49 , 2700.0 , 1281.5 , 71.2 , 80.9 , 76.0 , 48 , 1491 , 42 , 1373 , 7737 , 67 , 57 , 0 , 0 , 301.7 , 220.6 , 70.6 , 81.1 , 12.0 , 10.5 , 12.0 , 291.2 , 12.0 , 67.0 , 4 , 1.0500 , 95.0 , 13786 ,
2024-09-02 23:31:50 , 2700.0 , 1281.5 , 69.7 , 79.1 , 78.0 , 48 , 1489 , 42 , 1375 , 7744 , 70 , 60 , 0 , 0 , 259.3 , 188.2 , 60.9 , 71.1 , 12.0 , 10.2 , 12.0 , 249.1 , 12.0 , 57.6 , 4 , 1.0500 , 95.0 , 13780 ,
2024-09-02 23:31:51 , 2700.0 , 1281.5 , 68.9 , 77.4 , 76.0 , 48 , 1488 , 42 , 1376 , 7738 , 23 , 12 , 0 , 0 , 268.2 , 195.7 , 62.3 , 72.5 , 12.0 , 10.2 , 12.0 , 258.0 , 12.0 , 59.6 , 4 , 1.0500 , 95.0 , 13780 ,
2024-09-02 23:31:52 , 2700.0 , 1281.5 , 68.7 , 77.5 , 78.0 , 48 , 1488 , 42 , 1376 , 7761 , 70 , 54 , 0 , 0 , 281.1 , 205.4 , 65.4 , 75.7 , 12.0 , 10.3 , 12.0 , 270.8 , 12.0 , 62.5 , 4 , 1.0500 , 95.0 , 13764 ,
2024-09-02 23:31:53 , 2700.0 , 1281.5 , 71.3 , 81.2 , 78.0 , 48 , 1489 , 42 , 1375 , 7751 , 68 , 58 , 0 , 0 , 316.0 , 232.2 , 73.4 , 83.8 , 12.0 , 10.4 , 12.0 , 305.6 , 12.0 , 70.2 , 4 , 1.0500 , 95.0 , 13764 ,
2024-09-02 23:31:54 , 2700.0 , 1281.5 , 66.2 , 73.6 , 78.0 , 48 , 1491 , 42 , 1375 , 7751 , 35 , 22 , 0 , 1 , 316.4 , 233.8 , 72.1 , 82.6 , 12.0 , 10.4 , 12.0 , 306.0 , 12.0 , 70.3 , 4 , 1.0500 , 95.0 , 13771 ,
2024-09-02 23:31:55 , 2700.0 , 1281.5 , 68.3 , 76.7 , 76.0 , 48 , 1492 , 42 , 1374 , 7747 , 73 , 62 , 0 , 0 , 271.4 , 198.0 , 63.2 , 73.4 , 12.0 , 10.2 , 12.0 , 261.2 , 12.0 , 60.3 , 4 , 1.0500 , 94.0 , 13769 ,
2024-09-02 23:31:56 , 2700.0 , 1281.5 , 68.6 , 78.2 , 78.0 , 48 , 1492 , 42 , 1373 , 7747 , 68 , 57 , 0 , 0 , 263.9 , 191.8 , 61.9 , 72.1 , 12.0 , 10.2 , 12.0 , 253.7 , 12.0 , 58.6 , 4 , 1.0500 , 95.0 , 13771 ,
2024-09-02 23:31:57 , 2700.0 , 1281.5 , 70.9 , 80.6 , 78.0 , 48 , 1490 , 42 , 1376 , 7751 , 65 , 46 , 0 , 2 , 301.1 , 220.6 , 70.1 , 80.5 , 12.0 , 10.4 , 12.0 , 290.7 , 12.0 , 66.9 , 4 , 1.0500 , 94.0 , 13773 ,
2024-09-02 23:31:58 , 2700.0 , 1281.5 , 69.5 , 78.4 , 78.0 , 48 , 1490 , 42 , 1376 , 7751 , 69 , 58 , 0 , 0 , 274.2 , 200.6 , 63.4 , 73.6 , 12.0 , 10.2 , 12.0 , 264.0 , 12.0 , 60.9 , 4 , 1.0500 , 95.0 , 13773 ,
2024-09-02 23:31:59 , 2700.0 , 1281.5 , 69.5 , 78.1 , 78.0 , 48 , 1489 , 42 , 1375 , 7759 , 65 , 51 , 0 , 0 , 295.1 , 216.6 , 68.3 , 78.5 , 12.0 , 10.2 , 12.0 , 284.9 , 12.0 , 65.6 , 4 , 1.0500 , 95.0 , 13769 ,
2024-09-02 23:32:00 , 2700.0 , 1281.5 , 68.7 , 77.3 , 76.0 , 48 , 1491 , 42 , 1376 , 7746 , 78 , 66 , 0 , 0 , 308.0 , 226.0 , 71.6 , 82.0 , 12.0 , 10.4 , 12.0 , 297.6 , 12.0 , 68.4 , 4 , 1.0500 , 95.0 , 13776 ,
2024-09-02 23:32:01 , 2700.0 , 1281.5 , 70.0 , 79.3 , 78.0 , 48 , 1490 , 42 , 1374 , 7738 , 67 , 56 , 0 , 0 , 296.5 , 216.5 , 69.7 , 80.0 , 12.0 , 10.4 , 12.0 , 286.1 , 12.0 , 65.9 , 4 , 1.0500 , 95.0 , 13775 ,
2024-09-02 23:32:02 , 2700.0 , 1281.5 , 68.9 , 77.7 , 78.0 , 48 , 1491 , 42 , 1374 , 7734 , 68 , 53 , 0 , 0 , 318.6 , 234.1 , 74.1 , 84.5 , 12.0 , 10.4 , 12.0 , 308.1 , 12.0 , 70.8 , 4 , 1.0500 , 95.0 , 13804 ,
2024-09-02 23:32:03 , 2700.0 , 1281.5 , 71.0 , 80.8 , 76.0 , 48 , 1491 , 42 , 1376 , 7734 , 61 , 52 , 0 , 0 , 290.8 , 211.8 , 68.6 , 79.0 , 12.0 , 10.4 , 12.0 , 280.4 , 12.0 , 64.6 , 4 , 1.0500 , 95.0 , 13800 ,
2024-09-02 23:32:04 , 2700.0 , 1281.5 , 69.2 , 78.7 , 78.0 , 48 , 1492 , 42 , 1377 , 7734 , 72 , 63 , 0 , 0 , 317.2 , 233.8 , 73.1 , 83.4 , 12.0 , 10.3 , 12.0 , 306.9 , 12.0 , 70.5 , 4 , 1.0500 , 94.0 , 13795 ,
2024-09-02 23:32:05 , 2700.0 , 1281.5 , 70.1 , 79.3 , 78.0 , 48 , 1491 , 42 , 1376 , 7740 , 68 , 53 , 0 , 0 , 319.3 , 235.0 , 74.0 , 84.4 , 12.0 , 10.3 , 12.0 , 309.0 , 12.0 , 71.0 , 4 , 1.0500 , 95.0 , 13792 ,
2024-09-02 23:32:06 , 2715.0 , 1281.5 , 61.8 , 70.4 , 72.0 , 48 , 1490 , 42 , 1376 , 7734 , 1 , 0 , 0 , 0 , 89.8 , 55.9 , 24.1 , 33.8 , 12.1 , 9.7 , 12.0 , 80.1 , 12.0 , 20.0 , 4 , 1.0500 , 94.0 , 13774 ,
2024-09-02 23:32:07 , 2700.0 , 1281.5 , 67.5 , 75.6 , 76.0 , 48 , 1487 , 42 , 1376 , 7734 , 68 , 56 , 0 , 0 , 316.4 , 233.6 , 72.5 , 82.8 , 12.0 , 10.3 , 12.0 , 306.1 , 12.0 , 70.3 , 4 , 1.0500 , 95.0 , 13710 ,
2024-09-02 23:32:08 , 2700.0 , 1281.5 , 69.4 , 78.7 , 76.0 , 48 , 1489 , 42 , 1374 , 7742 , 68 , 58 , 0 , 0 , 319.5 , 234.6 , 74.5 , 84.9 , 12.0 , 10.4 , 12.0 , 309.1 , 12.0 , 71.0 , 4 , 1.0500 , 95.0 , 13707 ,
2024-09-02 23:32:09 , 2700.0 , 1281.5 , 66.1 , 73.3 , 78.0 , 48 , 1489 , 42 , 1375 , 7759 , 71 , 61 , 0 , 0 , 228.5 , 163.2 , 55.2 , 65.2 , 12.0 , 10.1 , 12.0 , 218.4 , 12.0 , 50.8 , 4 , 1.0500 , 95.0 , 13707 ,
2024-09-02 23:32:10 , 2700.0 , 1281.5 , 67.5 , 75.6 , 76.0 , 48 , 1491 , 42 , 1375 , 7762 , 68 , 58 , 0 , 0 , 252.3 , 182.7 , 59.4 , 69.6 , 12.0 , 10.2 , 12.0 , 242.1 , 12.0 , 56.1 , 4 , 1.0500 , 95.0 , 13773 ,
2024-09-02 23:32:11 , 2700.0 , 1281.5 , 68.9 , 77.6 , 78.0 , 48 , 1491 , 42 , 1374 , 7761 , 73 , 62 , 0 , 0 , 297.3 , 217.8 , 69.1 , 79.5 , 12.0 , 10.4 , 12.0 , 286.9 , 12.0 , 66.1 , 4 , 1.0500 , 95.0 , 13890 ,
2024-09-02 23:32:12 , 2700.0 , 1281.5 , 65.8 , 73.0 , 78.0 , 48 , 1493 , 42 , 1375 , 7756 , 69 , 60 , 0 , 0 , 254.7 , 185.2 , 59.4 , 69.5 , 12.0 , 10.2 , 12.0 , 244.6 , 12.0 , 56.6 , 4 , 1.0500 , 95.0 , 13885 ,
2024-09-02 23:32:13 , 2700.0 , 1281.5 , 70.7 , 79.6 , 76.0 , 48 , 1493 , 42 , 1375 , 7756 , 72 , 61 , 0 , 0 , 291.7 , 212.9 , 68.4 , 78.8 , 12.0 , 10.4 , 12.0 , 281.3 , 12.0 , 64.8 , 4 , 1.0500 , 95.0 , 13927 ,
2024-09-02 23:32:14 , 2700.0 , 1281.5 , 69.9 , 79.2 , 78.0 , 48 , 1493 , 42 , 1375 , 7762 , 72 , 62 , 0 , 0 , 308.4 , 226.7 , 71.3 , 81.6 , 12.0 , 10.4 , 12.0 , 298.0 , 12.0 , 68.5 , 4 , 1.0500 , 95.0 , 13919 ,
2024-09-02 23:32:15 , 2700.0 , 1281.5 , 65.3 , 72.7 , 76.0 , 48 , 1491 , 42 , 1376 , 7756 , 55 , 45 , 0 , 0 , 160.7 , 109.1 , 41.7 , 51.6 , 12.0 , 9.9 , 12.0 , 150.8 , 12.0 , 35.7 , 4 , 1.0500 , 95.0 , 13912 ,
2024-09-02 23:32:16 , 2700.0 , 1281.5 , 71.9 , 82.3 , 78.0 , 48 , 1491 , 42 , 1376 , 7756 , 67 , 57 , 0 , 0 , 280.1 , 203.6 , 66.1 , 76.5 , 12.0 , 10.4 , 12.0 , 269.7 , 12.0 , 62.2 , 4 , 1.0500 ,

It is TOO BAD that volunteers get NO CREDIT for their 1000's of seconds of machine time when the failure is caused by the researcher submitting the broken work.
The same bug has been reported here for some time.

Some times ago, i proposed a "minimal credits" for bugged wus, a symbolical "premium" for volunteers that runs wus (and, sometimes, reports problems).
Obviously, no answer.

It is TOO BAD that volunteers get NO CREDIT for their 1000's of seconds of machine time when the failure is caused by the researcher submitting the broken work.
The same bug has been reported here for some time.

Some times ago, i proposed a "minimal credits" for bugged wus, a symbolical "premium" for volunteers that runs wus (and, sometimes, reports problems).
Obviously, no answer.

What is crazy is when they use a DEPRECATED function and then say that it will only work with the latest drivers.
I am glad they released this screwed up binary on Ralph rather than releasing this junk on Rosetta.

What is crazy is when they use a DEPRECATED function and then say that it will only work with the latest drivers.
I am glad they released this screwed up binary on Ralph rather than releasing this junk on Rosetta.

In large projects it may be not so easy to change deprecated function to a new one as this may lead to a bunch of bugs which could be discovered much later. It may require long regression testing

Function is only deprecated because software developers of this library decided to prepare a new function that may accommodate new features that could be unnecessary for Rosetta

---

--
I crunch for Ukraine

What is crazy is when they use a DEPRECATED function and then say that it will only work with the latest drivers.
I am glad they released this screwed up binary on Ralph rather than releasing this junk on Rosetta.

In large projects it may be not so easy to change deprecated function to a new one as this may lead to a bunch of bugs which could be discovered much later. It may require long regression testing

Function is only deprecated because software developers of this library decided to prepare a new function that may accommodate new features that could be unnecessary for Rosetta

The developer dumped another 1,000 RoseTTAFold All-Atom work units, and it appears that the error messages are exactly the same. I wonder if any of the RoseTTAFold All-Atom work units ever passed.

Seems like the developer could code in some diagnostic messages to help configure systems if that is the problem.


Stderr output

<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
Z = torch.cross(Xn,Yn)
22:04:48 (2044): called boinc_finish(0)

</stderr_txt>
]]>

I STILL get this error on all my pc's!!

Stderr output
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<message>
The access code is invalid.
(0xc) - exit code 12 (0xc)</message>
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv2rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
Z = torch.cross(Xn,Yn)
usage: predict.py [-h] [-checkpoint CHECKPOINT] [-pdb PDB] [-silent SILENT] [-tmpl_chain TMPL_CHAIN]
[-tmpl_conf TMPL_CONF] [-movescale MOVESCALE] [-n_pred N_PRED] [-n_cycle N_CYCLE] [-no_extra_l1]
[-no_atom_frames] [-outcsv OUTCSV]
predict.py: error: unrecognized arguments: -z .venv_m_pred_alpha_358.zip

</stderr_txt>
]]>

I STILL get this error on all my pc's!!

Stderr output
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<message>
The access code is invalid.
(0xc) - exit code 12 (0xc)</message>
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv2rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
Z = torch.cross(Xn,Yn)
usage: predict.py [-h] [-checkpoint CHECKPOINT] [-pdb PDB] [-silent SILENT] [-tmpl_chain TMPL_CHAIN]
[-tmpl_conf TMPL_CONF] [-movescale MOVESCALE] [-n_pred N_PRED] [-n_cycle N_CYCLE] [-no_extra_l1]
[-no_atom_frames] [-outcsv OUTCSV]
predict.py: error: unrecognized arguments: -z .venv_m_pred_alpha_358.zip

</stderr_txt>
]]>

That is a different error to what others have been reporting.
Theirs don't error out, they complete normally but fail to Validate.

Yours are erroring out after a few seconds. You need to reset the project and the next time work is available it will re-download all the files and should start processing them then instead of erroring out.

---

Grant
Darwin NT

Still same on my side, 27 tasks finished after 22-27 minutes and reported normally, but end up with validate error.
NVIDIA RTX 4070 Super, newest driver 560.94 installed.


Stderr output
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
Z = torch.cross(Xn,Yn)
15:03:30 (18660): called boinc_finish(0)

</stderr_txt>
]]>

Since we don't have a new application, there's either a problem with the new Tasks (even though they process and complete without error), or the most likely problem- an issue with the Validator itself.
Considering the whole project was down for weeks, a problem with the Validator itself gets my vote.

---

Grant
Darwin NT

Since we don't have a new application, there's either a problem with the new Tasks (even though they process and complete without error), or the most likely problem- an issue with the Validator itself.
Considering the whole project was down for weeks, a problem with the Validator itself gets my vote.

The WU are VALIDATING now.
27 consecutive successes.

The stderr.txt file looks very similar to before.


Stderr output

<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
Z = torch.cross(Xn,Yn)
22:37:04 (25992): called boinc_finish(0)

</stderr_txt>
]]>

Same std_err output, same application, so it was probably an error with the Validators.
At least it's sorted now. (hopefully they can re-run all the ones that failed to validate before through the Validators again).

---

Grant
Darwin NT