[S 04/14 20:00] PVFS2 Server on node da13 version 2.7.1pre1-2008-04-14-194430 starting... [D 20:00:38.073288] Logging all (mask 18446744073709551615) [D 20:00:38.073341] PINT_encode_initialize [D 20:00:38.073351] lebf_initialize [D 20:00:38.073361] check_req_size [D 20:00:38.073368] encode_common [D 20:00:38.073378] lebf_encode_req [D 20:00:38.073388] lebf_encode_rel [D 20:00:38.073398] check_resp_size [D 20:00:38.073407] encode_common [D 20:00:38.073416] lebf_encode_resp [D 20:00:38.073425] lebf_encode_rel [D 20:00:38.073435] check_req_size [D 20:00:38.073442] encode_common [D 20:00:38.073451] lebf_encode_req [D 20:00:38.073457] lebf_encode_rel [D 20:00:38.073467] check_resp_size [D 20:00:38.073474] encode_common [D 20:00:38.073482] lebf_encode_resp [D 20:00:38.073489] lebf_encode_rel [D 20:00:38.073498] check_req_size [D 20:00:38.073505] encode_common [D 20:00:38.073514] lebf_encode_req [D 20:00:38.073523] PINT_do_request_commit: commit node 0x7ffff4bcdfe0 [D 20:00:38.073532] node stored at 0 [D 20:00:38.073541] clearing tree [D 20:00:38.073548] lebf_encode_rel [D 20:00:38.073557] check_resp_size [D 20:00:38.073565] encode_common [D 20:00:38.073573] lebf_encode_resp [D 20:00:38.073580] lebf_encode_rel [D 20:00:38.073589] check_req_size [D 20:00:38.073596] encode_common [D 20:00:38.073605] lebf_encode_req [D 20:00:38.073612] lebf_encode_rel [D 20:00:38.073621] check_resp_size [D 20:00:38.073628] encode_common [D 20:00:38.073637] lebf_encode_resp [D 20:00:38.073644] lebf_encode_rel [D 20:00:38.073653] check_req_size [D 20:00:38.073660] encode_common [D 20:00:38.073669] lebf_encode_req [D 20:00:38.073678] lebf_encode_rel [D 20:00:38.073688] check_resp_size [D 20:00:38.073695] encode_common [D 20:00:38.073703] lebf_encode_resp [D 20:00:38.073710] lebf_encode_rel [D 20:00:38.073719] check_req_size [D 20:00:38.073726] encode_common [D 20:00:38.073735] lebf_encode_req [D 20:00:38.073742] lebf_encode_rel [D 20:00:38.073751] check_resp_size [D 20:00:38.073758] encode_common [D 20:00:38.073766] lebf_encode_resp [D 20:00:38.073773] lebf_encode_rel [D 20:00:38.073782] check_req_size [D 20:00:38.073789] encode_common [D 20:00:38.073798] lebf_encode_req [D 20:00:38.073805] lebf_encode_rel [D 20:00:38.073814] check_resp_size [D 20:00:38.073821] encode_common [D 20:00:38.073830] lebf_encode_resp [D 20:00:38.073836] lebf_encode_rel [D 20:00:38.073845] check_req_size [D 20:00:38.073852] encode_common [D 20:00:38.073861] lebf_encode_req [D 20:00:38.073868] lebf_encode_rel [D 20:00:38.073877] check_resp_size [D 20:00:38.073884] encode_common [D 20:00:38.073893] lebf_encode_resp [D 20:00:38.073899] lebf_encode_rel [D 20:00:38.073908] check_req_size [D 20:00:38.073915] encode_common [D 20:00:38.073924] lebf_encode_req [D 20:00:38.073931] lebf_encode_rel [D 20:00:38.073940] check_resp_size [D 20:00:38.073947] encode_common [D 20:00:38.073955] lebf_encode_resp [D 20:00:38.073962] lebf_encode_rel [D 20:00:38.073971] check_req_size [D 20:00:38.073978] encode_common [D 20:00:38.073987] lebf_encode_req [D 20:00:38.073993] lebf_encode_rel [D 20:00:38.074002] check_resp_size [D 20:00:38.074009] encode_common [D 20:00:38.074018] lebf_encode_resp [D 20:00:38.074024] lebf_encode_rel [D 20:00:38.074033] check_req_size [D 20:00:38.074040] encode_common [D 20:00:38.074049] lebf_encode_req [D 20:00:38.074056] lebf_encode_rel [D 20:00:38.074065] check_resp_size [D 20:00:38.074072] encode_common [D 20:00:38.074081] lebf_encode_resp [D 20:00:38.074088] lebf_encode_rel [D 20:00:38.074096] check_req_size [D 20:00:38.074103] encode_common [D 20:00:38.074112] lebf_encode_req [D 20:00:38.074119] lebf_encode_rel [D 20:00:38.074128] check_resp_size [D 20:00:38.074135] encode_common [D 20:00:38.074143] lebf_encode_resp [D 20:00:38.074150] lebf_encode_rel [D 20:00:38.074159] check_req_size [D 20:00:38.074166] encode_common [D 20:00:38.074175] lebf_encode_req [D 20:00:38.074181] lebf_encode_rel [D 20:00:38.074190] check_resp_size [D 20:00:38.074197] encode_common [D 20:00:38.074205] lebf_encode_resp [D 20:00:38.074212] lebf_encode_rel [D 20:00:38.074221] check_resp_size [D 20:00:38.074229] encode_common [D 20:00:38.074238] lebf_encode_resp [D 20:00:38.074247] lebf_encode_rel [D 20:00:38.074256] check_req_size [D 20:00:38.074263] encode_common [D 20:00:38.074272] lebf_encode_req [D 20:00:38.074279] lebf_encode_rel [D 20:00:38.074288] check_resp_size [D 20:00:38.074295] encode_common [D 20:00:38.074303] lebf_encode_resp [D 20:00:38.074310] lebf_encode_rel [D 20:00:38.074319] check_req_size [D 20:00:38.074326] encode_common [D 20:00:38.074334] lebf_encode_req [D 20:00:38.074341] lebf_encode_rel [D 20:00:38.074350] check_resp_size [D 20:00:38.074357] encode_common [D 20:00:38.074366] lebf_encode_resp [D 20:00:38.074372] lebf_encode_rel [D 20:00:38.074381] check_req_size [D 20:00:38.074388] encode_common [D 20:00:38.074397] lebf_encode_req [D 20:00:38.074403] lebf_encode_rel [D 20:00:38.074412] check_resp_size [D 20:00:38.074419] encode_common [D 20:00:38.074428] lebf_encode_resp [D 20:00:38.074434] lebf_encode_rel [D 20:00:38.074443] check_req_size [D 20:00:38.074450] encode_common [D 20:00:38.074459] lebf_encode_req [D 20:00:38.074465] lebf_encode_rel [D 20:00:38.074474] check_resp_size [D 20:00:38.074481] encode_common [D 20:00:38.074490] lebf_encode_resp [D 20:00:38.074497] lebf_encode_rel [D 20:00:38.074506] check_req_size [D 20:00:38.074513] encode_common [D 20:00:38.074521] lebf_encode_req [D 20:00:38.074528] lebf_encode_rel [D 20:00:38.074537] check_resp_size [D 20:00:38.074544] encode_common [D 20:00:38.074552] lebf_encode_resp [D 20:00:38.074559] lebf_encode_rel [D 20:00:38.074568] check_req_size [D 20:00:38.074575] encode_common [D 20:00:38.074584] lebf_encode_req [D 20:00:38.074590] lebf_encode_rel [D 20:00:38.074599] check_resp_size [D 20:00:38.074606] encode_common [D 20:00:38.074615] lebf_encode_resp [D 20:00:38.074622] lebf_encode_rel [D 20:00:38.074631] check_req_size [D 20:00:38.074638] encode_common [D 20:00:38.074646] lebf_encode_req [D 20:00:38.074653] lebf_encode_rel [D 20:00:38.074662] check_resp_size [D 20:00:38.074669] encode_common [D 20:00:38.074677] lebf_encode_resp [D 20:00:38.074684] lebf_encode_rel [D 20:00:38.074693] check_req_size [D 20:00:38.074700] encode_common [D 20:00:38.074709] lebf_encode_req [D 20:00:38.074715] lebf_encode_rel [D 20:00:38.074724] check_resp_size [D 20:00:38.074731] encode_common [D 20:00:38.074740] lebf_encode_resp [D 20:00:38.074747] lebf_encode_rel [D 20:00:38.074756] check_req_size [D 20:00:38.074763] encode_common [D 20:00:38.074771] lebf_encode_req [D 20:00:38.074778] lebf_encode_rel [D 20:00:38.074787] check_resp_size [D 20:00:38.074794] encode_common [D 20:00:38.074802] lebf_encode_resp [D 20:00:38.074809] lebf_encode_rel [D 20:00:38.074818] check_req_size [D 20:00:38.074825] encode_common [D 20:00:38.074834] lebf_encode_req [D 20:00:38.074840] lebf_encode_rel [D 20:00:38.074849] check_resp_size [D 20:00:38.074856] encode_common [D 20:00:38.074865] lebf_encode_resp [D 20:00:38.074872] lebf_encode_rel [D 20:00:38.074880] check_req_size [D 20:00:38.074892] encode_common [D 20:00:38.074901] lebf_encode_req [D 20:00:38.074908] lebf_encode_rel [D 20:00:38.074917] check_resp_size [D 20:00:38.074923] encode_common [D 20:00:38.074932] lebf_encode_resp [D 20:00:38.074939] lebf_encode_rel [D 20:00:38.074948] check_req_size [D 20:00:38.074955] encode_common [D 20:00:38.074963] lebf_encode_req [D 20:00:38.074970] lebf_encode_rel [D 20:00:38.074979] check_resp_size [D 20:00:38.074986] encode_common [D 20:00:38.074994] lebf_encode_resp [D 20:00:38.075001] lebf_encode_rel [D 20:00:38.075010] check_req_size [D 20:00:38.075017] encode_common [D 20:00:38.075025] lebf_encode_req [D 20:00:38.075033] lebf_encode_rel [D 20:00:38.075041] check_resp_size [D 20:00:38.075048] encode_common [D 20:00:38.075057] lebf_encode_resp [D 20:00:38.075064] lebf_encode_rel [D 20:00:38.075073] check_req_size [D 20:00:38.075080] encode_common [D 20:00:38.075088] lebf_encode_req [D 20:00:38.075095] lebf_encode_rel [D 20:00:38.075104] check_resp_size [D 20:00:38.075111] encode_common [D 20:00:38.075120] lebf_encode_resp [D 20:00:38.075127] lebf_encode_rel [D 20:00:38.075136] check_req_size [D 20:00:38.075143] encode_common [D 20:00:38.075152] lebf_encode_req [D 20:00:38.075159] lebf_encode_rel [D 20:00:38.075167] check_resp_size [D 20:00:38.075174] encode_common [D 20:00:38.075183] lebf_encode_resp [D 20:00:38.075190] lebf_encode_rel [D 20:00:38.075199] check_req_size [D 20:00:38.075206] encode_common [D 20:00:38.075214] lebf_encode_req [D 20:00:38.075221] lebf_encode_rel [D 20:00:38.075230] check_resp_size [D 20:00:38.075237] encode_common [D 20:00:38.075246] lebf_encode_resp [D 20:00:38.075252] lebf_encode_rel [D 20:00:38.075262] check_req_size [D 20:00:38.075268] encode_common [D 20:00:38.075277] lebf_encode_req [D 20:00:38.075285] PINT_do_request_commit: commit node 0x7ffff4bcdfe0 [D 20:00:38.075291] node stored at 0 [D 20:00:38.075297] clearing tree [D 20:00:38.075303] lebf_encode_rel [D 20:00:38.075312] check_resp_size [D 20:00:38.075319] encode_common [D 20:00:38.075328] lebf_encode_resp [D 20:00:38.075334] lebf_encode_rel [D 20:00:38.075343] check_req_size [D 20:00:38.075350] encode_common [D 20:00:38.075359] lebf_encode_req [D 20:00:38.075366] lebf_encode_rel [D 20:00:38.075375] check_resp_size [D 20:00:38.075382] encode_common [D 20:00:38.075390] lebf_encode_resp [D 20:00:38.075397] lebf_encode_rel [D 20:00:38.075407] Passing ib://da13:3346 as BMI listen address. [D 20:00:38.075438] BMI_ib_initialize: init. [D 20:00:38.075447] openib_ib_initialize: init. [D 20:00:38.075961] openib_ib_initialize: max 65408 completion queue entries. [D 20:00:38.076299] BMI_ib_initialize: done. [D 20:00:38.076311] Server using shm key hint: 10 [D 20:00:38.076320] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 11 [D 20:00:38.076329] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 12 [D 20:00:38.077180] dbpf_thread_initialize: initialized [D 20:00:38.077232] [SYNC_COALESCE]: dbpf_sync_context_init for context 0 called [D 20:00:38.077266] dbpf_collection_lookup of coll: da13-test [D 20:00:38.077347] dbpf using default db cache size. [D 20:00:38.077358] dbpf using shm key: 646567233 [D 20:00:38.077783] collection lookup: version is 0.1.3 [D 20:00:38.077900] [SYNC_COALESCE]: dbpf_sync_context_init for context 1 called [D 20:00:38.077917] dbpf collection 136 - Setting handle timeout to 360000000 microseconds [D 20:00:38.077983] - set handle re-use timeout to 360 seconds (ret=0) [D 20:00:38.077993] dbpf collection 136 - Setting cache keywords of attribute cache to dh, [D 20:00:38.078003] Setting dbpf_attr_cache keywords to: dh, [D 20:00:38.078012] dbpf collection 136 - Setting cache size of attribute cache to 511 [D 20:00:38.078023] dbpf collection 136 - Setting maximum elements of attribute cache to 1024 [D 20:00:38.078030] dbpf collection 136 - Initialize collection attr. cache [D 20:00:38.078036] There are 1 cacheable keywords registered [D 20:00:38.078061] dbpf_attr_cache_initialize: initialized [D 20:00:38.078068] dbpf collection 136 - Setting collection handle ranges to 4-1431655767,1431655768-2863311531 [D 20:00:38.078084] op_queue add: 0x63d2d0 [D 20:00:38.078115] dbpf_thread_function started [D 20:00:38.078132] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) [D 20:00:38.078187] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1) [D 20:00:38.078199] op_queue add: 0x63d2d0 [D 20:00:38.079941] op_queue add: 0x63d710 [D 20:00:38.079987] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) [D 20:00:38.079998] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_ITERATE_HANDLES) (ret: 1) [D 20:00:38.080006] op_queue add: 0x63d710 [D 20:00:38.095931] dbpf collection 136 - Setting HIGH_WATERMARK to 8 [D 20:00:38.095942] dbpf collection 136 - Setting LOW_WATERMARK to 1 [D 20:00:38.095949] dbpf collection 136 - Enabling sync mode [D 20:00:38.095956] dbpf collection 136 - Disabling immediate completion [D 20:00:38.095962] File system da13-test using handles: 4-1431655767,1431655768-2863311531 [D 20:00:38.095969] Sync on metadata update for da13-test: yes [D 20:00:38.095975] Sync on I/O data update for da13-test: no [D 20:00:38.095981] Export options for da13-test: RootSquash no AllSquash no ReadOnly no AnonUID 0 AnonGID 0 [D 20:00:38.095992] fp_multiqueue_setinfo: data sync mode on coll_id 136 set to 0 [D 20:00:38.096002] [SYNC_COALESCE]: dbpf_sync_context_destroy for context 1 called [D 20:00:38.096010] Storage Init Complete (aio-threaded) [D 20:00:38.096017] 1 filesystem(s) initialized [D 20:00:38.096193] server_post_unexpected_recv [D 20:00:38.096206] [SM Frame PUSH]: (0x6457d0) frame: 0x6458d0 [D 20:00:38.096215] [SM Locating]: (0x6457d0) op-id: 999 [D 20:00:38.096222] server_op_state_get_machine 999 [D 20:00:38.096229] [SM Locating]: (0x6457d0) located: unexpected_sm:post_unexpected [D 20:00:38.096237] [SM frame get]: (0x6457d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096246] [SM Entering]: (0x6457d0) unexpected_sm:post_unexpected (status: -1229879416) [D 20:00:38.096256] [SM frame get]: (0x6457d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096267] [SM Exiting]: (0x6457d0) unexpected_sm:post_unexpected (error code: -1229879416), (action: DEFERRED) [D 20:00:38.096275] server_post_unexpected_recv [D 20:00:38.096285] [SM Frame PUSH]: (0x646a90) frame: 0x646b90 [D 20:00:38.096292] [SM Locating]: (0x646a90) op-id: 999 [D 20:00:38.096298] server_op_state_get_machine 999 [D 20:00:38.096304] [SM Locating]: (0x646a90) located: unexpected_sm:post_unexpected [D 20:00:38.096311] [SM frame get]: (0x646a90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096318] [SM Entering]: (0x646a90) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096324] [SM frame get]: (0x646a90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096332] [SM Exiting]: (0x646a90) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096339] server_post_unexpected_recv [D 20:00:38.096349] [SM Frame PUSH]: (0x647d50) frame: 0x647e50 [D 20:00:38.096356] [SM Locating]: (0x647d50) op-id: 999 [D 20:00:38.096362] server_op_state_get_machine 999 [D 20:00:38.096368] [SM Locating]: (0x647d50) located: unexpected_sm:post_unexpected [D 20:00:38.096374] [SM frame get]: (0x647d50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096381] [SM Entering]: (0x647d50) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096388] [SM frame get]: (0x647d50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096400] [SM Exiting]: (0x647d50) unexpected_sm:post_unexpected (error code: 6119360), (action: DEFERRED) [D 20:00:38.096408] server_post_unexpected_recv [D 20:00:38.096419] [SM Frame PUSH]: (0x649010) frame: 0x649110 [D 20:00:38.096426] [SM Locating]: (0x649010) op-id: 999 [D 20:00:38.096432] server_op_state_get_machine 999 [D 20:00:38.096438] [SM Locating]: (0x649010) located: unexpected_sm:post_unexpected [D 20:00:38.096445] [SM frame get]: (0x649010) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096452] [SM Entering]: (0x649010) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096458] [SM frame get]: (0x649010) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096466] [SM Exiting]: (0x649010) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096473] server_post_unexpected_recv [D 20:00:38.096486] [SM Frame PUSH]: (0x64a2d0) frame: 0x64a3d0 [D 20:00:38.096494] [SM Locating]: (0x64a2d0) op-id: 999 [D 20:00:38.096500] server_op_state_get_machine 999 [D 20:00:38.096506] [SM Locating]: (0x64a2d0) located: unexpected_sm:post_unexpected [D 20:00:38.096512] [SM frame get]: (0x64a2d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096519] [SM Entering]: (0x64a2d0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096526] [SM frame get]: (0x64a2d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096534] [SM Exiting]: (0x64a2d0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096541] server_post_unexpected_recv [D 20:00:38.096551] [SM Frame PUSH]: (0x64b590) frame: 0x64b690 [D 20:00:38.096558] [SM Locating]: (0x64b590) op-id: 999 [D 20:00:38.096564] server_op_state_get_machine 999 [D 20:00:38.096570] [SM Locating]: (0x64b590) located: unexpected_sm:post_unexpected [D 20:00:38.096577] [SM frame get]: (0x64b590) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096584] [SM Entering]: (0x64b590) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096590] [SM frame get]: (0x64b590) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096598] [SM Exiting]: (0x64b590) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096605] server_post_unexpected_recv [D 20:00:38.096614] [SM Frame PUSH]: (0x64c850) frame: 0x64c950 [D 20:00:38.096621] [SM Locating]: (0x64c850) op-id: 999 [D 20:00:38.096628] server_op_state_get_machine 999 [D 20:00:38.096634] [SM Locating]: (0x64c850) located: unexpected_sm:post_unexpected [D 20:00:38.096640] [SM frame get]: (0x64c850) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096647] [SM Entering]: (0x64c850) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096654] [SM frame get]: (0x64c850) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096661] [SM Exiting]: (0x64c850) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096669] server_post_unexpected_recv [D 20:00:38.096678] [SM Frame PUSH]: (0x64db10) frame: 0x64dc10 [D 20:00:38.096685] [SM Locating]: (0x64db10) op-id: 999 [D 20:00:38.096692] server_op_state_get_machine 999 [D 20:00:38.096698] [SM Locating]: (0x64db10) located: unexpected_sm:post_unexpected [D 20:00:38.096704] [SM frame get]: (0x64db10) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096711] [SM Entering]: (0x64db10) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096718] [SM frame get]: (0x64db10) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096725] [SM Exiting]: (0x64db10) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096733] server_post_unexpected_recv [D 20:00:38.096742] [SM Frame PUSH]: (0x64edd0) frame: 0x64eed0 [D 20:00:38.096750] [SM Locating]: (0x64edd0) op-id: 999 [D 20:00:38.096756] server_op_state_get_machine 999 [D 20:00:38.096762] [SM Locating]: (0x64edd0) located: unexpected_sm:post_unexpected [D 20:00:38.096768] [SM frame get]: (0x64edd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096775] [SM Entering]: (0x64edd0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096782] [SM frame get]: (0x64edd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096792] [SM Exiting]: (0x64edd0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096800] server_post_unexpected_recv [D 20:00:38.096809] [SM Frame PUSH]: (0x650090) frame: 0x650190 [D 20:00:38.096816] [SM Locating]: (0x650090) op-id: 999 [D 20:00:38.096822] server_op_state_get_machine 999 [D 20:00:38.096828] [SM Locating]: (0x650090) located: unexpected_sm:post_unexpected [D 20:00:38.096835] [SM frame get]: (0x650090) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096842] [SM Entering]: (0x650090) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096848] [SM frame get]: (0x650090) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096856] [SM Exiting]: (0x650090) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096863] server_post_unexpected_recv [D 20:00:38.096872] [SM Frame PUSH]: (0x651350) frame: 0x651450 [D 20:00:38.096879] [SM Locating]: (0x651350) op-id: 999 [D 20:00:38.096886] server_op_state_get_machine 999 [D 20:00:38.096891] [SM Locating]: (0x651350) located: unexpected_sm:post_unexpected [D 20:00:38.096898] [SM frame get]: (0x651350) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096905] [SM Entering]: (0x651350) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096912] [SM frame get]: (0x651350) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096919] [SM Exiting]: (0x651350) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096927] server_post_unexpected_recv [D 20:00:38.096937] [SM Frame PUSH]: (0x652610) frame: 0x652710 [D 20:00:38.096944] [SM Locating]: (0x652610) op-id: 999 [D 20:00:38.096950] server_op_state_get_machine 999 [D 20:00:38.096956] [SM Locating]: (0x652610) located: unexpected_sm:post_unexpected [D 20:00:38.096962] [SM frame get]: (0x652610) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096969] [SM Entering]: (0x652610) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.096976] [SM frame get]: (0x652610) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.096983] [SM Exiting]: (0x652610) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.096991] server_post_unexpected_recv [D 20:00:38.097000] [SM Frame PUSH]: (0x6538d0) frame: 0x6539d0 [D 20:00:38.097007] [SM Locating]: (0x6538d0) op-id: 999 [D 20:00:38.097013] server_op_state_get_machine 999 [D 20:00:38.097019] [SM Locating]: (0x6538d0) located: unexpected_sm:post_unexpected [D 20:00:38.097026] [SM frame get]: (0x6538d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097033] [SM Entering]: (0x6538d0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097040] [SM frame get]: (0x6538d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097047] [SM Exiting]: (0x6538d0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097054] server_post_unexpected_recv [D 20:00:38.097064] [SM Frame PUSH]: (0x654b90) frame: 0x654c90 [D 20:00:38.097071] [SM Locating]: (0x654b90) op-id: 999 [D 20:00:38.097077] server_op_state_get_machine 999 [D 20:00:38.097083] [SM Locating]: (0x654b90) located: unexpected_sm:post_unexpected [D 20:00:38.097090] [SM frame get]: (0x654b90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097097] [SM Entering]: (0x654b90) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097104] [SM frame get]: (0x654b90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097111] [SM Exiting]: (0x654b90) unexpected_sm:post_unexpected (error code: 4113), (action: DEFERRED) [D 20:00:38.097118] server_post_unexpected_recv [D 20:00:38.097128] [SM Frame PUSH]: (0x655e50) frame: 0x655f50 [D 20:00:38.097135] [SM Locating]: (0x655e50) op-id: 999 [D 20:00:38.097141] server_op_state_get_machine 999 [D 20:00:38.097147] [SM Locating]: (0x655e50) located: unexpected_sm:post_unexpected [D 20:00:38.097154] [SM frame get]: (0x655e50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097161] [SM Entering]: (0x655e50) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097168] [SM frame get]: (0x655e50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097178] [SM Exiting]: (0x655e50) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097185] server_post_unexpected_recv [D 20:00:38.097195] [SM Frame PUSH]: (0x657110) frame: 0x657210 [D 20:00:38.097202] [SM Locating]: (0x657110) op-id: 999 [D 20:00:38.097208] server_op_state_get_machine 999 [D 20:00:38.097214] [SM Locating]: (0x657110) located: unexpected_sm:post_unexpected [D 20:00:38.097221] [SM frame get]: (0x657110) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097228] [SM Entering]: (0x657110) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097235] [SM frame get]: (0x657110) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097242] [SM Exiting]: (0x657110) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097249] server_post_unexpected_recv [D 20:00:38.097260] [SM Frame PUSH]: (0x6583d0) frame: 0x6584d0 [D 20:00:38.097267] [SM Locating]: (0x6583d0) op-id: 999 [D 20:00:38.097273] server_op_state_get_machine 999 [D 20:00:38.097279] [SM Locating]: (0x6583d0) located: unexpected_sm:post_unexpected [D 20:00:38.097286] [SM frame get]: (0x6583d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097293] [SM Entering]: (0x6583d0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097299] [SM frame get]: (0x6583d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097307] [SM Exiting]: (0x6583d0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097314] server_post_unexpected_recv [D 20:00:38.097324] [SM Frame PUSH]: (0x659690) frame: 0x659790 [D 20:00:38.097331] [SM Locating]: (0x659690) op-id: 999 [D 20:00:38.097337] server_op_state_get_machine 999 [D 20:00:38.097343] [SM Locating]: (0x659690) located: unexpected_sm:post_unexpected [D 20:00:38.097350] [SM frame get]: (0x659690) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097357] [SM Entering]: (0x659690) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097363] [SM frame get]: (0x659690) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097371] [SM Exiting]: (0x659690) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097378] server_post_unexpected_recv [D 20:00:38.097387] [SM Frame PUSH]: (0x65a950) frame: 0x65aa50 [D 20:00:38.097394] [SM Locating]: (0x65a950) op-id: 999 [D 20:00:38.097400] server_op_state_get_machine 999 [D 20:00:38.097406] [SM Locating]: (0x65a950) located: unexpected_sm:post_unexpected [D 20:00:38.097413] [SM frame get]: (0x65a950) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097420] [SM Entering]: (0x65a950) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097427] [SM frame get]: (0x65a950) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097434] [SM Exiting]: (0x65a950) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097441] server_post_unexpected_recv [D 20:00:38.097451] [SM Frame PUSH]: (0x65bc10) frame: 0x65bd10 [D 20:00:38.097458] [SM Locating]: (0x65bc10) op-id: 999 [D 20:00:38.097464] server_op_state_get_machine 999 [D 20:00:38.097470] [SM Locating]: (0x65bc10) located: unexpected_sm:post_unexpected [D 20:00:38.097477] [SM frame get]: (0x65bc10) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097484] [SM Entering]: (0x65bc10) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097491] [SM frame get]: (0x65bc10) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097498] [SM Exiting]: (0x65bc10) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097505] server_post_unexpected_recv [D 20:00:38.097517] [SM Frame PUSH]: (0x65ced0) frame: 0x65cfd0 [D 20:00:38.097524] [SM Locating]: (0x65ced0) op-id: 999 [D 20:00:38.097531] server_op_state_get_machine 999 [D 20:00:38.097537] [SM Locating]: (0x65ced0) located: unexpected_sm:post_unexpected [D 20:00:38.097543] [SM frame get]: (0x65ced0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097550] [SM Entering]: (0x65ced0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097557] [SM frame get]: (0x65ced0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097564] [SM Exiting]: (0x65ced0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097572] server_post_unexpected_recv [D 20:00:38.097581] [SM Frame PUSH]: (0x65e190) frame: 0x65e290 [D 20:00:38.097588] [SM Locating]: (0x65e190) op-id: 999 [D 20:00:38.097594] server_op_state_get_machine 999 [D 20:00:38.097600] [SM Locating]: (0x65e190) located: unexpected_sm:post_unexpected [D 20:00:38.097607] [SM frame get]: (0x65e190) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097614] [SM Entering]: (0x65e190) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097621] [SM frame get]: (0x65e190) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097628] [SM Exiting]: (0x65e190) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097635] server_post_unexpected_recv [D 20:00:38.097645] [SM Frame PUSH]: (0x65f450) frame: 0x65f550 [D 20:00:38.097652] [SM Locating]: (0x65f450) op-id: 999 [D 20:00:38.097658] server_op_state_get_machine 999 [D 20:00:38.097664] [SM Locating]: (0x65f450) located: unexpected_sm:post_unexpected [D 20:00:38.097671] [SM frame get]: (0x65f450) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097678] [SM Entering]: (0x65f450) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097684] [SM frame get]: (0x65f450) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097692] [SM Exiting]: (0x65f450) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097699] server_post_unexpected_recv [D 20:00:38.097708] [SM Frame PUSH]: (0x660710) frame: 0x660810 [D 20:00:38.097716] [SM Locating]: (0x660710) op-id: 999 [D 20:00:38.097722] server_op_state_get_machine 999 [D 20:00:38.097729] [SM Locating]: (0x660710) located: unexpected_sm:post_unexpected [D 20:00:38.097735] [SM frame get]: (0x660710) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097742] [SM Entering]: (0x660710) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097749] [SM frame get]: (0x660710) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097756] [SM Exiting]: (0x660710) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097763] server_post_unexpected_recv [D 20:00:38.097773] [SM Frame PUSH]: (0x6619d0) frame: 0x661ad0 [D 20:00:38.097780] [SM Locating]: (0x6619d0) op-id: 999 [D 20:00:38.097786] server_op_state_get_machine 999 [D 20:00:38.097792] [SM Locating]: (0x6619d0) located: unexpected_sm:post_unexpected [D 20:00:38.097799] [SM frame get]: (0x6619d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097806] [SM Entering]: (0x6619d0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097813] [SM frame get]: (0x6619d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097820] [SM Exiting]: (0x6619d0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097827] server_post_unexpected_recv [D 20:00:38.097837] [SM Frame PUSH]: (0x662c90) frame: 0x662d90 [D 20:00:38.097844] [SM Locating]: (0x662c90) op-id: 999 [D 20:00:38.097850] server_op_state_get_machine 999 [D 20:00:38.097856] [SM Locating]: (0x662c90) located: unexpected_sm:post_unexpected [D 20:00:38.097863] [SM frame get]: (0x662c90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097870] [SM Entering]: (0x662c90) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097877] [SM frame get]: (0x662c90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097884] [SM Exiting]: (0x662c90) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097891] server_post_unexpected_recv [D 20:00:38.097903] [SM Frame PUSH]: (0x663f50) frame: 0x664050 [D 20:00:38.097910] [SM Locating]: (0x663f50) op-id: 999 [D 20:00:38.097916] server_op_state_get_machine 999 [D 20:00:38.097922] [SM Locating]: (0x663f50) located: unexpected_sm:post_unexpected [D 20:00:38.097929] [SM frame get]: (0x663f50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097936] [SM Entering]: (0x663f50) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.097943] [SM frame get]: (0x663f50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.097950] [SM Exiting]: (0x663f50) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.097957] server_post_unexpected_recv [D 20:00:38.097970] [SM Frame PUSH]: (0x665210) frame: 0x665310 [D 20:00:38.097977] [SM Locating]: (0x665210) op-id: 999 [D 20:00:38.097984] server_op_state_get_machine 999 [D 20:00:38.097990] [SM Locating]: (0x665210) located: unexpected_sm:post_unexpected [D 20:00:38.097996] [SM frame get]: (0x665210) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098003] [SM Entering]: (0x665210) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098010] [SM frame get]: (0x665210) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098018] [SM Exiting]: (0x665210) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098025] server_post_unexpected_recv [D 20:00:38.098036] [SM Frame PUSH]: (0x6664d0) frame: 0x6665d0 [D 20:00:38.098043] [SM Locating]: (0x6664d0) op-id: 999 [D 20:00:38.098049] server_op_state_get_machine 999 [D 20:00:38.098055] [SM Locating]: (0x6664d0) located: unexpected_sm:post_unexpected [D 20:00:38.098062] [SM frame get]: (0x6664d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098069] [SM Entering]: (0x6664d0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098075] [SM frame get]: (0x6664d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098083] [SM Exiting]: (0x6664d0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098090] server_post_unexpected_recv [D 20:00:38.098099] [SM Frame PUSH]: (0x667790) frame: 0x667890 [D 20:00:38.098106] [SM Locating]: (0x667790) op-id: 999 [D 20:00:38.098113] server_op_state_get_machine 999 [D 20:00:38.098119] [SM Locating]: (0x667790) located: unexpected_sm:post_unexpected [D 20:00:38.098126] [SM frame get]: (0x667790) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098133] [SM Entering]: (0x667790) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098140] [SM frame get]: (0x667790) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098147] [SM Exiting]: (0x667790) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098155] server_post_unexpected_recv [D 20:00:38.098164] [SM Frame PUSH]: (0x668a50) frame: 0x668b50 [D 20:00:38.098171] [SM Locating]: (0x668a50) op-id: 999 [D 20:00:38.098177] server_op_state_get_machine 999 [D 20:00:38.098184] [SM Locating]: (0x668a50) located: unexpected_sm:post_unexpected [D 20:00:38.098190] [SM frame get]: (0x668a50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098197] [SM Entering]: (0x668a50) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098204] [SM frame get]: (0x668a50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098211] [SM Exiting]: (0x668a50) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098219] server_post_unexpected_recv [D 20:00:38.098229] [SM Frame PUSH]: (0x669d10) frame: 0x669e10 [D 20:00:38.098236] [SM Locating]: (0x669d10) op-id: 999 [D 20:00:38.098243] server_op_state_get_machine 999 [D 20:00:38.098249] [SM Locating]: (0x669d10) located: unexpected_sm:post_unexpected [D 20:00:38.098255] [SM frame get]: (0x669d10) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098262] [SM Entering]: (0x669d10) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098269] [SM frame get]: (0x669d10) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098276] [SM Exiting]: (0x669d10) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098283] server_post_unexpected_recv [D 20:00:38.098295] [SM Frame PUSH]: (0x66afd0) frame: 0x66b0d0 [D 20:00:38.098302] [SM Locating]: (0x66afd0) op-id: 999 [D 20:00:38.098309] server_op_state_get_machine 999 [D 20:00:38.098315] [SM Locating]: (0x66afd0) located: unexpected_sm:post_unexpected [D 20:00:38.098321] [SM frame get]: (0x66afd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098328] [SM Entering]: (0x66afd0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098335] [SM frame get]: (0x66afd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098343] [SM Exiting]: (0x66afd0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098350] server_post_unexpected_recv [D 20:00:38.098359] [SM Frame PUSH]: (0x66c290) frame: 0x66c390 [D 20:00:38.098367] [SM Locating]: (0x66c290) op-id: 999 [D 20:00:38.098373] server_op_state_get_machine 999 [D 20:00:38.098379] [SM Locating]: (0x66c290) located: unexpected_sm:post_unexpected [D 20:00:38.098385] [SM frame get]: (0x66c290) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098392] [SM Entering]: (0x66c290) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098399] [SM frame get]: (0x66c290) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098406] [SM Exiting]: (0x66c290) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098414] server_post_unexpected_recv [D 20:00:38.098423] [SM Frame PUSH]: (0x66d550) frame: 0x66d650 [D 20:00:38.098430] [SM Locating]: (0x66d550) op-id: 999 [D 20:00:38.098437] server_op_state_get_machine 999 [D 20:00:38.098443] [SM Locating]: (0x66d550) located: unexpected_sm:post_unexpected [D 20:00:38.098449] [SM frame get]: (0x66d550) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098457] [SM Entering]: (0x66d550) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098463] [SM frame get]: (0x66d550) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098471] [SM Exiting]: (0x66d550) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098478] server_post_unexpected_recv [D 20:00:38.098487] [SM Frame PUSH]: (0x66e810) frame: 0x66e910 [D 20:00:38.098494] [SM Locating]: (0x66e810) op-id: 999 [D 20:00:38.098501] server_op_state_get_machine 999 [D 20:00:38.098507] [SM Locating]: (0x66e810) located: unexpected_sm:post_unexpected [D 20:00:38.098513] [SM frame get]: (0x66e810) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098521] [SM Entering]: (0x66e810) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098528] [SM frame get]: (0x66e810) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098535] [SM Exiting]: (0x66e810) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098542] server_post_unexpected_recv [D 20:00:38.098552] [SM Frame PUSH]: (0x66fad0) frame: 0x66fbd0 [D 20:00:38.098559] [SM Locating]: (0x66fad0) op-id: 999 [D 20:00:38.098565] server_op_state_get_machine 999 [D 20:00:38.098571] [SM Locating]: (0x66fad0) located: unexpected_sm:post_unexpected [D 20:00:38.098578] [SM frame get]: (0x66fad0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098585] [SM Entering]: (0x66fad0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098591] [SM frame get]: (0x66fad0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098599] [SM Exiting]: (0x66fad0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098606] server_post_unexpected_recv [D 20:00:38.098615] [SM Frame PUSH]: (0x670d90) frame: 0x670e90 [D 20:00:38.098622] [SM Locating]: (0x670d90) op-id: 999 [D 20:00:38.098629] server_op_state_get_machine 999 [D 20:00:38.098635] [SM Locating]: (0x670d90) located: unexpected_sm:post_unexpected [D 20:00:38.098641] [SM frame get]: (0x670d90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098649] [SM Entering]: (0x670d90) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098655] [SM frame get]: (0x670d90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098665] [SM Exiting]: (0x670d90) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098673] server_post_unexpected_recv [D 20:00:38.098683] [SM Frame PUSH]: (0x672050) frame: 0x672150 [D 20:00:38.098690] [SM Locating]: (0x672050) op-id: 999 [D 20:00:38.098696] server_op_state_get_machine 999 [D 20:00:38.098702] [SM Locating]: (0x672050) located: unexpected_sm:post_unexpected [D 20:00:38.098709] [SM frame get]: (0x672050) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098716] [SM Entering]: (0x672050) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098722] [SM frame get]: (0x672050) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098730] [SM Exiting]: (0x672050) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098737] server_post_unexpected_recv [D 20:00:38.098747] [SM Frame PUSH]: (0x673310) frame: 0x673410 [D 20:00:38.098754] [SM Locating]: (0x673310) op-id: 999 [D 20:00:38.098760] server_op_state_get_machine 999 [D 20:00:38.098766] [SM Locating]: (0x673310) located: unexpected_sm:post_unexpected [D 20:00:38.098773] [SM frame get]: (0x673310) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098780] [SM Entering]: (0x673310) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098786] [SM frame get]: (0x673310) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098794] [SM Exiting]: (0x673310) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098801] server_post_unexpected_recv [D 20:00:38.098812] [SM Frame PUSH]: (0x6745d0) frame: 0x6746d0 [D 20:00:38.098819] [SM Locating]: (0x6745d0) op-id: 999 [D 20:00:38.098825] server_op_state_get_machine 999 [D 20:00:38.098831] [SM Locating]: (0x6745d0) located: unexpected_sm:post_unexpected [D 20:00:38.098838] [SM frame get]: (0x6745d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098845] [SM Entering]: (0x6745d0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098852] [SM frame get]: (0x6745d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098859] [SM Exiting]: (0x6745d0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098866] server_post_unexpected_recv [D 20:00:38.098876] [SM Frame PUSH]: (0x675890) frame: 0x675990 [D 20:00:38.098882] [SM Locating]: (0x675890) op-id: 999 [D 20:00:38.098892] server_op_state_get_machine 999 [D 20:00:38.098898] [SM Locating]: (0x675890) located: unexpected_sm:post_unexpected [D 20:00:38.098904] [SM frame get]: (0x675890) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098912] [SM Entering]: (0x675890) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098919] [SM frame get]: (0x675890) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098926] [SM Exiting]: (0x675890) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098934] server_post_unexpected_recv [D 20:00:38.098943] [SM Frame PUSH]: (0x676b50) frame: 0x676c50 [D 20:00:38.098950] [SM Locating]: (0x676b50) op-id: 999 [D 20:00:38.098956] server_op_state_get_machine 999 [D 20:00:38.098962] [SM Locating]: (0x676b50) located: unexpected_sm:post_unexpected [D 20:00:38.098969] [SM frame get]: (0x676b50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098976] [SM Entering]: (0x676b50) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.098982] [SM frame get]: (0x676b50) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.098990] [SM Exiting]: (0x676b50) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.098997] server_post_unexpected_recv [D 20:00:38.099006] [SM Frame PUSH]: (0x677e10) frame: 0x677f10 [D 20:00:38.099013] [SM Locating]: (0x677e10) op-id: 999 [D 20:00:38.099020] server_op_state_get_machine 999 [D 20:00:38.099026] [SM Locating]: (0x677e10) located: unexpected_sm:post_unexpected [D 20:00:38.099032] [SM frame get]: (0x677e10) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099039] [SM Entering]: (0x677e10) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.099046] [SM frame get]: (0x677e10) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099056] [SM Exiting]: (0x677e10) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.099064] server_post_unexpected_recv [D 20:00:38.099073] [SM Frame PUSH]: (0x6790d0) frame: 0x6791d0 [D 20:00:38.099080] [SM Locating]: (0x6790d0) op-id: 999 [D 20:00:38.099086] server_op_state_get_machine 999 [D 20:00:38.099092] [SM Locating]: (0x6790d0) located: unexpected_sm:post_unexpected [D 20:00:38.099099] [SM frame get]: (0x6790d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099106] [SM Entering]: (0x6790d0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.099112] [SM frame get]: (0x6790d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099119] [SM Exiting]: (0x6790d0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.099127] server_post_unexpected_recv [D 20:00:38.099136] [SM Frame PUSH]: (0x67a390) frame: 0x67a490 [D 20:00:38.099143] [SM Locating]: (0x67a390) op-id: 999 [D 20:00:38.099149] server_op_state_get_machine 999 [D 20:00:38.099155] [SM Locating]: (0x67a390) located: unexpected_sm:post_unexpected [D 20:00:38.099162] [SM frame get]: (0x67a390) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099169] [SM Entering]: (0x67a390) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.099175] [SM frame get]: (0x67a390) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099183] [SM Exiting]: (0x67a390) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.099190] server_post_unexpected_recv [D 20:00:38.099199] [SM Frame PUSH]: (0x67b650) frame: 0x67b750 [D 20:00:38.099206] [SM Locating]: (0x67b650) op-id: 999 [D 20:00:38.099213] server_op_state_get_machine 999 [D 20:00:38.099219] [SM Locating]: (0x67b650) located: unexpected_sm:post_unexpected [D 20:00:38.099225] [SM frame get]: (0x67b650) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099232] [SM Entering]: (0x67b650) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.099239] [SM frame get]: (0x67b650) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099246] [SM Exiting]: (0x67b650) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.099253] server_post_unexpected_recv [D 20:00:38.099263] [SM Frame PUSH]: (0x67c910) frame: 0x67ca10 [D 20:00:38.099270] [SM Locating]: (0x67c910) op-id: 999 [D 20:00:38.099276] server_op_state_get_machine 999 [D 20:00:38.099282] [SM Locating]: (0x67c910) located: unexpected_sm:post_unexpected [D 20:00:38.099288] [SM frame get]: (0x67c910) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099295] [SM Entering]: (0x67c910) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.099302] [SM frame get]: (0x67c910) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099309] [SM Exiting]: (0x67c910) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.099317] server_post_unexpected_recv [D 20:00:38.099326] [SM Frame PUSH]: (0x67dbd0) frame: 0x67dcd0 [D 20:00:38.099333] [SM Locating]: (0x67dbd0) op-id: 999 [D 20:00:38.099340] server_op_state_get_machine 999 [D 20:00:38.099346] [SM Locating]: (0x67dbd0) located: unexpected_sm:post_unexpected [D 20:00:38.099352] [SM frame get]: (0x67dbd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099359] [SM Entering]: (0x67dbd0) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.099366] [SM frame get]: (0x67dbd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099373] [SM Exiting]: (0x67dbd0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.099381] server_post_unexpected_recv [D 20:00:38.099390] [SM Frame PUSH]: (0x67ee90) frame: 0x67ef90 [D 20:00:38.099400] [SM Locating]: (0x67ee90) op-id: 999 [D 20:00:38.099407] server_op_state_get_machine 999 [D 20:00:38.099413] [SM Locating]: (0x67ee90) located: unexpected_sm:post_unexpected [D 20:00:38.099419] [SM frame get]: (0x67ee90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099426] [SM Entering]: (0x67ee90) unexpected_sm:post_unexpected (status: 0) [D 20:00:38.099433] [SM frame get]: (0x67ee90) op-id: 999 index: 0 base-frm: 0 [D 20:00:38.099440] [SM Exiting]: (0x67ee90) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:38.099459] Initialization completed successfully. [D 20:00:38.099466] server_state_machine_alloc_noreq 19 [D 20:00:38.099480] [SM Frame PUSH]: (0x680150) frame: 0x680250 [D 20:00:38.099487] [SM Locating]: (0x680150) op-id: 19 [D 20:00:38.099493] server_op_state_get_machine 19 [D 20:00:38.099501] [SM Locating]: (0x680150) located: perf_update_sm:do_work [D 20:00:38.099508] [SM frame get]: (0x680150) op-id: 19 index: 0 base-frm: 0 [D 20:00:38.099515] [SM frame get]: (0x680150) op-id: 19 index: 0 base-frm: 0 [D 20:00:38.099521] server_state_machine_start_noreq 0x680150 [D 20:00:38.099528] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 6820552) [P 20:00:38.099572] Start times (hr:min:sec): 20:00:38.096 0 0 0 0 0 [P 20:00:38.099579] Intervals (hr:min:sec) : 00:00:00.003 0 0 0 0 0 [P 20:00:38.099586] ------------------------------------------------------------------------------------------------------------- [P 20:00:38.099592] bytes read : 0 0 0 0 0 0 [P 20:00:38.099598] bytes written : 0 0 0 0 0 0 [P 20:00:38.099605] metadata reads : 0 0 0 0 0 0 [P 20:00:38.099611] metadata writes : 0 0 0 0 0 0 [P 20:00:38.099617] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:38.099623] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:38.099630] request scheduler : 0 0 0 0 0 0 [D 20:00:38.099645] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:38.099653] server_state_machine_alloc_noreq 27 [D 20:00:38.099663] [SM Frame PUSH]: (0x681490) frame: 0x681590 [D 20:00:38.099670] [SM Locating]: (0x681490) op-id: 27 [D 20:00:38.099677] server_op_state_get_machine 27 [D 20:00:38.099684] [SM Locating]: (0x681490) located: job_timer_sm:do_work [D 20:00:38.099691] [SM frame get]: (0x681490) op-id: 27 index: 0 base-frm: 0 [D 20:00:38.099698] [SM frame get]: (0x681490) op-id: 27 index: 0 base-frm: 0 [D 20:00:38.099705] server_state_machine_start_noreq 0x681490 [D 20:00:38.099711] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 6825480) [D 20:00:38.099723] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) [S 04/14 20:00] PVFS2 Server ready. [D 20:00:39.139958] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:39.140003] Start times (hr:min:sec): 20:00:38.099 20:00:38.096 0 0 0 0 [P 20:00:39.140010] Intervals (hr:min:sec) : 00:00:01.040 00:00:00.003 0 0 0 0 [P 20:00:39.140017] ------------------------------------------------------------------------------------------------------------- [P 20:00:39.140023] bytes read : 0 0 0 0 0 0 [P 20:00:39.140030] bytes written : 0 0 0 0 0 0 [P 20:00:39.140036] metadata reads : 0 0 0 0 0 0 [P 20:00:39.140042] metadata writes : 0 0 0 0 0 0 [P 20:00:39.140048] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:39.140054] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:39.140061] request scheduler : 0 0 0 0 0 0 [D 20:00:39.140069] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:39.140078] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:39.140086] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:40.179981] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:40.180032] Start times (hr:min:sec): 20:00:39.140 20:00:38.099 20:00:38.096 0 0 0 [P 20:00:40.180039] Intervals (hr:min:sec) : 00:00:01.040 00:00:01.041 00:00:00.003 0 0 0 [P 20:00:40.180045] ------------------------------------------------------------------------------------------------------------- [P 20:00:40.180052] bytes read : 0 0 0 0 0 0 [P 20:00:40.180058] bytes written : 0 0 0 0 0 0 [P 20:00:40.180064] metadata reads : 0 0 0 0 0 0 [P 20:00:40.180070] metadata writes : 0 0 0 0 0 0 [P 20:00:40.180076] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:40.180083] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:40.180089] request scheduler : 0 0 0 0 0 0 [D 20:00:40.180098] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:40.180106] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:40.180114] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:41.220006] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:41.220062] Start times (hr:min:sec): 20:00:40.180 20:00:39.140 20:00:38.099 20:00:38.096 0 0 [P 20:00:41.220069] Intervals (hr:min:sec) : 00:00:01.040 00:00:01.040 00:00:01.041 00:00:00.003 0 0 [P 20:00:41.220075] ------------------------------------------------------------------------------------------------------------- [P 20:00:41.220082] bytes read : 0 0 0 0 0 0 [P 20:00:41.220088] bytes written : 0 0 0 0 0 0 [P 20:00:41.220094] metadata reads : 0 0 0 0 0 0 [P 20:00:41.220100] metadata writes : 0 0 0 0 0 0 [P 20:00:41.220106] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:41.220113] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:41.220119] request scheduler : 0 0 0 0 0 0 [D 20:00:41.220127] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:41.220135] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:41.220144] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:41.509341] openib_post_rr: 10.1.4.56:39990 bh 0 wr_id 683b70. [D 20:00:41.509377] openib_post_rr: 10.1.4.56:39990 bh 1 wr_id 683ba0. [D 20:00:41.509385] openib_post_rr: 10.1.4.56:39990 bh 2 wr_id 683bd0. [D 20:00:41.509392] openib_post_rr: 10.1.4.56:39990 bh 3 wr_id 683c00. [D 20:00:41.509399] openib_post_rr: 10.1.4.56:39990 bh 4 wr_id 683c30. [D 20:00:41.509406] openib_post_rr: 10.1.4.56:39990 bh 5 wr_id 683c60. [D 20:00:41.509413] openib_post_rr: 10.1.4.56:39990 bh 6 wr_id 683c90. [D 20:00:41.509420] openib_post_rr: 10.1.4.56:39990 bh 7 wr_id 683cc0. [D 20:00:41.509426] openib_post_rr: 10.1.4.56:39990 bh 8 wr_id 683cf0. [D 20:00:41.509433] openib_post_rr: 10.1.4.56:39990 bh 9 wr_id 683d20. [D 20:00:41.509440] openib_post_rr: 10.1.4.56:39990 bh 10 wr_id 683d50. [D 20:00:41.509447] openib_post_rr: 10.1.4.56:39990 bh 11 wr_id 683d80. [D 20:00:41.509453] openib_post_rr: 10.1.4.56:39990 bh 12 wr_id 683db0. [D 20:00:41.509460] openib_post_rr: 10.1.4.56:39990 bh 13 wr_id 683de0. [D 20:00:41.509466] openib_post_rr: 10.1.4.56:39990 bh 14 wr_id 683e10. [D 20:00:41.509473] openib_post_rr: 10.1.4.56:39990 bh 15 wr_id 683e40. [D 20:00:41.509480] openib_post_rr: 10.1.4.56:39990 bh 16 wr_id 683e70. [D 20:00:41.509486] openib_post_rr: 10.1.4.56:39990 bh 17 wr_id 683ea0. [D 20:00:41.509493] openib_post_rr: 10.1.4.56:39990 bh 18 wr_id 683ed0. [D 20:00:41.509500] openib_post_rr: 10.1.4.56:39990 bh 19 wr_id 683f00. [D 20:00:41.509506] openib_post_rr: 10.1.4.56:39990 bh 20 wr_id 683f30. [D 20:00:41.509513] openib_post_rr: 10.1.4.56:39990 bh 21 wr_id 683f60. [D 20:00:41.509520] openib_post_rr: 10.1.4.56:39990 bh 22 wr_id 683f90. [D 20:00:41.509526] openib_post_rr: 10.1.4.56:39990 bh 23 wr_id 683fc0. [D 20:00:41.509533] openib_post_rr: 10.1.4.56:39990 bh 24 wr_id 683ff0. [D 20:00:41.509540] openib_post_rr: 10.1.4.56:39990 bh 25 wr_id 684020. [D 20:00:41.509546] openib_post_rr: 10.1.4.56:39990 bh 26 wr_id 684050. [D 20:00:41.509553] openib_post_rr: 10.1.4.56:39990 bh 27 wr_id 684080. [D 20:00:41.509560] openib_post_rr: 10.1.4.56:39990 bh 28 wr_id 6840b0. [D 20:00:41.509567] openib_post_rr: 10.1.4.56:39990 bh 29 wr_id 6840e0. [D 20:00:41.509573] openib_post_rr: 10.1.4.56:39990 bh 30 wr_id 684110. [D 20:00:41.509580] openib_post_rr: 10.1.4.56:39990 bh 31 wr_id 684140. [D 20:00:41.509587] openib_post_rr: 10.1.4.56:39990 bh 32 wr_id 684170. [D 20:00:41.509594] openib_post_rr: 10.1.4.56:39990 bh 33 wr_id 6841a0. [D 20:00:41.509600] openib_post_rr: 10.1.4.56:39990 bh 34 wr_id 6841d0. [D 20:00:41.509607] openib_post_rr: 10.1.4.56:39990 bh 35 wr_id 684200. [D 20:00:41.509614] openib_post_rr: 10.1.4.56:39990 bh 36 wr_id 684230. [D 20:00:41.509620] openib_post_rr: 10.1.4.56:39990 bh 37 wr_id 684260. [D 20:00:41.509627] openib_post_rr: 10.1.4.56:39990 bh 38 wr_id 684290. [D 20:00:41.509634] openib_post_rr: 10.1.4.56:39990 bh 39 wr_id 6842c0. [D 20:00:41.509641] openib_post_rr: 10.1.4.56:39990 bh 40 wr_id 6842f0. [D 20:00:41.509647] openib_post_rr: 10.1.4.56:39990 bh 41 wr_id 684320. [D 20:00:41.509654] openib_post_rr: 10.1.4.56:39990 bh 42 wr_id 684350. [D 20:00:41.509661] openib_post_rr: 10.1.4.56:39990 bh 43 wr_id 684380. [D 20:00:41.509667] openib_post_rr: 10.1.4.56:39990 bh 44 wr_id 6843b0. [D 20:00:41.509674] openib_post_rr: 10.1.4.56:39990 bh 45 wr_id 6843e0. [D 20:00:41.509680] openib_post_rr: 10.1.4.56:39990 bh 46 wr_id 684410. [D 20:00:41.509687] openib_post_rr: 10.1.4.56:39990 bh 47 wr_id 684440. [D 20:00:41.509694] openib_post_rr: 10.1.4.56:39990 bh 48 wr_id 684470. [D 20:00:41.509700] openib_post_rr: 10.1.4.56:39990 bh 49 wr_id 6844a0. [D 20:00:41.509707] openib_post_rr: 10.1.4.56:39990 bh 50 wr_id 6844d0. [D 20:00:41.509714] openib_post_rr: 10.1.4.56:39990 bh 51 wr_id 684500. [D 20:00:41.509721] openib_post_rr: 10.1.4.56:39990 bh 52 wr_id 684530. [D 20:00:41.509727] openib_post_rr: 10.1.4.56:39990 bh 53 wr_id 684560. [D 20:00:41.509734] openib_post_rr: 10.1.4.56:39990 bh 54 wr_id 684590. [D 20:00:41.509741] openib_post_rr: 10.1.4.56:39990 bh 55 wr_id 6845c0. [D 20:00:41.509748] openib_post_rr: 10.1.4.56:39990 bh 56 wr_id 6845f0. [D 20:00:41.509754] openib_post_rr: 10.1.4.56:39990 bh 57 wr_id 684620. [D 20:00:41.509761] openib_post_rr: 10.1.4.56:39990 bh 58 wr_id 684650. [D 20:00:41.509768] openib_post_rr: 10.1.4.56:39990 bh 59 wr_id 684680. [D 20:00:41.509775] openib_post_rr: 10.1.4.56:39990 bh 60 wr_id 6846b0. [D 20:00:41.509781] openib_post_rr: 10.1.4.56:39990 bh 61 wr_id 6846e0. [D 20:00:41.509788] openib_post_rr: 10.1.4.56:39990 bh 62 wr_id 684710. [D 20:00:41.509794] openib_post_rr: 10.1.4.56:39990 bh 63 wr_id 684740. [D 20:00:41.509801] openib_post_rr: 10.1.4.56:39990 bh 64 wr_id 684770. [D 20:00:41.509808] openib_post_rr: 10.1.4.56:39990 bh 65 wr_id 6847a0. [D 20:00:41.509814] openib_post_rr: 10.1.4.56:39990 bh 66 wr_id 6847d0. [D 20:00:41.509821] openib_post_rr: 10.1.4.56:39990 bh 67 wr_id 684800. [D 20:00:41.509827] openib_post_rr: 10.1.4.56:39990 bh 68 wr_id 684830. [D 20:00:41.509834] openib_post_rr: 10.1.4.56:39990 bh 69 wr_id 684860. [D 20:00:41.509840] openib_post_rr: 10.1.4.56:39990 bh 70 wr_id 684890. [D 20:00:41.509847] openib_post_rr: 10.1.4.56:39990 bh 71 wr_id 6848c0. [D 20:00:41.509854] openib_post_rr: 10.1.4.56:39990 bh 72 wr_id 6848f0. [D 20:00:41.509860] openib_post_rr: 10.1.4.56:39990 bh 73 wr_id 684920. [D 20:00:41.509867] openib_post_rr: 10.1.4.56:39990 bh 74 wr_id 684950. [D 20:00:41.509873] openib_post_rr: 10.1.4.56:39990 bh 75 wr_id 684980. [D 20:00:41.509880] openib_post_rr: 10.1.4.56:39990 bh 76 wr_id 6849b0. [D 20:00:41.509886] openib_post_rr: 10.1.4.56:39990 bh 77 wr_id 6849e0. [D 20:00:41.509893] openib_post_rr: 10.1.4.56:39990 bh 78 wr_id 684a10. [D 20:00:41.509900] openib_post_rr: 10.1.4.56:39990 bh 79 wr_id 684a40. [D 20:00:41.509906] openib_post_rr: 10.1.4.56:39990 bh 80 wr_id 684a70. [D 20:00:41.509913] openib_post_rr: 10.1.4.56:39990 bh 81 wr_id 684aa0. [D 20:00:41.509920] openib_post_rr: 10.1.4.56:39990 bh 82 wr_id 684ad0. [D 20:00:41.509926] openib_post_rr: 10.1.4.56:39990 bh 83 wr_id 684b00. [D 20:00:41.509933] openib_post_rr: 10.1.4.56:39990 bh 84 wr_id 684b30. [D 20:00:41.509939] openib_post_rr: 10.1.4.56:39990 bh 85 wr_id 684b60. [D 20:00:41.509946] openib_post_rr: 10.1.4.56:39990 bh 86 wr_id 684b90. [D 20:00:41.509953] openib_post_rr: 10.1.4.56:39990 bh 87 wr_id 684bc0. [D 20:00:41.509959] openib_post_rr: 10.1.4.56:39990 bh 88 wr_id 684bf0. [D 20:00:41.509966] openib_post_rr: 10.1.4.56:39990 bh 89 wr_id 684c20. [D 20:00:41.509972] openib_post_rr: 10.1.4.56:39990 bh 90 wr_id 684c50. [D 20:00:41.509979] openib_post_rr: 10.1.4.56:39990 bh 91 wr_id 684c80. [D 20:00:41.509985] openib_post_rr: 10.1.4.56:39990 bh 92 wr_id 684cb0. [D 20:00:41.509992] openib_post_rr: 10.1.4.56:39990 bh 93 wr_id 684ce0. [D 20:00:41.509999] openib_post_rr: 10.1.4.56:39990 bh 94 wr_id 684d10. [D 20:00:41.510005] openib_post_rr: 10.1.4.56:39990 bh 95 wr_id 684d40. [D 20:00:41.510012] openib_post_rr: 10.1.4.56:39990 bh 96 wr_id 684d70. [D 20:00:41.510018] openib_post_rr: 10.1.4.56:39990 bh 97 wr_id 684da0. [D 20:00:41.510025] openib_post_rr: 10.1.4.56:39990 bh 98 wr_id 684dd0. [D 20:00:41.510032] openib_post_rr: 10.1.4.56:39990 bh 99 wr_id 684e00. [D 20:00:41.512481] openib_check_cq: completed recv wr_id 683b70. [D 20:00:41.512500] openib_post_rr: 10.1.4.56:39990 bh 0 wr_id 683b70. [D 20:00:41.516018] [SM Entering]: (0x6457d0) unexpected_sm:map_request (status: 0) [D 20:00:41.516031] [SM frame get]: (0x6457d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.516039] server_post_unexpected_recv [D 20:00:41.516050] [SM Frame PUSH]: (0x646960) frame: 0x686bb0 [D 20:00:41.516057] [SM Locating]: (0x646960) op-id: 999 [D 20:00:41.516064] server_op_state_get_machine 999 [D 20:00:41.516070] [SM Locating]: (0x646960) located: unexpected_sm:post_unexpected [D 20:00:41.516077] [SM frame get]: (0x646960) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.516084] [SM Entering]: (0x646960) unexpected_sm:post_unexpected (status: 0) [D 20:00:41.516091] [SM frame get]: (0x646960) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.516100] [SM Exiting]: (0x646960) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:41.516108] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:41.516115] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.516121] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 1 [D 20:00:41.516128] [SM frame get]: (0x6457d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.516134] server_state_machine_start 0x6457d0 [D 20:00:41.516140] PINT_decode [D 20:00:41.516150] lebf_decode_req [D 20:00:41.516157] [SM Locating]: (0x6457d0) op-id: 13 [D 20:00:41.516163] server_op_state_get_machine 13 [D 20:00:41.516172] [SM push_state]: (0x6457d0) op-id: 13 stk-ptr: 0 base-frm: 0 [D 20:00:41.516179] [SM Locating]: (0x6457d0) located: prelude_sm:setup [D 20:00:41.516188] [SM Entering]: (0x6457d0) prelude_sm:setup (status: 0) [D 20:00:41.516197] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [D 20:00:41.516204] [SM Exiting]: (0x6457d0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:41.516211] [SM Exiting]: (0x6457d0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:41.516218] [SM push_state]: (0x6457d0) op-id: 13 stk-ptr: 1 base-frm: 0 [D 20:00:41.516225] [SM Entering]: (0x6457d0) prelude_work_sm:req_sched (status: 0) [D 20:00:41.516232] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [D 20:00:41.516239] (0x6458d0) getconfig (prelude sm) state: req_sched -[A 20:00:41.531923] troy.scl@10.1.4.56:39990 H=0 S=0x6458d0: getconfig: request [D 20:00:41.531942] [SM Exiting]: (0x6457d0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:41.531952] [SM Entering]: (0x6457d0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:41.531959] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [A 20:00:41.537737] troy.scl@10.1.4.56:39990 H=0 S=0x6458d0: getconfig: start [D 20:00:41.537748] (0x6458d0) getconfig (prelude sm) state: getattr_if_needed [D 20:00:41.537758] [SM Exiting]: (0x6457d0) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:41.537767] [SM Entering]: (0x6457d0) prelude_work_sm:perm_check (status: 0) [D 20:00:41.537779] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [D 20:00:41.537788] PVFS operation "getconfig" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:41.537799] Final permission check for "getconfig" set error code to 0 [D 20:00:41.537810] (0x6458d0) getconfig (prelude sm) state: perm_check (status = 0) [D 20:00:41.537818] [SM Exiting]: (0x6457d0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:41.537827] [SM pop_state]: (0x6457d0) op-id: 13 stk-ptr: 2 base-frm: 0 [D 20:00:41.537836] [SM pop_state]: (0x6457d0) op-id: 13 stk-ptr: 1 base-frm: 0 [D 20:00:41.537845] [SM Entering]: (0x6457d0) get_config_sm:init (status: 0) [D 20:00:41.537855] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [D 20:00:41.537864] [SM Exiting]: (0x6457d0) get_config_sm:init (error code: 0), (action: COMPLETE) [D 20:00:41.537873] [SM push_state]: (0x6457d0) op-id: 13 stk-ptr: 0 base-frm: 0 [D 20:00:41.537882] [SM Entering]: (0x6457d0) final_response_sm:release (status: 0) [D 20:00:41.537890] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [D 20:00:41.537899] (s_op 0x6458d0) getconfig (FR sm) state: release: (error_code = 0) [D 20:00:41.537910] [SM Exiting]: (0x6457d0) final_response_sm:release (error code: 0), (action: COMPLETE) [D 20:00:41.537919] [SM Entering]: (0x6457d0) final_response_sm:send_resp (status: 0) [D 20:00:41.537928] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [D 20:00:41.537936] (s_op 0x6458d0) getconfig (FR sm) state: send_resp (status = 0) [D 20:00:41.537945] PINT_encode [D 20:00:41.537953] encode_common [D 20:00:41.538096] lebf_encode_resp [D 20:00:41.538136] BMI_post_send_list: addr: 6843072, count: 1, total_size: 936, tag: 1 [D 20:00:41.538148] element 0: offset: 0x696330, size: 936 [D 20:00:41.538207] post_sr: 10.1.4.56:39990 bh 0 len 952 wr_id 6828a0. -[D 20:00:41.538245] openib_check_cq: completed send wr_id 6828a0. [D 20:00:41.538222] [SM Exiting]: (0x6457d0) final_response_sm:send_resp (error code: 0), (action: DEFERRED) [D 20:00:41.538274] [SM Entering]: (0x6457d0) final_response_sm:cleanup (status: 0) [D 20:00:41.538281] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [D 20:00:41.538288] (s_op 0x6458d0) getconfig (FR sm) state: cleanup -[D 20:00:41.542303] openib_check_cq: completed recv wr_id 683ba0. [D 20:00:41.542316] openib_post_rr: 10.1.4.56:39990 bh 1 wr_id 683ba0. [A 20:00:41.544113] troy.scl@10.1.4.56:39990 H=0 S=0x6458d0: getconfig: finish (Success) [D 20:00:41.544126] PINT_encode_release [D 20:00:41.544133] lebf_encode_rel [D 20:00:41.544140] [SM Exiting]: (0x6457d0) final_response_sm:cleanup (error code: 0), (action: COMPLETE) [D 20:00:41.544147] [SM pop_state]: (0x6457d0) op-id: 13 stk-ptr: 1 base-frm: 0 [D 20:00:41.544154] [SM Entering]: (0x6457d0) get_config_sm:cleanup (status: 0) [D 20:00:41.544161] [SM frame get]: (0x6457d0) op-id: 13 index: 0 base-frm: 0 [D 20:00:41.544168] server_state_machine_complete 0x6457d0 [D 20:00:41.544174] PINT_decode_release [D 20:00:41.544180] lebf_decode_rel [D 20:00:41.544186] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:00:41.544193] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.544200] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 0 [D 20:00:41.544207] [SM Exiting]: (0x6457d0) get_config_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:00:41.544214] server_state_machine_terminate 0x6457d0 [D 20:00:41.544224] [SM Entering]: (0x646a90) unexpected_sm:map_request (status: 0) [D 20:00:41.544231] [SM frame get]: (0x646a90) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.544238] server_post_unexpected_recv [D 20:00:41.544247] [SM Frame PUSH]: (0x647c20) frame: 0x6457d0 [D 20:00:41.544253] [SM Locating]: (0x647c20) op-id: 999 [D 20:00:41.544260] server_op_state_get_machine 999 [D 20:00:41.544266] [SM Locating]: (0x647c20) located: unexpected_sm:post_unexpected [D 20:00:41.544273] [SM frame get]: (0x647c20) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.544280] [SM Entering]: (0x647c20) unexpected_sm:post_unexpected (status: 0) [D 20:00:41.544286] [SM frame get]: (0x647c20) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.544295] [SM Exiting]: (0x647c20) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:41.544303] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:41.544309] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.544315] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 1 [D 20:00:41.544322] [SM frame get]: (0x646a90) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.544328] server_state_machine_start 0x646a90 [D 20:00:41.544334] PINT_decode [D 20:00:41.544341] lebf_decode_req [D 20:00:41.544347] [SM Locating]: (0x646a90) op-id: 6 [D 20:00:41.544353] server_op_state_get_machine 6 [D 20:00:41.544360] [SM push_state]: (0x646a90) op-id: 6 stk-ptr: 0 base-frm: 0 [D 20:00:41.544366] [SM Locating]: (0x646a90) located: prelude_sm:setup [D 20:00:41.544373] [SM Entering]: (0x646a90) prelude_sm:setup (status: 0) [D 20:00:41.544380] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.544389] [SM Exiting]: (0x646a90) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:41.544396] [SM Exiting]: (0x646a90) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:41.544403] [SM push_state]: (0x646a90) op-id: 6 stk-ptr: 1 base-frm: 0 [D 20:00:41.544410] [SM Entering]: (0x646a90) prelude_work_sm:req_sched (status: 0) [D 20:00:41.544417] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.544423] (0x646b90) lookup_path (prelude sm) state: req_sched [A 20:00:41.550105] troy.scl@10.1.4.56:39990 H=138 S=0x646b90: lookup_path: request [D 20:00:41.550118] REQ SCHED POSTING, handle: 138, queue_element: 0x694160 [D 20:00:41.550126] REQ SCHED SCHEDULING, handle: 138, queue_element: 0x694160 [D 20:00:41.550133] [SM Exiting]: (0x646a90) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:41.550140] [SM Entering]: (0x646a90) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:41.550147] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [A 20:00:41.555977] troy.scl@10.1.4.56:39990 H=138 S=0x646b90: lookup_path: start [D 20:00:41.555988] (0x646b90) lookup_path (prelude sm) state: getattr_if_needed [D 20:00:41.555996] About to retrieve attributes for handle 138 [D 20:00:41.556013] op_queue add: 0x694520 [D 20:00:41.556023] [SM Exiting]: (0x646a90) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:41.556041] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:41.556053] dbpf_open_cache_get: called [D 20:00:41.556065] dbpf_open_cache_get: resetting entry from free list. [D 20:00:41.556077] dbpf_open_cache open_fd: opening fd 138 [D 20:00:41.556090] dbpf_open_cache open_fd: filename: //scratch/1/pvfs-da13-test/00000088/bstreams/00000010/0000008a.bstream [D 20:00:41.556119] dbpf_open_cache_get: could not open (ret=-1073742082) [D 20:00:41.556151] ATTRIB: retrieved attributes from DISK for key 138 uid = 0, mode = 511, type = 4, dfile_count = 0, dist_size = 0 b_size = 0 [D 20:00:41.556162] dbpf_attr_cache_insert: inserting 138 (b_size is 0) [D 20:00:41.556170] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:41.556189] op_queue add: 0x694520 [D 20:00:41.556207] [SM Entering]: (0x646a90) prelude_work_sm:perm_check (status: 0) [D 20:00:41.556218] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.556225] PVFS operation "lookup_path" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:41.556234] Final permission check for "lookup_path" set error code to 0 [D 20:00:41.556241] (0x646b90) lookup_path (prelude sm) state: perm_check (status = 0) [D 20:00:41.556247] [SM Exiting]: (0x646a90) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:41.556254] [SM pop_state]: (0x646a90) op-id: 6 stk-ptr: 2 base-frm: 0 [D 20:00:41.556270] [SM pop_state]: (0x646a90) op-id: 6 stk-ptr: 1 base-frm: 0 [D 20:00:41.556276] [SM Entering]: (0x646a90) lookup_sm:init (status: 0) [D 20:00:41.556285] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.556293] STARTING LOOKUP REQUEST (path:qtest.F08)(fs_id:136)(handle:138)(attrmask:127)(# of segments:1) [D 20:00:41.556301] [SM Exiting]: (0x646a90) lookup_sm:init (error code: 0), (action: COMPLETE) [D 20:00:41.556323] [SM Entering]: (0x646a90) lookup_sm:read_object_metadata (status: 0) [D 20:00:41.556335] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.556343] dspace_getattr fast path attr cache hit on 138 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:41.556352] [SM Exiting]: (0x646a90) lookup_sm:read_object_metadata (error code: 0), (action: COMPLETE) [D 20:00:41.556359] [SM Entering]: (0x646a90) lookup_sm:verify_object_metadata (status: 0) [D 20:00:41.556366] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.556374] attrs = (owner = 0, group = 0, perms = 777, type = 4) [D 20:00:41.556381] - check_mode called --- (uid=2840,gid=4001,access_type=1) [D 20:00:41.556387] - object attributes --- (uid=0,gid=0,mode=511) [D 20:00:41.556397] - checking if uid (2840) is root ... [D 20:00:41.556403] - no [D 20:00:41.556409] - checking if owner (0) matches uid (2840)... [D 20:00:41.556415] - no [D 20:00:41.556421] - checking if permissions (511) allows access type (1) by others... [D 20:00:41.556429] - yes [D 20:00:41.556442] object is a directory; will be looking for handle for segment "qtest.F08" in a bit [D 20:00:41.556448] [SM Exiting]: (0x646a90) lookup_sm:verify_object_metadata (error code: 0), (action: COMPLETE) [D 20:00:41.556460] [SM Entering]: (0x646a90) lookup_sm:read_directory_entry_handle (status: 0) [D 20:00:41.556467] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.556473] reading dirent handle value from handle 138 [D 20:00:41.556484] *** Trove KeyVal Read of de [D 20:00:41.556491] dbpf_cache_elem_lookup: cache elem matching 138 returned (num_elems=1) [D 20:00:41.556501] op_queue add: 0x694520 [D 20:00:41.556510] [SM Exiting]: (0x646a90) lookup_sm:read_directory_entry_handle (error code: 0), (action: DEFERRED) [D 20:00:41.556524] [DBPF THREAD]: [KEYVAL -1]: -1 [D 20:00:41.556532] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (KEYVAL_READ) [D 20:00:41.556578] dbpf_cache_elem_lookup: cache elem matching 138 returned (num_elems=1) [D 20:00:41.556589] ** CANNOT cache data retrieved (key is de) [D 20:00:41.556597] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (KEYVAL_READ) (ret: 1) [D 20:00:41.556604] op_queue add: 0x694520 [D 20:00:41.556621] [SM Entering]: (0x646a90) lookup_sm:read_directory_entry (status: 0) [D 20:00:41.556633] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.556640] reading from dirent handle = 4, segment = qtest.F08 (len=9) [D 20:00:41.556648] *** Trove KeyVal Read of qtest.F08 [D 20:00:41.556655] op_queue add: 0x694520 [D 20:00:41.556663] [SM Exiting]: (0x646a90) lookup_sm:read_directory_entry (error code: 0), (action: DEFERRED) [D 20:00:41.556675] [DBPF THREAD]: [KEYVAL -1]: -2 [D 20:00:41.556688] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (KEYVAL_READ) [D 20:00:41.556708] warning: keyval read error on handle 4 and key= qtest.F08 (DB_NOTFOUND: No matching key/data pair found) [D 20:00:41.556718] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (KEYVAL_READ) (ret: -1073742082) [D 20:00:41.556725] op_queue add: 0x694520 [D 20:00:41.556761] [SM Entering]: (0x646a90) lookup_sm:setup_resp (status: 0) [D 20:00:41.556770] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.556777] lookup error in previous step [D 20:00:41.556783] sending 'error' response with 0 handle(s) and 0 attr(s) [A 20:00:41.562473] troy.scl@10.1.4.56:39990 H=138 S=0x646b90: lookup_path: path: qtest.F08, lookup failed [D 20:00:41.562485] [SM Exiting]: (0x646a90) lookup_sm:setup_resp (error code: -1073742082), (action: COMPLETE) [D 20:00:41.562494] [SM push_state]: (0x646a90) op-id: 6 stk-ptr: 0 base-frm: 0 [D 20:00:41.562502] [SM Entering]: (0x646a90) final_response_sm:release (status: 0) [D 20:00:41.562509] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.562520] (s_op 0x646b90) lookup_path (FR sm) state: release: (error_code = -1073742082) [D 20:00:41.562528] REQ SCHED RELEASING, handle: 138, queue_element: 0x694160 [D 20:00:41.562537] [SM Exiting]: (0x646a90) final_response_sm:release (error code: 0), (action: COMPLETE) [D 20:00:41.562544] [SM Entering]: (0x646a90) final_response_sm:send_resp (status: 0) [D 20:00:41.562551] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.562558] (s_op 0x646b90) lookup_path (FR sm) state: send_resp (status = -1073741826) [D 20:00:41.562565] PINT_encode [D 20:00:41.562571] encode_common [D 20:00:41.562687] lebf_encode_resp [D 20:00:41.562698] BMI_post_send_list: addr: 6843072, count: 1, total_size: 16, tag: 2 [D 20:00:41.562706] element 0: offset: 0x6a64a0, size: 16 [D 20:00:41.562714] post_sr: 10.1.4.56:39990 bh 1 len 32 wr_id 6828d0. [D 20:00:41.562724] [SM Exiting]: (0x646a90) final_response_sm:send_resp (error code: 0), (action: DEFERRED) -[D 20:00:41.562752] openib_check_cq: completed send wr_id 6828d0. [D 20:00:41.562770] [SM Entering]: (0x646a90) final_response_sm:cleanup (status: 0) [D 20:00:41.562780] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.562787] (s_op 0x646b90) lookup_path (FR sm) state: cleanup [D 20:00:41.565034] openib_check_cq: completed recv wr_id 683bd0. [D 20:00:41.565046] openib_post_rr: 10.1.4.56:39990 bh 2 wr_id 683bd0. [A 20:00:41.568596] troy.scl@10.1.4.56:39990 H=138 S=0x646b90: lookup_path: finish (No such file or directory) [D 20:00:41.568608] PINT_encode_release [D 20:00:41.568615] lebf_encode_rel [D 20:00:41.568621] [SM Exiting]: (0x646a90) final_response_sm:cleanup (error code: 0), (action: COMPLETE) [D 20:00:41.568629] [SM pop_state]: (0x646a90) op-id: 6 stk-ptr: 1 base-frm: 0 [D 20:00:41.568636] [SM Entering]: (0x646a90) lookup_sm:cleanup (status: 0) [D 20:00:41.568642] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.568649] [SM frame get]: (0x646a90) op-id: 6 index: 0 base-frm: 0 [D 20:00:41.568656] server_state_machine_complete 0x646a90 [D 20:00:41.568662] PINT_decode_release [D 20:00:41.568668] lebf_decode_rel [D 20:00:41.568673] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:00:41.568680] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.568686] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 0 [D 20:00:41.568693] [SM Exiting]: (0x646a90) lookup_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:00:41.568700] server_state_machine_terminate 0x646a90 [D 20:00:41.568709] [SM Entering]: (0x647d50) unexpected_sm:map_request (status: 0) [D 20:00:41.568717] [SM frame get]: (0x647d50) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.568724] server_post_unexpected_recv [D 20:00:41.568732] [SM Frame PUSH]: (0x648ee0) frame: 0x646a90 [D 20:00:41.568738] [SM Locating]: (0x648ee0) op-id: 999 [D 20:00:41.568745] server_op_state_get_machine 999 [D 20:00:41.568751] [SM Locating]: (0x648ee0) located: unexpected_sm:post_unexpected [D 20:00:41.568757] [SM frame get]: (0x648ee0) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.568764] [SM Entering]: (0x648ee0) unexpected_sm:post_unexpected (status: 0) [D 20:00:41.568771] [SM frame get]: (0x648ee0) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.568779] [SM Exiting]: (0x648ee0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:41.568786] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:41.568793] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.568799] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 1 [D 20:00:41.568806] [SM frame get]: (0x647d50) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.568812] server_state_machine_start 0x647d50 [D 20:00:41.568818] PINT_decode [D 20:00:41.568825] lebf_decode_req [D 20:00:41.568833] [SM Locating]: (0x647d50) op-id: 4 [D 20:00:41.568840] server_op_state_get_machine 4 [D 20:00:41.568847] [SM push_state]: (0x647d50) op-id: 4 stk-ptr: 0 base-frm: 0 [D 20:00:41.568854] [SM Locating]: (0x647d50) located: prelude_sm:setup [D 20:00:41.568861] [SM Entering]: (0x647d50) prelude_sm:setup (status: 0) [D 20:00:41.568868] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.568876] [SM Exiting]: (0x647d50) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:41.568884] [SM Exiting]: (0x647d50) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:41.568891] [SM push_state]: (0x647d50) op-id: 4 stk-ptr: 1 base-frm: 0 [D 20:00:41.568897] [SM Entering]: (0x647d50) prelude_work_sm:req_sched (status: 0) [D 20:00:41.568904] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.568910] (0x647e50) getattr (prelude sm) state: req_sched [A 20:00:41.574717] troy.scl@10.1.4.56:39990 H=138 S=0x647e50: getattr: request [D 20:00:41.574729] REQ SCHED POSTING, handle: 138, queue_element: 0x6943a0 [D 20:00:41.574737] REQ SCHED SCHEDULING, handle: 138, queue_element: 0x6943a0 [D 20:00:41.574744] [SM Exiting]: (0x647d50) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:41.574752] [SM Entering]: (0x647d50) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:41.574759] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [A 20:00:41.782853] troy.scl@10.1.4.56:39990 H=138 S=0x647e50: getattr: start [D 20:00:41.782863] (0x647e50) getattr (prelude sm) state: getattr_if_needed [D 20:00:41.782871] About to retrieve attributes for handle 138 [D 20:00:41.782879] dspace_getattr fast path attr cache hit on 138 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:41.782888] [SM Exiting]: (0x647d50) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:41.782895] [SM Entering]: (0x647d50) prelude_work_sm:perm_check (status: 0) [D 20:00:41.782902] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.782909] PVFS operation "getattr" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:41.782918] Final permission check for "getattr" set error code to 0 [D 20:00:41.782924] (0x647e50) getattr (prelude sm) state: perm_check (status = 0) [D 20:00:41.782931] [SM Exiting]: (0x647d50) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:41.782938] [SM pop_state]: (0x647d50) op-id: 4 stk-ptr: 2 base-frm: 0 [D 20:00:41.782945] [SM pop_state]: (0x647d50) op-id: 4 stk-ptr: 1 base-frm: 0 [D 20:00:41.782952] [SM Entering]: (0x647d50) get_attr_sm:setup_op (status: 0) [D 20:00:41.782962] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.782969] [SM Exiting]: (0x647d50) get_attr_sm:setup_op (error code: 0), (action: COMPLETE) [D 20:00:41.782976] [SM push_state]: (0x647d50) op-id: 4 stk-ptr: 0 base-frm: 0 [D 20:00:41.782983] [SM Entering]: (0x647d50) get_attr_work_sm:verify_attribs (status: 0) [D 20:00:41.782991] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.782999] No version found! Using mtime 1208221235 [A 20:00:41.788975] troy.scl@10.1.4.56:39990 H=138 S=0x647e50: getattr: type: directory [D 20:00:41.788986] getattr: dirent_count not needed. [D 20:00:41.788993] getattr: dfile_count needed. [D 20:00:41.788999] [SM Exiting]: (0x647d50) get_attr_work_sm:verify_attribs (error code: 10), (action: COMPLETE) [D 20:00:41.789007] [SM Entering]: (0x647d50) get_attr_work_sm:get_dirdata_handle (status: 0) [D 20:00:41.789014] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.789022] *** Trove KeyVal Read of de [D 20:00:41.789029] dbpf_cache_elem_lookup: cache elem matching 138 returned (num_elems=1) [D 20:00:41.789037] op_queue add: 0x694670 [D 20:00:41.789045] [SM Exiting]: (0x647d50) get_attr_work_sm:get_dirdata_handle (error code: 10), (action: DEFERRED) [D 20:00:41.789058] [DBPF THREAD]: [KEYVAL -1]: -3 [D 20:00:41.789066] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (KEYVAL_READ) [D 20:00:41.789080] dbpf_cache_elem_lookup: cache elem matching 138 returned (num_elems=1) [D 20:00:41.789088] ** CANNOT cache data retrieved (key is de) [D 20:00:41.789094] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (KEYVAL_READ) (ret: 1) [D 20:00:41.789101] op_queue add: 0x694670 [D 20:00:41.789137] [SM Entering]: (0x647d50) get_attr_work_sm:get_dirent_count (status: 0) [D 20:00:41.789146] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.789153] [SM Exiting]: (0x647d50) get_attr_work_sm:get_dirent_count (error code: 11), (action: COMPLETE) [D 20:00:41.789161] [SM Entering]: (0x647d50) get_attr_work_sm:get_dir_hint (status: 0) [D 20:00:41.789167] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.789174] trying to getxattr of user.pvfs2.dist_name,user.pvfs2.dist_params,user.pvfs2.num_dfiles of dir handle (coll_id = 136, handle = 138 [D 20:00:41.789189] op_queue add: 0x694670 [D 20:00:41.789198] [SM Exiting]: (0x647d50) get_attr_work_sm:get_dir_hint (error code: 0), (action: DEFERRED) [D 20:00:41.789251] [DBPF THREAD]: [KEYVAL -1]: -4 [D 20:00:41.789261] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (KEYVAL_READ_LIST) [D 20:00:41.789272] keyval get user.pvfs2.dist_name failed with error DB_NOTFOUND: No matching key/data pair found [D 20:00:41.789283] keyval get user.pvfs2.dist_params failed with error DB_NOTFOUND: No matching key/data pair found [D 20:00:41.789293] keyval get user.pvfs2.num_dfiles failed with error DB_NOTFOUND: No matching key/data pair found [D 20:00:41.789301] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (KEYVAL_READ_LIST) (ret: -1073742082) [D 20:00:41.789308] op_queue add: 0x694670 [D 20:00:41.789325] [SM Entering]: (0x647d50) get_attr_work_sm:interpret_dir_hint (status: 0) [D 20:00:41.789335] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.789341] getattr: job status code = -1073742082 [D 20:00:41.789348] getattr: dir hint dfile_count: 0 [D 20:00:41.789354] [SM Exiting]: (0x647d50) get_attr_work_sm:interpret_dir_hint (error code: 0), (action: COMPLETE) [D 20:00:41.789361] [SM Entering]: (0x647d50) get_attr_work_sm:setup_resp (status: 0) [D 20:00:41.789368] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.789374] - RETURNING retrieved attrs: [owner = 0, group = 0 perms = 777, type = 4, atime = 1208221235, mtime = 1208221235 ctime = 1208221235, dist_size = 0] [D 20:00:41.789383] server returning dirent_count = 0 dfile_count = 0 dist_name_len = 0 dist_params_len = 0 [D 20:00:41.789390] @ End DIRECTORY attributes: sending status 0 (error = 0) [D 20:00:41.789397] [SM Exiting]: (0x647d50) get_attr_work_sm:setup_resp (error code: 0), (action: COMPLETE) [D 20:00:41.789403] [SM pop_state]: (0x647d50) op-id: 4 stk-ptr: 1 base-frm: 0 [D 20:00:41.789410] [SM push_state]: (0x647d50) op-id: 4 stk-ptr: 0 base-frm: 0 [D 20:00:41.789416] [SM Entering]: (0x647d50) final_response_sm:release (status: 0) [D 20:00:41.789423] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.789434] (s_op 0x647e50) getattr (FR sm) state: release: (error_code = 0) [D 20:00:41.789443] REQ SCHED RELEASING, handle: 138, queue_element: 0x6943a0 [D 20:00:41.789451] [SM Exiting]: (0x647d50) final_response_sm:release (error code: 0), (action: COMPLETE) [D 20:00:41.789458] [SM Entering]: (0x647d50) final_response_sm:send_resp (status: 0) [D 20:00:41.789465] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.789471] (s_op 0x647e50) getattr (FR sm) state: send_resp (status = 0) [D 20:00:41.789478] PINT_encode [D 20:00:41.789484] encode_common [D 20:00:41.789656] lebf_encode_resp [D 20:00:41.789668] BMI_post_send_list: addr: 6843072, count: 1, total_size: 108, tag: 3 [D 20:00:41.789676] element 0: offset: 0x6a88e0, size: 108 [D 20:00:41.789690] post_sr: 10.1.4.56:39990 bh 2 len 124 wr_id 682900. [D 20:00:41.789699] [SM Exiting]: (0x647d50) final_response_sm:send_resp (error code: 0), (action: DEFERRED) -[D 20:00:41.789714] openib_check_cq: completed send wr_id 682900. [D 20:00:41.789758] [SM Entering]: (0x647d50) final_response_sm:cleanup (status: 0) [D 20:00:41.789769] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.789776] (s_op 0x647e50) getattr (FR sm) state: cleanup [D 20:00:41.791527] openib_check_cq: completed recv wr_id 683c00. [D 20:00:41.791537] openib_post_rr: 10.1.4.56:39990 bh 3 wr_id 683c00. [A 20:00:41.795597] troy.scl@10.1.4.56:39990 H=138 S=0x647e50: getattr: finish (Success) [D 20:00:41.795608] PINT_encode_release [D 20:00:41.795615] lebf_encode_rel [D 20:00:41.795621] [SM Exiting]: (0x647d50) final_response_sm:cleanup (error code: 0), (action: COMPLETE) [D 20:00:41.795629] [SM pop_state]: (0x647d50) op-id: 4 stk-ptr: 1 base-frm: 0 [D 20:00:41.795636] [SM Entering]: (0x647d50) get_attr_sm:cleanup (status: 0) [D 20:00:41.795642] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.795650] [SM frame get]: (0x647d50) op-id: 4 index: 0 base-frm: 0 [D 20:00:41.795656] server_state_machine_complete 0x647d50 [D 20:00:41.795662] PINT_decode_release [D 20:00:41.795668] lebf_decode_rel [D 20:00:41.795674] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:00:41.795681] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.795687] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 0 [D 20:00:41.795694] [SM Exiting]: (0x647d50) get_attr_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:00:41.795701] server_state_machine_terminate 0x647d50 [D 20:00:41.795710] [SM Entering]: (0x649010) unexpected_sm:map_request (status: 0) [D 20:00:41.795717] [SM frame get]: (0x649010) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.795724] server_post_unexpected_recv [D 20:00:41.795732] [SM Frame PUSH]: (0x64a1a0) frame: 0x647d50 [D 20:00:41.795739] [SM Locating]: (0x64a1a0) op-id: 999 [D 20:00:41.795745] server_op_state_get_machine 999 [D 20:00:41.795751] [SM Locating]: (0x64a1a0) located: unexpected_sm:post_unexpected [D 20:00:41.795757] [SM frame get]: (0x64a1a0) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.795764] [SM Entering]: (0x64a1a0) unexpected_sm:post_unexpected (status: 0) [D 20:00:41.795771] [SM frame get]: (0x64a1a0) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.795779] [SM Exiting]: (0x64a1a0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:41.795786] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:41.795793] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.795799] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 1 [D 20:00:41.795806] [SM frame get]: (0x649010) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.795812] server_state_machine_start 0x649010 [D 20:00:41.795818] PINT_decode [D 20:00:41.795824] lebf_decode_req [D 20:00:41.795831] [SM Locating]: (0x649010) op-id: 1 [D 20:00:41.795837] server_op_state_get_machine 1 [D 20:00:41.795843] [SM push_state]: (0x649010) op-id: 1 stk-ptr: 0 base-frm: 0 [D 20:00:41.795850] [SM Locating]: (0x649010) located: prelude_sm:setup [D 20:00:41.795856] [SM Entering]: (0x649010) prelude_sm:setup (status: 0) [D 20:00:41.795863] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.795869] [SM Exiting]: (0x649010) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:41.795876] [SM Exiting]: (0x649010) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:41.795883] [SM push_state]: (0x649010) op-id: 1 stk-ptr: 1 base-frm: 0 [D 20:00:41.795890] [SM Entering]: (0x649010) prelude_work_sm:req_sched (status: 0) [D 20:00:41.795896] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.795903] (0x649110) create (prelude sm) state: req_sched [A 20:00:41.801843] troy.scl@10.1.4.56:39990 H=0 S=0x649110: create: request [D 20:00:41.801855] [SM Exiting]: (0x649010) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:41.801863] [SM Entering]: (0x649010) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:41.801870] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [A 20:00:41.807839] troy.scl@10.1.4.56:39990 H=0 S=0x649110: create: start [D 20:00:41.807850] (0x649110) create (prelude sm) state: getattr_if_needed [D 20:00:41.807857] [SM Exiting]: (0x649010) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:41.807865] [SM Entering]: (0x649010) prelude_work_sm:perm_check (status: 0) [D 20:00:41.807872] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.807879] PVFS operation "create" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:41.807887] Final permission check for "create" set error code to 0 [D 20:00:41.807894] (0x649110) create (prelude sm) state: perm_check (status = 0) [D 20:00:41.807900] [SM Exiting]: (0x649010) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:41.807907] [SM pop_state]: (0x649010) op-id: 1 stk-ptr: 2 base-frm: 0 [D 20:00:41.807913] [SM pop_state]: (0x649010) op-id: 1 stk-ptr: 1 base-frm: 0 [D 20:00:41.807920] [SM Entering]: (0x649010) create_sm:create (status: 0) [D 20:00:41.807927] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.807938] op_queue add: 0x6944e0 [D 20:00:41.807946] [SYNC_COALESCE]: enqueue called [D 20:00:41.807952] [SYNC_COALESCE]: enqueue 1 counter sync:1 non_sync:0 [D 20:00:41.807961] [SM Exiting]: (0x649010) create_sm:create (error code: 0), (action: DEFERRED) [D 20:00:41.807973] [SYNC_COALESCE]: dequeue called [D 20:00:41.807981] [SYNC_COALESCE]: dequeue 1 counter sync:0 non_sync:0 [D 20:00:41.807988] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_CREATE) [D 20:00:41.808000] [1 extents] -- new_handle is 146 (cur_extent is 4 - 1431655767) [D 20:00:41.808087] dbpf_attr_cache_insert: inserting 146 (b_size is 0) [D 20:00:41.808098] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_CREATE) (ret: 1) [D 20:00:41.808105] [SYNC_COALESCE]: sync_coalesce called, handle: 0, cid: 0 [D 20:00:41.808112] [SYNC_COALESCE]: sync_context: 0x5bfec0 [D 20:00:41.808118] [SYNC_COALESCE]: sync context type is DSPACE [D 20:00:41.808124] [SYNC_COALESCE]: high or low watermark reached: coalesced: 0 queued: 0 [D 20:00:41.808131] [SYNC_COALESCE]: syncing now! [D 20:00:41.808139] [SYNC_COALESCE]: coalesce 1 sync start in coalesce_queue:0 pending:0 [D 20:00:41.808261] [SYNC_COALESCE]: coalesce 1 sync stop [D 20:00:41.808272] [SYNC_COALESCE]: moving op 0x6944e0 with handle: 0 to completion queue [D 20:00:41.808280] op_queue add: 0x6944e0 [D 20:00:41.808309] [SM Entering]: (0x649010) create_sm:setup_resp (status: 0) [D 20:00:41.808319] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.808326] Handle created: 146 [A 20:00:41.814336] troy.scl@10.1.4.56:39990 H=0 S=0x649110: create: new handle: 146, type metafile. [D 20:00:41.814348] [SM Exiting]: (0x649010) create_sm:setup_resp (error code: 0), (action: COMPLETE) [D 20:00:41.814357] [SM push_state]: (0x649010) op-id: 1 stk-ptr: 0 base-frm: 0 [D 20:00:41.814364] [SM Entering]: (0x649010) final_response_sm:release (status: 0) [D 20:00:41.814371] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.814378] (s_op 0x649110) create (FR sm) state: release: (error_code = 0) [D 20:00:41.814387] [SM Exiting]: (0x649010) final_response_sm:release (error code: 0), (action: COMPLETE) [D 20:00:41.814394] [SM Entering]: (0x649010) final_response_sm:send_resp (status: 0) [D 20:00:41.814401] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.814413] (s_op 0x649110) create (FR sm) state: send_resp (status = 0) [D 20:00:41.814420] PINT_encode [D 20:00:41.814426] encode_common [D 20:00:41.814432] lebf_encode_resp [D 20:00:41.814439] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 4 [D 20:00:41.814447] element 0: offset: 0x694020, size: 24 [D 20:00:41.814454] post_sr: 10.1.4.56:39990 bh 3 len 40 wr_id 682930. [D 20:00:41.814463] [SM Exiting]: (0x649010) final_response_sm:send_resp (error code: 0), (action: DEFERRED) -[D 20:00:41.814499] openib_check_cq: completed send wr_id 682930. [D 20:00:41.814517] [SM Entering]: (0x649010) final_response_sm:cleanup (status: 0) [D 20:00:41.814527] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.814533] (s_op 0x649110) create (FR sm) state: cleanup [D 20:00:41.815706] openib_check_cq: completed recv wr_id 683c30. [D 20:00:41.815716] openib_post_rr: 10.1.4.56:39990 bh 4 wr_id 683c30. [A 20:00:41.820582] troy.scl@10.1.4.56:39990 H=0 S=0x649110: create: finish (Success) [D 20:00:41.820594] PINT_encode_release [D 20:00:41.820602] lebf_encode_rel [D 20:00:41.820608] [SM Exiting]: (0x649010) final_response_sm:cleanup (error code: 0), (action: COMPLETE) [D 20:00:41.820616] [SM pop_state]: (0x649010) op-id: 1 stk-ptr: 1 base-frm: 0 [D 20:00:41.820623] [SM Entering]: (0x649010) create_sm:cleanup (status: 0) [D 20:00:41.820630] [SM frame get]: (0x649010) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.820637] server_state_machine_complete 0x649010 [D 20:00:41.820643] PINT_decode_release [D 20:00:41.820652] lebf_decode_rel [D 20:00:41.820660] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:00:41.820667] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.820674] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 0 [D 20:00:41.820681] [SM Exiting]: (0x649010) create_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:00:41.820693] server_state_machine_terminate 0x649010 [D 20:00:41.820702] [SM Entering]: (0x64a2d0) unexpected_sm:map_request (status: 0) [D 20:00:41.820709] [SM frame get]: (0x64a2d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.820716] server_post_unexpected_recv [D 20:00:41.820724] [SM Frame PUSH]: (0x64b460) frame: 0x649010 [D 20:00:41.820731] [SM Locating]: (0x64b460) op-id: 999 [D 20:00:41.820737] server_op_state_get_machine 999 [D 20:00:41.820744] [SM Locating]: (0x64b460) located: unexpected_sm:post_unexpected [D 20:00:41.820752] [SM frame get]: (0x64b460) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.820759] [SM Entering]: (0x64b460) unexpected_sm:post_unexpected (status: 0) [D 20:00:41.820766] [SM frame get]: (0x64b460) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.820774] [SM Exiting]: (0x64b460) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:41.820782] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:41.820796] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.820803] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 1 [D 20:00:41.820810] [SM frame get]: (0x64a2d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.820817] server_state_machine_start 0x64a2d0 [D 20:00:41.820823] PINT_decode [D 20:00:41.820829] lebf_decode_req [D 20:00:41.820835] [SM Locating]: (0x64a2d0) op-id: 1 [D 20:00:41.820847] server_op_state_get_machine 1 [D 20:00:41.820853] [SM push_state]: (0x64a2d0) op-id: 1 stk-ptr: 0 base-frm: 0 [D 20:00:41.820860] [SM Locating]: (0x64a2d0) located: prelude_sm:setup [D 20:00:41.820867] [SM Entering]: (0x64a2d0) prelude_sm:setup (status: 0) [D 20:00:41.820878] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.820885] [SM Exiting]: (0x64a2d0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:41.820892] [SM Exiting]: (0x64a2d0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:41.820902] [SM push_state]: (0x64a2d0) op-id: 1 stk-ptr: 1 base-frm: 0 [D 20:00:41.820910] [SM Entering]: (0x64a2d0) prelude_work_sm:req_sched (status: 0) [D 20:00:41.820931] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.820940] (0x64a3d0) create (prelude sm) state: req_sched [A 20:00:41.826829] troy.scl@10.1.4.56:39990 H=0 S=0x64a3d0: create: request [D 20:00:41.826841] [SM Exiting]: (0x64a2d0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:41.826850] [SM Entering]: (0x64a2d0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:41.826857] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [A 20:00:41.832825] troy.scl@10.1.4.56:39990 H=0 S=0x64a3d0: create: start [D 20:00:41.832836] (0x64a3d0) create (prelude sm) state: getattr_if_needed [D 20:00:41.832845] [SM Exiting]: (0x64a2d0) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:41.832852] [SM Entering]: (0x64a2d0) prelude_work_sm:perm_check (status: 0) [D 20:00:41.832859] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.832866] PVFS operation "create" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:41.832875] Final permission check for "create" set error code to 0 [D 20:00:41.832881] (0x64a3d0) create (prelude sm) state: perm_check (status = 0) [D 20:00:41.832888] [SM Exiting]: (0x64a2d0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:41.832895] [SM pop_state]: (0x64a2d0) op-id: 1 stk-ptr: 2 base-frm: 0 [D 20:00:41.832906] [SM pop_state]: (0x64a2d0) op-id: 1 stk-ptr: 1 base-frm: 0 [D 20:00:41.832912] [SM Entering]: (0x64a2d0) create_sm:create (status: 0) [D 20:00:41.832919] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.832928] op_queue add: 0x6a8510 [D 20:00:41.832940] [SYNC_COALESCE]: enqueue called [D 20:00:41.832946] [SYNC_COALESCE]: enqueue 1 counter sync:1 non_sync:0 [D 20:00:41.832954] [SM Exiting]: (0x64a2d0) create_sm:create (error code: 0), (action: DEFERRED) [D 20:00:41.832969] [SYNC_COALESCE]: dequeue called [D 20:00:41.832977] [SYNC_COALESCE]: dequeue 1 counter sync:0 non_sync:0 [D 20:00:41.832983] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_CREATE) [D 20:00:41.832992] [1 extents] -- new_handle is 2863311531 (cur_extent is 1431655768 - 2863311531) [D 20:00:41.833020] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 0) [D 20:00:41.833035] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_CREATE) (ret: 1) [D 20:00:41.833043] [SYNC_COALESCE]: sync_coalesce called, handle: 0, cid: 0 [D 20:00:41.833050] [SYNC_COALESCE]: sync_context: 0x5bfec0 [D 20:00:41.833056] [SYNC_COALESCE]: sync context type is DSPACE [D 20:00:41.833062] [SYNC_COALESCE]: high or low watermark reached: coalesced: 0 queued: 0 [D 20:00:41.833069] [SYNC_COALESCE]: syncing now! [D 20:00:41.833074] [SYNC_COALESCE]: coalesce 1 sync start in coalesce_queue:0 pending:0 [D 20:00:41.833138] [SYNC_COALESCE]: coalesce 1 sync stop [D 20:00:41.833148] [SYNC_COALESCE]: moving op 0x6a8510 with handle: 0 to completion queue [D 20:00:41.833169] op_queue add: 0x6a8510 [D 20:00:41.833204] [SM Entering]: (0x64a2d0) create_sm:setup_resp (status: 0) [D 20:00:41.833213] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.833220] Handle created: 2863311531 [A 20:00:41.839198] troy.scl@10.1.4.56:39990 H=0 S=0x64a3d0: create: new handle: 2863311531, type datafile. [D 20:00:41.839209] [SM Exiting]: (0x64a2d0) create_sm:setup_resp (error code: 0), (action: COMPLETE) [D 20:00:41.839219] [SM push_state]: (0x64a2d0) op-id: 1 stk-ptr: 0 base-frm: 0 [D 20:00:41.839226] [SM Entering]: (0x64a2d0) final_response_sm:release (status: 0) [D 20:00:41.839232] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.839239] (s_op 0x64a3d0) create (FR sm) state: release: (error_code = 0) [D 20:00:41.839248] [SM Exiting]: (0x64a2d0) final_response_sm:release (error code: 0), (action: COMPLETE) [D 20:00:41.839255] [SM Entering]: (0x64a2d0) final_response_sm:send_resp (status: 0) [D 20:00:41.839262] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.839275] (s_op 0x64a3d0) create (FR sm) state: send_resp (status = 0) [D 20:00:41.839281] PINT_encode [D 20:00:41.839287] encode_common [D 20:00:41.839293] lebf_encode_resp [D 20:00:41.839300] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 5 [D 20:00:41.839307] element 0: offset: 0x694400, size: 24 [D 20:00:41.839315] post_sr: 10.1.4.56:39990 bh 4 len 40 wr_id 682960. [D 20:00:41.839324] [SM Exiting]: (0x64a2d0) final_response_sm:send_resp (error code: 0), (action: DEFERRED) -[D 20:00:41.839360] openib_check_cq: completed send wr_id 682960. [D 20:00:41.839378] [SM Entering]: (0x64a2d0) final_response_sm:cleanup (status: 0) [D 20:00:41.839387] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.839394] (s_op 0x64a3d0) create (FR sm) state: cleanup [D 20:00:41.840582] openib_check_cq: completed recv wr_id 683c60. [D 20:00:41.840593] openib_post_rr: 10.1.4.56:39990 bh 5 wr_id 683c60. [A 20:00:41.845320] troy.scl@10.1.4.56:39990 H=0 S=0x64a3d0: create: finish (Success) [D 20:00:41.845331] PINT_encode_release [D 20:00:41.845338] lebf_encode_rel [D 20:00:41.845344] [SM Exiting]: (0x64a2d0) final_response_sm:cleanup (error code: 0), (action: COMPLETE) [D 20:00:41.845352] [SM pop_state]: (0x64a2d0) op-id: 1 stk-ptr: 1 base-frm: 0 [D 20:00:41.845358] [SM Entering]: (0x64a2d0) create_sm:cleanup (status: 0) [D 20:00:41.845365] [SM frame get]: (0x64a2d0) op-id: 1 index: 0 base-frm: 0 [D 20:00:41.845372] server_state_machine_complete 0x64a2d0 [D 20:00:41.845378] PINT_decode_release [D 20:00:41.845384] lebf_decode_rel [D 20:00:41.845389] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:00:41.845396] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.845402] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 0 [D 20:00:41.845409] [SM Exiting]: (0x64a2d0) create_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:00:41.845417] server_state_machine_terminate 0x64a2d0 [D 20:00:41.845425] [SM Entering]: (0x64b590) unexpected_sm:map_request (status: 0) [D 20:00:41.845433] [SM frame get]: (0x64b590) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.845439] server_post_unexpected_recv [D 20:00:41.845448] [SM Frame PUSH]: (0x64c720) frame: 0x64a2d0 [D 20:00:41.845454] [SM Locating]: (0x64c720) op-id: 999 [D 20:00:41.845461] server_op_state_get_machine 999 [D 20:00:41.845467] [SM Locating]: (0x64c720) located: unexpected_sm:post_unexpected [D 20:00:41.845473] [SM frame get]: (0x64c720) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.845480] [SM Entering]: (0x64c720) unexpected_sm:post_unexpected (status: 0) [D 20:00:41.845487] [SM frame get]: (0x64c720) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.845495] [SM Exiting]: (0x64c720) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:41.845502] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:41.845508] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:41.845515] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 1 [D 20:00:41.845521] [SM frame get]: (0x64b590) op-id: 999 index: 0 base-frm: 0 [D 20:00:41.845528] server_state_machine_start 0x64b590 [D 20:00:41.845534] PINT_decode [D 20:00:41.845540] lebf_decode_req [D 20:00:41.845549] [SM Locating]: (0x64b590) op-id: 5 [D 20:00:41.845555] server_op_state_get_machine 5 [D 20:00:41.845562] [SM push_state]: (0x64b590) op-id: 5 stk-ptr: 0 base-frm: 0 [D 20:00:41.845568] [SM Locating]: (0x64b590) located: prelude_sm:setup [D 20:00:41.845574] [SM Entering]: (0x64b590) prelude_sm:setup (status: 0) [D 20:00:41.845581] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:41.845588] [SM Exiting]: (0x64b590) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:41.845595] [SM Exiting]: (0x64b590) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:41.845602] [SM push_state]: (0x64b590) op-id: 5 stk-ptr: 1 base-frm: 0 [D 20:00:41.845608] [SM Entering]: (0x64b590) prelude_work_sm:req_sched (status: 0) [D 20:00:41.845615] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:41.845621] (0x64b690) setattr (prelude sm) state: req_sched [A 20:00:41.851565] troy.scl@10.1.4.56:39990 H=146 S=0x64b690: setattr: request [D 20:00:41.851578] REQ SCHED POSTING, handle: 146, queue_element: 0x6a87a0 [D 20:00:41.851586] REQ SCHED SCHEDULING, handle: 146, queue_element: 0x6a87a0 [D 20:00:41.851593] [SM Exiting]: (0x64b590) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:41.851600] [SM Entering]: (0x64b590) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:41.851607] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [A 20:00:42.062951] troy.scl@10.1.4.56:39990 H=146 S=0x64b690: setattr: start [D 20:00:42.062962] (0x64b690) setattr (prelude sm) state: getattr_if_needed [D 20:00:42.062969] About to retrieve attributes for handle 146 [D 20:00:42.062977] dspace_getattr fast path attr cache hit on 146 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:42.062986] [SM Exiting]: (0x64b590) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:42.062994] [SM Entering]: (0x64b590) prelude_work_sm:perm_check (status: 0) [D 20:00:42.063001] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.063008] PVFS operation "setattr" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:42.063019] Final permission check for "setattr" set error code to 0 [D 20:00:42.063026] (0x64b690) setattr (prelude sm) state: perm_check (status = 0) [D 20:00:42.063033] [SM Exiting]: (0x64b590) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:42.063040] [SM pop_state]: (0x64b590) op-id: 5 stk-ptr: 2 base-frm: 0 [D 20:00:42.063047] [SM pop_state]: (0x64b590) op-id: 5 stk-ptr: 1 base-frm: 0 [D 20:00:42.063053] [SM Entering]: (0x64b590) set_attr_sm:verify_attribs (status: 0) [D 20:00:42.063060] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.063069] attrs read from dspace: [owner = 0, group = 0, perms = 0, type = 1] [D 20:00:42.063077] attrs read from request: [owner = 2840, group = 4001, perms = 600, type = 1] [D 20:00:42.063084] handle 146 refers to a metafile [D 20:00:42.063090] *** dspace has dfile count 0 and req has dfile count 1 [D 20:00:42.063096] *** dspace has dist size 0 and req has dist size 56 [D 20:00:42.063102] *** using dfile_count of 1 [D 20:00:42.063108] *** using dist_size of 56 [D 20:00:42.063114] [SM Exiting]: (0x64b590) set_attr_sm:verify_attribs (error code: 7), (action: COMPLETE) [D 20:00:42.063121] [SM Entering]: (0x64b590) set_attr_sm:write_metafile_datafile_handles_if_required (status: 0) [D 20:00:42.063128] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.063135] request has dfile_count of 1 | dspace has 1 [D 20:00:42.063141] metafile has 1 datafiles associated with it [D 20:00:42.063147] writing dh [146,136,len 8] [D 20:00:42.063156] op_queue add: 0x6943a0 [D 20:00:42.063163] [SYNC_COALESCE]: enqueue called [D 20:00:42.063169] [SYNC_COALESCE]: enqueue 0 counter sync:0 non_sync:1 [D 20:00:42.063177] [SM Exiting]: (0x64b590) set_attr_sm:write_metafile_datafile_handles_if_required (error code: 0), (action: DEFERRED) [D 20:00:42.063190] [SYNC_COALESCE]: dequeue called [D 20:00:42.063197] [SYNC_COALESCE]: dequeue 0 counter sync:0 non_sync:0 [D 20:00:42.063204] [DBPF THREAD]: [KEYVAL -1]: -5 [D 20:00:42.063210] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (KEYVAL_WRITE) [D 20:00:42.063216] dbpf_keyval_write_op_svc: handle: 146, key: dh [D 20:00:42.063224] keyval_db->put(handle= 146, key= dh (3)) size=11 [D 20:00:42.063237] *** Trove KeyVal Write of dh [D 20:00:42.063245] dbpf_cache_elem_lookup: cache elem matching 146 returned (num_elems=3) [D 20:00:42.063251] dbpf_cache_elem_lookup: cache elem matching 146 returned (num_elems=3) [D 20:00:42.063258] Setting data 0x686b90 based on key 146 and key_str dh (data_sz=8) [D 20:00:42.063266] *** cached keyval data written (key is dh) [D 20:00:42.063272] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (KEYVAL_WRITE) (ret: 1) [D 20:00:42.063278] [SYNC_COALESCE]: sync_coalesce called, handle: 146, cid: 0 [D 20:00:42.063285] [SYNC_COALESCE]: sync not needed, moving to completion queue: 146 [D 20:00:42.063291] op_queue add: 0x6943a0 [D 20:00:42.063308] [SM Entering]: (0x64b590) set_attr_sm:write_metafile_distribution_if_required (status: 0) [D 20:00:42.063317] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.063324] metafile distribution size = 56 [D 20:00:42.063331] writing md [146,136,len 56] [D 20:00:42.063340] op_queue add: 0x6943a0 [D 20:00:42.063346] [SYNC_COALESCE]: enqueue called [D 20:00:42.063353] [SYNC_COALESCE]: enqueue 0 counter sync:1 non_sync:0 [D 20:00:42.063360] [SM Exiting]: (0x64b590) set_attr_sm:write_metafile_distribution_if_required (error code: 0), (action: DEFERRED) [D 20:00:42.063372] [SYNC_COALESCE]: dequeue called [D 20:00:42.063380] [SYNC_COALESCE]: dequeue 0 counter sync:0 non_sync:0 [D 20:00:42.063386] [DBPF THREAD]: [KEYVAL -1]: -6 [D 20:00:42.063392] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (KEYVAL_WRITE) [D 20:00:42.063399] dbpf_keyval_write_op_svc: handle: 146, key: md [D 20:00:42.063405] keyval_db->put(handle= 146, key= md (3)) size=11 [D 20:00:42.063417] *** Trove KeyVal Write of md [D 20:00:42.063425] dbpf_cache_elem_lookup: cache elem matching 146 returned (num_elems=3) [D 20:00:42.063431] dbpf_cache_elem_lookup: cache elem matching 146 returned (num_elems=3) [D 20:00:42.063438] ** CANNOT cache data written (key is md) [D 20:00:42.063445] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (KEYVAL_WRITE) (ret: 1) [D 20:00:42.063451] [SYNC_COALESCE]: sync_coalesce called, handle: 146, cid: 0 [D 20:00:42.063457] [SYNC_COALESCE]: sync_context: 0x5bfac0 [D 20:00:42.063464] [SYNC_COALESCE]: sync context type is KEYVAL [D 20:00:42.063469] [SYNC_COALESCE]: high or low watermark reached: coalesced: 0 queued: 0 [D 20:00:42.063476] [SYNC_COALESCE]: syncing now! [D 20:00:42.063481] [SYNC_COALESCE]: coalesce 0 sync start in coalesce_queue:0 pending:0 [D 20:00:42.063614] [SYNC_COALESCE]: coalesce 0 sync stop [D 20:00:42.063624] [SYNC_COALESCE]: moving op 0x6943a0 with handle: 146 to completion queue [D 20:00:42.063632] op_queue add: 0x6943a0 [D 20:00:42.063675] [SM Entering]: (0x64b590) set_attr_sm:setobj_attrib (status: 0) [D 20:00:42.063684] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.063691] request has dfile_count of 1 | dspace has 1 [D 20:00:42.063697] writing count of 1 to disk [D 20:00:42.063704] WRITING attrs: [owner = 2840, group = 4001 perms = 600, type = 1, atime = 1208221242, mtime = 0 ctime = 1208221242 | dfile_count = 1 | dist_size = 56] [D 20:00:42.063715] op_queue add: 0x6944a0 [D 20:00:42.063722] [SYNC_COALESCE]: enqueue called [D 20:00:42.063728] [SYNC_COALESCE]: enqueue 1 counter sync:1 non_sync:0 [D 20:00:42.063736] [SM Exiting]: (0x64b590) set_attr_sm:setobj_attrib (error code: 0), (action: DEFERRED) [D 20:00:42.063807] [SYNC_COALESCE]: dequeue called [D 20:00:42.063816] [SYNC_COALESCE]: dequeue 1 counter sync:0 non_sync:0 [D 20:00:42.063824] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_SETATTR) [D 20:00:42.063841] dbpf_cache_elem_lookup: cache elem matching 146 returned (num_elems=3) [D 20:00:42.063850] Updating cached attributes for key 146 [D 20:00:42.063856] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_SETATTR) (ret: 1) [D 20:00:42.063863] [SYNC_COALESCE]: sync_coalesce called, handle: 146, cid: 0 [D 20:00:42.063869] [SYNC_COALESCE]: sync_context: 0x5bfec0 [D 20:00:42.063875] [SYNC_COALESCE]: sync context type is DSPACE [D 20:00:42.063881] [SYNC_COALESCE]: high or low watermark reached: coalesced: 0 queued: 0 [D 20:00:42.063888] [SYNC_COALESCE]: syncing now! [D 20:00:42.063894] [SYNC_COALESCE]: coalesce 1 sync start in coalesce_queue:0 pending:0 [D 20:00:42.064027] [SYNC_COALESCE]: coalesce 1 sync stop [D 20:00:42.064036] [SYNC_COALESCE]: moving op 0x6944a0 with handle: 146 to completion queue [D 20:00:42.064043] op_queue add: 0x6944a0 [D 20:00:42.064067] [SM push_state]: (0x64b590) op-id: 5 stk-ptr: 0 base-frm: 0 [D 20:00:42.064105] [SM Entering]: (0x64b590) final_response_sm:release (status: 0) [D 20:00:42.064112] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.064119] (s_op 0x64b690) setattr (FR sm) state: release: (error_code = 0) [D 20:00:42.064127] REQ SCHED RELEASING, handle: 146, queue_element: 0x6a87a0 [D 20:00:42.064135] [SM Exiting]: (0x64b590) final_response_sm:release (error code: 0), (action: COMPLETE) [D 20:00:42.064143] [SM Entering]: (0x64b590) final_response_sm:send_resp (status: 0) [D 20:00:42.064150] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.064174] (s_op 0x64b690) setattr (FR sm) state: send_resp (status = 0) [D 20:00:42.064181] PINT_encode [D 20:00:42.064187] encode_common [D 20:00:42.064193] lebf_encode_resp [D 20:00:42.064200] BMI_post_send_list: addr: 6843072, count: 1, total_size: 16, tag: 6 [D 20:00:42.064207] element 0: offset: 0x6a8620, size: 16 [D 20:00:42.064216] post_sr: 10.1.4.56:39990 bh 5 len 32 wr_id 682990. [D 20:00:42.064225] [SM Exiting]: (0x64b590) final_response_sm:send_resp (error code: 0), (action: DEFERRED) -[D 20:00:42.064240] openib_check_cq: completed send wr_id 682990. [D 20:00:42.065457] openib_check_cq: completed recv wr_id 683c90. [D 20:00:42.065467] openib_post_rr: 10.1.4.56:39990 bh 6 wr_id 683c90. [D 20:00:42.069478] [SM Entering]: (0x64b590) final_response_sm:cleanup (status: 0) [D 20:00:42.069488] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.069497] (s_op 0x64b690) setattr (FR sm) state: cleanup [A 20:00:42.075442] troy.scl@10.1.4.56:39990 H=146 S=0x64b690: setattr: finish (Success) [D 20:00:42.075454] PINT_encode_release [D 20:00:42.075462] lebf_encode_rel [D 20:00:42.075469] [SM Exiting]: (0x64b590) final_response_sm:cleanup (error code: 0), (action: COMPLETE) [D 20:00:42.075477] [SM pop_state]: (0x64b590) op-id: 5 stk-ptr: 1 base-frm: 0 [D 20:00:42.075484] [SM Entering]: (0x64b590) set_attr_sm:cleanup (status: 0) [D 20:00:42.075491] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.075498] [SM frame get]: (0x64b590) op-id: 5 index: 0 base-frm: 0 [D 20:00:42.075504] server_state_machine_complete 0x64b590 [D 20:00:42.075511] PINT_decode_release [D 20:00:42.075518] lebf_decode_rel [D 20:00:42.075524] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:00:42.075531] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:42.075537] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 0 [D 20:00:42.075551] [SM Exiting]: (0x64b590) set_attr_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:00:42.075558] server_state_machine_terminate 0x64b590 [D 20:00:42.075565] [SM Entering]: (0x64c850) unexpected_sm:map_request (status: 0) [D 20:00:42.075572] [SM frame get]: (0x64c850) op-id: 999 index: 0 base-frm: 0 [D 20:00:42.075579] server_post_unexpected_recv [D 20:00:42.075588] [SM Frame PUSH]: (0x6a87a0) frame: 0x64b590 [D 20:00:42.075595] [SM Locating]: (0x6a87a0) op-id: 999 [D 20:00:42.075601] server_op_state_get_machine 999 [D 20:00:42.075610] [SM Locating]: (0x6a87a0) located: unexpected_sm:post_unexpected [D 20:00:42.075616] [SM frame get]: (0x6a87a0) op-id: 999 index: 0 base-frm: 0 [D 20:00:42.075623] [SM Entering]: (0x6a87a0) unexpected_sm:post_unexpected (status: 0) [D 20:00:42.075630] [SM frame get]: (0x6a87a0) op-id: 999 index: 0 base-frm: 0 [D 20:00:42.075639] [SM Exiting]: (0x6a87a0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:42.075647] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:42.075654] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:42.075660] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 1 [D 20:00:42.075673] [SM frame get]: (0x64c850) op-id: 999 index: 0 base-frm: 0 [D 20:00:42.075680] server_state_machine_start 0x64c850 [D 20:00:42.075686] PINT_decode [D 20:00:42.075692] lebf_decode_req [D 20:00:42.075703] [SM Locating]: (0x64c850) op-id: 7 [D 20:00:42.075709] server_op_state_get_machine 7 [D 20:00:42.075715] [SM push_state]: (0x64c850) op-id: 7 stk-ptr: 0 base-frm: 0 [D 20:00:42.075722] [SM Locating]: (0x64c850) located: prelude_sm:setup [D 20:00:42.075728] [SM Entering]: (0x64c850) prelude_sm:setup (status: 0) [D 20:00:42.075735] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.075742] [SM Exiting]: (0x64c850) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:42.075754] [SM Exiting]: (0x64c850) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:42.075761] [SM push_state]: (0x64c850) op-id: 7 stk-ptr: 1 base-frm: 0 [D 20:00:42.075767] [SM Entering]: (0x64c850) prelude_work_sm:req_sched (status: 0) [D 20:00:42.075774] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.075794] (0x64c950) crdirent (prelude sm) state: req_sched [A 20:00:42.081817] troy.scl@10.1.4.56:39990 H=138 S=0x64c950: crdirent: request [D 20:00:42.081830] REQ SCHED POSTING, handle: 138, queue_element: 0x694540 [D 20:00:42.081838] REQ SCHED SCHEDULING, handle: 138, queue_element: 0x694540 [D 20:00:42.081845] [SM Exiting]: (0x64c850) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:42.081853] [SM Entering]: (0x64c850) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:42.081859] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [A 20:00:42.087813] troy.scl@10.1.4.56:39990 H=138 S=0x64c950: crdirent: start [D 20:00:42.087823] (0x64c950) crdirent (prelude sm) state: getattr_if_needed [D 20:00:42.087831] About to retrieve attributes for handle 138 [D 20:00:42.087839] dspace_getattr fast path attr cache hit on 138 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:42.087847] [SM Exiting]: (0x64c850) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:42.087855] [SM Entering]: (0x64c850) prelude_work_sm:perm_check (status: 0) [D 20:00:42.087862] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.087869] PVFS operation "crdirent" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:42.087878] - check_mode called --- (uid=2840,gid=4001,access_type=2) [D 20:00:42.087884] - object attributes --- (uid=0,gid=0,mode=511) [D 20:00:42.087891] - checking if uid (2840) is root ... [D 20:00:42.087897] - no [D 20:00:42.087903] - checking if owner (0) matches uid (2840)... [D 20:00:42.087909] - no [D 20:00:42.087914] - checking if permissions (511) allows access type (2) by others... [D 20:00:42.087921] - yes [D 20:00:42.087927] - check_mode called --- (uid=2840,gid=4001,access_type=1) [D 20:00:42.087933] - object attributes --- (uid=0,gid=0,mode=511) [D 20:00:42.087939] - checking if uid (2840) is root ... [D 20:00:42.087945] - no [D 20:00:42.087951] - checking if owner (0) matches uid (2840)... [D 20:00:42.087957] - no [D 20:00:42.087963] - checking if permissions (511) allows access type (1) by others... [D 20:00:42.087969] - yes [D 20:00:42.087975] Final permission check for "crdirent" set error code to 0 [D 20:00:42.087981] (0x64c950) crdirent (prelude sm) state: perm_check (status = 0) [D 20:00:42.087988] [SM Exiting]: (0x64c850) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:42.087995] [SM pop_state]: (0x64c850) op-id: 7 stk-ptr: 2 base-frm: 0 [D 20:00:42.088001] [SM pop_state]: (0x64c850) op-id: 7 stk-ptr: 1 base-frm: 0 [D 20:00:42.088008] [SM Entering]: (0x64c850) crdirent_sm:setup_op (status: 0) [D 20:00:42.088015] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [A 20:00:42.094059] troy.scl@10.1.4.56:39990 H=138 S=0x64c950: crdirent: crdirent entry: qtest.F08 points to 146 [D 20:00:42.094071] [SM Exiting]: (0x64c850) crdirent_sm:setup_op (error code: 0), (action: COMPLETE) [D 20:00:42.094082] [SM Entering]: (0x64c850) crdirent_sm:validate (status: 0) [D 20:00:42.094090] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.094099] got crdirent for qtest.F08 (with handle 146) in 138 [D 20:00:42.094108] [SM Exiting]: (0x64c850) crdirent_sm:validate (error code: 0), (action: COMPLETE) [D 20:00:42.094117] [SM Entering]: (0x64c850) crdirent_sm:read_directory_entry_handle (status: 0) [D 20:00:42.094125] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.094136] *** Trove KeyVal Read of de [D 20:00:42.094144] dbpf_cache_elem_lookup: cache elem matching 138 returned (num_elems=3) [D 20:00:42.094153] op_queue add: 0x6948b0 [D 20:00:42.094164] [SM Exiting]: (0x64c850) crdirent_sm:read_directory_entry_handle (error code: 0), (action: DEFERRED) [D 20:00:42.094169] [DBPF THREAD]: [KEYVAL -1]: -7 [D 20:00:42.094180] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (KEYVAL_READ) [D 20:00:42.094193] dbpf_cache_elem_lookup: cache elem matching 138 returned (num_elems=3) [D 20:00:42.094202] ** CANNOT cache data retrieved (key is de) [D 20:00:42.094208] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (KEYVAL_READ) (ret: 1) [D 20:00:42.094215] op_queue add: 0x6948b0 [D 20:00:42.094233] [SM Entering]: (0x64c850) crdirent_sm:write_directory_entry (status: 0) [D 20:00:42.094251] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.094260] writing new directory entry for qtest.F08 (handle = 146) to dirdata dspace 4 [D 20:00:42.094270] op_queue add: 0x6948b0 [D 20:00:42.094278] [SYNC_COALESCE]: enqueue called [D 20:00:42.094286] [SYNC_COALESCE]: enqueue 0 counter sync:1 non_sync:0 [D 20:00:42.094296] [SM Exiting]: (0x64c850) crdirent_sm:write_directory_entry (error code: 0), (action: DEFERRED) [D 20:00:42.094301] [SYNC_COALESCE]: dequeue called [D 20:00:42.094311] [SYNC_COALESCE]: dequeue 0 counter sync:0 non_sync:0 [D 20:00:42.094318] [DBPF THREAD]: [KEYVAL -1]: -8 [D 20:00:42.094324] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (KEYVAL_WRITE) [D 20:00:42.094331] dbpf_keyval_write_op_svc: handle: 4, key: qtest.F08 [D 20:00:42.094337] keyval_db->put(handle= 4, key= qtest.F08 (10)) size=18 [D 20:00:42.094352] *** Trove KeyVal Write of qtest.F08 [D 20:00:42.094363] [DBPF KEYVAL]: handle_info count increment: handle: 4, value: 1 [D 20:00:42.094376] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (KEYVAL_WRITE) (ret: 1) [D 20:00:42.094384] [SYNC_COALESCE]: sync_coalesce called, handle: 4, cid: 0 [D 20:00:42.094391] [SYNC_COALESCE]: sync_context: 0x5bfac0 [D 20:00:42.094397] [SYNC_COALESCE]: sync context type is KEYVAL [D 20:00:42.094403] [SYNC_COALESCE]: high or low watermark reached: coalesced: 0 queued: 0 [D 20:00:42.094410] [SYNC_COALESCE]: syncing now! [D 20:00:42.094415] [SYNC_COALESCE]: coalesce 0 sync start in coalesce_queue:0 pending:0 [D 20:00:42.094478] [SYNC_COALESCE]: coalesce 0 sync stop [D 20:00:42.094488] [SYNC_COALESCE]: moving op 0x6948b0 with handle: 4 to completion queue [D 20:00:42.094495] op_queue add: 0x6948b0 [D 20:00:42.094516] [SM Entering]: (0x64c850) crdirent_sm:check_for_req_dir_update (status: 0) [D 20:00:42.094531] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.094540] [SM Exiting]: (0x64c850) crdirent_sm:check_for_req_dir_update (error code: 132), (action: COMPLETE) [D 20:00:42.094549] [SM Entering]: (0x64c850) crdirent_sm:update_directory_attr (status: 0) [D 20:00:42.094558] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.094569] op_queue add: 0x6943a0 [D 20:00:42.094577] [SYNC_COALESCE]: enqueue called [D 20:00:42.094585] [SYNC_COALESCE]: enqueue 1 counter sync:1 non_sync:0 [D 20:00:42.094595] [SM Exiting]: (0x64c850) crdirent_sm:update_directory_attr (error code: 132), (action: DEFERRED) [D 20:00:42.094600] [SYNC_COALESCE]: dequeue called [D 20:00:42.094610] [SYNC_COALESCE]: dequeue 1 counter sync:0 non_sync:0 [D 20:00:42.094618] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_SETATTR) [D 20:00:42.094630] dbpf_cache_elem_lookup: cache elem matching 138 returned (num_elems=3) [D 20:00:42.094638] Updating cached attributes for key 138 [D 20:00:42.094645] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_SETATTR) (ret: 1) [D 20:00:42.094652] [SYNC_COALESCE]: sync_coalesce called, handle: 138, cid: 0 [D 20:00:42.094658] [SYNC_COALESCE]: sync_context: 0x5bfec0 [D 20:00:42.094664] [SYNC_COALESCE]: sync context type is DSPACE [D 20:00:42.094670] [SYNC_COALESCE]: high or low watermark reached: coalesced: 0 queued: 0 [D 20:00:42.094677] [SYNC_COALESCE]: syncing now! [D 20:00:42.094682] [SYNC_COALESCE]: coalesce 1 sync start in coalesce_queue:0 pending:0 [D 20:00:42.094737] [SYNC_COALESCE]: coalesce 1 sync stop [D 20:00:42.094747] [SYNC_COALESCE]: moving op 0x6943a0 with handle: 138 to completion queue [D 20:00:42.094754] op_queue add: 0x6943a0 [D 20:00:42.094775] [SM push_state]: (0x64c850) op-id: 7 stk-ptr: 0 base-frm: 0 [D 20:00:42.094789] [SM Entering]: (0x64c850) final_response_sm:release (status: 0) [D 20:00:42.094798] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.094807] (s_op 0x64c950) crdirent (FR sm) state: release: (error_code = 0) [D 20:00:42.094816] REQ SCHED RELEASING, handle: 138, queue_element: 0x694540 [D 20:00:42.094826] [SM Exiting]: (0x64c850) final_response_sm:release (error code: 0), (action: COMPLETE) [D 20:00:42.094835] [SM Entering]: (0x64c850) final_response_sm:send_resp (status: 0) [D 20:00:42.094844] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.094853] (s_op 0x64c950) crdirent (FR sm) state: send_resp (status = 0) [D 20:00:42.094861] PINT_encode [D 20:00:42.094869] encode_common [D 20:00:42.094877] lebf_encode_resp [D 20:00:42.094886] BMI_post_send_list: addr: 6843072, count: 1, total_size: 16, tag: 7 [D 20:00:42.094905] element 0: offset: 0x686b90, size: 16 [D 20:00:42.094915] post_sr: 10.1.4.56:39990 bh 6 len 32 wr_id 6829c0. [D 20:00:42.094925] [SM Exiting]: (0x64c850) final_response_sm:send_resp (error code: 0), (action: DEFERRED) -[D 20:00:42.094942] openib_check_cq: completed send wr_id 6829c0. [D 20:00:42.094984] [SM Entering]: (0x64c850) final_response_sm:cleanup (status: 0) [D 20:00:42.094995] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.095002] (s_op 0x64c950) crdirent (FR sm) state: cleanup [A 20:00:42.101055] troy.scl@10.1.4.56:39990 H=138 S=0x64c950: crdirent: finish (Success) [D 20:00:42.101066] PINT_encode_release [D 20:00:42.101073] lebf_encode_rel [D 20:00:42.101080] [SM Exiting]: (0x64c850) final_response_sm:cleanup (error code: 0), (action: COMPLETE) [D 20:00:42.101087] [SM pop_state]: (0x64c850) op-id: 7 stk-ptr: 1 base-frm: 0 [D 20:00:42.101094] [SM Entering]: (0x64c850) crdirent_sm:cleanup (status: 0) [D 20:00:42.101101] [SM frame get]: (0x64c850) op-id: 7 index: 0 base-frm: 0 [D 20:00:42.101107] server_state_machine_complete 0x64c850 [D 20:00:42.101114] PINT_decode_release [D 20:00:42.101120] lebf_decode_rel [D 20:00:42.101126] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:00:42.101132] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:42.101138] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 0 [D 20:00:42.101145] [SM Exiting]: (0x64c850) crdirent_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:00:42.101153] server_state_machine_terminate 0x64c850 [D 20:00:42.308033] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:42.308098] Start times (hr:min:sec): 20:00:41.220 20:00:40.180 20:00:39.140 20:00:38.099 20:00:38.096 0 [P 20:00:42.308105] Intervals (hr:min:sec) : 00:00:01.088 00:00:01.040 00:00:01.040 00:00:01.041 00:00:00.003 0 [P 20:00:42.308112] ------------------------------------------------------------------------------------------------------------- [P 20:00:42.308119] bytes read : 0 0 0 0 0 0 [P 20:00:42.308125] bytes written : 0 0 0 0 0 0 [P 20:00:42.308131] metadata reads : 12 0 0 0 0 0 [P 20:00:42.308138] metadata writes : 7 0 0 0 0 0 [P 20:00:42.308144] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:42.308150] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:42.308157] request scheduler : 0 0 0 0 0 0 [D 20:00:42.308166] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:42.308174] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:42.308183] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:43.241621] openib_check_cq: completed recv wr_id 683cc0. [D 20:00:43.241635] openib_post_rr: 10.1.4.56:39990 bh 7 wr_id 683cc0. [D 20:00:43.244058] [SM Entering]: (0x64db10) unexpected_sm:map_request (status: 0) [D 20:00:43.244070] [SM frame get]: (0x64db10) op-id: 999 index: 0 base-frm: 0 [D 20:00:43.244077] server_post_unexpected_recv [D 20:00:43.244084] [SM Frame PUSH]: (0x64eca0) frame: 0x64c850 [D 20:00:43.244091] [SM Locating]: (0x64eca0) op-id: 999 [D 20:00:43.244097] server_op_state_get_machine 999 [D 20:00:43.244103] [SM Locating]: (0x64eca0) located: unexpected_sm:post_unexpected [D 20:00:43.244110] [SM frame get]: (0x64eca0) op-id: 999 index: 0 base-frm: 0 [D 20:00:43.244117] [SM Entering]: (0x64eca0) unexpected_sm:post_unexpected (status: 0) [D 20:00:43.244123] [SM frame get]: (0x64eca0) op-id: 999 index: 0 base-frm: 0 [D 20:00:43.244131] [SM Exiting]: (0x64eca0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:43.244139] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:43.244145] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:43.244152] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 1 [D 20:00:43.244158] [SM frame get]: (0x64db10) op-id: 999 index: 0 base-frm: 0 [D 20:00:43.244165] server_state_machine_start 0x64db10 [D 20:00:43.244171] PINT_decode [D 20:00:43.244177] lebf_decode_req [D 20:00:43.244186] [SM Locating]: (0x64db10) op-id: 3 [D 20:00:43.244193] server_op_state_get_machine 3 [D 20:00:43.244199] [SM push_state]: (0x64db10) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:43.244205] [SM Locating]: (0x64db10) located: prelude_sm:setup [D 20:00:43.244212] [SM Entering]: (0x64db10) prelude_sm:setup (status: 0) [D 20:00:43.244219] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [D 20:00:43.244228] [SM Exiting]: (0x64db10) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:43.244235] [SM Exiting]: (0x64db10) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:43.244242] [SM push_state]: (0x64db10) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:43.244249] [SM Entering]: (0x64db10) prelude_work_sm:req_sched (status: 0) [D 20:00:43.244256] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [D 20:00:43.244262] (0x64dc10) io (prelude sm) state: req_sched [A 20:00:43.250057] troy.scl@10.1.4.56:39990 H=2863311531 S=0x64dc10: io: request [D 20:00:43.250070] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6943a0 [D 20:00:43.250078] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6943a0 [D 20:00:43.250086] [SM Exiting]: (0x64db10) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:43.250093] [SM Entering]: (0x64db10) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:43.250104] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [A 20:00:43.255927] troy.scl@10.1.4.56:39990 H=2863311531 S=0x64dc10: io: start [D 20:00:43.255938] (0x64dc10) io (prelude sm) state: getattr_if_needed [D 20:00:43.255946] About to retrieve attributes for handle 2863311531 [D 20:00:43.255954] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:43.255963] [SM Exiting]: (0x64db10) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:43.255975] [SM Entering]: (0x64db10) prelude_work_sm:perm_check (status: 0) [D 20:00:43.255982] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [D 20:00:43.255989] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:43.255998] Final permission check for "io" set error code to 0 [D 20:00:43.256004] (0x64dc10) io (prelude sm) state: perm_check (status = 0) [D 20:00:43.256010] [SM Exiting]: (0x64db10) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:43.256017] [SM pop_state]: (0x64db10) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:43.256024] [SM pop_state]: (0x64db10) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:43.256036] [SM Entering]: (0x64db10) io_sm:send_positive_ack (status: 0) [D 20:00:43.256043] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [D 20:00:43.256052] PINT_encode [D 20:00:43.256060] encode_common [D 20:00:43.256072] lebf_encode_resp [D 20:00:43.256079] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 8 [D 20:00:43.256086] element 0: offset: 0x694140, size: 24 [D 20:00:43.256096] post_sr: 10.1.4.56:39990 bh 7 len 40 wr_id 6829f0. [D 20:00:43.256108] [SM Exiting]: (0x64db10) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:43.256126] openib_check_cq: completed send wr_id 6829f0. [D 20:00:43.260060] [SM Entering]: (0x64db10) io_sm:start_flow (status: 0) [D 20:00:43.260072] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [D 20:00:43.260080] io_start_flow() issuing flow to write data. [D 20:00:43.260087] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:43.260094] file_req_offset: 0, aggregate_size: 33554432, handle: 2863311531 [D 20:00:43.260104] PINT_new_request_states n=1 [D 20:00:43.260113] flowproto posting 0x6aadd0 [D 20:00:43.260130] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:43.260141] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:00:43.261366] ========================================================= [D 20:00:43.261390] PINT_process_request [D 20:00:43.261400] tiling 2 copies [D 20:00:43.261407] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:43.261415] lvl 0 el 0 blk 0 by 0 [D 20:00:43.261421] to 0 ta 0 fi 33554432 [D 20:00:43.261428] basic type or contiguous data [D 20:00:43.261434] Request disp [D 20:00:43.261440] contig_offset = 0 contig_size = 67108864 lvl_flag = 1 [D 20:00:43.261447] calling distribute [D 20:00:43.261453] PINT_distribute [D 20:00:43.261459] of 0 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:43.261467] begin iteration loff: 0 [D 20:00:43.261474] segment exceeds byte limit [D 20:00:43.261479] file being extended [D 20:00:43.261485] process a segment [D 20:00:43.261491] of 0 sz 2097152 [D 20:00:43.261498] add a segment 0 [D 20:00:43.261504] end iteration [D 20:00:43.261510] done with segments or bytes [D 20:00:43.261516] of 2097152 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:43.261522] next loff: 2097152 next poff: 2097152 [D 20:00:43.261529] return value 2097152 [D 20:00:43.261535] finished [D 20:00:43.261541] retval = 2097152 [D 20:00:43.261547] exiting distribute returned less than expected [D 20:00:43.261553] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 2097152 fo 33554432 eof 0 [D 20:00:43.261560] ========================================================= [D 20:00:43.261569] BMI_post_recv: addr: 6843072, offset: 0x2aaaac6bb010, size: 2097152, tag: 8 [D 20:00:43.261582] flowproto posted 0x6aadd0 [D 20:00:43.261589] Job flows in progress (post time): 1 [D 20:00:43.261597] [SM Exiting]: (0x64db10) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:43.364062] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:43.364133] Start times (hr:min:sec): 20:00:42.308 20:00:41.220 20:00:40.180 20:00:39.140 20:00:38.099 20:00:38.096 [P 20:00:43.364141] Intervals (hr:min:sec) : 00:00:01.056 00:00:01.088 00:00:01.040 00:00:01.040 00:00:01.041 00:00:00.003 [P 20:00:43.364147] ------------------------------------------------------------------------------------------------------------- [P 20:00:43.364154] bytes read : 0 0 0 0 0 0 [P 20:00:43.364160] bytes written : 0 0 0 0 0 0 [P 20:00:43.364166] metadata reads : 13 12 0 0 0 0 [P 20:00:43.364172] metadata writes : 7 7 0 0 0 0 [P 20:00:43.364179] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:43.364185] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:43.364191] request scheduler : 1 0 0 0 0 0 [D 20:00:43.364201] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:43.364209] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:43.364218] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:44.061053] openib_check_cq: completed recv wr_id 683cf0. [D 20:00:44.061068] openib_post_rr: 10.1.4.56:39990 bh 8 wr_id 683cf0. [D 20:00:44.061108] [SM Entering]: (0x64edd0) unexpected_sm:map_request (status: 0) [D 20:00:44.061120] [SM frame get]: (0x64edd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:44.061128] server_post_unexpected_recv [D 20:00:44.061140] [SM Frame PUSH]: (0x64ff60) frame: 0x6ad840 [D 20:00:44.061147] [SM Locating]: (0x64ff60) op-id: 999 [D 20:00:44.061153] server_op_state_get_machine 999 [D 20:00:44.061159] [SM Locating]: (0x64ff60) located: unexpected_sm:post_unexpected [D 20:00:44.061166] [SM frame get]: (0x64ff60) op-id: 999 index: 0 base-frm: 0 [D 20:00:44.061173] [SM Entering]: (0x64ff60) unexpected_sm:post_unexpected (status: 0) [D 20:00:44.061180] [SM frame get]: (0x64ff60) op-id: 999 index: 0 base-frm: 0 [D 20:00:44.061189] [SM Exiting]: (0x64ff60) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:44.061196] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:44.061203] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:44.061210] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 2 [D 20:00:44.061217] [SM frame get]: (0x64edd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:44.061223] server_state_machine_start 0x64edd0 [D 20:00:44.061230] PINT_decode [D 20:00:44.061237] lebf_decode_req [D 20:00:44.061246] [SM Locating]: (0x64edd0) op-id: 3 [D 20:00:44.061253] server_op_state_get_machine 3 [D 20:00:44.061259] [SM push_state]: (0x64edd0) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:44.061266] [SM Locating]: (0x64edd0) located: prelude_sm:setup [D 20:00:44.061273] [SM Entering]: (0x64edd0) prelude_sm:setup (status: 0) [D 20:00:44.061280] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.061287] [SM Exiting]: (0x64edd0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:44.061294] [SM Exiting]: (0x64edd0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:44.061301] [SM push_state]: (0x64edd0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:44.061308] [SM Entering]: (0x64edd0) prelude_work_sm:req_sched (status: 0) [D 20:00:44.061314] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.061321] (0x64eed0) io (prelude sm) state: req_sched [A 20:00:44.067240] troy.scl@10.1.4.56:39990 H=2863311531 S=0x64eed0: io: request [D 20:00:44.067254] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:44.067262] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x682620 [D 20:00:44.067268] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x682620 [D 20:00:44.067275] [SM Exiting]: (0x64edd0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:44.067283] [SM Entering]: (0x64edd0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:44.067290] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [A 20:00:44.073108] troy.scl@10.1.4.56:39990 H=2863311531 S=0x64eed0: io: start [D 20:00:44.073120] (0x64eed0) io (prelude sm) state: getattr_if_needed [D 20:00:44.073128] About to retrieve attributes for handle 2863311531 [D 20:00:44.073138] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:44.073147] [SM Exiting]: (0x64edd0) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:44.073158] [SM Entering]: (0x64edd0) prelude_work_sm:perm_check (status: 0) [D 20:00:44.073165] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.073173] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:44.073182] Final permission check for "io" set error code to 0 [D 20:00:44.073189] (0x64eed0) io (prelude sm) state: perm_check (status = 0) [D 20:00:44.073196] [SM Exiting]: (0x64edd0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:44.073203] [SM pop_state]: (0x64edd0) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:44.073210] [SM pop_state]: (0x64edd0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:44.073220] [SM Entering]: (0x64edd0) io_sm:send_positive_ack (status: 0) [D 20:00:44.073227] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.073234] PINT_encode [D 20:00:44.073241] encode_common [D 20:00:44.073248] lebf_encode_resp [D 20:00:44.073256] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 9 [D 20:00:44.073263] element 0: offset: 0x6948d0, size: 24 [D 20:00:44.073273] post_sr: 10.1.4.56:39990 bh 8 len 40 wr_id 682a20. [D 20:00:44.073288] [SM Exiting]: (0x64edd0) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:44.073307] openib_check_cq: completed send wr_id 682a20. [D 20:00:44.076081] [SM Entering]: (0x64edd0) io_sm:start_flow (status: 0) [D 20:00:44.076093] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.076101] io_start_flow() issuing flow to write data. [D 20:00:44.076108] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:44.076115] file_req_offset: 33554432, aggregate_size: 33554432, handle: 2863311531 [D 20:00:44.076123] PINT_new_request_states n=1 [D 20:00:44.076131] flowproto posting 0x6aecb0 [D 20:00:44.076146] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:44.076154] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:00:44.077506] ========================================================= [D 20:00:44.077533] PINT_process_request [D 20:00:44.077540] tiling 3 copies [D 20:00:44.077546] skipping ahead to target_offset [D 20:00:44.077553] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:44.077560] lvl 0 el 0 blk 0 by 0 [D 20:00:44.077567] to 0 ta 33554432 fi 67108864 [D 20:00:44.077574] basic type or contiguous data [D 20:00:44.077580] Request disp [D 20:00:44.077586] contig_offset = 0 contig_size = 100663296 lvl_flag = 1 [D 20:00:44.077592] process logical skip [D 20:00:44.077599] retval = 33554432 [D 20:00:44.077604] exiting logical skip because distribute indicates done [D 20:00:44.077611] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:44.077618] lvl 0 el 0 blk 0 by 33554432 [D 20:00:44.077624] to 33554432 ta 33554432 fi 67108864 [D 20:00:44.077630] basic type or contiguous data [D 20:00:44.077636] Request disp [D 20:00:44.077642] contig_offset = 33554432 contig_size = 67108864 lvl_flag = 1 [D 20:00:44.077649] calling distribute [D 20:00:44.077655] PINT_distribute [D 20:00:44.077661] of 33554432 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:44.077668] begin iteration loff: 33554432 [D 20:00:44.077675] segment exceeds byte limit [D 20:00:44.077681] file being extended [D 20:00:44.077687] process a segment [D 20:00:44.077693] of 33554432 sz 2097152 [D 20:00:44.077700] add a segment 0 [D 20:00:44.077705] end iteration [D 20:00:44.077711] done with segments or bytes [D 20:00:44.077717] of 35651584 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:44.077724] next loff: 35651584 next poff: 35651584 [D 20:00:44.077731] return value 2097152 [D 20:00:44.077737] finished [D 20:00:44.077743] retval = 2097152 [D 20:00:44.077749] exiting distribute returned less than expected [D 20:00:44.077755] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 35651584 fo 67108864 eof 0 [D 20:00:44.077762] ========================================================= [D 20:00:44.077769] BMI_post_recv: addr: 6843072, offset: 0x2aaaac8bc010, size: 2097152, tag: 9 [D 20:00:44.077782] flowproto posted 0x6aecb0 [D 20:00:44.077789] Job flows in progress (post time): 2 [D 20:00:44.077796] [SM Exiting]: (0x64edd0) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:44.388087] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:44.388158] Start times (hr:min:sec): 20:00:43.364 20:00:42.308 20:00:41.220 20:00:40.180 20:00:39.140 20:00:38.099 [P 20:00:44.388165] Intervals (hr:min:sec) : 00:00:01.024 00:00:01.056 00:00:01.088 00:00:01.040 00:00:01.040 00:00:01.041 [P 20:00:44.388171] ------------------------------------------------------------------------------------------------------------- [P 20:00:44.388178] bytes read : 0 0 0 0 0 0 [P 20:00:44.388184] bytes written : 0 0 0 0 0 0 [P 20:00:44.388190] metadata reads : 14 13 12 0 0 0 [P 20:00:44.388196] metadata writes : 7 7 7 0 0 0 [P 20:00:44.388203] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:44.388209] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:44.388215] request scheduler : 2 1 0 0 0 0 [D 20:00:44.388225] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:44.388234] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:44.388242] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:44.821279] openib_check_cq: completed recv wr_id 683d20. [D 20:00:44.821294] openib_post_rr: 10.1.4.56:39990 bh 9 wr_id 683d20. [D 20:00:44.821332] [SM Entering]: (0x650090) unexpected_sm:map_request (status: 0) [D 20:00:44.821343] [SM frame get]: (0x650090) op-id: 999 index: 0 base-frm: 0 [D 20:00:44.821351] server_post_unexpected_recv [D 20:00:44.821364] [SM Frame PUSH]: (0x651220) frame: 0x6b18f0 [D 20:00:44.821371] [SM Locating]: (0x651220) op-id: 999 [D 20:00:44.821377] server_op_state_get_machine 999 [D 20:00:44.821383] [SM Locating]: (0x651220) located: unexpected_sm:post_unexpected [D 20:00:44.821390] [SM frame get]: (0x651220) op-id: 999 index: 0 base-frm: 0 [D 20:00:44.821397] [SM Entering]: (0x651220) unexpected_sm:post_unexpected (status: 0) [D 20:00:44.821404] [SM frame get]: (0x651220) op-id: 999 index: 0 base-frm: 0 [D 20:00:44.821412] [SM Exiting]: (0x651220) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:44.821420] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:44.821426] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:44.821433] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 3 [D 20:00:44.821439] [SM frame get]: (0x650090) op-id: 999 index: 0 base-frm: 0 [D 20:00:44.821446] server_state_machine_start 0x650090 [D 20:00:44.821452] PINT_decode [D 20:00:44.821459] lebf_decode_req [D 20:00:44.821469] [SM Locating]: (0x650090) op-id: 3 [D 20:00:44.821476] server_op_state_get_machine 3 [D 20:00:44.821482] [SM push_state]: (0x650090) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:44.821488] [SM Locating]: (0x650090) located: prelude_sm:setup [D 20:00:44.821495] [SM Entering]: (0x650090) prelude_sm:setup (status: 0) [D 20:00:44.821502] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.821509] [SM Exiting]: (0x650090) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:44.821516] [SM Exiting]: (0x650090) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:44.821522] [SM push_state]: (0x650090) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:44.821529] [SM Entering]: (0x650090) prelude_work_sm:req_sched (status: 0) [D 20:00:44.821535] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.821542] (0x650190) io (prelude sm) state: req_sched [A 20:00:44.827453] troy.scl@10.1.4.56:39990 H=2863311531 S=0x650190: io: request [D 20:00:44.827467] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:44.827475] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6b2c40 [D 20:00:44.827481] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6b2c40 [D 20:00:44.827488] [SM Exiting]: (0x650090) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:44.827495] [SM Entering]: (0x650090) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:44.827502] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [A 20:00:44.833319] troy.scl@10.1.4.56:39990 H=2863311531 S=0x650190: io: start [D 20:00:44.833330] (0x650190) io (prelude sm) state: getattr_if_needed [D 20:00:44.833339] About to retrieve attributes for handle 2863311531 [D 20:00:44.833349] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:44.833358] [SM Exiting]: (0x650090) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:44.833370] [SM Entering]: (0x650090) prelude_work_sm:perm_check (status: 0) [D 20:00:44.833377] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.833385] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:44.833394] Final permission check for "io" set error code to 0 [D 20:00:44.833401] (0x650190) io (prelude sm) state: perm_check (status = 0) [D 20:00:44.833408] [SM Exiting]: (0x650090) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:44.833415] [SM pop_state]: (0x650090) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:44.833422] [SM pop_state]: (0x650090) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:44.833432] [SM Entering]: (0x650090) io_sm:send_positive_ack (status: 0) [D 20:00:44.833439] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.833446] PINT_encode [D 20:00:44.833452] encode_common [D 20:00:44.833459] lebf_encode_resp [D 20:00:44.833467] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 10 [D 20:00:44.833475] element 0: offset: 0x6b17b0, size: 24 [D 20:00:44.833483] post_sr: 10.1.4.56:39990 bh 9 len 40 wr_id 682a50. [D 20:00:44.833493] [SM Exiting]: (0x650090) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:44.833511] openib_check_cq: completed send wr_id 682a50. [D 20:00:44.836099] [SM Entering]: (0x650090) io_sm:start_flow (status: 0) [D 20:00:44.836111] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [D 20:00:44.836120] io_start_flow() issuing flow to write data. [D 20:00:44.836126] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:44.836133] file_req_offset: 67108864, aggregate_size: 33554432, handle: 2863311531 [D 20:00:44.836141] PINT_new_request_states n=1 [D 20:00:44.836148] flowproto posting 0x6b2df0 [D 20:00:44.836160] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:44.836168] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:00:44.837616] ========================================================= [D 20:00:44.837642] PINT_process_request [D 20:00:44.837649] tiling 4 copies [D 20:00:44.837655] skipping ahead to target_offset [D 20:00:44.837661] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:44.837669] lvl 0 el 0 blk 0 by 0 [D 20:00:44.837675] to 0 ta 67108864 fi 100663296 [D 20:00:44.837682] basic type or contiguous data [D 20:00:44.837688] Request disp [D 20:00:44.837694] contig_offset = 0 contig_size = 134217728 lvl_flag = 1 [D 20:00:44.837700] process logical skip [D 20:00:44.837706] retval = 67108864 [D 20:00:44.837712] exiting logical skip because distribute indicates done [D 20:00:44.837718] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:44.837726] lvl 0 el 0 blk 0 by 67108864 [D 20:00:44.837732] to 67108864 ta 67108864 fi 100663296 [D 20:00:44.837738] basic type or contiguous data [D 20:00:44.837744] Request disp [D 20:00:44.837750] contig_offset = 67108864 contig_size = 67108864 lvl_flag = 1 [D 20:00:44.837757] calling distribute [D 20:00:44.837763] PINT_distribute [D 20:00:44.837768] of 67108864 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:44.837776] begin iteration loff: 67108864 [D 20:00:44.837783] segment exceeds byte limit [D 20:00:44.837789] file being extended [D 20:00:44.837795] process a segment [D 20:00:44.837800] of 67108864 sz 2097152 [D 20:00:44.837807] add a segment 0 [D 20:00:44.837813] end iteration [D 20:00:44.837819] done with segments or bytes [D 20:00:44.837825] of 69206016 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:44.837831] next loff: 69206016 next poff: 69206016 [D 20:00:44.837838] return value 2097152 [D 20:00:44.837845] finished [D 20:00:44.837850] retval = 2097152 [D 20:00:44.837856] exiting distribute returned less than expected [D 20:00:44.837862] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 69206016 fo 100663296 eof 0 [D 20:00:44.837870] ========================================================= [D 20:00:44.837876] BMI_post_recv: addr: 6843072, offset: 0x2aaaacabd010, size: 2097152, tag: 10 [D 20:00:44.837888] flowproto posted 0x6b2df0 [D 20:00:44.837896] Job flows in progress (post time): 3 [D 20:00:44.837903] [SM Exiting]: (0x650090) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:45.460120] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:45.460192] Start times (hr:min:sec): 20:00:44.388 20:00:43.364 20:00:42.308 20:00:41.220 20:00:40.180 20:00:39.140 [P 20:00:45.460199] Intervals (hr:min:sec) : 00:00:01.072 00:00:01.024 00:00:01.056 00:00:01.088 00:00:01.040 00:00:01.040 [P 20:00:45.460206] ------------------------------------------------------------------------------------------------------------- [P 20:00:45.460212] bytes read : 0 0 0 0 0 0 [P 20:00:45.460218] bytes written : 0 0 0 0 0 0 [P 20:00:45.460225] metadata reads : 15 14 13 12 0 0 [P 20:00:45.460231] metadata writes : 7 7 7 7 0 0 [P 20:00:45.460237] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:45.460243] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:45.460249] request scheduler : 3 2 1 0 0 0 [D 20:00:45.460259] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:45.460267] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:45.460276] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:45.593762] openib_check_cq: completed recv wr_id 683d50. [D 20:00:45.593778] openib_post_rr: 10.1.4.56:39990 bh 10 wr_id 683d50. [D 20:00:45.593812] [SM Entering]: (0x651350) unexpected_sm:map_request (status: 0) [D 20:00:45.593824] [SM frame get]: (0x651350) op-id: 999 index: 0 base-frm: 0 [D 20:00:45.593832] server_post_unexpected_recv [D 20:00:45.593844] [SM Frame PUSH]: (0x6524e0) frame: 0x6b5b70 [D 20:00:45.593851] [SM Locating]: (0x6524e0) op-id: 999 [D 20:00:45.593857] server_op_state_get_machine 999 [D 20:00:45.593863] [SM Locating]: (0x6524e0) located: unexpected_sm:post_unexpected [D 20:00:45.593870] [SM frame get]: (0x6524e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:45.593877] [SM Entering]: (0x6524e0) unexpected_sm:post_unexpected (status: 0) [D 20:00:45.593884] [SM frame get]: (0x6524e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:45.593892] [SM Exiting]: (0x6524e0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:45.593900] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:45.593907] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:45.593913] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 4 [D 20:00:45.593920] [SM frame get]: (0x651350) op-id: 999 index: 0 base-frm: 0 [D 20:00:45.593926] server_state_machine_start 0x651350 [D 20:00:45.593933] PINT_decode [D 20:00:45.593940] lebf_decode_req [D 20:00:45.593949] [SM Locating]: (0x651350) op-id: 3 [D 20:00:45.593956] server_op_state_get_machine 3 [D 20:00:45.593962] [SM push_state]: (0x651350) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:45.593969] [SM Locating]: (0x651350) located: prelude_sm:setup [D 20:00:45.593976] [SM Entering]: (0x651350) prelude_sm:setup (status: 0) [D 20:00:45.593982] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [D 20:00:45.593989] [SM Exiting]: (0x651350) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:45.593996] [SM Exiting]: (0x651350) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:45.594003] [SM push_state]: (0x651350) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:45.594010] [SM Entering]: (0x651350) prelude_work_sm:req_sched (status: 0) [D 20:00:45.594017] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [D 20:00:45.594023] (0x651450) io (prelude sm) state: req_sched [A 20:00:45.599908] troy.scl@10.1.4.56:39990 H=2863311531 S=0x651450: io: request [D 20:00:45.599921] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:45.599929] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6b6ec0 [D 20:00:45.599936] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6b6ec0 [D 20:00:45.599943] [SM Exiting]: (0x651350) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:45.599950] [SM Entering]: (0x651350) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:45.599957] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [A 20:00:45.605773] troy.scl@10.1.4.56:39990 H=2863311531 S=0x651450: io: start [D 20:00:45.605785] (0x651450) io (prelude sm) state: getattr_if_needed [D 20:00:45.605793] About to retrieve attributes for handle 2863311531 [D 20:00:45.605802] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:45.605811] [SM Exiting]: (0x651350) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:45.605819] [SM Entering]: (0x651350) prelude_work_sm:perm_check (status: 0) [D 20:00:45.605827] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [D 20:00:45.605835] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:45.605843] Final permission check for "io" set error code to 0 [D 20:00:45.605850] (0x651450) io (prelude sm) state: perm_check (status = 0) [D 20:00:45.605861] [SM Exiting]: (0x651350) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:45.605868] [SM pop_state]: (0x651350) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:45.605874] [SM pop_state]: (0x651350) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:45.605881] [SM Entering]: (0x651350) io_sm:send_positive_ack (status: 0) [D 20:00:45.605888] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [D 20:00:45.605897] PINT_encode [D 20:00:45.605903] encode_common [D 20:00:45.605910] lebf_encode_resp [D 20:00:45.605918] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 11 [D 20:00:45.605931] element 0: offset: 0x6b5a30, size: 24 [D 20:00:45.605939] post_sr: 10.1.4.56:39990 bh 10 len 40 wr_id 682a80. [D 20:00:45.605949] [SM Exiting]: (0x651350) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:45.605969] openib_check_cq: completed send wr_id 682a80. [D 20:00:45.608118] [SM Entering]: (0x651350) io_sm:start_flow (status: 0) [D 20:00:45.608129] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [D 20:00:45.608138] io_start_flow() issuing flow to write data. [D 20:00:45.608145] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:45.608151] file_req_offset: 100663296, aggregate_size: 33554432, handle: 2863311531 [D 20:00:45.608159] PINT_new_request_states n=1 [D 20:00:45.608167] flowproto posting 0x6b7070 [D 20:00:45.608180] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:45.608187] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:45.609648] ========================================================= [D 20:00:45.609676] PINT_process_request [D 20:00:45.609683] tiling 5 copies [D 20:00:45.609690] skipping ahead to target_offset [D 20:00:45.609696] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:45.609704] lvl 0 el 0 blk 0 by 0 [D 20:00:45.609710] to 0 ta 100663296 fi 134217728 [D 20:00:45.609717] basic type or contiguous data [D 20:00:45.609723] Request disp [D 20:00:45.609729] contig_offset = 0 contig_size = 167772160 lvl_flag = 1 [D 20:00:45.609736] process logical skip [D 20:00:45.609741] retval = 100663296 [D 20:00:45.609747] exiting logical skip because distribute indicates done [D 20:00:45.609754] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:45.609761] lvl 0 el 0 blk 0 by 100663296 [D 20:00:45.609767] to 100663296 ta 100663296 fi 134217728 [D 20:00:45.609773] basic type or contiguous data [D 20:00:45.609779] Request disp [D 20:00:45.609785] contig_offset = 100663296 contig_size = 67108864 lvl_flag = 1 [D 20:00:45.609792] calling distribute [D 20:00:45.609798] PINT_distribute [D 20:00:45.609804] of 100663296 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:45.609812] begin iteration loff: 100663296 [D 20:00:45.609818] segment exceeds byte limit [D 20:00:45.609824] file being extended [D 20:00:45.609830] process a segment [D 20:00:45.609836] of 100663296 sz 2097152 [D 20:00:45.609842] add a segment 0 [D 20:00:45.609849] end iteration [D 20:00:45.609854] done with segments or bytes [D 20:00:45.609860] of 102760448 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:45.609867] next loff: 102760448 next poff: 102760448 [D 20:00:45.609874] return value 2097152 [D 20:00:45.609880] finished [D 20:00:45.609886] retval = 2097152 [D 20:00:45.609892] exiting distribute returned less than expected [D 20:00:45.609898] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 102760448 fo 134217728 eof 0 [D 20:00:45.609905] ========================================================= [D 20:00:45.609912] BMI_post_recv: addr: 6843072, offset: 0x2aaaaccbe010, size: 2097152, tag: 11 [D 20:00:45.609930] flowproto posted 0x6b7070 [D 20:00:45.609938] Job flows in progress (post time): 4 [D 20:00:45.609946] [SM Exiting]: (0x651350) io_sm:start_flow (error code: 0), (action: DEFERRED) -[D 20:00:46.387055] openib_check_cq: completed recv wr_id 683d80. [D 20:00:46.387071] openib_post_rr: 10.1.4.56:39990 bh 11 wr_id 683d80. [D 20:00:46.387110] [SM Entering]: (0x652610) unexpected_sm:map_request (status: 0) [D 20:00:46.387122] [SM frame get]: (0x652610) op-id: 999 index: 0 base-frm: 0 [D 20:00:46.387130] server_post_unexpected_recv [D 20:00:46.387143] [SM Frame PUSH]: (0x6537a0) frame: 0x6b9e30 [D 20:00:46.387150] [SM Locating]: (0x6537a0) op-id: 999 [D 20:00:46.387157] server_op_state_get_machine 999 [D 20:00:46.387165] [SM Locating]: (0x6537a0) located: unexpected_sm:post_unexpected [D 20:00:46.387171] [SM frame get]: (0x6537a0) op-id: 999 index: 0 base-frm: 0 [D 20:00:46.387178] [SM Entering]: (0x6537a0) unexpected_sm:post_unexpected (status: 0) [D 20:00:46.387185] [SM frame get]: (0x6537a0) op-id: 999 index: 0 base-frm: 0 [D 20:00:46.387194] [SM Exiting]: (0x6537a0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:46.387201] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:46.387208] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:46.387215] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 5 [D 20:00:46.387222] [SM frame get]: (0x652610) op-id: 999 index: 0 base-frm: 0 [D 20:00:46.387228] server_state_machine_start 0x652610 [D 20:00:46.387235] PINT_decode [D 20:00:46.387242] lebf_decode_req [D 20:00:46.387254] [SM Locating]: (0x652610) op-id: 3 [D 20:00:46.387262] server_op_state_get_machine 3 [D 20:00:46.387268] [SM push_state]: (0x652610) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:46.387275] [SM Locating]: (0x652610) located: prelude_sm:setup [D 20:00:46.387282] [SM Entering]: (0x652610) prelude_sm:setup (status: 0) [D 20:00:46.387288] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [D 20:00:46.387295] [SM Exiting]: (0x652610) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:46.387302] [SM Exiting]: (0x652610) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:46.387310] [SM push_state]: (0x652610) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:46.387316] [SM Entering]: (0x652610) prelude_work_sm:req_sched (status: 0) [D 20:00:46.387323] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [D 20:00:46.387329] (0x652710) io (prelude sm) state: req_sched [A 20:00:46.393353] troy.scl@10.1.4.56:39990 H=2863311531 S=0x652710: io: request [D 20:00:46.393367] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:46.393375] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6bb180 [D 20:00:46.393381] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6bb180 [D 20:00:46.393388] [SM Exiting]: (0x652610) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:46.393396] [SM Entering]: (0x652610) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:46.393404] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [A 20:00:46.399091] troy.scl@10.1.4.56:39990 H=2863311531 S=0x652710: io: start [D 20:00:46.399102] (0x652710) io (prelude sm) state: getattr_if_needed [D 20:00:46.399111] About to retrieve attributes for handle 2863311531 [D 20:00:46.399121] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:46.399130] [SM Exiting]: (0x652610) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:46.399142] [SM Entering]: (0x652610) prelude_work_sm:perm_check (status: 0) [D 20:00:46.399149] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [D 20:00:46.399157] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:46.399169] Final permission check for "io" set error code to 0 [D 20:00:46.399176] (0x652710) io (prelude sm) state: perm_check (status = 0) [D 20:00:46.399182] [SM Exiting]: (0x652610) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:46.399189] [SM pop_state]: (0x652610) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:46.399196] [SM pop_state]: (0x652610) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:46.399208] [SM Entering]: (0x652610) io_sm:send_positive_ack (status: 0) [D 20:00:46.399215] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [D 20:00:46.399222] PINT_encode [D 20:00:46.399228] encode_common [D 20:00:46.399235] lebf_encode_resp [D 20:00:46.399244] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 12 [D 20:00:46.399251] element 0: offset: 0x6b9cf0, size: 24 [D 20:00:46.399260] post_sr: 10.1.4.56:39990 bh 11 len 40 wr_id 682ab0. [D 20:00:46.399270] [SM Exiting]: (0x652610) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:46.399290] openib_check_cq: completed send wr_id 682ab0. [D 20:00:46.400138] [SM Entering]: (0x652610) io_sm:start_flow (status: 0) [D 20:00:46.400150] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [D 20:00:46.400158] io_start_flow() issuing flow to write data. [D 20:00:46.400165] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:46.400171] file_req_offset: 134217728, aggregate_size: 33554432, handle: 2863311531 [D 20:00:46.400180] PINT_new_request_states n=1 [D 20:00:46.400187] flowproto posting 0x6bb330 [D 20:00:46.400199] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:46.400207] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:46.401702] ========================================================= [D 20:00:46.401728] PINT_process_request [D 20:00:46.401736] tiling 6 copies [D 20:00:46.401742] skipping ahead to target_offset [D 20:00:46.401749] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:46.401756] lvl 0 el 0 blk 0 by 0 [D 20:00:46.401762] to 0 ta 134217728 fi 167772160 [D 20:00:46.401770] basic type or contiguous data [D 20:00:46.401776] Request disp [D 20:00:46.401782] contig_offset = 0 contig_size = 201326592 lvl_flag = 1 [D 20:00:46.401788] process logical skip [D 20:00:46.401794] retval = 134217728 [D 20:00:46.401800] exiting logical skip because distribute indicates done [D 20:00:46.401806] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:46.401814] lvl 0 el 0 blk 0 by 134217728 [D 20:00:46.401820] to 134217728 ta 134217728 fi 167772160 [D 20:00:46.401826] basic type or contiguous data [D 20:00:46.401832] Request disp [D 20:00:46.401838] contig_offset = 134217728 contig_size = 67108864 lvl_flag = 1 [D 20:00:46.401845] calling distribute [D 20:00:46.401851] PINT_distribute [D 20:00:46.401857] of 134217728 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:46.401864] begin iteration loff: 134217728 [D 20:00:46.401871] segment exceeds byte limit [D 20:00:46.401877] file being extended [D 20:00:46.401883] process a segment [D 20:00:46.401889] of 134217728 sz 2097152 [D 20:00:46.401895] add a segment 0 [D 20:00:46.401901] end iteration [D 20:00:46.401907] done with segments or bytes [D 20:00:46.401913] of 136314880 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:46.401920] next loff: 136314880 next poff: 136314880 [D 20:00:46.401926] return value 2097152 [D 20:00:46.401933] finished [D 20:00:46.401938] retval = 2097152 [D 20:00:46.401944] exiting distribute returned less than expected [D 20:00:46.401950] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 136314880 fo 167772160 eof 0 [D 20:00:46.401958] ========================================================= [D 20:00:46.401964] BMI_post_recv: addr: 6843072, offset: 0x2aaaacebf010, size: 2097152, tag: 12 [D 20:00:46.401981] flowproto posted 0x6bb330 [D 20:00:46.401988] Job flows in progress (post time): 5 [D 20:00:46.401996] [SM Exiting]: (0x652610) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:46.504141] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:46.504219] Start times (hr:min:sec): 20:00:45.460 20:00:44.388 20:00:43.364 20:00:42.308 20:00:41.220 20:00:40.180 [P 20:00:46.504226] Intervals (hr:min:sec) : 00:00:01.044 00:00:01.072 00:00:01.024 00:00:01.056 00:00:01.088 00:00:01.040 [P 20:00:46.504233] ------------------------------------------------------------------------------------------------------------- [P 20:00:46.504239] bytes read : 0 0 0 0 0 0 [P 20:00:46.504245] bytes written : 0 0 0 0 0 0 [P 20:00:46.504251] metadata reads : 17 15 14 13 12 0 [P 20:00:46.504258] metadata writes : 7 7 7 7 7 0 [P 20:00:46.504264] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:46.504270] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:46.504276] request scheduler : 5 3 2 1 0 0 [D 20:00:46.504286] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:46.504294] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:46.504303] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:47.175498] openib_check_cq: completed recv wr_id 683db0. [D 20:00:47.175513] openib_post_rr: 10.1.4.56:39990 bh 12 wr_id 683db0. [D 20:00:47.175547] [SM Entering]: (0x6538d0) unexpected_sm:map_request (status: 0) [D 20:00:47.175559] [SM frame get]: (0x6538d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:47.175567] server_post_unexpected_recv [D 20:00:47.175578] [SM Frame PUSH]: (0x654a60) frame: 0x6be0f0 [D 20:00:47.175585] [SM Locating]: (0x654a60) op-id: 999 [D 20:00:47.175592] server_op_state_get_machine 999 [D 20:00:47.175598] [SM Locating]: (0x654a60) located: unexpected_sm:post_unexpected [D 20:00:47.175604] [SM frame get]: (0x654a60) op-id: 999 index: 0 base-frm: 0 [D 20:00:47.175611] [SM Entering]: (0x654a60) unexpected_sm:post_unexpected (status: 0) [D 20:00:47.175618] [SM frame get]: (0x654a60) op-id: 999 index: 0 base-frm: 0 [D 20:00:47.175627] [SM Exiting]: (0x654a60) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:47.175634] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:47.175641] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:47.175647] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 6 [D 20:00:47.175654] [SM frame get]: (0x6538d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:47.175661] server_state_machine_start 0x6538d0 [D 20:00:47.175667] PINT_decode [D 20:00:47.175674] lebf_decode_req [D 20:00:47.175683] [SM Locating]: (0x6538d0) op-id: 3 [D 20:00:47.175690] server_op_state_get_machine 3 [D 20:00:47.175696] [SM push_state]: (0x6538d0) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:47.175703] [SM Locating]: (0x6538d0) located: prelude_sm:setup [D 20:00:47.175710] [SM Entering]: (0x6538d0) prelude_sm:setup (status: 0) [D 20:00:47.175716] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.175723] [SM Exiting]: (0x6538d0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:47.175730] [SM Exiting]: (0x6538d0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:47.175737] [SM push_state]: (0x6538d0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:47.175744] [SM Entering]: (0x6538d0) prelude_work_sm:req_sched (status: 0) [D 20:00:47.175751] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.175757] (0x6539d0) io (prelude sm) state: req_sched [A 20:00:47.181547] troy.scl@10.1.4.56:39990 H=2863311531 S=0x6539d0: io: request [D 20:00:47.181561] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:47.181569] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6bf5b0 [D 20:00:47.181575] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6bf5b0 [D 20:00:47.181582] [SM Exiting]: (0x6538d0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:47.181590] [SM Entering]: (0x6538d0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:47.181597] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [A 20:00:47.187286] troy.scl@10.1.4.56:39990 H=2863311531 S=0x6539d0: io: start [D 20:00:47.187298] (0x6539d0) io (prelude sm) state: getattr_if_needed [D 20:00:47.187307] About to retrieve attributes for handle 2863311531 [D 20:00:47.187316] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:47.187325] [SM Exiting]: (0x6538d0) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:47.187337] [SM Entering]: (0x6538d0) prelude_work_sm:perm_check (status: 0) [D 20:00:47.187344] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.187351] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:47.187360] Final permission check for "io" set error code to 0 [D 20:00:47.187366] (0x6539d0) io (prelude sm) state: perm_check (status = 0) [D 20:00:47.187373] [SM Exiting]: (0x6538d0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:47.187380] [SM pop_state]: (0x6538d0) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:47.187387] [SM pop_state]: (0x6538d0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:47.187399] [SM Entering]: (0x6538d0) io_sm:send_positive_ack (status: 0) [D 20:00:47.187406] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.187413] PINT_encode [D 20:00:47.187419] encode_common [D 20:00:47.187426] lebf_encode_resp [D 20:00:47.187435] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 13 [D 20:00:47.187442] element 0: offset: 0x6bdfb0, size: 24 [D 20:00:47.187451] post_sr: 10.1.4.56:39990 bh 12 len 40 wr_id 682ae0. [D 20:00:47.187465] [SM Exiting]: (0x6538d0) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:47.187484] openib_check_cq: completed send wr_id 682ae0. [D 20:00:47.188157] [SM Entering]: (0x6538d0) io_sm:start_flow (status: 0) [D 20:00:47.188169] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.188177] io_start_flow() issuing flow to write data. [D 20:00:47.188184] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:47.188190] file_req_offset: 167772160, aggregate_size: 33554432, handle: 2863311531 [D 20:00:47.188199] PINT_new_request_states n=1 [D 20:00:47.188206] flowproto posting 0x6bf6b0 [D 20:00:47.188220] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:47.188227] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:00:47.189730] ========================================================= [D 20:00:47.189758] PINT_process_request [D 20:00:47.189765] tiling 7 copies [D 20:00:47.189771] skipping ahead to target_offset [D 20:00:47.189777] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:47.189785] lvl 0 el 0 blk 0 by 0 [D 20:00:47.189791] to 0 ta 167772160 fi 201326592 [D 20:00:47.189798] basic type or contiguous data [D 20:00:47.189804] Request disp [D 20:00:47.189810] contig_offset = 0 contig_size = 234881024 lvl_flag = 1 [D 20:00:47.189817] process logical skip [D 20:00:47.189823] retval = 167772160 [D 20:00:47.189828] exiting logical skip because distribute indicates done [D 20:00:47.189834] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:47.189842] lvl 0 el 0 blk 0 by 167772160 [D 20:00:47.189848] to 167772160 ta 167772160 fi 201326592 [D 20:00:47.189854] basic type or contiguous data [D 20:00:47.189860] Request disp [D 20:00:47.189866] contig_offset = 167772160 contig_size = 67108864 lvl_flag = 1 [D 20:00:47.189873] calling distribute [D 20:00:47.189879] PINT_distribute [D 20:00:47.189884] of 167772160 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:47.189892] begin iteration loff: 167772160 [D 20:00:47.189899] segment exceeds byte limit [D 20:00:47.189905] file being extended [D 20:00:47.189911] process a segment [D 20:00:47.189916] of 167772160 sz 2097152 [D 20:00:47.189923] add a segment 0 [D 20:00:47.189928] end iteration [D 20:00:47.189934] done with segments or bytes [D 20:00:47.189940] of 169869312 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:47.189947] next loff: 169869312 next poff: 169869312 [D 20:00:47.189953] return value 2097152 [D 20:00:47.189959] finished [D 20:00:47.189965] retval = 2097152 [D 20:00:47.189971] exiting distribute returned less than expected [D 20:00:47.189977] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 169869312 fo 201326592 eof 0 [D 20:00:47.189984] ========================================================= [D 20:00:47.189991] BMI_post_recv: addr: 6843072, offset: 0x2aaaad0c0010, size: 2097152, tag: 13 [D 20:00:47.190010] flowproto posted 0x6bf6b0 [D 20:00:47.190018] Job flows in progress (post time): 6 [D 20:00:47.190026] [SM Exiting]: (0x6538d0) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:47.604167] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:47.604239] Start times (hr:min:sec): 20:00:46.504 20:00:45.460 20:00:44.388 20:00:43.364 20:00:42.308 20:00:41.220 [P 20:00:47.604247] Intervals (hr:min:sec) : 00:00:01.100 00:00:01.044 00:00:01.072 00:00:01.024 00:00:01.056 00:00:01.088 [P 20:00:47.604253] ------------------------------------------------------------------------------------------------------------- [P 20:00:47.604260] bytes read : 0 0 0 0 0 0 [P 20:00:47.604266] bytes written : 0 0 0 0 0 0 [P 20:00:47.604272] metadata reads : 18 17 15 14 13 12 [P 20:00:47.604278] metadata writes : 7 7 7 7 7 7 [P 20:00:47.604285] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:47.604291] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:47.604297] request scheduler : 6 5 3 2 1 0 [D 20:00:47.604307] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:47.604315] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:47.604324] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:47.869341] openib_check_cq: completed recv wr_id 683de0. [D 20:00:47.869356] openib_post_rr: 10.1.4.56:39990 bh 13 wr_id 683de0. [D 20:00:47.869394] [SM Entering]: (0x654b90) unexpected_sm:map_request (status: 0) [D 20:00:47.869406] [SM frame get]: (0x654b90) op-id: 999 index: 0 base-frm: 0 [D 20:00:47.869414] server_post_unexpected_recv [D 20:00:47.869427] [SM Frame PUSH]: (0x655d20) frame: 0x6c2ac0 [D 20:00:47.869434] [SM Locating]: (0x655d20) op-id: 999 [D 20:00:47.869441] server_op_state_get_machine 999 [D 20:00:47.869447] [SM Locating]: (0x655d20) located: unexpected_sm:post_unexpected [D 20:00:47.869453] [SM frame get]: (0x655d20) op-id: 999 index: 0 base-frm: 0 [D 20:00:47.869461] [SM Entering]: (0x655d20) unexpected_sm:post_unexpected (status: 0) [D 20:00:47.869468] [SM frame get]: (0x655d20) op-id: 999 index: 0 base-frm: 0 [D 20:00:47.869476] [SM Exiting]: (0x655d20) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:47.869483] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:47.869490] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:47.869497] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 7 [D 20:00:47.869504] [SM frame get]: (0x654b90) op-id: 999 index: 0 base-frm: 0 [D 20:00:47.869510] server_state_machine_start 0x654b90 [D 20:00:47.869517] PINT_decode [D 20:00:47.869524] lebf_decode_req [D 20:00:47.869532] [SM Locating]: (0x654b90) op-id: 3 [D 20:00:47.869539] server_op_state_get_machine 3 [D 20:00:47.869545] [SM push_state]: (0x654b90) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:47.869552] [SM Locating]: (0x654b90) located: prelude_sm:setup [D 20:00:47.869558] [SM Entering]: (0x654b90) prelude_sm:setup (status: 0) [D 20:00:47.869565] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.869573] [SM Exiting]: (0x654b90) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:47.869579] [SM Exiting]: (0x654b90) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:47.869586] [SM push_state]: (0x654b90) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:47.869593] [SM Entering]: (0x654b90) prelude_work_sm:req_sched (status: 0) [D 20:00:47.869600] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.869606] (0x654c90) io (prelude sm) state: req_sched [A 20:00:47.875419] troy.scl@10.1.4.56:39990 H=2863311531 S=0x654c90: io: request [D 20:00:47.875433] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:47.875441] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6c0210 [D 20:00:47.875448] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6c0210 [D 20:00:47.875455] [SM Exiting]: (0x654b90) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:47.875462] [SM Entering]: (0x654b90) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:47.875469] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [A 20:00:47.881408] troy.scl@10.1.4.56:39990 H=2863311531 S=0x654c90: io: start [D 20:00:47.881420] (0x654c90) io (prelude sm) state: getattr_if_needed [D 20:00:47.881428] About to retrieve attributes for handle 2863311531 [D 20:00:47.881438] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:47.881447] [SM Exiting]: (0x654b90) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:47.881458] [SM Entering]: (0x654b90) prelude_work_sm:perm_check (status: 0) [D 20:00:47.881465] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.881473] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:47.881482] Final permission check for "io" set error code to 0 [D 20:00:47.881489] (0x654c90) io (prelude sm) state: perm_check (status = 0) [D 20:00:47.881496] [SM Exiting]: (0x654b90) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:47.881503] [SM pop_state]: (0x654b90) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:47.881510] [SM pop_state]: (0x654b90) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:47.881520] [SM Entering]: (0x654b90) io_sm:send_positive_ack (status: 0) [D 20:00:47.881527] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.881534] PINT_encode [D 20:00:47.881541] encode_common [D 20:00:47.881553] lebf_encode_resp [D 20:00:47.881561] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 14 [D 20:00:47.881569] element 0: offset: 0x6bfd60, size: 24 [D 20:00:47.881578] post_sr: 10.1.4.56:39990 bh 13 len 40 wr_id 682b10. [D 20:00:47.881588] [SM Exiting]: (0x654b90) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:47.881607] openib_check_cq: completed send wr_id 682b10. [D 20:00:47.884175] [SM Entering]: (0x654b90) io_sm:start_flow (status: 0) [D 20:00:47.884187] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [D 20:00:47.884196] io_start_flow() issuing flow to write data. [D 20:00:47.884202] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:47.884209] file_req_offset: 201326592, aggregate_size: 33554432, handle: 2863311531 [D 20:00:47.884217] PINT_new_request_states n=1 [D 20:00:47.884225] flowproto posting 0x6c0290 [D 20:00:47.884241] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:47.884249] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:00:47.885745] ========================================================= [D 20:00:47.885772] PINT_process_request [D 20:00:47.885779] tiling 8 copies [D 20:00:47.885785] skipping ahead to target_offset [D 20:00:47.885791] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:47.885798] lvl 0 el 0 blk 0 by 0 [D 20:00:47.885805] to 0 ta 201326592 fi 234881024 [D 20:00:47.885811] basic type or contiguous data [D 20:00:47.885818] Request disp [D 20:00:47.885824] contig_offset = 0 contig_size = 268435456 lvl_flag = 1 [D 20:00:47.885830] process logical skip [D 20:00:47.885836] retval = 201326592 [D 20:00:47.885842] exiting logical skip because distribute indicates done [D 20:00:47.885848] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:47.885855] lvl 0 el 0 blk 0 by 201326592 [D 20:00:47.885861] to 201326592 ta 201326592 fi 234881024 [D 20:00:47.885867] basic type or contiguous data [D 20:00:47.885874] Request disp [D 20:00:47.885879] contig_offset = 201326592 contig_size = 67108864 lvl_flag = 1 [D 20:00:47.885886] calling distribute [D 20:00:47.885892] PINT_distribute [D 20:00:47.885898] of 201326592 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:47.885905] begin iteration loff: 201326592 [D 20:00:47.885912] segment exceeds byte limit [D 20:00:47.885918] file being extended [D 20:00:47.885924] process a segment [D 20:00:47.885930] of 201326592 sz 2097152 [D 20:00:47.885936] add a segment 0 [D 20:00:47.885942] end iteration [D 20:00:47.885948] done with segments or bytes [D 20:00:47.885954] of 203423744 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:47.885961] next loff: 203423744 next poff: 203423744 [D 20:00:47.885967] return value 2097152 [D 20:00:47.885973] finished [D 20:00:47.885979] retval = 2097152 [D 20:00:47.885985] exiting distribute returned less than expected [D 20:00:47.885991] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 203423744 fo 234881024 eof 0 [D 20:00:47.885998] ========================================================= [D 20:00:47.886005] BMI_post_recv: addr: 6843072, offset: 0x2aaaad2c1010, size: 2097152, tag: 14 [D 20:00:47.886021] flowproto posted 0x6c0290 [D 20:00:47.886029] Job flows in progress (post time): 7 [D 20:00:47.886037] [SM Exiting]: (0x654b90) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:48.612192] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:48.612263] Start times (hr:min:sec): 20:00:47.604 20:00:46.504 20:00:45.460 20:00:44.388 20:00:43.364 20:00:42.308 [P 20:00:48.612271] Intervals (hr:min:sec) : 00:00:01.008 00:00:01.100 00:00:01.044 00:00:01.072 00:00:01.024 00:00:01.056 [P 20:00:48.612277] ------------------------------------------------------------------------------------------------------------- [P 20:00:48.612284] bytes read : 0 0 0 0 0 0 [P 20:00:48.612290] bytes written : 0 0 0 0 0 0 [P 20:00:48.612296] metadata reads : 19 18 17 15 14 13 [P 20:00:48.612303] metadata writes : 7 7 7 7 7 7 [P 20:00:48.612309] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:48.612315] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:48.612321] request scheduler : 7 6 5 3 2 1 [D 20:00:48.612331] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:48.612339] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:48.612348] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:48.685644] openib_check_cq: completed recv wr_id 683e10. [D 20:00:48.685660] openib_post_rr: 10.1.4.56:39990 bh 14 wr_id 683e10. [D 20:00:48.685694] [SM Entering]: (0x655e50) unexpected_sm:map_request (status: 0) [D 20:00:48.685706] [SM frame get]: (0x655e50) op-id: 999 index: 0 base-frm: 0 [D 20:00:48.685714] server_post_unexpected_recv [D 20:00:48.685727] [SM Frame PUSH]: (0x656fe0) frame: 0x6c6840 [D 20:00:48.685734] [SM Locating]: (0x656fe0) op-id: 999 [D 20:00:48.685741] server_op_state_get_machine 999 [D 20:00:48.685747] [SM Locating]: (0x656fe0) located: unexpected_sm:post_unexpected [D 20:00:48.685754] [SM frame get]: (0x656fe0) op-id: 999 index: 0 base-frm: 0 [D 20:00:48.685761] [SM Entering]: (0x656fe0) unexpected_sm:post_unexpected (status: 0) [D 20:00:48.685767] [SM frame get]: (0x656fe0) op-id: 999 index: 0 base-frm: 0 [D 20:00:48.685776] [SM Exiting]: (0x656fe0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:48.685784] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:48.685791] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:48.685797] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 8 [D 20:00:48.685804] [SM frame get]: (0x655e50) op-id: 999 index: 0 base-frm: 0 [D 20:00:48.685811] server_state_machine_start 0x655e50 [D 20:00:48.685817] PINT_decode [D 20:00:48.685824] lebf_decode_req [D 20:00:48.685834] [SM Locating]: (0x655e50) op-id: 3 [D 20:00:48.685840] server_op_state_get_machine 3 [D 20:00:48.685847] [SM push_state]: (0x655e50) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:48.685853] [SM Locating]: (0x655e50) located: prelude_sm:setup [D 20:00:48.685860] [SM Entering]: (0x655e50) prelude_sm:setup (status: 0) [D 20:00:48.685867] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [D 20:00:48.685874] [SM Exiting]: (0x655e50) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:48.685881] [SM Exiting]: (0x655e50) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:48.685888] [SM push_state]: (0x655e50) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:48.685895] [SM Entering]: (0x655e50) prelude_work_sm:req_sched (status: 0) [D 20:00:48.685901] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [D 20:00:48.685908] (0x655f50) io (prelude sm) state: req_sched [A 20:00:48.691850] troy.scl@10.1.4.56:39990 H=2863311531 S=0x655f50: io: request [D 20:00:48.691864] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:48.691871] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6c7cd0 [D 20:00:48.691878] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6c7cd0 [D 20:00:48.691885] [SM Exiting]: (0x655e50) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:48.691893] [SM Entering]: (0x655e50) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:48.691899] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [A 20:00:48.697715] troy.scl@10.1.4.56:39990 H=2863311531 S=0x655f50: io: start [D 20:00:48.697726] (0x655f50) io (prelude sm) state: getattr_if_needed [D 20:00:48.697735] About to retrieve attributes for handle 2863311531 [D 20:00:48.697745] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:48.697755] [SM Exiting]: (0x655e50) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:48.697765] [SM Entering]: (0x655e50) prelude_work_sm:perm_check (status: 0) [D 20:00:48.697772] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [D 20:00:48.697780] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:48.697789] Final permission check for "io" set error code to 0 [D 20:00:48.697806] (0x655f50) io (prelude sm) state: perm_check (status = 0) [D 20:00:48.697814] [SM Exiting]: (0x655e50) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:48.697821] [SM pop_state]: (0x655e50) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:48.697831] [SM pop_state]: (0x655e50) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:48.697838] [SM Entering]: (0x655e50) io_sm:send_positive_ack (status: 0) [D 20:00:48.697845] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [D 20:00:48.697852] PINT_encode [D 20:00:48.697864] encode_common [D 20:00:48.697871] lebf_encode_resp [D 20:00:48.697878] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 15 [D 20:00:48.697885] element 0: offset: 0x6bf860, size: 24 [D 20:00:48.697899] post_sr: 10.1.4.56:39990 bh 14 len 40 wr_id 682b40. [D 20:00:48.697909] [SM Exiting]: (0x655e50) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:48.697953] openib_check_cq: completed send wr_id 682b40. [D 20:00:48.700201] [SM Entering]: (0x655e50) io_sm:start_flow (status: 0) [D 20:00:48.700213] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [D 20:00:48.700221] io_start_flow() issuing flow to write data. [D 20:00:48.700228] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:48.700234] file_req_offset: 234881024, aggregate_size: 33554432, handle: 2863311531 [D 20:00:48.700243] PINT_new_request_states n=1 [D 20:00:48.700250] flowproto posting 0x6c7f30 [D 20:00:48.700267] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:48.700275] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:00:48.701773] ========================================================= [D 20:00:48.701801] PINT_process_request [D 20:00:48.701808] tiling 9 copies [D 20:00:48.701815] skipping ahead to target_offset [D 20:00:48.701821] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:48.701829] lvl 0 el 0 blk 0 by 0 [D 20:00:48.701835] to 0 ta 234881024 fi 268435456 [D 20:00:48.701842] basic type or contiguous data [D 20:00:48.701848] Request disp [D 20:00:48.701854] contig_offset = 0 contig_size = 301989888 lvl_flag = 1 [D 20:00:48.701861] process logical skip [D 20:00:48.701867] retval = 234881024 [D 20:00:48.701873] exiting logical skip because distribute indicates done [D 20:00:48.701879] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:48.701886] lvl 0 el 0 blk 0 by 234881024 [D 20:00:48.701892] to 234881024 ta 234881024 fi 268435456 [D 20:00:48.701899] basic type or contiguous data [D 20:00:48.701904] Request disp [D 20:00:48.701910] contig_offset = 234881024 contig_size = 67108864 lvl_flag = 1 [D 20:00:48.701916] calling distribute [D 20:00:48.701922] PINT_distribute [D 20:00:48.701928] of 234881024 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:48.701936] begin iteration loff: 234881024 [D 20:00:48.701942] segment exceeds byte limit [D 20:00:48.701948] file being extended [D 20:00:48.701954] process a segment [D 20:00:48.701960] of 234881024 sz 2097152 [D 20:00:48.701966] add a segment 0 [D 20:00:48.701972] end iteration [D 20:00:48.701978] done with segments or bytes [D 20:00:48.701984] of 236978176 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:48.701991] next loff: 236978176 next poff: 236978176 [D 20:00:48.701997] return value 2097152 [D 20:00:48.702003] finished [D 20:00:48.702009] retval = 2097152 [D 20:00:48.702015] exiting distribute returned less than expected [D 20:00:48.702021] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 236978176 fo 268435456 eof 0 [D 20:00:48.702028] ========================================================= [D 20:00:48.702035] BMI_post_recv: addr: 6843072, offset: 0x2aaaad4c2010, size: 2097152, tag: 15 [D 20:00:48.702053] flowproto posted 0x6c7f30 [D 20:00:48.702061] Job flows in progress (post time): 8 [D 20:00:48.702069] [SM Exiting]: (0x655e50) io_sm:start_flow (error code: 0), (action: DEFERRED) -[D 20:00:49.357080] openib_check_cq: completed recv wr_id 683e40. [D 20:00:49.357095] openib_post_rr: 10.1.4.56:39990 bh 15 wr_id 683e40. [D 20:00:49.357133] [SM Entering]: (0x657110) unexpected_sm:map_request (status: 0) [D 20:00:49.357145] [SM frame get]: (0x657110) op-id: 999 index: 0 base-frm: 0 [D 20:00:49.357153] server_post_unexpected_recv [D 20:00:49.357167] [SM Frame PUSH]: (0x6582a0) frame: 0x6caaa0 [D 20:00:49.357174] [SM Locating]: (0x6582a0) op-id: 999 [D 20:00:49.357180] server_op_state_get_machine 999 [D 20:00:49.357187] [SM Locating]: (0x6582a0) located: unexpected_sm:post_unexpected [D 20:00:49.357193] [SM frame get]: (0x6582a0) op-id: 999 index: 0 base-frm: 0 [D 20:00:49.357200] [SM Entering]: (0x6582a0) unexpected_sm:post_unexpected (status: 0) [D 20:00:49.357207] [SM frame get]: (0x6582a0) op-id: 999 index: 0 base-frm: 0 [D 20:00:49.357216] [SM Exiting]: (0x6582a0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:49.357224] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:49.357231] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:49.357237] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 9 [D 20:00:49.357244] [SM frame get]: (0x657110) op-id: 999 index: 0 base-frm: 0 [D 20:00:49.357250] server_state_machine_start 0x657110 [D 20:00:49.357256] PINT_decode [D 20:00:49.357263] lebf_decode_req [D 20:00:49.357273] [SM Locating]: (0x657110) op-id: 3 [D 20:00:49.357280] server_op_state_get_machine 3 [D 20:00:49.357286] [SM push_state]: (0x657110) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:49.357293] [SM Locating]: (0x657110) located: prelude_sm:setup [D 20:00:49.357300] [SM Entering]: (0x657110) prelude_sm:setup (status: 0) [D 20:00:49.357306] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [D 20:00:49.357313] [SM Exiting]: (0x657110) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:49.357320] [SM Exiting]: (0x657110) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:49.357327] [SM push_state]: (0x657110) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:49.357334] [SM Entering]: (0x657110) prelude_work_sm:req_sched (status: 0) [D 20:00:49.357340] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [D 20:00:49.357347] (0x657210) io (prelude sm) state: req_sched [A 20:00:49.363358] troy.scl@10.1.4.56:39990 H=2863311531 S=0x657210: io: request [D 20:00:49.363372] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:49.363380] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6cc030 [D 20:00:49.363386] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6cc030 [D 20:00:49.363393] [SM Exiting]: (0x657110) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:49.363401] [SM Entering]: (0x657110) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:49.363408] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [A 20:00:49.369474] troy.scl@10.1.4.56:39990 H=2863311531 S=0x657210: io: start [D 20:00:49.369485] (0x657210) io (prelude sm) state: getattr_if_needed [D 20:00:49.369494] About to retrieve attributes for handle 2863311531 [D 20:00:49.369503] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:49.369512] [SM Exiting]: (0x657110) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:49.369524] [SM Entering]: (0x657110) prelude_work_sm:perm_check (status: 0) [D 20:00:49.369531] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [D 20:00:49.369538] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:49.369547] Final permission check for "io" set error code to 0 [D 20:00:49.369564] (0x657210) io (prelude sm) state: perm_check (status = 0) [D 20:00:49.369571] [SM Exiting]: (0x657110) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:49.369586] [SM pop_state]: (0x657110) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:49.369592] [SM pop_state]: (0x657110) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:49.369599] [SM Entering]: (0x657110) io_sm:send_positive_ack (status: 0) [D 20:00:49.369606] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [D 20:00:49.369613] PINT_encode [D 20:00:49.369620] encode_common [D 20:00:49.369627] lebf_encode_resp [D 20:00:49.369634] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 16 [D 20:00:49.369641] element 0: offset: 0x6c7d90, size: 24 [D 20:00:49.369656] post_sr: 10.1.4.56:39990 bh 15 len 40 wr_id 682b70. [D 20:00:49.369666] [SM Exiting]: (0x657110) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:49.369705] openib_check_cq: completed send wr_id 682b70. [D 20:00:49.372219] [SM Entering]: (0x657110) io_sm:start_flow (status: 0) [D 20:00:49.372231] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [D 20:00:49.372240] io_start_flow() issuing flow to write data. [D 20:00:49.372246] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:49.372253] file_req_offset: 268435456, aggregate_size: 33554432, handle: 2863311531 [D 20:00:49.372261] PINT_new_request_states n=1 [D 20:00:49.372268] flowproto posting 0x6cc2b0 [D 20:00:49.372284] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:49.372292] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:00:49.373794] ========================================================= [D 20:00:49.373823] PINT_process_request [D 20:00:49.373830] tiling 10 copies [D 20:00:49.373837] skipping ahead to target_offset [D 20:00:49.373843] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:49.373851] lvl 0 el 0 blk 0 by 0 [D 20:00:49.373857] to 0 ta 268435456 fi 301989888 [D 20:00:49.373864] basic type or contiguous data [D 20:00:49.373870] Request disp [D 20:00:49.373876] contig_offset = 0 contig_size = 335544320 lvl_flag = 1 [D 20:00:49.373883] process logical skip [D 20:00:49.373889] retval = 268435456 [D 20:00:49.373895] exiting logical skip because distribute indicates done [D 20:00:49.373901] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:49.373908] lvl 0 el 0 blk 0 by 268435456 [D 20:00:49.373915] to 268435456 ta 268435456 fi 301989888 [D 20:00:49.373921] basic type or contiguous data [D 20:00:49.373927] Request disp [D 20:00:49.373933] contig_offset = 268435456 contig_size = 67108864 lvl_flag = 1 [D 20:00:49.373939] calling distribute [D 20:00:49.373945] PINT_distribute [D 20:00:49.373951] of 268435456 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:49.373959] begin iteration loff: 268435456 [D 20:00:49.373965] segment exceeds byte limit [D 20:00:49.373971] file being extended [D 20:00:49.373977] process a segment [D 20:00:49.373983] of 268435456 sz 2097152 [D 20:00:49.373989] add a segment 0 [D 20:00:49.373995] end iteration [D 20:00:49.374001] done with segments or bytes [D 20:00:49.374006] of 270532608 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:49.374013] next loff: 270532608 next poff: 270532608 [D 20:00:49.374020] return value 2097152 [D 20:00:49.374026] finished [D 20:00:49.374032] retval = 2097152 [D 20:00:49.374037] exiting distribute returned less than expected [D 20:00:49.374044] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 270532608 fo 301989888 eof 0 [D 20:00:49.374051] ========================================================= [D 20:00:49.374057] BMI_post_recv: addr: 6843072, offset: 0x2aaaad6c3010, size: 2097152, tag: 16 [D 20:00:49.374075] flowproto posted 0x6cc2b0 [D 20:00:49.374083] Job flows in progress (post time): 9 [D 20:00:49.374092] [SM Exiting]: (0x657110) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:49.684220] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:49.684297] Start times (hr:min:sec): 20:00:48.612 20:00:47.604 20:00:46.504 20:00:45.460 20:00:44.388 20:00:43.364 [P 20:00:49.684305] Intervals (hr:min:sec) : 00:00:01.072 00:00:01.008 00:00:01.100 00:00:01.044 00:00:01.072 00:00:01.024 [P 20:00:49.684312] ------------------------------------------------------------------------------------------------------------- [P 20:00:49.684318] bytes read : 0 0 0 0 0 0 [P 20:00:49.684324] bytes written : 0 0 0 0 0 0 [P 20:00:49.684330] metadata reads : 21 19 18 17 15 14 [P 20:00:49.684337] metadata writes : 7 7 7 7 7 7 [P 20:00:49.684343] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:49.684349] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:49.684355] request scheduler : 9 7 6 5 3 2 [D 20:00:49.684366] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:49.684374] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:49.684382] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:50.361350] openib_check_cq: completed recv wr_id 683e70. [D 20:00:50.361366] openib_post_rr: 10.1.4.56:39990 bh 16 wr_id 683e70. [D 20:00:50.361387] [SM Entering]: (0x6583d0) unexpected_sm:map_request (status: 0) [D 20:00:50.361400] [SM frame get]: (0x6583d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:50.361408] server_post_unexpected_recv [D 20:00:50.361416] [SM Frame PUSH]: (0x659560) frame: 0x6ceca0 [D 20:00:50.361423] [SM Locating]: (0x659560) op-id: 999 [D 20:00:50.361429] server_op_state_get_machine 999 [D 20:00:50.361435] [SM Locating]: (0x659560) located: unexpected_sm:post_unexpected [D 20:00:50.361442] [SM frame get]: (0x659560) op-id: 999 index: 0 base-frm: 0 [D 20:00:50.361449] [SM Entering]: (0x659560) unexpected_sm:post_unexpected (status: 0) [D 20:00:50.361456] [SM frame get]: (0x659560) op-id: 999 index: 0 base-frm: 0 [D 20:00:50.361464] [SM Exiting]: (0x659560) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:50.361472] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:50.361479] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:50.361485] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 10 [D 20:00:50.361492] [SM frame get]: (0x6583d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:50.361498] server_state_machine_start 0x6583d0 [D 20:00:50.361505] PINT_decode [D 20:00:50.361512] lebf_decode_req [D 20:00:50.361521] [SM Locating]: (0x6583d0) op-id: 3 [D 20:00:50.361528] server_op_state_get_machine 3 [D 20:00:50.361534] [SM push_state]: (0x6583d0) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:50.361541] [SM Locating]: (0x6583d0) located: prelude_sm:setup [D 20:00:50.361548] [SM Entering]: (0x6583d0) prelude_sm:setup (status: 0) [D 20:00:50.361554] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:50.361562] [SM Exiting]: (0x6583d0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:50.361568] [SM Exiting]: (0x6583d0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:50.361575] [SM push_state]: (0x6583d0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:50.361582] [SM Entering]: (0x6583d0) prelude_work_sm:req_sched (status: 0) [D 20:00:50.361589] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:50.361595] (0x6584d0) io (prelude sm) state: req_sched [A 20:00:50.367565] troy.scl@10.1.4.56:39990 H=2863311531 S=0x6584d0: io: request [D 20:00:50.367579] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:50.367587] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6cfff0 [D 20:00:50.367593] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6cfff0 [D 20:00:50.367600] [SM Exiting]: (0x6583d0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:50.367607] [SM Entering]: (0x6583d0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:50.367614] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [A 20:00:50.373553] troy.scl@10.1.4.56:39990 H=2863311531 S=0x6584d0: io: start [D 20:00:50.373565] (0x6584d0) io (prelude sm) state: getattr_if_needed [D 20:00:50.373573] About to retrieve attributes for handle 2863311531 [D 20:00:50.373583] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:50.373593] [SM Exiting]: (0x6583d0) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:50.373603] [SM Entering]: (0x6583d0) prelude_work_sm:perm_check (status: 0) [D 20:00:50.373610] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:50.373618] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:50.373627] Final permission check for "io" set error code to 0 [D 20:00:50.373633] (0x6584d0) io (prelude sm) state: perm_check (status = 0) [D 20:00:50.373640] [SM Exiting]: (0x6583d0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:50.373647] [SM pop_state]: (0x6583d0) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:50.373654] [SM pop_state]: (0x6583d0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:50.373664] [SM Entering]: (0x6583d0) io_sm:send_positive_ack (status: 0) [D 20:00:50.373671] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:50.373678] PINT_encode [D 20:00:50.373684] encode_common [D 20:00:50.373696] lebf_encode_resp [D 20:00:50.373703] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 17 [D 20:00:50.373711] element 0: offset: 0x6c7f10, size: 24 [D 20:00:50.373720] post_sr: 10.1.4.56:39990 bh 16 len 40 wr_id 682ba0. [D 20:00:50.373730] [SM Exiting]: (0x6583d0) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:50.373748] openib_check_cq: completed send wr_id 682ba0. [D 20:00:50.376238] [SM Entering]: (0x6583d0) io_sm:start_flow (status: 0) [D 20:00:50.376250] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:50.376259] io_start_flow() issuing flow to write data. [D 20:00:50.376265] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:50.376272] file_req_offset: 301989888, aggregate_size: 33554432, handle: 2863311531 [D 20:00:50.376280] PINT_new_request_states n=1 [D 20:00:50.376288] flowproto posting 0x6d01a0 [D 20:00:50.376300] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:50.376308] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:00:50.377772] ========================================================= [D 20:00:50.377800] PINT_process_request [D 20:00:50.377807] tiling 11 copies [D 20:00:50.377813] skipping ahead to target_offset [D 20:00:50.377820] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:50.377827] lvl 0 el 0 blk 0 by 0 [D 20:00:50.377834] to 0 ta 301989888 fi 335544320 [D 20:00:50.377841] basic type or contiguous data [D 20:00:50.377847] Request disp [D 20:00:50.377853] contig_offset = 0 contig_size = 369098752 lvl_flag = 1 [D 20:00:50.377859] process logical skip [D 20:00:50.377865] retval = 301989888 [D 20:00:50.377871] exiting logical skip because distribute indicates done [D 20:00:50.377877] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:50.377884] lvl 0 el 0 blk 0 by 301989888 [D 20:00:50.377891] to 301989888 ta 301989888 fi 335544320 [D 20:00:50.377897] basic type or contiguous data [D 20:00:50.377903] Request disp [D 20:00:50.377909] contig_offset = 301989888 contig_size = 67108864 lvl_flag = 1 [D 20:00:50.377915] calling distribute [D 20:00:50.377921] PINT_distribute [D 20:00:50.377927] of 301989888 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:50.377935] begin iteration loff: 301989888 [D 20:00:50.377941] segment exceeds byte limit [D 20:00:50.377947] file being extended [D 20:00:50.377953] process a segment [D 20:00:50.377959] of 301989888 sz 2097152 [D 20:00:50.377965] add a segment 0 [D 20:00:50.377972] end iteration [D 20:00:50.377977] done with segments or bytes [D 20:00:50.377983] of 304087040 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:50.377990] next loff: 304087040 next poff: 304087040 [D 20:00:50.377996] return value 2097152 [D 20:00:50.378003] finished [D 20:00:50.378008] retval = 2097152 [D 20:00:50.378014] exiting distribute returned less than expected [D 20:00:50.378020] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 304087040 fo 335544320 eof 0 [D 20:00:50.378028] ========================================================= [D 20:00:50.378034] BMI_post_recv: addr: 6843072, offset: 0x2aaaad8c4010, size: 2097152, tag: 17 [D 20:00:50.378052] flowproto posted 0x6d01a0 [D 20:00:50.378060] Job flows in progress (post time): 10 [D 20:00:50.378068] [SM Exiting]: (0x6583d0) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:50.688244] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:50.688321] Start times (hr:min:sec): 20:00:49.684 20:00:48.612 20:00:47.604 20:00:46.504 20:00:45.460 20:00:44.388 [P 20:00:50.688328] Intervals (hr:min:sec) : 00:00:01.004 00:00:01.072 00:00:01.008 00:00:01.100 00:00:01.044 00:00:01.072 [P 20:00:50.688335] ------------------------------------------------------------------------------------------------------------- [P 20:00:50.688341] bytes read : 0 0 0 0 0 0 [P 20:00:50.688347] bytes written : 0 0 0 0 0 0 [P 20:00:50.688353] metadata reads : 22 21 19 18 17 15 [P 20:00:50.688359] metadata writes : 7 7 7 7 7 7 [P 20:00:50.688366] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:50.688372] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:50.688378] request scheduler : 10 9 7 6 5 3 [D 20:00:50.688388] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:50.688396] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:50.688405] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:51.492998] openib_check_cq: completed recv wr_id 683ea0. [D 20:00:51.493014] openib_post_rr: 10.1.4.56:39990 bh 17 wr_id 683ea0. [D 20:00:51.493053] [SM Entering]: (0x659690) unexpected_sm:map_request (status: 0) [D 20:00:51.493065] [SM frame get]: (0x659690) op-id: 999 index: 0 base-frm: 0 [D 20:00:51.493074] server_post_unexpected_recv [D 20:00:51.493081] [SM Frame PUSH]: (0x65a820) frame: 0x6d2f00 [D 20:00:51.493088] [SM Locating]: (0x65a820) op-id: 999 [D 20:00:51.493094] server_op_state_get_machine 999 [D 20:00:51.493100] [SM Locating]: (0x65a820) located: unexpected_sm:post_unexpected [D 20:00:51.493107] [SM frame get]: (0x65a820) op-id: 999 index: 0 base-frm: 0 [D 20:00:51.493114] [SM Entering]: (0x65a820) unexpected_sm:post_unexpected (status: 0) [D 20:00:51.493121] [SM frame get]: (0x65a820) op-id: 999 index: 0 base-frm: 0 [D 20:00:51.493133] [SM Exiting]: (0x65a820) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:51.493141] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:51.493148] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:51.493155] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 11 [D 20:00:51.493161] [SM frame get]: (0x659690) op-id: 999 index: 0 base-frm: 0 [D 20:00:51.493168] server_state_machine_start 0x659690 [D 20:00:51.493174] PINT_decode [D 20:00:51.493181] lebf_decode_req [D 20:00:51.493190] [SM Locating]: (0x659690) op-id: 3 [D 20:00:51.493197] server_op_state_get_machine 3 [D 20:00:51.493204] [SM push_state]: (0x659690) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:51.493210] [SM Locating]: (0x659690) located: prelude_sm:setup [D 20:00:51.493217] [SM Entering]: (0x659690) prelude_sm:setup (status: 0) [D 20:00:51.493224] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [D 20:00:51.493231] [SM Exiting]: (0x659690) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:51.493238] [SM Exiting]: (0x659690) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:51.493245] [SM push_state]: (0x659690) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:51.493251] [SM Entering]: (0x659690) prelude_work_sm:req_sched (status: 0) [D 20:00:51.493258] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [D 20:00:51.493265] (0x659790) io (prelude sm) state: req_sched [A 20:00:51.499321] troy.scl@10.1.4.56:39990 H=2863311531 S=0x659790: io: request [D 20:00:51.499336] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:51.499344] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6d4250 [D 20:00:51.499351] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6d4250 [D 20:00:51.499358] [SM Exiting]: (0x659690) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:51.499365] [SM Entering]: (0x659690) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:51.499372] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [A 20:00:51.505312] troy.scl@10.1.4.56:39990 H=2863311531 S=0x659790: io: start [D 20:00:51.505324] (0x659790) io (prelude sm) state: getattr_if_needed [D 20:00:51.505332] About to retrieve attributes for handle 2863311531 [D 20:00:51.505342] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:51.505351] [SM Exiting]: (0x659690) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:51.505363] [SM Entering]: (0x659690) prelude_work_sm:perm_check (status: 0) [D 20:00:51.505370] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [D 20:00:51.505378] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:51.505387] Final permission check for "io" set error code to 0 [D 20:00:51.505394] (0x659790) io (prelude sm) state: perm_check (status = 0) [D 20:00:51.505400] [SM Exiting]: (0x659690) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:51.505408] [SM pop_state]: (0x659690) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:51.505414] [SM pop_state]: (0x659690) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:51.505426] [SM Entering]: (0x659690) io_sm:send_positive_ack (status: 0) [D 20:00:51.505433] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [D 20:00:51.505440] PINT_encode [D 20:00:51.505446] encode_common [D 20:00:51.505453] lebf_encode_resp [D 20:00:51.505462] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 18 [D 20:00:51.505470] element 0: offset: 0x6c7b80, size: 24 [D 20:00:51.505478] post_sr: 10.1.4.56:39990 bh 17 len 40 wr_id 682bd0. [D 20:00:51.505493] [SM Exiting]: (0x659690) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:51.505512] openib_check_cq: completed send wr_id 682bd0. [D 20:00:51.508260] [SM Entering]: (0x659690) io_sm:start_flow (status: 0) [D 20:00:51.508271] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [D 20:00:51.508281] io_start_flow() issuing flow to write data. [D 20:00:51.508288] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:51.508294] file_req_offset: 335544320, aggregate_size: 33554432, handle: 2863311531 [D 20:00:51.508303] PINT_new_request_states n=1 [D 20:00:51.508311] flowproto posting 0x6d4400 [D 20:00:51.508324] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:51.508335] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:00:51.512319] ========================================================= [D 20:00:51.512352] PINT_process_request [D 20:00:51.512360] tiling 12 copies [D 20:00:51.512366] skipping ahead to target_offset [D 20:00:51.512372] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:51.512380] lvl 0 el 0 blk 0 by 0 [D 20:00:51.512387] to 0 ta 335544320 fi 369098752 [D 20:00:51.512394] basic type or contiguous data [D 20:00:51.512400] Request disp [D 20:00:51.512406] contig_offset = 0 contig_size = 402653184 lvl_flag = 1 [D 20:00:51.512413] process logical skip [D 20:00:51.512419] retval = 335544320 [D 20:00:51.512425] exiting logical skip because distribute indicates done [D 20:00:51.512431] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:51.512438] lvl 0 el 0 blk 0 by 335544320 [D 20:00:51.512444] to 335544320 ta 335544320 fi 369098752 [D 20:00:51.512451] basic type or contiguous data [D 20:00:51.512457] Request disp [D 20:00:51.512463] contig_offset = 335544320 contig_size = 67108864 lvl_flag = 1 [D 20:00:51.512469] calling distribute [D 20:00:51.512476] PINT_distribute [D 20:00:51.512482] of 335544320 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:51.512489] begin iteration loff: 335544320 [D 20:00:51.512496] segment exceeds byte limit [D 20:00:51.512502] file being extended [D 20:00:51.512508] process a segment [D 20:00:51.512513] of 335544320 sz 2097152 [D 20:00:51.512519] add a segment 0 [D 20:00:51.512525] end iteration [D 20:00:51.512531] done with segments or bytes [D 20:00:51.512537] of 337641472 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:51.512544] next loff: 337641472 next poff: 337641472 [D 20:00:51.512550] return value 2097152 [D 20:00:51.512556] finished [D 20:00:51.512562] retval = 2097152 [D 20:00:51.512568] exiting distribute returned less than expected [D 20:00:51.512574] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 337641472 fo 369098752 eof 0 [D 20:00:51.512581] ========================================================= [D 20:00:51.512588] BMI_post_recv: addr: 6843072, offset: 0x2aaaadac5010, size: 2097152, tag: 18 [D 20:00:51.512601] flowproto posted 0x6d4400 [D 20:00:51.512609] Job flows in progress (post time): 11 [D 20:00:51.512617] [SM Exiting]: (0x659690) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:51.720271] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:51.720347] Start times (hr:min:sec): 20:00:50.688 20:00:49.684 20:00:48.612 20:00:47.604 20:00:46.504 20:00:45.460 [P 20:00:51.720354] Intervals (hr:min:sec) : 00:00:01.032 00:00:01.004 00:00:01.072 00:00:01.008 00:00:01.100 00:00:01.044 [P 20:00:51.720361] ------------------------------------------------------------------------------------------------------------- [P 20:00:51.720367] bytes read : 0 0 0 0 0 0 [P 20:00:51.720373] bytes written : 0 0 0 0 0 0 [P 20:00:51.720379] metadata reads : 23 22 21 19 18 17 [P 20:00:51.720386] metadata writes : 7 7 7 7 7 7 [P 20:00:51.720392] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:51.720398] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:51.720404] request scheduler : 11 10 9 7 6 5 [D 20:00:51.720414] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:51.720423] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:51.720431] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:52.636917] openib_check_cq: completed recv wr_id 683ed0. [D 20:00:52.636931] openib_post_rr: 10.1.4.56:39990 bh 18 wr_id 683ed0. [D 20:00:52.636971] [SM Entering]: (0x65a950) unexpected_sm:map_request (status: 0) [D 20:00:52.636982] [SM frame get]: (0x65a950) op-id: 999 index: 0 base-frm: 0 [D 20:00:52.636990] server_post_unexpected_recv [D 20:00:52.637002] [SM Frame PUSH]: (0x65bae0) frame: 0x6d71a0 [D 20:00:52.637009] [SM Locating]: (0x65bae0) op-id: 999 [D 20:00:52.637016] server_op_state_get_machine 999 [D 20:00:52.637022] [SM Locating]: (0x65bae0) located: unexpected_sm:post_unexpected [D 20:00:52.637028] [SM frame get]: (0x65bae0) op-id: 999 index: 0 base-frm: 0 [D 20:00:52.637036] [SM Entering]: (0x65bae0) unexpected_sm:post_unexpected (status: 0) [D 20:00:52.637042] [SM frame get]: (0x65bae0) op-id: 999 index: 0 base-frm: 0 [D 20:00:52.637051] [SM Exiting]: (0x65bae0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:52.637058] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:52.637065] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:52.637071] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 12 [D 20:00:52.637079] [SM frame get]: (0x65a950) op-id: 999 index: 0 base-frm: 0 [D 20:00:52.637085] server_state_machine_start 0x65a950 [D 20:00:52.637091] PINT_decode [D 20:00:52.637099] lebf_decode_req [D 20:00:52.637108] [SM Locating]: (0x65a950) op-id: 3 [D 20:00:52.637115] server_op_state_get_machine 3 [D 20:00:52.637121] [SM push_state]: (0x65a950) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:52.637128] [SM Locating]: (0x65a950) located: prelude_sm:setup [D 20:00:52.637135] [SM Entering]: (0x65a950) prelude_sm:setup (status: 0) [D 20:00:52.637142] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [D 20:00:52.637149] [SM Exiting]: (0x65a950) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:52.637156] [SM Exiting]: (0x65a950) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:52.637163] [SM push_state]: (0x65a950) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:52.637170] [SM Entering]: (0x65a950) prelude_work_sm:req_sched (status: 0) [D 20:00:52.637177] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [D 20:00:52.637183] (0x65aa50) io (prelude sm) state: req_sched [A 20:00:52.643198] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65aa50: io: request [D 20:00:52.643212] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:52.643220] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6d84f0 [D 20:00:52.643227] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6d84f0 [D 20:00:52.643234] [SM Exiting]: (0x65a950) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:52.643241] [SM Entering]: (0x65a950) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:52.643248] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [A 20:00:52.649190] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65aa50: io: start [D 20:00:52.649202] (0x65aa50) io (prelude sm) state: getattr_if_needed [D 20:00:52.649210] About to retrieve attributes for handle 2863311531 [D 20:00:52.649220] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:52.649229] [SM Exiting]: (0x65a950) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:52.649240] [SM Entering]: (0x65a950) prelude_work_sm:perm_check (status: 0) [D 20:00:52.649247] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [D 20:00:52.649254] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:52.649263] Final permission check for "io" set error code to 0 [D 20:00:52.649270] (0x65aa50) io (prelude sm) state: perm_check (status = 0) [D 20:00:52.649276] [SM Exiting]: (0x65a950) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:52.649283] [SM pop_state]: (0x65a950) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:52.649290] [SM pop_state]: (0x65a950) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:52.649302] [SM Entering]: (0x65a950) io_sm:send_positive_ack (status: 0) [D 20:00:52.649309] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [D 20:00:52.649315] PINT_encode [D 20:00:52.649322] encode_common [D 20:00:52.649329] lebf_encode_resp [D 20:00:52.649342] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 19 [D 20:00:52.649349] element 0: offset: 0x6d7060, size: 24 [D 20:00:52.649358] post_sr: 10.1.4.56:39990 bh 18 len 40 wr_id 682c00. [D 20:00:52.649372] [SM Exiting]: (0x65a950) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:52.649390] openib_check_cq: completed send wr_id 682c00. [D 20:00:52.652294] [SM Entering]: (0x65a950) io_sm:start_flow (status: 0) [D 20:00:52.652306] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [D 20:00:52.652315] io_start_flow() issuing flow to write data. [D 20:00:52.652321] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:52.652327] file_req_offset: 369098752, aggregate_size: 33554432, handle: 2863311531 [D 20:00:52.652336] PINT_new_request_states n=1 [D 20:00:52.652344] flowproto posting 0x6d86a0 [D 20:00:52.652359] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:52.652367] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:00:52.653889] ========================================================= [D 20:00:52.653915] PINT_process_request [D 20:00:52.653922] tiling 13 copies [D 20:00:52.653929] skipping ahead to target_offset [D 20:00:52.653935] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:52.653943] lvl 0 el 0 blk 0 by 0 [D 20:00:52.653949] to 0 ta 369098752 fi 402653184 [D 20:00:52.653956] basic type or contiguous data [D 20:00:52.653962] Request disp [D 20:00:52.653968] contig_offset = 0 contig_size = 436207616 lvl_flag = 1 [D 20:00:52.653974] process logical skip [D 20:00:52.653980] retval = 369098752 [D 20:00:52.653986] exiting logical skip because distribute indicates done [D 20:00:52.653992] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:52.653999] lvl 0 el 0 blk 0 by 369098752 [D 20:00:52.654005] to 369098752 ta 369098752 fi 402653184 [D 20:00:52.654012] basic type or contiguous data [D 20:00:52.654018] Request disp [D 20:00:52.654023] contig_offset = 369098752 contig_size = 67108864 lvl_flag = 1 [D 20:00:52.654030] calling distribute [D 20:00:52.654036] PINT_distribute [D 20:00:52.654042] of 369098752 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:52.654050] begin iteration loff: 369098752 [D 20:00:52.654056] segment exceeds byte limit [D 20:00:52.654062] file being extended [D 20:00:52.654068] process a segment [D 20:00:52.654074] of 369098752 sz 2097152 [D 20:00:52.654080] add a segment 0 [D 20:00:52.654086] end iteration [D 20:00:52.654091] done with segments or bytes [D 20:00:52.654097] of 371195904 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:52.654104] next loff: 371195904 next poff: 371195904 [D 20:00:52.654110] return value 2097152 [D 20:00:52.654116] finished [D 20:00:52.654122] retval = 2097152 [D 20:00:52.654128] exiting distribute returned less than expected [D 20:00:52.654134] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 371195904 fo 402653184 eof 0 [D 20:00:52.654141] ========================================================= [D 20:00:52.654148] BMI_post_recv: addr: 6843072, offset: 0x2aaaadcc6010, size: 2097152, tag: 19 [D 20:00:52.654166] flowproto posted 0x6d86a0 [D 20:00:52.654174] Job flows in progress (post time): 12 [D 20:00:52.654183] [SM Exiting]: (0x65a950) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:52.756296] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:52.756368] Start times (hr:min:sec): 20:00:51.720 20:00:50.688 20:00:49.684 20:00:48.612 20:00:47.604 20:00:46.504 [P 20:00:52.756375] Intervals (hr:min:sec) : 00:00:01.036 00:00:01.032 00:00:01.004 00:00:01.072 00:00:01.008 00:00:01.100 [P 20:00:52.756381] ------------------------------------------------------------------------------------------------------------- [P 20:00:52.756388] bytes read : 0 0 0 0 0 0 [P 20:00:52.756394] bytes written : 0 0 0 0 0 0 [P 20:00:52.756400] metadata reads : 24 23 22 21 19 18 [P 20:00:52.756406] metadata writes : 7 7 7 7 7 7 [P 20:00:52.756413] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:52.756419] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:52.756425] request scheduler : 12 11 10 9 7 6 [D 20:00:52.756434] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:52.756442] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:52.756451] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:53.796319] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:53.796386] Start times (hr:min:sec): 20:00:52.756 20:00:51.720 20:00:50.688 20:00:49.684 20:00:48.612 20:00:47.604 [P 20:00:53.796393] Intervals (hr:min:sec) : 00:00:01.040 00:00:01.036 00:00:01.032 00:00:01.004 00:00:01.072 00:00:01.008 [P 20:00:53.796400] ------------------------------------------------------------------------------------------------------------- [P 20:00:53.796406] bytes read : 0 0 0 0 0 0 [P 20:00:53.796412] bytes written : 0 0 0 0 0 0 [P 20:00:53.796418] metadata reads : 24 24 23 22 21 19 [P 20:00:53.796424] metadata writes : 7 7 7 7 7 7 [P 20:00:53.796430] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:53.796436] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:53.796442] request scheduler : 12 12 11 10 9 7 [D 20:00:53.796451] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:53.796459] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:53.796467] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:53.881813] openib_check_cq: completed recv wr_id 683f00. [D 20:00:53.881828] openib_post_rr: 10.1.4.56:39990 bh 19 wr_id 683f00. [D 20:00:53.881861] [SM Entering]: (0x65bc10) unexpected_sm:map_request (status: 0) [D 20:00:53.881873] [SM frame get]: (0x65bc10) op-id: 999 index: 0 base-frm: 0 [D 20:00:53.881881] server_post_unexpected_recv [D 20:00:53.881894] [SM Frame PUSH]: (0x65cda0) frame: 0x6db460 [D 20:00:53.881901] [SM Locating]: (0x65cda0) op-id: 999 [D 20:00:53.881907] server_op_state_get_machine 999 [D 20:00:53.881913] [SM Locating]: (0x65cda0) located: unexpected_sm:post_unexpected [D 20:00:53.881920] [SM frame get]: (0x65cda0) op-id: 999 index: 0 base-frm: 0 [D 20:00:53.881927] [SM Entering]: (0x65cda0) unexpected_sm:post_unexpected (status: 0) [D 20:00:53.881934] [SM frame get]: (0x65cda0) op-id: 999 index: 0 base-frm: 0 [D 20:00:53.881942] [SM Exiting]: (0x65cda0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:53.881949] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:53.881957] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:53.881963] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 13 [D 20:00:53.881970] [SM frame get]: (0x65bc10) op-id: 999 index: 0 base-frm: 0 [D 20:00:53.881976] server_state_machine_start 0x65bc10 [D 20:00:53.881983] PINT_decode [D 20:00:53.881990] lebf_decode_req [D 20:00:53.882000] [SM Locating]: (0x65bc10) op-id: 3 [D 20:00:53.882007] server_op_state_get_machine 3 [D 20:00:53.882013] [SM push_state]: (0x65bc10) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:53.882019] [SM Locating]: (0x65bc10) located: prelude_sm:setup [D 20:00:53.882027] [SM Entering]: (0x65bc10) prelude_sm:setup (status: 0) [D 20:00:53.882033] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [D 20:00:53.882040] [SM Exiting]: (0x65bc10) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:53.882047] [SM Exiting]: (0x65bc10) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:53.882054] [SM push_state]: (0x65bc10) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:53.882061] [SM Entering]: (0x65bc10) prelude_work_sm:req_sched (status: 0) [D 20:00:53.882067] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [D 20:00:53.882074] (0x65bd10) io (prelude sm) state: req_sched [A 20:00:53.888146] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65bd10: io: request [D 20:00:53.888161] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:53.888169] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6dc7b0 [D 20:00:53.888176] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6dc7b0 [D 20:00:53.888183] [SM Exiting]: (0x65bc10) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:53.888190] [SM Entering]: (0x65bc10) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:53.888197] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [A 20:00:53.894137] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65bd10: io: start [D 20:00:53.894149] (0x65bd10) io (prelude sm) state: getattr_if_needed [D 20:00:53.894157] About to retrieve attributes for handle 2863311531 [D 20:00:53.894167] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:53.894176] [SM Exiting]: (0x65bc10) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:53.894187] [SM Entering]: (0x65bc10) prelude_work_sm:perm_check (status: 0) [D 20:00:53.894194] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [D 20:00:53.894202] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:53.894211] Final permission check for "io" set error code to 0 [D 20:00:53.894217] (0x65bd10) io (prelude sm) state: perm_check (status = 0) [D 20:00:53.894223] [SM Exiting]: (0x65bc10) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:53.894231] [SM pop_state]: (0x65bc10) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:53.894238] [SM pop_state]: (0x65bc10) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:53.894249] [SM Entering]: (0x65bc10) io_sm:send_positive_ack (status: 0) [D 20:00:53.894256] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [D 20:00:53.894263] PINT_encode [D 20:00:53.894270] encode_common [D 20:00:53.894282] lebf_encode_resp [D 20:00:53.894289] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 20 [D 20:00:53.894297] element 0: offset: 0x6db320, size: 24 [D 20:00:53.894306] post_sr: 10.1.4.56:39990 bh 19 len 40 wr_id 682c30. [D 20:00:53.894316] [SM Exiting]: (0x65bc10) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:53.894335] openib_check_cq: completed send wr_id 682c30. [D 20:00:53.896325] [SM Entering]: (0x65bc10) io_sm:start_flow (status: 0) [D 20:00:53.896337] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [D 20:00:53.896345] io_start_flow() issuing flow to write data. [D 20:00:53.896352] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:53.896358] file_req_offset: 402653184, aggregate_size: 33554432, handle: 2863311531 [D 20:00:53.896367] PINT_new_request_states n=1 [D 20:00:53.896374] flowproto posting 0x6dc960 [D 20:00:53.896390] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:53.896398] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:00:53.897865] ========================================================= [D 20:00:53.897892] PINT_process_request [D 20:00:53.897899] tiling 14 copies [D 20:00:53.897906] skipping ahead to target_offset [D 20:00:53.897912] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:53.897920] lvl 0 el 0 blk 0 by 0 [D 20:00:53.897926] to 0 ta 402653184 fi 436207616 [D 20:00:53.897933] basic type or contiguous data [D 20:00:53.897939] Request disp [D 20:00:53.897945] contig_offset = 0 contig_size = 469762048 lvl_flag = 1 [D 20:00:53.897951] process logical skip [D 20:00:53.897957] retval = 402653184 [D 20:00:53.897963] exiting logical skip because distribute indicates done [D 20:00:53.897969] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:53.897976] lvl 0 el 0 blk 0 by 402653184 [D 20:00:53.897982] to 402653184 ta 402653184 fi 436207616 [D 20:00:53.897989] basic type or contiguous data [D 20:00:53.897995] Request disp [D 20:00:53.898000] contig_offset = 402653184 contig_size = 67108864 lvl_flag = 1 [D 20:00:53.898007] calling distribute [D 20:00:53.898013] PINT_distribute [D 20:00:53.898019] of 402653184 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:53.898026] begin iteration loff: 402653184 [D 20:00:53.898033] segment exceeds byte limit [D 20:00:53.898039] file being extended [D 20:00:53.898045] process a segment [D 20:00:53.898051] of 402653184 sz 2097152 [D 20:00:53.898057] add a segment 0 [D 20:00:53.898063] end iteration [D 20:00:53.898069] done with segments or bytes [D 20:00:53.898074] of 404750336 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:53.898081] next loff: 404750336 next poff: 404750336 [D 20:00:53.898087] return value 2097152 [D 20:00:53.898094] finished [D 20:00:53.898100] retval = 2097152 [D 20:00:53.898106] exiting distribute returned less than expected [D 20:00:53.898112] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 404750336 fo 436207616 eof 0 [D 20:00:53.898119] ========================================================= [D 20:00:53.898126] BMI_post_recv: addr: 6843072, offset: 0x2aaaadec7010, size: 2097152, tag: 20 [D 20:00:53.898145] flowproto posted 0x6dc960 [D 20:00:53.898152] Job flows in progress (post time): 13 [D 20:00:53.898161] [SM Exiting]: (0x65bc10) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:54.832347] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:54.832419] Start times (hr:min:sec): 20:00:53.796 20:00:52.756 20:00:51.720 20:00:50.688 20:00:49.684 20:00:48.612 [P 20:00:54.832426] Intervals (hr:min:sec) : 00:00:01.036 00:00:01.040 00:00:01.036 00:00:01.032 00:00:01.004 00:00:01.072 [P 20:00:54.832433] ------------------------------------------------------------------------------------------------------------- [P 20:00:54.832439] bytes read : 0 0 0 0 0 0 [P 20:00:54.832445] bytes written : 0 0 0 0 0 0 [P 20:00:54.832452] metadata reads : 25 24 24 23 22 21 [P 20:00:54.832458] metadata writes : 7 7 7 7 7 7 [P 20:00:54.832464] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:54.832470] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:54.832476] request scheduler : 13 12 12 11 10 9 [D 20:00:54.832486] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:54.832494] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:54.832503] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:55.132161] openib_check_cq: completed recv wr_id 683f30. [D 20:00:55.132178] openib_post_rr: 10.1.4.56:39990 bh 20 wr_id 683f30. [D 20:00:55.132220] [SM Entering]: (0x65ced0) unexpected_sm:map_request (status: 0) [D 20:00:55.132232] [SM frame get]: (0x65ced0) op-id: 999 index: 0 base-frm: 0 [D 20:00:55.132240] server_post_unexpected_recv [D 20:00:55.132253] [SM Frame PUSH]: (0x65e060) frame: 0x6df720 [D 20:00:55.132260] [SM Locating]: (0x65e060) op-id: 999 [D 20:00:55.132267] server_op_state_get_machine 999 [D 20:00:55.132273] [SM Locating]: (0x65e060) located: unexpected_sm:post_unexpected [D 20:00:55.132279] [SM frame get]: (0x65e060) op-id: 999 index: 0 base-frm: 0 [D 20:00:55.132286] [SM Entering]: (0x65e060) unexpected_sm:post_unexpected (status: 0) [D 20:00:55.132293] [SM frame get]: (0x65e060) op-id: 999 index: 0 base-frm: 0 [D 20:00:55.132302] [SM Exiting]: (0x65e060) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:55.132309] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:55.132316] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:55.132323] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 14 [D 20:00:55.132330] [SM frame get]: (0x65ced0) op-id: 999 index: 0 base-frm: 0 [D 20:00:55.132336] server_state_machine_start 0x65ced0 [D 20:00:55.132342] PINT_decode [D 20:00:55.132352] lebf_decode_req [D 20:00:55.132362] [SM Locating]: (0x65ced0) op-id: 3 [D 20:00:55.132369] server_op_state_get_machine 3 [D 20:00:55.132375] [SM push_state]: (0x65ced0) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:55.132381] [SM Locating]: (0x65ced0) located: prelude_sm:setup [D 20:00:55.132388] [SM Entering]: (0x65ced0) prelude_sm:setup (status: 0) [D 20:00:55.132395] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [D 20:00:55.132402] [SM Exiting]: (0x65ced0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:55.132409] [SM Exiting]: (0x65ced0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:55.132416] [SM push_state]: (0x65ced0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:55.132423] [SM Entering]: (0x65ced0) prelude_work_sm:req_sched (status: 0) [D 20:00:55.132429] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [D 20:00:55.132436] (0x65cfd0) io (prelude sm) state: req_sched [A 20:00:55.138217] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65cfd0: io: request [D 20:00:55.138232] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:55.138240] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6e0be0 [D 20:00:55.138246] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6e0be0 [D 20:00:55.138253] [SM Exiting]: (0x65ced0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:55.138261] [SM Entering]: (0x65ced0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:55.138267] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [A 20:00:55.144083] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65cfd0: io: start [D 20:00:55.144094] (0x65cfd0) io (prelude sm) state: getattr_if_needed [D 20:00:55.144103] About to retrieve attributes for handle 2863311531 [D 20:00:55.144112] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:55.144122] [SM Exiting]: (0x65ced0) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:55.144133] [SM Entering]: (0x65ced0) prelude_work_sm:perm_check (status: 0) [D 20:00:55.144140] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [D 20:00:55.144148] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:55.144156] Final permission check for "io" set error code to 0 [D 20:00:55.144164] (0x65cfd0) io (prelude sm) state: perm_check (status = 0) [D 20:00:55.144171] [SM Exiting]: (0x65ced0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:55.144178] [SM pop_state]: (0x65ced0) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:55.144185] [SM pop_state]: (0x65ced0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:55.144195] [SM Entering]: (0x65ced0) io_sm:send_positive_ack (status: 0) [D 20:00:55.144201] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [D 20:00:55.144208] PINT_encode [D 20:00:55.144215] encode_common [D 20:00:55.144221] lebf_encode_resp [D 20:00:55.144236] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 21 [D 20:00:55.144243] element 0: offset: 0x6df5e0, size: 24 [D 20:00:55.144252] post_sr: 10.1.4.56:39990 bh 20 len 40 wr_id 682c60. [D 20:00:55.144262] [SM Exiting]: (0x65ced0) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:00:55.144280] openib_check_cq: completed send wr_id 682c60. [D 20:00:55.144351] [SM Entering]: (0x65ced0) io_sm:start_flow (status: 0) [D 20:00:55.144362] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [D 20:00:55.144371] io_start_flow() issuing flow to write data. [D 20:00:55.144377] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:55.144384] file_req_offset: 436207616, aggregate_size: 33554432, handle: 2863311531 [D 20:00:55.144392] PINT_new_request_states n=1 [D 20:00:55.144401] flowproto posting 0x6e0ce0 [D 20:00:55.144433] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:55.144442] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:00:55.148425] ========================================================= [D 20:00:55.148457] PINT_process_request [D 20:00:55.148464] tiling 15 copies [D 20:00:55.148471] skipping ahead to target_offset [D 20:00:55.148477] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:55.148484] lvl 0 el 0 blk 0 by 0 [D 20:00:55.148491] to 0 ta 436207616 fi 469762048 [D 20:00:55.148497] basic type or contiguous data [D 20:00:55.148503] Request disp [D 20:00:55.148509] contig_offset = 0 contig_size = 503316480 lvl_flag = 1 [D 20:00:55.148516] process logical skip [D 20:00:55.148522] retval = 436207616 [D 20:00:55.148528] exiting logical skip because distribute indicates done [D 20:00:55.148534] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:55.148541] lvl 0 el 0 blk 0 by 436207616 [D 20:00:55.148547] to 436207616 ta 436207616 fi 469762048 [D 20:00:55.148554] basic type or contiguous data [D 20:00:55.148560] Request disp [D 20:00:55.148565] contig_offset = 436207616 contig_size = 67108864 lvl_flag = 1 [D 20:00:55.148572] calling distribute [D 20:00:55.148578] PINT_distribute [D 20:00:55.148584] of 436207616 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:55.148592] begin iteration loff: 436207616 [D 20:00:55.148598] segment exceeds byte limit [D 20:00:55.148604] file being extended [D 20:00:55.148610] process a segment [D 20:00:55.148616] of 436207616 sz 2097152 [D 20:00:55.148623] add a segment 0 [D 20:00:55.148629] end iteration [D 20:00:55.148634] done with segments or bytes [D 20:00:55.148640] of 438304768 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:55.148647] next loff: 438304768 next poff: 438304768 [D 20:00:55.148653] return value 2097152 [D 20:00:55.148660] finished [D 20:00:55.148665] retval = 2097152 [D 20:00:55.148671] exiting distribute returned less than expected [D 20:00:55.148677] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 438304768 fo 469762048 eof 0 [D 20:00:55.148685] ========================================================= [D 20:00:55.148691] BMI_post_recv: addr: 6843072, offset: 0x2aaaae0c8010, size: 2097152, tag: 21 [D 20:00:55.148709] flowproto posted 0x6e0ce0 [D 20:00:55.148717] Job flows in progress (post time): 14 [D 20:00:55.148726] [SM Exiting]: (0x65ced0) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:55.876372] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:00:55.876444] Start times (hr:min:sec): 20:00:54.832 20:00:53.796 20:00:52.756 20:00:51.720 20:00:50.688 20:00:49.684 [P 20:00:55.876451] Intervals (hr:min:sec) : 00:00:01.044 00:00:01.036 00:00:01.040 00:00:01.036 00:00:01.032 00:00:01.004 [P 20:00:55.876458] ------------------------------------------------------------------------------------------------------------- [P 20:00:55.876464] bytes read : 0 0 0 0 0 0 [P 20:00:55.876470] bytes written : 0 0 0 0 0 0 [P 20:00:55.876477] metadata reads : 26 25 24 24 23 22 [P 20:00:55.876483] metadata writes : 7 7 7 7 7 7 [P 20:00:55.876489] metadata dspace ops : 0 0 0 0 0 0 [P 20:00:55.876495] metadata keyval ops : 0 0 0 0 0 0 [P 20:00:55.876501] request scheduler : 14 13 12 12 11 10 [D 20:00:55.876512] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:00:55.876520] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:00:55.876528] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) -[D 20:00:56.320915] openib_check_cq: completed recv wr_id 683f60. [D 20:00:56.320930] openib_post_rr: 10.1.4.56:39990 bh 21 wr_id 683f60. [D 20:00:56.320964] [SM Entering]: (0x65e190) unexpected_sm:map_request (status: 0) [D 20:00:56.320976] [SM frame get]: (0x65e190) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.320984] server_post_unexpected_recv [D 20:00:56.320998] [SM Frame PUSH]: (0x65f320) frame: 0x6e40f0 [D 20:00:56.321005] [SM Locating]: (0x65f320) op-id: 999 [D 20:00:56.321012] server_op_state_get_machine 999 [D 20:00:56.321018] [SM Locating]: (0x65f320) located: unexpected_sm:post_unexpected [D 20:00:56.321024] [SM frame get]: (0x65f320) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.321032] [SM Entering]: (0x65f320) unexpected_sm:post_unexpected (status: 0) [D 20:00:56.321038] [SM frame get]: (0x65f320) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.321047] [SM Exiting]: (0x65f320) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:56.321054] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:56.321061] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:56.321068] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 15 [D 20:00:56.321074] [SM frame get]: (0x65e190) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.321081] server_state_machine_start 0x65e190 [D 20:00:56.321087] PINT_decode [D 20:00:56.321094] lebf_decode_req [D 20:00:56.321104] [SM Locating]: (0x65e190) op-id: 3 [D 20:00:56.321110] server_op_state_get_machine 3 [D 20:00:56.321117] [SM push_state]: (0x65e190) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:56.321123] [SM Locating]: (0x65e190) located: prelude_sm:setup [D 20:00:56.321130] [SM Entering]: (0x65e190) prelude_sm:setup (status: 0) [D 20:00:56.321137] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.321144] [SM Exiting]: (0x65e190) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:56.321151] [SM Exiting]: (0x65e190) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:56.321158] [SM push_state]: (0x65e190) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.321164] [SM Entering]: (0x65e190) prelude_work_sm:req_sched (status: 0) [D 20:00:56.321171] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.321177] (0x65e290) io (prelude sm) state: req_sched [D 20:00:56.323658] openib_check_cq: completed recv wr_id 683f90. [D 20:00:56.323671] openib_post_rr: 10.1.4.56:39990 bh 22 wr_id 683f90. [D 20:00:56.326591] openib_check_cq: completed recv wr_id 683fc0. [D 20:00:56.326606] openib_post_rr: 10.1.4.56:39990 bh 23 wr_id 683fc0. [A 20:00:56.327319] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65e290: io: request [D 20:00:56.327334] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:56.327342] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6e19b0 [D 20:00:56.327349] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6e19b0 [D 20:00:56.327356] [SM Exiting]: (0x65e190) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:56.327363] [SM Entering]: (0x65e190) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:56.327370] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.329520] openib_check_cq: completed recv wr_id 683ff0. [D 20:00:56.329532] openib_post_rr: 10.1.4.56:39990 bh 24 wr_id 683ff0. [D 20:00:56.332445] openib_check_cq: completed recv wr_id 684020. [D 20:00:56.332458] openib_post_rr: 10.1.4.56:39990 bh 25 wr_id 684020. [A 20:00:56.333183] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65e290: io: start [D 20:00:56.333195] (0x65e290) io (prelude sm) state: getattr_if_needed [D 20:00:56.333202] About to retrieve attributes for handle 2863311531 [D 20:00:56.333212] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=0) [D 20:00:56.333221] [SM Exiting]: (0x65e190) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:00:56.333228] [SM Entering]: (0x65e190) prelude_work_sm:perm_check (status: 0) [D 20:00:56.333236] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.333243] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:00:56.333252] Final permission check for "io" set error code to 0 [D 20:00:56.333258] (0x65e290) io (prelude sm) state: perm_check (status = 0) [D 20:00:56.333265] [SM Exiting]: (0x65e190) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:00:56.333272] [SM pop_state]: (0x65e190) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:00:56.333279] [SM pop_state]: (0x65e190) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.333285] [SM Entering]: (0x65e190) io_sm:send_positive_ack (status: 0) [D 20:00:56.333292] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.333298] PINT_encode [D 20:00:56.333305] encode_common [D 20:00:56.333311] lebf_encode_resp [D 20:00:56.333319] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 22 [D 20:00:56.333326] element 0: offset: 0x6e1a30, size: 24 [D 20:00:56.333336] post_sr: 10.1.4.56:39990 bh 21 len 40 wr_id 682c90. [D 20:00:56.333349] [SM Exiting]: (0x65e190) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:00:56.333352] openib_check_cq: completed send wr_id 682c90. [D 20:00:56.333363] [SM Entering]: (0x65f450) unexpected_sm:map_request (status: 0) [D 20:00:56.333372] [SM frame get]: (0x65f450) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.333378] server_post_unexpected_recv [D 20:00:56.333391] [SM Frame PUSH]: (0x6605e0) frame: 0x6e65b0 [D 20:00:56.333398] [SM Locating]: (0x6605e0) op-id: 999 [D 20:00:56.333404] server_op_state_get_machine 999 [D 20:00:56.333410] [SM Locating]: (0x6605e0) located: unexpected_sm:post_unexpected [D 20:00:56.333417] [SM frame get]: (0x6605e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.333424] [SM Entering]: (0x6605e0) unexpected_sm:post_unexpected (status: 0) [D 20:00:56.333431] [SM frame get]: (0x6605e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.333438] [SM Exiting]: (0x6605e0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:56.333446] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:56.333452] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:56.333458] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 16 [D 20:00:56.333465] [SM frame get]: (0x65f450) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.333472] server_state_machine_start 0x65f450 [D 20:00:56.333478] PINT_decode [D 20:00:56.333484] lebf_decode_req [D 20:00:56.333491] [SM Locating]: (0x65f450) op-id: 3 [D 20:00:56.333498] server_op_state_get_machine 3 [D 20:00:56.333504] [SM push_state]: (0x65f450) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:56.333510] [SM Locating]: (0x65f450) located: prelude_sm:setup [D 20:00:56.333517] [SM Entering]: (0x65f450) prelude_sm:setup (status: 0) [D 20:00:56.333523] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.333530] [SM Exiting]: (0x65f450) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:56.333537] [SM Exiting]: (0x65f450) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:56.333544] [SM push_state]: (0x65f450) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.333551] [SM Entering]: (0x65f450) prelude_work_sm:req_sched (status: 0) [D 20:00:56.333557] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.333564] (0x65f550) io (prelude sm) state: req_sched [D 20:00:56.335408] openib_check_cq: completed recv wr_id 684050. [D 20:00:56.335420] openib_post_rr: 10.1.4.56:39990 bh 26 wr_id 684050. [D 20:00:56.338335] openib_check_cq: completed recv wr_id 684080. [D 20:00:56.338348] openib_post_rr: 10.1.4.56:39990 bh 27 wr_id 684080. [A 20:00:56.339555] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65f550: io: request [D 20:00:56.339568] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:56.339576] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x6e5270 [D 20:00:56.339583] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x6e5270 [D 20:00:56.339589] [SM Exiting]: (0x65f450) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:56.339597] [SM Entering]: (0x65f450) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:56.339603] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.341263] openib_check_cq: completed recv wr_id 6840b0. [D 20:00:56.341276] openib_post_rr: 10.1.4.56:39990 bh 28 wr_id 6840b0. [D 20:00:56.344194] openib_check_cq: completed recv wr_id 6840e0. [D 20:00:56.344206] openib_post_rr: 10.1.4.56:39990 bh 29 wr_id 6840e0. [D 20:00:56.347106] openib_check_cq: completed recv wr_id 684110. [D 20:00:56.347117] openib_post_rr: 10.1.4.56:39990 bh 30 wr_id 684110. [D 20:00:56.350031] openib_check_cq: completed recv wr_id 684140. [D 20:00:56.350043] openib_post_rr: 10.1.4.56:39990 bh 31 wr_id 684140. [D 20:00:56.352963] openib_check_cq: completed recv wr_id 684170. [D 20:00:56.352974] openib_post_rr: 10.1.4.56:39990 bh 32 wr_id 684170. [D 20:00:56.355868] openib_check_cq: completed recv wr_id 6841a0. [D 20:00:56.355879] openib_post_rr: 10.1.4.56:39990 bh 33 wr_id 6841a0. [D 20:00:56.358795] openib_check_cq: completed recv wr_id 6841d0. [D 20:00:56.358807] openib_post_rr: 10.1.4.56:39990 bh 34 wr_id 6841d0. [D 20:00:56.361727] openib_check_cq: completed recv wr_id 684200. [D 20:00:56.361740] openib_post_rr: 10.1.4.56:39990 bh 35 wr_id 684200. [D 20:00:56.364657] openib_check_cq: completed recv wr_id 684230. [D 20:00:56.364668] openib_post_rr: 10.1.4.56:39990 bh 36 wr_id 684230. -[D 20:00:56.370905] openib_check_cq: completed recv wr_id 684260. [D 20:00:56.370915] openib_post_rr: 10.1.4.56:39990 bh 37 wr_id 684260. [D 20:00:56.370924] post_sr: 10.1.4.56:39990 bh 22 len 48 wr_id 682cc0. [D 20:00:56.370938] openib_check_cq: completed send wr_id 682cc0. [D 20:00:56.376870] openib_check_cq: completed recv wr_id 684290. [D 20:00:56.376880] openib_post_rr: 10.1.4.56:39990 bh 38 wr_id 684290. [D 20:00:56.376888] post_sr: 10.1.4.56:39990 bh 23 len 48 wr_id 682cf0. [D 20:00:56.376902] openib_check_cq: completed send wr_id 682cf0. [D 20:00:56.377108] openib_check_cq: completed recv wr_id 6842c0. [D 20:00:56.377126] openib_post_rr: 10.1.4.56:39990 bh 39 wr_id 6842c0. [D 20:00:56.377137] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:00:56.377148] get_data_sync_mode returning 0 [D 20:00:56.377163] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.377171] dbpf_bstream_rw_list: mem_offset: 0x2aaaac6bb010, mem_size: 2097152 [D 20:00:56.377178] dbpf_bstream_rw_list: stream_offset: 0, stream_size: 2097152 [D 20:00:56.377185] dbpf_open_cache_get: called [D 20:00:56.377192] dbpf_open_cache_get: resetting entry from free list. [D 20:00:56.377200] dbpf_open_cache open_fd: opening fd 2863311531 [D 20:00:56.377214] dbpf_open_cache open_fd: filename: //scratch/1/pvfs-da13-test/00000088/bstreams/00000043/aaaaaaab.bstream [D 20:00:56.377327] dbpf_open_cache_get: moving to used list. [D 20:00:56.377339] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:56.377376] DBPF I/O ops in progress: 1 [D 20:00:56.377385] lio_listio called with the following aiocbs: [D 20:00:56.377393] aiocb_ptr_array[0]: fd: 13, off: 0, bytes: 2097152, buf: 0x2aaaac6bb010, type: 1 [D 20:00:56.377531] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.380485] ========================================================= [D 20:00:56.380553] PINT_process_request [D 20:00:56.380563] tiling 2 copies [D 20:00:56.380571] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.380579] lvl 0 el 0 blk 0 by 2097152 [D 20:00:56.380585] to 2097152 ta 0 fi 33554432 [D 20:00:56.380593] basic type or contiguous data [D 20:00:56.380600] Request disp [D 20:00:56.380607] contig_offset = 2097152 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.380614] calling distribute [D 20:00:56.380620] PINT_distribute [D 20:00:56.380627] of 2097152 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 2097152 exfl 1 [D 20:00:56.380636] begin iteration loff: 2097152 [D 20:00:56.380644] segment exceeds byte limit [D 20:00:56.380650] file being extended [D 20:00:56.380656] process a segment [D 20:00:56.380662] of 2097152 sz 2097152 [D 20:00:56.380669] add a segment 0 [D 20:00:56.380675] end iteration [D 20:00:56.380681] done with segments or bytes [D 20:00:56.380687] of 4194304 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.380694] next loff: 4194304 next poff: 4194304 [D 20:00:56.380701] return value 2097152 [D 20:00:56.380709] finished [D 20:00:56.380715] retval = 2097152 [D 20:00:56.380721] exiting distribute returned less than expected [D 20:00:56.380728] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 4194304 fo 33554432 eof 0 [D 20:00:56.380735] ========================================================= [D 20:00:56.380743] BMI_post_recv: addr: 6843072, offset: 0x2aaaae2c9010, size: 2097152, tag: 8 [D 20:00:56.382445] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.382495] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.382511] dbpf_open_cache_put: cached entry. [D 20:00:56.382518] dbpf_open_cache_put: move to unused list. [D 20:00:56.382525] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.382532] op_queue add: 0x6e5fe0 [D 20:00:56.382544] DBPF I/O ops in progress: 0 [D 20:00:56.382552] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:00:56.382629] openib_check_cq: completed recv wr_id 6842f0. [D 20:00:56.382641] openib_post_rr: 10.1.4.56:39990 bh 40 wr_id 6842f0. [D 20:00:56.382652] post_sr: 10.1.4.56:39990 bh 24 len 48 wr_id 682d20. [D 20:00:56.382668] openib_check_cq: completed send wr_id 682d20. [D 20:00:56.382865] openib_check_cq: completed recv wr_id 684320. [D 20:00:56.382873] openib_post_rr: 10.1.4.56:39990 bh 41 wr_id 684320. [D 20:00:56.382883] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:00:56.382892] get_data_sync_mode returning 0 [D 20:00:56.382901] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.382908] dbpf_bstream_rw_list: mem_offset: 0x2aaaac8bc010, mem_size: 2097152 [D 20:00:56.382915] dbpf_bstream_rw_list: stream_offset: 33554432, stream_size: 2097152 [D 20:00:56.382922] dbpf_open_cache_get: called [D 20:00:56.382928] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.382935] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.382949] DBPF I/O ops in progress: 1 [D 20:00:56.382956] lio_listio called with the following aiocbs: [D 20:00:56.382962] aiocb_ptr_array[0]: fd: 13, off: 33554432, bytes: 2097152, buf: 0x2aaaac8bc010, type: 1 [D 20:00:56.382980] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.387409] ========================================================= [D 20:00:56.387433] PINT_process_request [D 20:00:56.387444] tiling 3 copies [D 20:00:56.387453] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.387461] lvl 0 el 0 blk 0 by 35651584 [D 20:00:56.387467] to 35651584 ta 33554432 fi 67108864 [D 20:00:56.387475] basic type or contiguous data [D 20:00:56.387481] Request disp [D 20:00:56.387487] contig_offset = 35651584 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.387508] calling distribute [D 20:00:56.387514] PINT_distribute [D 20:00:56.387520] of 35651584 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 35651584 exfl 1 [D 20:00:56.387528] begin iteration loff: 35651584 [D 20:00:56.387542] segment exceeds byte limit [D 20:00:56.387548] file being extended [D 20:00:56.387554] process a segment [D 20:00:56.387560] of 35651584 sz 2097152 [D 20:00:56.387567] add a segment 0 [D 20:00:56.387575] end iteration [D 20:00:56.387581] done with segments or bytes [D 20:00:56.387587] of 37748736 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.387594] next loff: 37748736 next poff: 37748736 [D 20:00:56.387630] return value 2097152 [D 20:00:56.387642] finished [D 20:00:56.387647] retval = 2097152 [D 20:00:56.387654] exiting distribute returned less than expected [D 20:00:56.387660] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 37748736 fo 67108864 eof 0 [D 20:00:56.387671] ========================================================= [D 20:00:56.387677] BMI_post_recv: addr: 6843072, offset: 0x2aaaae4ca010, size: 2097152, tag: 9 [D 20:00:56.388284] openib_check_cq: completed recv wr_id 684350. [D 20:00:56.388297] openib_post_rr: 10.1.4.56:39990 bh 42 wr_id 684350. [D 20:00:56.388307] post_sr: 10.1.4.56:39990 bh 25 len 48 wr_id 682d50. [D 20:00:56.388322] openib_check_cq: completed send wr_id 682d50. [D 20:00:56.387362] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.388393] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.388408] dbpf_open_cache_put: cached entry. [D 20:00:56.388415] dbpf_open_cache_put: move to unused list. [D 20:00:56.388422] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.388429] op_queue add: 0x6e5fe0 [D 20:00:56.388450] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:00:56.388467] DBPF I/O ops in progress: 0 [D 20:00:56.388520] openib_check_cq: completed recv wr_id 684380. [D 20:00:56.388530] openib_post_rr: 10.1.4.56:39990 bh 43 wr_id 684380. [D 20:00:56.388544] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:00:56.388553] get_data_sync_mode returning 0 [D 20:00:56.388562] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.388573] dbpf_bstream_rw_list: mem_offset: 0x2aaaacabd010, mem_size: 2097152 [D 20:00:56.388580] dbpf_bstream_rw_list: stream_offset: 67108864, stream_size: 2097152 [D 20:00:56.388587] dbpf_open_cache_get: called [D 20:00:56.388594] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.388601] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.388623] DBPF I/O ops in progress: 1 [D 20:00:56.388631] lio_listio called with the following aiocbs: [D 20:00:56.388638] aiocb_ptr_array[0]: fd: 13, off: 67108864, bytes: 2097152, buf: 0x2aaaacabd010, type: 1 [D 20:00:56.388650] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.390271] ========================================================= [D 20:00:56.390333] PINT_process_request [D 20:00:56.390342] tiling 4 copies [D 20:00:56.390349] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.390358] lvl 0 el 0 blk 0 by 69206016 [D 20:00:56.390365] to 69206016 ta 67108864 fi 100663296 [D 20:00:56.390373] basic type or contiguous data [D 20:00:56.390379] Request disp [D 20:00:56.390386] contig_offset = 69206016 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.390393] calling distribute [D 20:00:56.390401] PINT_distribute [D 20:00:56.390408] of 69206016 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 69206016 exfl 1 [D 20:00:56.390417] begin iteration loff: 69206016 [D 20:00:56.390424] segment exceeds byte limit [D 20:00:56.390430] file being extended [D 20:00:56.390438] process a segment [D 20:00:56.390444] of 69206016 sz 2097152 [D 20:00:56.390451] add a segment 0 [D 20:00:56.390457] end iteration [D 20:00:56.390463] done with segments or bytes [D 20:00:56.390469] of 71303168 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.390476] next loff: 71303168 next poff: 71303168 [D 20:00:56.390482] return value 2097152 [D 20:00:56.390490] finished [D 20:00:56.390497] retval = 2097152 [D 20:00:56.390503] exiting distribute returned less than expected [D 20:00:56.390509] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 71303168 fo 100663296 eof 0 [D 20:00:56.390517] ========================================================= [D 20:00:56.390525] BMI_post_recv: addr: 6843072, offset: 0x2aaaae6cb010, size: 2097152, tag: 10 [D 20:00:56.393619] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.393658] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.393669] dbpf_open_cache_put: cached entry. [D 20:00:56.393676] dbpf_open_cache_put: move to unused list. [D 20:00:56.393683] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.393690] op_queue add: 0x6e5fe0 [D 20:00:56.393712] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:00:56.393728] DBPF I/O ops in progress: 0 [D 20:00:56.393823] openib_check_cq: completed recv wr_id 6843b0. [D 20:00:56.393838] openib_post_rr: 10.1.4.56:39990 bh 44 wr_id 6843b0. [D 20:00:56.393849] post_sr: 10.1.4.56:39990 bh 26 len 48 wr_id 682d80. [D 20:00:56.393865] openib_check_cq: completed send wr_id 682d80. [D 20:00:56.394061] openib_check_cq: completed recv wr_id 6843e0. [D 20:00:56.394071] openib_post_rr: 10.1.4.56:39990 bh 45 wr_id 6843e0. [D 20:00:56.394081] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:00:56.394089] get_data_sync_mode returning 0 [D 20:00:56.394104] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.394111] dbpf_bstream_rw_list: mem_offset: 0x2aaaaccbe010, mem_size: 2097152 [D 20:00:56.394118] dbpf_bstream_rw_list: stream_offset: 100663296, stream_size: 2097152 [D 20:00:56.394125] dbpf_open_cache_get: called [D 20:00:56.394136] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.394142] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.394158] DBPF I/O ops in progress: 1 [D 20:00:56.394166] lio_listio called with the following aiocbs: [D 20:00:56.394172] aiocb_ptr_array[0]: fd: 13, off: 100663296, bytes: 2097152, buf: 0x2aaaaccbe010, type: 1 [D 20:00:56.394186] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.397022] ========================================================= [D 20:00:56.397084] PINT_process_request [D 20:00:56.397093] tiling 5 copies [D 20:00:56.397101] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.397108] lvl 0 el 0 blk 0 by 102760448 [D 20:00:56.397115] to 102760448 ta 100663296 fi 134217728 [D 20:00:56.397123] basic type or contiguous data [D 20:00:56.397130] Request disp [D 20:00:56.397136] contig_offset = 102760448 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.397143] calling distribute [D 20:00:56.397149] PINT_distribute [D 20:00:56.397155] of 102760448 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 102760448 exfl 1 [D 20:00:56.397165] begin iteration loff: 102760448 [D 20:00:56.397173] segment exceeds byte limit [D 20:00:56.397179] file being extended [D 20:00:56.397185] process a segment [D 20:00:56.397191] of 102760448 sz 2097152 [D 20:00:56.397198] add a segment 0 [D 20:00:56.397204] end iteration [D 20:00:56.397210] done with segments or bytes [D 20:00:56.397216] of 104857600 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.397223] next loff: 104857600 next poff: 104857600 [D 20:00:56.397230] return value 2097152 [D 20:00:56.397237] finished [D 20:00:56.397243] retval = 2097152 [D 20:00:56.397250] exiting distribute returned less than expected [D 20:00:56.397256] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 104857600 fo 134217728 eof 0 [D 20:00:56.397263] ========================================================= [D 20:00:56.397271] BMI_post_recv: addr: 6843072, offset: 0x2aaaae8cc010, size: 2097152, tag: 11 [D 20:00:56.398676] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.398707] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.398718] dbpf_open_cache_put: cached entry. [D 20:00:56.398724] dbpf_open_cache_put: move to unused list. [D 20:00:56.398731] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.398738] op_queue add: 0x6e5fe0 [D 20:00:56.398760] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:56.398777] DBPF I/O ops in progress: 0 [D 20:00:56.399342] openib_check_cq: completed recv wr_id 684410. [D 20:00:56.399356] openib_post_rr: 10.1.4.56:39990 bh 46 wr_id 684410. [D 20:00:56.399367] post_sr: 10.1.4.56:39990 bh 27 len 48 wr_id 682db0. [D 20:00:56.399383] openib_check_cq: completed send wr_id 682db0. [D 20:00:56.399579] openib_check_cq: completed recv wr_id 684440. [D 20:00:56.399589] openib_post_rr: 10.1.4.56:39990 bh 47 wr_id 684440. [D 20:00:56.399600] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:00:56.399608] get_data_sync_mode returning 0 [D 20:00:56.399618] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.399626] dbpf_bstream_rw_list: mem_offset: 0x2aaaacebf010, mem_size: 2097152 [D 20:00:56.399632] dbpf_bstream_rw_list: stream_offset: 134217728, stream_size: 2097152 [D 20:00:56.399639] dbpf_open_cache_get: called [D 20:00:56.399651] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.399658] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.399674] DBPF I/O ops in progress: 1 [D 20:00:56.399683] lio_listio called with the following aiocbs: [D 20:00:56.399689] aiocb_ptr_array[0]: fd: 13, off: 134217728, bytes: 2097152, buf: 0x2aaaacebf010, type: 1 [D 20:00:56.399702] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.402559] ========================================================= [D 20:00:56.402626] PINT_process_request [D 20:00:56.402634] tiling 6 copies [D 20:00:56.402642] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.402650] lvl 0 el 0 blk 0 by 136314880 [D 20:00:56.402657] to 136314880 ta 134217728 fi 167772160 [D 20:00:56.402664] basic type or contiguous data [D 20:00:56.402671] Request disp [D 20:00:56.402677] contig_offset = 136314880 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.402685] calling distribute [D 20:00:56.402691] PINT_distribute [D 20:00:56.402698] of 136314880 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 136314880 exfl 1 [D 20:00:56.402707] begin iteration loff: 136314880 [D 20:00:56.402715] segment exceeds byte limit [D 20:00:56.402722] file being extended [D 20:00:56.402728] process a segment [D 20:00:56.402734] of 136314880 sz 2097152 [D 20:00:56.402740] add a segment 0 [D 20:00:56.402747] end iteration [D 20:00:56.402753] done with segments or bytes [D 20:00:56.402758] of 138412032 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.402765] next loff: 138412032 next poff: 138412032 [D 20:00:56.402772] return value 2097152 [D 20:00:56.402779] finished [D 20:00:56.402786] retval = 2097152 [D 20:00:56.402792] exiting distribute returned less than expected [D 20:00:56.402798] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 138412032 fo 167772160 eof 0 [D 20:00:56.402806] ========================================================= [D 20:00:56.402814] BMI_post_recv: addr: 6843072, offset: 0x2aaaaeacd010, size: 2097152, tag: 12 [D 20:00:56.404203] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.404230] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.404241] dbpf_open_cache_put: cached entry. [D 20:00:56.404248] dbpf_open_cache_put: move to unused list. [D 20:00:56.404254] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.404261] op_queue add: 0x6e5fe0 [D 20:00:56.404282] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:56.404299] DBPF I/O ops in progress: 0 [D 20:00:56.404870] openib_check_cq: completed recv wr_id 684470. [D 20:00:56.404884] openib_post_rr: 10.1.4.56:39990 bh 48 wr_id 684470. [D 20:00:56.404895] post_sr: 10.1.4.56:39990 bh 28 len 48 wr_id 682de0. [D 20:00:56.404910] openib_check_cq: completed send wr_id 682de0. [D 20:00:56.405111] openib_check_cq: completed recv wr_id 6844a0. [D 20:00:56.405122] openib_post_rr: 10.1.4.56:39990 bh 49 wr_id 6844a0. [D 20:00:56.405132] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:00:56.405141] get_data_sync_mode returning 0 [D 20:00:56.405150] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.405158] dbpf_bstream_rw_list: mem_offset: 0x2aaaad0c0010, mem_size: 2097152 [D 20:00:56.405165] dbpf_bstream_rw_list: stream_offset: 167772160, stream_size: 2097152 [D 20:00:56.405172] dbpf_open_cache_get: called [D 20:00:56.405183] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.405190] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.405206] DBPF I/O ops in progress: 1 [D 20:00:56.405214] lio_listio called with the following aiocbs: [D 20:00:56.405221] aiocb_ptr_array[0]: fd: 13, off: 167772160, bytes: 2097152, buf: 0x2aaaad0c0010, type: 1 [D 20:00:56.405239] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.408068] ========================================================= [D 20:00:56.408132] PINT_process_request [D 20:00:56.408139] tiling 7 copies [D 20:00:56.408148] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.408156] lvl 0 el 0 blk 0 by 169869312 [D 20:00:56.408163] to 169869312 ta 167772160 fi 201326592 [D 20:00:56.408171] basic type or contiguous data [D 20:00:56.408177] Request disp [D 20:00:56.408184] contig_offset = 169869312 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.408192] calling distribute [D 20:00:56.408199] PINT_distribute [D 20:00:56.408206] of 169869312 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 169869312 exfl 1 [D 20:00:56.408215] begin iteration loff: 169869312 [D 20:00:56.408223] segment exceeds byte limit [D 20:00:56.408229] file being extended [D 20:00:56.408235] process a segment [D 20:00:56.408241] of 169869312 sz 2097152 [D 20:00:56.408248] add a segment 0 [D 20:00:56.408254] end iteration [D 20:00:56.408261] done with segments or bytes [D 20:00:56.408267] of 171966464 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.408274] next loff: 171966464 next poff: 171966464 [D 20:00:56.408281] return value 2097152 [D 20:00:56.408289] finished [D 20:00:56.408295] retval = 2097152 [D 20:00:56.408301] exiting distribute returned less than expected [D 20:00:56.408307] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 171966464 fo 201326592 eof 0 [D 20:00:56.408314] ========================================================= [D 20:00:56.408322] BMI_post_recv: addr: 6843072, offset: 0x2aaaaecce010, size: 2097152, tag: 13 [D 20:00:56.409750] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.409779] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.409790] dbpf_open_cache_put: cached entry. [D 20:00:56.409797] dbpf_open_cache_put: move to unused list. [D 20:00:56.409803] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.409810] op_queue add: 0x6e5fe0 [D 20:00:56.409831] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:00:56.409848] DBPF I/O ops in progress: 0 [D 20:00:56.410364] openib_check_cq: completed recv wr_id 6844d0. [D 20:00:56.410378] openib_post_rr: 10.1.4.56:39990 bh 50 wr_id 6844d0. [D 20:00:56.410388] post_sr: 10.1.4.56:39990 bh 29 len 48 wr_id 682e10. [D 20:00:56.410404] openib_check_cq: completed send wr_id 682e10. [D 20:00:56.410601] openib_check_cq: completed recv wr_id 684500. [D 20:00:56.410611] openib_post_rr: 10.1.4.56:39990 bh 51 wr_id 684500. [D 20:00:56.410622] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:00:56.410630] get_data_sync_mode returning 0 [D 20:00:56.410640] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.410648] dbpf_bstream_rw_list: mem_offset: 0x2aaaad2c1010, mem_size: 2097152 [D 20:00:56.410655] dbpf_bstream_rw_list: stream_offset: 201326592, stream_size: 2097152 [D 20:00:56.410662] dbpf_open_cache_get: called [D 20:00:56.410673] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.410680] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.410696] DBPF I/O ops in progress: 1 [D 20:00:56.410704] lio_listio called with the following aiocbs: [D 20:00:56.410710] aiocb_ptr_array[0]: fd: 13, off: 201326592, bytes: 2097152, buf: 0x2aaaad2c1010, type: 1 [D 20:00:56.410737] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.413597] ========================================================= [D 20:00:56.413661] PINT_process_request [D 20:00:56.413669] tiling 8 copies [D 20:00:56.413677] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.413685] lvl 0 el 0 blk 0 by 203423744 [D 20:00:56.413691] to 203423744 ta 201326592 fi 234881024 [D 20:00:56.413699] basic type or contiguous data [D 20:00:56.413705] Request disp [D 20:00:56.413711] contig_offset = 203423744 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.413718] calling distribute [D 20:00:56.413724] PINT_distribute [D 20:00:56.413730] of 203423744 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 203423744 exfl 1 [D 20:00:56.413740] begin iteration loff: 203423744 [D 20:00:56.413748] segment exceeds byte limit [D 20:00:56.413754] file being extended [D 20:00:56.413760] process a segment [D 20:00:56.413766] of 203423744 sz 2097152 [D 20:00:56.413773] add a segment 0 [D 20:00:56.413780] end iteration [D 20:00:56.413786] done with segments or bytes [D 20:00:56.413792] of 205520896 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.413800] next loff: 205520896 next poff: 205520896 [D 20:00:56.413806] return value 2097152 [D 20:00:56.413813] finished [D 20:00:56.413819] retval = 2097152 [D 20:00:56.413825] exiting distribute returned less than expected [D 20:00:56.413832] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 205520896 fo 234881024 eof 0 [D 20:00:56.413839] ========================================================= [D 20:00:56.413846] BMI_post_recv: addr: 6843072, offset: 0x2aaaaeecf010, size: 2097152, tag: 14 [D 20:00:56.415230] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.415257] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.415268] dbpf_open_cache_put: cached entry. [D 20:00:56.415275] dbpf_open_cache_put: move to unused list. [D 20:00:56.415281] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.415289] op_queue add: 0x6e5fe0 [D 20:00:56.415310] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:00:56.415326] DBPF I/O ops in progress: 0 [D 20:00:56.415822] openib_check_cq: completed recv wr_id 684530. [D 20:00:56.415836] openib_post_rr: 10.1.4.56:39990 bh 52 wr_id 684530. [D 20:00:56.415846] post_sr: 10.1.4.56:39990 bh 30 len 48 wr_id 682e40. [D 20:00:56.415862] openib_check_cq: completed send wr_id 682e40. [D 20:00:56.416069] openib_check_cq: completed recv wr_id 684560. [D 20:00:56.416081] openib_post_rr: 10.1.4.56:39990 bh 53 wr_id 684560. [D 20:00:56.416091] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:00:56.416100] get_data_sync_mode returning 0 [D 20:00:56.416110] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.416117] dbpf_bstream_rw_list: mem_offset: 0x2aaaad4c2010, mem_size: 2097152 [D 20:00:56.416124] dbpf_bstream_rw_list: stream_offset: 234881024, stream_size: 2097152 [D 20:00:56.416131] dbpf_open_cache_get: called [D 20:00:56.416142] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.416148] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.416163] DBPF I/O ops in progress: 1 [D 20:00:56.416172] lio_listio called with the following aiocbs: [D 20:00:56.416179] aiocb_ptr_array[0]: fd: 13, off: 234881024, bytes: 2097152, buf: 0x2aaaad4c2010, type: 1 [D 20:00:56.416192] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.419040] ========================================================= [D 20:00:56.419109] PINT_process_request [D 20:00:56.419117] tiling 9 copies [D 20:00:56.419125] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.419134] lvl 0 el 0 blk 0 by 236978176 [D 20:00:56.419140] to 236978176 ta 234881024 fi 268435456 [D 20:00:56.419147] basic type or contiguous data [D 20:00:56.419154] Request disp [D 20:00:56.419160] contig_offset = 236978176 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.419167] calling distribute [D 20:00:56.419174] PINT_distribute [D 20:00:56.419181] of 236978176 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 236978176 exfl 1 [D 20:00:56.419189] begin iteration loff: 236978176 [D 20:00:56.419196] segment exceeds byte limit [D 20:00:56.419202] file being extended [D 20:00:56.419209] process a segment [D 20:00:56.419214] of 236978176 sz 2097152 [D 20:00:56.419221] add a segment 0 [D 20:00:56.419228] end iteration [D 20:00:56.419234] done with segments or bytes [D 20:00:56.419240] of 239075328 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.419247] next loff: 239075328 next poff: 239075328 [D 20:00:56.419254] return value 2097152 [D 20:00:56.419260] finished [D 20:00:56.419266] retval = 2097152 [D 20:00:56.419272] exiting distribute returned less than expected [D 20:00:56.419278] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 239075328 fo 268435456 eof 0 [D 20:00:56.419286] ========================================================= [D 20:00:56.419293] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf0d0010, size: 2097152, tag: 15 [D 20:00:56.420703] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.420733] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.420744] dbpf_open_cache_put: cached entry. [D 20:00:56.420751] dbpf_open_cache_put: move to unused list. [D 20:00:56.420757] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.420764] op_queue add: 0x6e5fe0 [D 20:00:56.420786] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:00:56.420803] DBPF I/O ops in progress: 0 [D 20:00:56.421314] openib_check_cq: completed recv wr_id 684590. [D 20:00:56.421328] openib_post_rr: 10.1.4.56:39990 bh 54 wr_id 684590. [D 20:00:56.421338] post_sr: 10.1.4.56:39990 bh 31 len 48 wr_id 682e70. [D 20:00:56.421354] openib_check_cq: completed send wr_id 682e70. [D 20:00:56.421550] openib_check_cq: completed recv wr_id 6845c0. [D 20:00:56.421561] openib_post_rr: 10.1.4.56:39990 bh 55 wr_id 6845c0. [D 20:00:56.421572] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:00:56.421580] get_data_sync_mode returning 0 [D 20:00:56.421590] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.421598] dbpf_bstream_rw_list: mem_offset: 0x2aaaad6c3010, mem_size: 2097152 [D 20:00:56.421605] dbpf_bstream_rw_list: stream_offset: 268435456, stream_size: 2097152 [D 20:00:56.421612] dbpf_open_cache_get: called [D 20:00:56.421618] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.421631] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.421647] DBPF I/O ops in progress: 1 [D 20:00:56.421654] lio_listio called with the following aiocbs: [D 20:00:56.421661] aiocb_ptr_array[0]: fd: 13, off: 268435456, bytes: 2097152, buf: 0x2aaaad6c3010, type: 1 [D 20:00:56.421673] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.426248] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.426295] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.426320] dbpf_open_cache_put: cached entry. [D 20:00:56.426328] dbpf_open_cache_put: move to unused list. [D 20:00:56.426344] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.426361] ========================================================= [D 20:00:56.426376] PINT_process_request [D 20:00:56.426383] tiling 10 copies [D 20:00:56.426393] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.426400] lvl 0 el 0 blk 0 by 270532608 [D 20:00:56.426407] to 270532608 ta 268435456 fi 301989888 [D 20:00:56.426414] basic type or contiguous data [D 20:00:56.426424] Request disp [D 20:00:56.426430] contig_offset = 270532608 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.426437] calling distribute [D 20:00:56.426443] PINT_distribute [D 20:00:56.426457] of 270532608 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 270532608 exfl 1 [D 20:00:56.426474] begin iteration loff: 270532608 [D 20:00:56.426482] segment exceeds byte limit [D 20:00:56.426488] file being extended [D 20:00:56.426494] process a segment [D 20:00:56.426500] of 270532608 sz 2097152 [D 20:00:56.426506] add a segment 0 [D 20:00:56.426518] end iteration [D 20:00:56.426524] done with segments or bytes [D 20:00:56.426530] of 272629760 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.426537] next loff: 272629760 next poff: 272629760 [D 20:00:56.426554] return value 2097152 [D 20:00:56.426560] finished [D 20:00:56.426566] retval = 2097152 [D 20:00:56.426572] exiting distribute returned less than expected [D 20:00:56.426587] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 272629760 fo 301989888 eof 0 [D 20:00:56.426600] ========================================================= [D 20:00:56.426608] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf2d1010, size: 2097152, tag: 16 [D 20:00:56.426783] openib_check_cq: completed recv wr_id 6845f0. [D 20:00:56.426796] openib_post_rr: 10.1.4.56:39990 bh 56 wr_id 6845f0. [D 20:00:56.426806] post_sr: 10.1.4.56:39990 bh 32 len 48 wr_id 682ea0. [D 20:00:56.426823] openib_check_cq: completed send wr_id 682ea0. [D 20:00:56.427018] openib_check_cq: completed recv wr_id 684620. [D 20:00:56.427028] openib_post_rr: 10.1.4.56:39990 bh 57 wr_id 684620. [D 20:00:56.427037] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:00:56.427046] get_data_sync_mode returning 0 [D 20:00:56.427057] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.427064] dbpf_bstream_rw_list: mem_offset: 0x2aaaad8c4010, mem_size: 2097152 [D 20:00:56.427071] dbpf_bstream_rw_list: stream_offset: 301989888, stream_size: 2097152 [D 20:00:56.427078] dbpf_open_cache_get: called [D 20:00:56.427084] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.427092] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.427114] DBPF I/O ops in progress: 2 [D 20:00:56.427122] lio_listio called with the following aiocbs: [D 20:00:56.427128] aiocb_ptr_array[0]: fd: 13, off: 301989888, bytes: 2097152, buf: 0x2aaaad8c4010, type: 1 [D 20:00:56.427141] issue_or_delay_io_operation: lio_listio posted 0x6ec190 (handle 2863311531, ret 0) [D 20:00:56.426352] op_queue add: 0x6e5fe0 [D 20:00:56.429020] ========================================================= [D 20:00:56.429046] PINT_process_request [D 20:00:56.429054] tiling 11 copies [D 20:00:56.429062] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.429071] lvl 0 el 0 blk 0 by 304087040 [D 20:00:56.429077] to 304087040 ta 301989888 fi 335544320 [D 20:00:56.429086] basic type or contiguous data [D 20:00:56.429093] Request disp [D 20:00:56.429100] contig_offset = 304087040 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.429108] calling distribute [D 20:00:56.429114] PINT_distribute [D 20:00:56.429120] of 304087040 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 304087040 exfl 1 [D 20:00:56.429129] begin iteration loff: 304087040 [D 20:00:56.429136] segment exceeds byte limit [D 20:00:56.429143] file being extended [D 20:00:56.429150] process a segment [D 20:00:56.429156] of 304087040 sz 2097152 [D 20:00:56.429163] add a segment 0 [D 20:00:56.429169] end iteration [D 20:00:56.429175] done with segments or bytes [D 20:00:56.429181] of 306184192 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.429189] next loff: 306184192 next poff: 306184192 [D 20:00:56.429196] return value 2097152 [D 20:00:56.429205] finished [D 20:00:56.429211] retval = 2097152 [D 20:00:56.429218] exiting distribute returned less than expected [D 20:00:56.429224] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 306184192 fo 335544320 eof 0 [D 20:00:56.429232] ========================================================= [D 20:00:56.429238] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf4d2010, size: 2097152, tag: 17 [D 20:00:56.432235] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:00:56.432269] openib_check_cq: completed recv wr_id 684650. [D 20:00:56.432287] DBPF I/O ops in progress: 1 [D 20:00:56.432287] openib_post_rr: 10.1.4.56:39990 bh 58 wr_id 684650. [D 20:00:56.432315] post_sr: 10.1.4.56:39990 bh 33 len 48 wr_id 682ed0. [D 20:00:56.432331] openib_check_cq: completed send wr_id 682ed0. [D 20:00:56.432503] openib_check_cq: completed recv wr_id 684680. [D 20:00:56.432513] openib_post_rr: 10.1.4.56:39990 bh 59 wr_id 684680. [D 20:00:56.432524] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:00:56.432533] get_data_sync_mode returning 0 [D 20:00:56.432543] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.432550] dbpf_bstream_rw_list: mem_offset: 0x2aaaadac5010, mem_size: 2097152 [D 20:00:56.432557] dbpf_bstream_rw_list: stream_offset: 335544320, stream_size: 2097152 [D 20:00:56.432564] dbpf_open_cache_get: called [D 20:00:56.432571] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.432579] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.432607] DBPF I/O ops in progress: 2 [D 20:00:56.432615] lio_listio called with the following aiocbs: [D 20:00:56.432622] aiocb_ptr_array[0]: fd: 13, off: 335544320, bytes: 2097152, buf: 0x2aaaadac5010, type: 1 [D 20:00:56.432638] issue_or_delay_io_operation: lio_listio posted 0x6e5fe0 (handle 2863311531, ret 0) [D 20:00:56.434095] --- aio_progress_notification called with handle 2863311531 (0x6ec190) [D 20:00:56.434161] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.434182] dbpf_open_cache_put: cached entry. [D 20:00:56.434191] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.434199] op_queue add: 0x6ec190 [D 20:00:56.434225] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:00:56.434247] DBPF I/O ops in progress: 1 [D 20:00:56.437517] ========================================================= [D 20:00:56.437561] PINT_process_request [D 20:00:56.437574] tiling 12 copies [D 20:00:56.437581] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.437589] lvl 0 el 0 blk 0 by 337641472 [D 20:00:56.437595] to 337641472 ta 335544320 fi 369098752 [D 20:00:56.437603] basic type or contiguous data [D 20:00:56.437609] Request disp [D 20:00:56.437615] contig_offset = 337641472 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.437623] calling distribute [D 20:00:56.437628] PINT_distribute [D 20:00:56.437635] of 337641472 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 337641472 exfl 1 [D 20:00:56.437643] begin iteration loff: 337641472 [D 20:00:56.437661] segment exceeds byte limit [D 20:00:56.437667] file being extended [D 20:00:56.437673] process a segment [D 20:00:56.437679] of 337641472 sz 2097152 [D 20:00:56.437686] add a segment 0 [D 20:00:56.437692] end iteration [D 20:00:56.437698] done with segments or bytes [D 20:00:56.437720] of 339738624 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.437727] next loff: 339738624 next poff: 339738624 [D 20:00:56.437734] return value 2097152 [D 20:00:56.437741] finished [D 20:00:56.437754] retval = 2097152 [D 20:00:56.437760] exiting distribute returned less than expected [D 20:00:56.437766] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 339738624 fo 369098752 eof 0 [D 20:00:56.437773] ========================================================= [D 20:00:56.437789] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf6d3010, size: 2097152, tag: 18 [D 20:00:56.437803] openib_check_cq: completed recv wr_id 6846b0. [D 20:00:56.437822] openib_post_rr: 10.1.4.56:39990 bh 60 wr_id 6846b0. [D 20:00:56.437864] post_sr: 10.1.4.56:39990 bh 34 len 48 wr_id 682f00. [D 20:00:56.437888] openib_check_cq: completed send wr_id 682f00. [D 20:00:56.437939] openib_check_cq: completed recv wr_id 6846e0. [D 20:00:56.437949] openib_post_rr: 10.1.4.56:39990 bh 61 wr_id 6846e0. [D 20:00:56.437974] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:00:56.437985] get_data_sync_mode returning 0 [D 20:00:56.437996] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.438005] dbpf_bstream_rw_list: mem_offset: 0x2aaaadcc6010, mem_size: 2097152 [D 20:00:56.438012] dbpf_bstream_rw_list: stream_offset: 369098752, stream_size: 2097152 [D 20:00:56.438019] dbpf_open_cache_get: called [D 20:00:56.438026] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.438034] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.438065] DBPF I/O ops in progress: 2 [D 20:00:56.438073] lio_listio called with the following aiocbs: [D 20:00:56.438079] aiocb_ptr_array[0]: fd: 13, off: 369098752, bytes: 2097152, buf: 0x2aaaadcc6010, type: 1 [D 20:00:56.438097] issue_or_delay_io_operation: lio_listio posted 0x6ec240 (handle 2863311531, ret 0) [D 20:00:56.437500] --- aio_progress_notification called with handle 2863311531 (0x6e5fe0) [D 20:00:56.440486] ========================================================= [D 20:00:56.440549] PINT_process_request [D 20:00:56.440559] tiling 13 copies [D 20:00:56.440567] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.440575] lvl 0 el 0 blk 0 by 371195904 [D 20:00:56.440581] to 371195904 ta 369098752 fi 402653184 [D 20:00:56.440589] basic type or contiguous data [D 20:00:56.440596] Request disp [D 20:00:56.440602] contig_offset = 371195904 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.440609] calling distribute [D 20:00:56.440615] PINT_distribute [D 20:00:56.440622] of 371195904 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 371195904 exfl 1 [D 20:00:56.440631] begin iteration loff: 371195904 [D 20:00:56.440639] segment exceeds byte limit [D 20:00:56.440645] file being extended [D 20:00:56.440652] process a segment [D 20:00:56.440658] of 371195904 sz 2097152 [D 20:00:56.440665] add a segment 0 [D 20:00:56.440671] end iteration [D 20:00:56.440677] done with segments or bytes [D 20:00:56.440684] of 373293056 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.440691] next loff: 373293056 next poff: 373293056 [D 20:00:56.440697] return value 2097152 [D 20:00:56.440704] finished [D 20:00:56.440710] retval = 2097152 [D 20:00:56.440716] exiting distribute returned less than expected [D 20:00:56.440722] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 373293056 fo 402653184 eof 0 [D 20:00:56.440730] ========================================================= [D 20:00:56.440737] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf8d4010, size: 2097152, tag: 19 [D 20:00:56.442865] --- aio_progress_notification called with handle 2863311531 (0x6ec240) [D 20:00:56.442905] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.442916] dbpf_open_cache_put: cached entry. [D 20:00:56.442923] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.442930] op_queue add: 0x6ec240 [D 20:00:56.442943] DBPF I/O ops in progress: 1 [D 20:00:56.442970] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:00:56.443127] openib_check_cq: completed recv wr_id 684710. [D 20:00:56.443138] openib_post_rr: 10.1.4.56:39990 bh 62 wr_id 684710. [D 20:00:56.443148] post_sr: 10.1.4.56:39990 bh 35 len 48 wr_id 682f30. [D 20:00:56.443164] openib_check_cq: completed send wr_id 682f30. [D 20:00:56.443361] openib_check_cq: completed recv wr_id 684740. [D 20:00:56.443369] openib_post_rr: 10.1.4.56:39990 bh 63 wr_id 684740. [D 20:00:56.443379] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:00:56.443388] get_data_sync_mode returning 0 [D 20:00:56.443397] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.443404] dbpf_bstream_rw_list: mem_offset: 0x2aaaadec7010, mem_size: 2097152 [D 20:00:56.443411] dbpf_bstream_rw_list: stream_offset: 402653184, stream_size: 2097152 [D 20:00:56.443418] dbpf_open_cache_get: called [D 20:00:56.443425] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.443431] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.443445] DBPF I/O ops in progress: 2 [D 20:00:56.443452] lio_listio called with the following aiocbs: [D 20:00:56.443458] aiocb_ptr_array[0]: fd: 13, off: 402653184, bytes: 2097152, buf: 0x2aaaadec7010, type: 1 [D 20:00:56.443471] issue_or_delay_io_operation: lio_listio posted 0x6ec240 (handle 2863311531, ret 0) [D 20:00:56.446182] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.446252] dbpf_open_cache_put: cached entry. [D 20:00:56.446260] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.446268] op_queue add: 0x6e5fe0 [D 20:00:56.446297] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:00:56.446315] DBPF I/O ops in progress: 1 [D 20:00:56.447901] ========================================================= [D 20:00:56.447922] PINT_process_request [D 20:00:56.447932] tiling 14 copies [D 20:00:56.447939] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.447947] lvl 0 el 0 blk 0 by 404750336 [D 20:00:56.447954] to 404750336 ta 402653184 fi 436207616 [D 20:00:56.447961] basic type or contiguous data [D 20:00:56.447967] Request disp [D 20:00:56.447984] contig_offset = 404750336 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.447991] calling distribute [D 20:00:56.447997] PINT_distribute [D 20:00:56.448003] of 404750336 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 404750336 exfl 1 [D 20:00:56.448011] begin iteration loff: 404750336 [D 20:00:56.448020] segment exceeds byte limit [D 20:00:56.448026] file being extended [D 20:00:56.448032] process a segment [D 20:00:56.448038] of 404750336 sz 2097152 [D 20:00:56.448054] add a segment 0 [D 20:00:56.448060] end iteration [D 20:00:56.448066] done with segments or bytes [D 20:00:56.448072] of 406847488 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.448088] next loff: 406847488 next poff: 406847488 [D 20:00:56.448095] return value 2097152 [D 20:00:56.448102] finished [D 20:00:56.448112] retval = 2097152 [D 20:00:56.448123] exiting distribute returned less than expected [D 20:00:56.448129] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 406847488 fo 436207616 eof 0 [D 20:00:56.448136] ========================================================= [D 20:00:56.448146] BMI_post_recv: addr: 6843072, offset: 0x2aaaafad5010, size: 2097152, tag: 20 [D 20:00:56.447860] --- aio_progress_notification called with handle 2863311531 (0x6ec240) [D 20:00:56.448419] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.448464] dbpf_open_cache_put: cached entry. [D 20:00:56.448501] dbpf_open_cache_put: move to unused list. [D 20:00:56.448539] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.448577] op_queue add: 0x6ec240 [D 20:00:56.448626] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:00:56.448672] DBPF I/O ops in progress: 0 [D 20:00:56.451345] openib_check_cq: completed recv wr_id 684770. [D 20:00:56.451359] openib_post_rr: 10.1.4.56:39990 bh 64 wr_id 684770. [D 20:00:56.451581] openib_check_cq: completed recv wr_id 6847a0. [D 20:00:56.451592] openib_post_rr: 10.1.4.56:39990 bh 65 wr_id 6847a0. [D 20:00:56.451602] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:00:56.451610] get_data_sync_mode returning 0 [D 20:00:56.451624] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.451632] dbpf_bstream_rw_list: mem_offset: 0x2aaaae0c8010, mem_size: 2097152 [D 20:00:56.451639] dbpf_bstream_rw_list: stream_offset: 436207616, stream_size: 2097152 [D 20:00:56.451646] dbpf_open_cache_get: called [D 20:00:56.451656] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.451663] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.451678] DBPF I/O ops in progress: 1 [D 20:00:56.451687] lio_listio called with the following aiocbs: [D 20:00:56.451693] aiocb_ptr_array[0]: fd: 13, off: 436207616, bytes: 2097152, buf: 0x2aaaae0c8010, type: 1 [D 20:00:56.451710] issue_or_delay_io_operation: lio_listio posted 0x6ec240 (handle 2863311531, ret 0) [D 20:00:56.453078] ========================================================= [D 20:00:56.453121] PINT_process_request [D 20:00:56.453131] tiling 15 copies [D 20:00:56.453139] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.453147] lvl 0 el 0 blk 0 by 438304768 [D 20:00:56.453154] to 438304768 ta 436207616 fi 469762048 [D 20:00:56.453162] basic type or contiguous data [D 20:00:56.453168] Request disp [D 20:00:56.453175] contig_offset = 438304768 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.453183] calling distribute [D 20:00:56.453189] PINT_distribute [D 20:00:56.453195] of 438304768 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 438304768 exfl 1 [D 20:00:56.453203] begin iteration loff: 438304768 [D 20:00:56.453211] segment exceeds byte limit [D 20:00:56.453217] file being extended [D 20:00:56.453224] process a segment [D 20:00:56.453230] of 438304768 sz 2097152 [D 20:00:56.453237] add a segment 0 [D 20:00:56.453243] end iteration [D 20:00:56.453249] done with segments or bytes [D 20:00:56.453255] of 440401920 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.453263] next loff: 440401920 next poff: 440401920 [D 20:00:56.453270] return value 2097152 [D 20:00:56.453277] finished [D 20:00:56.453283] retval = 2097152 [D 20:00:56.453290] exiting distribute returned less than expected [D 20:00:56.453296] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 440401920 fo 469762048 eof 0 [D 20:00:56.453304] ========================================================= [D 20:00:56.453310] BMI_post_recv: addr: 6843072, offset: 0x2aaaafcd6010, size: 2097152, tag: 21 [D 20:00:56.455169] openib_check_cq: completed recv wr_id 6847d0. [D 20:00:56.455210] openib_post_rr: 10.1.4.56:39990 bh 66 wr_id 6847d0. [D 20:00:56.455224] post_sr: 10.1.4.56:39990 bh 36 len 48 wr_id 682f60. [D 20:00:56.455248] openib_check_cq: completed send wr_id 682f60. [D 20:00:56.456319] --- aio_progress_notification called with handle 2863311531 (0x6ec240) [D 20:00:56.456342] openib_check_cq: completed recv wr_id 684800. [D 20:00:56.456390] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.456394] openib_post_rr: 10.1.4.56:39990 bh 67 wr_id 684800. [D 20:00:56.456471] dbpf_open_cache_put: cached entry. [D 20:00:56.456481] post_sr: 10.1.4.56:39990 bh 37 len 48 wr_id 682f90. [D 20:00:56.456543] dbpf_open_cache_put: move to unused list. [D 20:00:56.456553] openib_check_cq: completed send wr_id 682f90. [D 20:00:56.456614] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.456681] op_queue add: 0x6ec240 [D 20:00:56.456732] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:00:56.456778] DBPF I/O ops in progress: 0 [D 20:00:56.457508] openib_check_cq: completed recv wr_id 684830. [D 20:00:56.457519] openib_post_rr: 10.1.4.56:39990 bh 68 wr_id 684830. [D 20:00:56.457528] post_sr: 10.1.4.56:39990 bh 38 len 48 wr_id 682fc0. [D 20:00:56.457542] openib_check_cq: completed send wr_id 682fc0. [D 20:00:56.458676] openib_check_cq: completed recv wr_id 684860. [D 20:00:56.458687] openib_post_rr: 10.1.4.56:39990 bh 69 wr_id 684860. [D 20:00:56.458695] post_sr: 10.1.4.56:39990 bh 39 len 48 wr_id 682ff0. [D 20:00:56.458708] openib_check_cq: completed send wr_id 682ff0. -[D 20:00:56.468850] openib_check_cq: completed recv wr_id 684890. [D 20:00:56.468859] openib_post_rr: 10.1.4.56:39990 bh 70 wr_id 684890. [D 20:00:56.468867] post_sr: 10.1.4.56:39990 bh 40 len 48 wr_id 683020. [D 20:00:56.468878] openib_check_cq: completed recv wr_id 6848c0. [D 20:00:56.468885] openib_post_rr: 10.1.4.56:39990 bh 71 wr_id 6848c0. [D 20:00:56.468893] post_sr: 10.1.4.56:39990 bh 41 len 48 wr_id 683050. [D 20:00:56.468900] openib_check_cq: completed recv wr_id 6848f0. [D 20:00:56.468908] openib_post_rr: 10.1.4.56:39990 bh 72 wr_id 6848f0. [D 20:00:56.468915] post_sr: 10.1.4.56:39990 bh 42 len 48 wr_id 683080. [D 20:00:56.468922] openib_check_cq: completed recv wr_id 684920. [D 20:00:56.468934] openib_post_rr: 10.1.4.56:39990 bh 73 wr_id 684920. [D 20:00:56.468947] post_sr: 10.1.4.56:39990 bh 43 len 48 wr_id 6830b0. [D 20:00:56.468955] openib_check_cq: completed recv wr_id 684950. [D 20:00:56.468965] openib_post_rr: 10.1.4.56:39990 bh 74 wr_id 684950. [D 20:00:56.468972] post_sr: 10.1.4.56:39990 bh 44 len 48 wr_id 6830e0. [D 20:00:56.468980] openib_check_cq: completed recv wr_id 684980. [D 20:00:56.468988] openib_post_rr: 10.1.4.56:39990 bh 75 wr_id 684980. [D 20:00:56.468996] openib_check_cq: completed recv wr_id 6849b0. [D 20:00:56.469003] openib_post_rr: 10.1.4.56:39990 bh 76 wr_id 6849b0. [D 20:00:56.469010] openib_check_cq: completed recv wr_id 6849e0. [D 20:00:56.469022] openib_post_rr: 10.1.4.56:39990 bh 77 wr_id 6849e0. [D 20:00:56.469029] post_sr: 10.1.4.56:39990 bh 45 len 48 wr_id 683110. [D 20:00:56.469037] openib_check_cq: completed recv wr_id 684a10. [D 20:00:56.469044] openib_post_rr: 10.1.4.56:39990 bh 78 wr_id 684a10. [D 20:00:56.469051] post_sr: 10.1.4.56:39990 bh 46 len 48 wr_id 683140. [D 20:00:56.469058] openib_check_cq: completed recv wr_id 684a40. [D 20:00:56.469083] openib_post_rr: 10.1.4.56:39990 bh 79 wr_id 684a40. [D 20:00:56.469092] post_sr: 10.1.4.56:39990 bh 47 len 48 wr_id 683170. [D 20:00:56.469100] openib_check_cq: completed send wr_id 683020. [D 20:00:56.469109] openib_check_cq: completed send wr_id 683050. [D 20:00:56.469116] openib_check_cq: completed send wr_id 683080. [D 20:00:56.469123] openib_check_cq: completed send wr_id 6830b0. [D 20:00:56.469130] openib_check_cq: completed send wr_id 6830e0. [D 20:00:56.469137] openib_check_cq: completed send wr_id 683110. [D 20:00:56.469144] openib_check_cq: completed send wr_id 683140. [D 20:00:56.469150] openib_check_cq: completed send wr_id 683170. [D 20:00:56.469163] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:00:56.469173] get_data_sync_mode returning 0 [D 20:00:56.469182] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.469195] dbpf_bstream_rw_list: mem_offset: 0x2aaaae2c9010, mem_size: 2097152 [D 20:00:56.469202] dbpf_bstream_rw_list: stream_offset: 2097152, stream_size: 2097152 [D 20:00:56.469209] dbpf_open_cache_get: called [D 20:00:56.469216] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.469226] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.469241] DBPF I/O ops in progress: 1 [D 20:00:56.469248] lio_listio called with the following aiocbs: [D 20:00:56.469258] aiocb_ptr_array[0]: fd: 13, off: 2097152, bytes: 2097152, buf: 0x2aaaae2c9010, type: 1 [D 20:00:56.469270] issue_or_delay_io_operation: lio_listio posted 0x6ec240 (handle 2863311531, ret 0) [D 20:00:56.470641] ========================================================= [D 20:00:56.470685] PINT_process_request [D 20:00:56.470694] tiling 2 copies [D 20:00:56.470702] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.470710] lvl 0 el 0 blk 0 by 4194304 [D 20:00:56.470717] to 4194304 ta 0 fi 33554432 [D 20:00:56.470724] basic type or contiguous data [D 20:00:56.470731] Request disp [D 20:00:56.470737] contig_offset = 4194304 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.470745] calling distribute [D 20:00:56.470751] PINT_distribute [D 20:00:56.470758] of 4194304 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 4194304 exfl 1 [D 20:00:56.470767] begin iteration loff: 4194304 [D 20:00:56.470775] segment exceeds byte limit [D 20:00:56.470781] file being extended [D 20:00:56.470787] process a segment [D 20:00:56.470793] of 4194304 sz 2097152 [D 20:00:56.470800] add a segment 0 [D 20:00:56.470806] end iteration [D 20:00:56.470812] done with segments or bytes [D 20:00:56.470818] of 6291456 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.470825] next loff: 6291456 next poff: 6291456 [D 20:00:56.470832] return value 2097152 [D 20:00:56.470839] finished [D 20:00:56.470845] retval = 2097152 [D 20:00:56.470852] exiting distribute returned less than expected [D 20:00:56.470858] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 6291456 fo 33554432 eof 0 [D 20:00:56.470865] ========================================================= [D 20:00:56.470872] BMI_post_recv: addr: 6843072, offset: 0x2aaaafed7010, size: 2097152, tag: 8 [D 20:00:56.470885] openib_check_cq: completed recv wr_id 684a70. [D 20:00:56.470895] openib_post_rr: 10.1.4.56:39990 bh 80 wr_id 684a70. [D 20:00:56.470906] post_sr: 10.1.4.56:39990 bh 48 len 48 wr_id 6831a0. [D 20:00:56.470925] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:00:56.470935] get_data_sync_mode returning 0 [D 20:00:56.470948] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.470955] dbpf_bstream_rw_list: mem_offset: 0x2aaaae4ca010, mem_size: 2097152 [D 20:00:56.470962] dbpf_bstream_rw_list: stream_offset: 35651584, stream_size: 2097152 [D 20:00:56.470969] dbpf_open_cache_get: called [D 20:00:56.470976] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.470984] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.471008] DBPF I/O ops in progress: 2 [D 20:00:56.471015] lio_listio called with the following aiocbs: [D 20:00:56.471022] aiocb_ptr_array[0]: fd: 13, off: 35651584, bytes: 2097152, buf: 0x2aaaae4ca010, type: 1 [D 20:00:56.471033] issue_or_delay_io_operation: lio_listio posted 0x6ec440 (handle 2863311531, ret 0) [D 20:00:56.473848] ========================================================= [D 20:00:56.473926] PINT_process_request [D 20:00:56.473936] tiling 3 copies [D 20:00:56.473944] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.473953] lvl 0 el 0 blk 0 by 37748736 [D 20:00:56.473960] to 37748736 ta 33554432 fi 67108864 [D 20:00:56.473967] basic type or contiguous data [D 20:00:56.473974] Request disp [D 20:00:56.473980] contig_offset = 37748736 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.473988] calling distribute [D 20:00:56.473994] PINT_distribute [D 20:00:56.474002] of 37748736 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 37748736 exfl 1 [D 20:00:56.474010] begin iteration loff: 37748736 [D 20:00:56.474017] segment exceeds byte limit [D 20:00:56.474023] file being extended [D 20:00:56.474030] process a segment [D 20:00:56.474036] of 37748736 sz 2097152 [D 20:00:56.474042] add a segment 0 [D 20:00:56.474049] end iteration [D 20:00:56.474056] done with segments or bytes [D 20:00:56.474062] of 39845888 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.474069] next loff: 39845888 next poff: 39845888 [D 20:00:56.474077] return value 2097152 [D 20:00:56.474084] finished [D 20:00:56.474090] retval = 2097152 [D 20:00:56.474096] exiting distribute returned less than expected [D 20:00:56.474102] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 39845888 fo 67108864 eof 0 [D 20:00:56.474109] ========================================================= [D 20:00:56.474118] BMI_post_recv: addr: 6843072, offset: 0x2aaab00d8010, size: 2097152, tag: 9 [D 20:00:56.474135] openib_check_cq: completed send wr_id 6831a0. [D 20:00:56.474146] openib_check_cq: completed recv wr_id 684aa0. [D 20:00:56.474159] openib_post_rr: 10.1.4.56:39990 bh 81 wr_id 684aa0. [D 20:00:56.474172] post_sr: 10.1.4.56:39990 bh 49 len 48 wr_id 6831d0. [D 20:00:56.474182] openib_check_cq: completed recv wr_id 684ad0. [D 20:00:56.474192] openib_post_rr: 10.1.4.56:39990 bh 82 wr_id 684ad0. [D 20:00:56.474200] openib_check_cq: completed recv wr_id 684b00. [D 20:00:56.474208] openib_post_rr: 10.1.4.56:39990 bh 83 wr_id 684b00. [D 20:00:56.474215] openib_check_cq: completed send wr_id 6831d0. [D 20:00:56.474254] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:00:56.474266] get_data_sync_mode returning 0 [D 20:00:56.474281] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.474289] dbpf_bstream_rw_list: mem_offset: 0x2aaaae6cb010, mem_size: 2097152 [D 20:00:56.474297] dbpf_bstream_rw_list: stream_offset: 69206016, stream_size: 2097152 [D 20:00:56.474304] dbpf_open_cache_get: called [D 20:00:56.474311] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.474318] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.474357] DBPF I/O ops in progress: 3 [D 20:00:56.474366] lio_listio called with the following aiocbs: [D 20:00:56.474372] aiocb_ptr_array[0]: fd: 13, off: 69206016, bytes: 2097152, buf: 0x2aaaae6cb010, type: 1 [D 20:00:56.474387] issue_or_delay_io_operation: lio_listio posted 0x6ec710 (handle 2863311531, ret 0) [D 20:00:56.477858] --- aio_progress_notification called with handle 2863311531 (0x6ec240) [D 20:00:56.478103] ========================================================= [D 20:00:56.478129] PINT_process_request [D 20:00:56.478141] tiling 4 copies [D 20:00:56.478149] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.478157] lvl 0 el 0 blk 0 by 71303168 [D 20:00:56.478163] to 71303168 ta 67108864 fi 100663296 [D 20:00:56.478171] basic type or contiguous data [D 20:00:56.478177] Request disp [D 20:00:56.478190] contig_offset = 71303168 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.478198] calling distribute [D 20:00:56.478205] PINT_distribute [D 20:00:56.478212] of 71303168 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 71303168 exfl 1 [D 20:00:56.478222] begin iteration loff: 71303168 [D 20:00:56.478230] segment exceeds byte limit [D 20:00:56.478236] file being extended [D 20:00:56.478243] process a segment [D 20:00:56.478248] of 71303168 sz 2097152 [D 20:00:56.478255] add a segment 0 [D 20:00:56.478262] end iteration [D 20:00:56.478268] done with segments or bytes [D 20:00:56.478274] of 73400320 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.478281] next loff: 73400320 next poff: 73400320 [D 20:00:56.478288] return value 2097152 [D 20:00:56.478296] finished [D 20:00:56.478302] retval = 2097152 [D 20:00:56.478308] exiting distribute returned less than expected [D 20:00:56.478314] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 73400320 fo 100663296 eof 0 [D 20:00:56.478322] ========================================================= [D 20:00:56.478329] BMI_post_recv: addr: 6843072, offset: 0x2aaab02d9010, size: 2097152, tag: 10 [D 20:00:56.478368] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:00:56.478380] get_data_sync_mode returning 0 [D 20:00:56.478396] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.478404] dbpf_bstream_rw_list: mem_offset: 0x2aaaae8cc010, mem_size: 2097152 [D 20:00:56.478411] dbpf_bstream_rw_list: stream_offset: 102760448, stream_size: 2097152 [D 20:00:56.478419] dbpf_open_cache_get: called [D 20:00:56.478426] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.478434] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.478470] DBPF I/O ops in progress: 4 [D 20:00:56.478479] lio_listio called with the following aiocbs: [D 20:00:56.478485] aiocb_ptr_array[0]: fd: 13, off: 102760448, bytes: 2097152, buf: 0x2aaaae8cc010, type: 1 [D 20:00:56.478500] issue_or_delay_io_operation: lio_listio posted 0x6ec860 (handle 2863311531, ret 0) [D 20:00:56.481865] ========================================================= [D 20:00:56.481947] PINT_process_request [D 20:00:56.481955] tiling 5 copies [D 20:00:56.481963] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.481971] lvl 0 el 0 blk 0 by 104857600 [D 20:00:56.481978] to 104857600 ta 100663296 fi 134217728 [D 20:00:56.481987] basic type or contiguous data [D 20:00:56.481994] Request disp [D 20:00:56.482001] contig_offset = 104857600 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.482010] calling distribute [D 20:00:56.482018] PINT_distribute [D 20:00:56.482026] of 104857600 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 104857600 exfl 1 [D 20:00:56.482034] begin iteration loff: 104857600 [D 20:00:56.482042] segment exceeds byte limit [D 20:00:56.482048] file being extended [D 20:00:56.482055] process a segment [D 20:00:56.482061] of 104857600 sz 2097152 [D 20:00:56.482067] add a segment 0 [D 20:00:56.482075] end iteration [D 20:00:56.482081] done with segments or bytes [D 20:00:56.482087] of 106954752 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.482095] next loff: 106954752 next poff: 106954752 [D 20:00:56.482102] return value 2097152 [D 20:00:56.482108] finished [D 20:00:56.482114] retval = 2097152 [D 20:00:56.482121] exiting distribute returned less than expected [D 20:00:56.482127] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 106954752 fo 134217728 eof 0 [D 20:00:56.482135] ========================================================= [D 20:00:56.482143] BMI_post_recv: addr: 6843072, offset: 0x2aaab04da010, size: 2097152, tag: 11 [D 20:00:56.488362] --- aio_progress_notification called with handle 2863311531 (0x6ec440) [D 20:00:56.488505] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.488519] dbpf_open_cache_put: cached entry. [D 20:00:56.488526] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.488533] op_queue add: 0x6ec440 [D 20:00:56.488552] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:00:56.488571] DBPF I/O ops in progress: 3 [D 20:00:56.488606] --- aio_progress_notification called with handle 2863311531 (0x6ec710) [D 20:00:56.488626] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.488641] dbpf_open_cache_put: cached entry. [D 20:00:56.488648] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.488655] op_queue add: 0x6ec710 [D 20:00:56.488671] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:00:56.488682] DBPF I/O ops in progress: 2 [D 20:00:56.488699] --- aio_progress_notification called with handle 2863311531 (0x6ec860) [D 20:00:56.488715] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.488728] dbpf_open_cache_put: cached entry. [D 20:00:56.488735] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.488741] op_queue add: 0x6ec860 [D 20:00:56.488757] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:56.488769] DBPF I/O ops in progress: 1 [D 20:00:56.488854] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.488900] dbpf_open_cache_put: cached entry. [D 20:00:56.488943] dbpf_open_cache_put: move to unused list. [D 20:00:56.488980] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.489018] op_queue add: 0x6ec240 [D 20:00:56.489063] DBPF I/O ops in progress: 0 [D 20:00:56.489066] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. -[D 20:00:56.493744] openib_check_cq: completed recv wr_id 684b30. [D 20:00:56.493761] openib_post_rr: 10.1.4.56:39990 bh 84 wr_id 684b30. [D 20:00:56.493773] post_sr: 10.1.4.56:39990 bh 50 len 48 wr_id 683200. [D 20:00:56.493782] openib_check_cq: completed recv wr_id 684b60. [D 20:00:56.493789] openib_post_rr: 10.1.4.56:39990 bh 85 wr_id 684b60. [D 20:00:56.493797] post_sr: 10.1.4.56:39990 bh 51 len 48 wr_id 683230. [D 20:00:56.493804] openib_check_cq: completed recv wr_id 684b90. [D 20:00:56.493811] openib_post_rr: 10.1.4.56:39990 bh 86 wr_id 684b90. [D 20:00:56.493819] post_sr: 10.1.4.56:39990 bh 52 len 48 wr_id 683260. [D 20:00:56.493826] openib_check_cq: completed recv wr_id 684bc0. [D 20:00:56.493834] openib_post_rr: 10.1.4.56:39990 bh 87 wr_id 684bc0. [D 20:00:56.493841] post_sr: 10.1.4.56:39990 bh 53 len 48 wr_id 683290. [D 20:00:56.493849] openib_check_cq: completed recv wr_id 684bf0. [D 20:00:56.493857] openib_post_rr: 10.1.4.56:39990 bh 88 wr_id 684bf0. [D 20:00:56.493864] openib_check_cq: completed send wr_id 683200. [D 20:00:56.493871] openib_check_cq: completed send wr_id 683230. [D 20:00:56.493878] openib_check_cq: completed send wr_id 683260. [D 20:00:56.493884] openib_check_cq: completed send wr_id 683290. [D 20:00:56.493896] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:00:56.493905] get_data_sync_mode returning 0 [D 20:00:56.493915] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.493922] dbpf_bstream_rw_list: mem_offset: 0x2aaaaeacd010, mem_size: 2097152 [D 20:00:56.493929] dbpf_bstream_rw_list: stream_offset: 136314880, stream_size: 2097152 [D 20:00:56.493936] dbpf_open_cache_get: called [D 20:00:56.493942] dbpf_open_cache_get: found bstream entry in unused list. [D 20:00:56.493949] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.493965] DBPF I/O ops in progress: 1 [D 20:00:56.493971] lio_listio called with the following aiocbs: [D 20:00:56.493977] aiocb_ptr_array[0]: fd: 13, off: 136314880, bytes: 2097152, buf: 0x2aaaaeacd010, type: 1 [D 20:00:56.493990] issue_or_delay_io_operation: lio_listio posted 0x6ec440 (handle 2863311531, ret 0) [D 20:00:56.497103] ========================================================= [D 20:00:56.497233] PINT_process_request [D 20:00:56.497244] tiling 6 copies [D 20:00:56.497253] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.497261] lvl 0 el 0 blk 0 by 138412032 [D 20:00:56.497268] to 138412032 ta 134217728 fi 167772160 [D 20:00:56.497277] basic type or contiguous data [D 20:00:56.497284] Request disp [D 20:00:56.497292] contig_offset = 138412032 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.497300] calling distribute [D 20:00:56.497307] PINT_distribute [D 20:00:56.497313] of 138412032 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 138412032 exfl 1 [D 20:00:56.497324] begin iteration loff: 138412032 [D 20:00:56.497332] segment exceeds byte limit [D 20:00:56.497339] file being extended [D 20:00:56.497346] process a segment [D 20:00:56.497352] of 138412032 sz 2097152 [D 20:00:56.497359] add a segment 0 [D 20:00:56.497366] end iteration [D 20:00:56.497373] done with segments or bytes [D 20:00:56.497379] of 140509184 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.497386] next loff: 140509184 next poff: 140509184 [D 20:00:56.497392] return value 2097152 [D 20:00:56.497399] finished [D 20:00:56.497406] retval = 2097152 [D 20:00:56.497412] exiting distribute returned less than expected [D 20:00:56.497419] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 140509184 fo 167772160 eof 0 [D 20:00:56.497426] ========================================================= [D 20:00:56.497433] BMI_post_recv: addr: 6843072, offset: 0x2aaab06db010, size: 2097152, tag: 12 [D 20:00:56.497454] openib_check_cq: completed recv wr_id 684c20. [D 20:00:56.497468] openib_post_rr: 10.1.4.56:39990 bh 89 wr_id 684c20. [D 20:00:56.497479] openib_check_cq: completed recv wr_id 684c50. [D 20:00:56.497489] openib_post_rr: 10.1.4.56:39990 bh 90 wr_id 684c50. [D 20:00:56.497497] openib_check_cq: completed recv wr_id 684c80. [D 20:00:56.497504] openib_post_rr: 10.1.4.56:39990 bh 91 wr_id 684c80. [D 20:00:56.497511] openib_check_cq: completed recv wr_id 684cb0. [D 20:00:56.497519] openib_post_rr: 10.1.4.56:39990 bh 92 wr_id 684cb0. [D 20:00:56.497527] openib_check_cq: completed recv wr_id 684ce0. [D 20:00:56.497534] openib_post_rr: 10.1.4.56:39990 bh 93 wr_id 684ce0. [D 20:00:56.497542] openib_check_cq: completed recv wr_id 684d10. [D 20:00:56.497549] openib_post_rr: 10.1.4.56:39990 bh 94 wr_id 684d10. [D 20:00:56.497556] openib_check_cq: completed recv wr_id 684d40. [D 20:00:56.497565] openib_post_rr: 10.1.4.56:39990 bh 95 wr_id 684d40. [D 20:00:56.497573] openib_check_cq: completed recv wr_id 684d70. [D 20:00:56.497580] openib_post_rr: 10.1.4.56:39990 bh 96 wr_id 684d70. [D 20:00:56.497624] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:00:56.497636] get_data_sync_mode returning 0 [D 20:00:56.497651] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.497659] dbpf_bstream_rw_list: mem_offset: 0x2aaaaecce010, mem_size: 2097152 [D 20:00:56.497667] dbpf_bstream_rw_list: stream_offset: 169869312, stream_size: 2097152 [D 20:00:56.497674] dbpf_open_cache_get: called [D 20:00:56.497682] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.497690] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.497722] DBPF I/O ops in progress: 2 [D 20:00:56.497730] lio_listio called with the following aiocbs: [D 20:00:56.497737] aiocb_ptr_array[0]: fd: 13, off: 169869312, bytes: 2097152, buf: 0x2aaaaecce010, type: 1 [D 20:00:56.497753] issue_or_delay_io_operation: lio_listio posted 0x6ec240 (handle 2863311531, ret 0) [D 20:00:56.501045] ========================================================= [D 20:00:56.501196] PINT_process_request [D 20:00:56.501206] tiling 7 copies [D 20:00:56.501214] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.501222] lvl 0 el 0 blk 0 by 171966464 [D 20:00:56.501229] to 171966464 ta 167772160 fi 201326592 [D 20:00:56.501238] basic type or contiguous data [D 20:00:56.501245] Request disp [D 20:00:56.501251] contig_offset = 171966464 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.501258] calling distribute [D 20:00:56.501264] PINT_distribute [D 20:00:56.501272] of 171966464 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 171966464 exfl 1 [D 20:00:56.501281] begin iteration loff: 171966464 [D 20:00:56.501288] segment exceeds byte limit [D 20:00:56.501294] file being extended [D 20:00:56.501301] process a segment [D 20:00:56.501307] of 171966464 sz 2097152 [D 20:00:56.501314] add a segment 0 [D 20:00:56.501321] end iteration [D 20:00:56.501327] done with segments or bytes [D 20:00:56.501334] of 174063616 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.501341] next loff: 174063616 next poff: 174063616 [D 20:00:56.501347] return value 2097152 [D 20:00:56.501354] finished [D 20:00:56.501360] retval = 2097152 [D 20:00:56.501367] exiting distribute returned less than expected [D 20:00:56.501374] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 174063616 fo 201326592 eof 0 [D 20:00:56.501381] ========================================================= [D 20:00:56.501390] BMI_post_recv: addr: 6843072, offset: 0x2aaab08dc010, size: 2097152, tag: 13 [D 20:00:56.501432] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:00:56.501444] get_data_sync_mode returning 0 [D 20:00:56.501461] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.501469] dbpf_bstream_rw_list: mem_offset: 0x2aaaaeecf010, mem_size: 2097152 [D 20:00:56.501476] dbpf_bstream_rw_list: stream_offset: 203423744, stream_size: 2097152 [D 20:00:56.501485] dbpf_open_cache_get: called [D 20:00:56.501492] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.501500] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.501528] DBPF I/O ops in progress: 3 [D 20:00:56.501536] lio_listio called with the following aiocbs: [D 20:00:56.501542] aiocb_ptr_array[0]: fd: 13, off: 203423744, bytes: 2097152, buf: 0x2aaaaeecf010, type: 1 [D 20:00:56.501558] issue_or_delay_io_operation: lio_listio posted 0x6ebf50 (handle 2863311531, ret 0) [D 20:00:56.504896] ========================================================= [D 20:00:56.505094] PINT_process_request [D 20:00:56.505107] tiling 8 copies [D 20:00:56.505116] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.505125] lvl 0 el 0 blk 0 by 205520896 [D 20:00:56.505132] to 205520896 ta 201326592 fi 234881024 [D 20:00:56.505139] basic type or contiguous data [D 20:00:56.505146] Request disp [D 20:00:56.505152] contig_offset = 205520896 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.505159] calling distribute [D 20:00:56.505165] PINT_distribute [D 20:00:56.505173] of 205520896 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 205520896 exfl 1 [D 20:00:56.505182] begin iteration loff: 205520896 [D 20:00:56.505189] segment exceeds byte limit [D 20:00:56.505195] file being extended [D 20:00:56.505201] process a segment [D 20:00:56.505207] of 205520896 sz 2097152 [D 20:00:56.505214] add a segment 0 [D 20:00:56.505221] end iteration [D 20:00:56.505227] done with segments or bytes [D 20:00:56.505234] of 207618048 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.505241] next loff: 207618048 next poff: 207618048 [D 20:00:56.505248] return value 2097152 [D 20:00:56.505256] finished [D 20:00:56.505262] retval = 2097152 [D 20:00:56.505268] exiting distribute returned less than expected [D 20:00:56.505274] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 207618048 fo 234881024 eof 0 [D 20:00:56.505281] ========================================================= [D 20:00:56.505288] BMI_post_recv: addr: 6843072, offset: 0x2aaab0add010, size: 2097152, tag: 14 [D 20:00:56.505331] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:00:56.505344] get_data_sync_mode returning 0 [D 20:00:56.505358] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.505365] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf0d0010, mem_size: 2097152 [D 20:00:56.505372] dbpf_bstream_rw_list: stream_offset: 236978176, stream_size: 2097152 [D 20:00:56.505380] dbpf_open_cache_get: called [D 20:00:56.505387] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.505394] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.505423] DBPF I/O ops in progress: 4 [D 20:00:56.505430] lio_listio called with the following aiocbs: [D 20:00:56.505436] aiocb_ptr_array[0]: fd: 13, off: 236978176, bytes: 2097152, buf: 0x2aaaaf0d0010, type: 1 [D 20:00:56.505451] issue_or_delay_io_operation: lio_listio posted 0x6ec710 (handle 2863311531, ret 0) [D 20:00:56.508094] ========================================================= [D 20:00:56.508160] PINT_process_request [D 20:00:56.508209] tiling 9 copies [D 20:00:56.508219] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.508227] lvl 0 el 0 blk 0 by 239075328 [D 20:00:56.508233] to 239075328 ta 234881024 fi 268435456 [D 20:00:56.508242] basic type or contiguous data [D 20:00:56.508249] Request disp [D 20:00:56.508255] contig_offset = 239075328 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.508263] calling distribute [D 20:00:56.508269] PINT_distribute [D 20:00:56.508276] of 239075328 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 239075328 exfl 1 [D 20:00:56.508284] begin iteration loff: 239075328 [D 20:00:56.508293] segment exceeds byte limit [D 20:00:56.508299] file being extended [D 20:00:56.508305] process a segment [D 20:00:56.508311] of 239075328 sz 2097152 [D 20:00:56.508318] add a segment 0 [D 20:00:56.508325] end iteration [D 20:00:56.508331] done with segments or bytes [D 20:00:56.508337] of 241172480 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.508344] next loff: 241172480 next poff: 241172480 [D 20:00:56.508351] return value 2097152 [D 20:00:56.508359] finished [D 20:00:56.508365] retval = 2097152 [D 20:00:56.508371] exiting distribute returned less than expected [D 20:00:56.508377] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 241172480 fo 268435456 eof 0 [D 20:00:56.508410] ========================================================= [D 20:00:56.508419] BMI_post_recv: addr: 6843072, offset: 0x2aaab0cde010, size: 2097152, tag: 15 [D 20:00:56.508438] openib_check_cq: completed recv wr_id 684da0. [D 20:00:56.508453] openib_post_rr: 10.1.4.56:39990 bh 97 wr_id 684da0. [D 20:00:56.508465] post_sr: 10.1.4.56:39990 bh 54 len 48 wr_id 6832c0. [D 20:00:56.508476] openib_check_cq: completed recv wr_id 684dd0. [D 20:00:56.508485] openib_post_rr: 10.1.4.56:39990 bh 98 wr_id 684dd0. [D 20:00:56.508492] openib_check_cq: completed recv wr_id 684e00. [D 20:00:56.508499] openib_post_rr: 10.1.4.56:39990 bh 99 wr_id 684e00. [D 20:00:56.508507] post_sr: 10.1.4.56:39990 bh 55 len 48 wr_id 6832f0. [D 20:00:56.508514] openib_check_cq: completed recv wr_id 683b70. [D 20:00:56.508522] openib_post_rr: 10.1.4.56:39990 bh 0 wr_id 683b70. [D 20:00:56.508530] post_sr: 10.1.4.56:39990 bh 56 len 48 wr_id 683320. [D 20:00:56.508537] openib_check_cq: completed recv wr_id 683ba0. [D 20:00:56.508547] openib_post_rr: 10.1.4.56:39990 bh 1 wr_id 683ba0. [D 20:00:56.508554] openib_check_cq: completed recv wr_id 683bd0. [D 20:00:56.508562] openib_post_rr: 10.1.4.56:39990 bh 2 wr_id 683bd0. [D 20:00:56.508570] openib_check_cq: completed recv wr_id 683c00. [D 20:00:56.508578] openib_post_rr: 10.1.4.56:39990 bh 3 wr_id 683c00. [D 20:00:56.508586] openib_check_cq: completed recv wr_id 683c30. [D 20:00:56.508594] openib_post_rr: 10.1.4.56:39990 bh 4 wr_id 683c30. [D 20:00:56.508601] openib_check_cq: completed recv wr_id 683c60. [D 20:00:56.508608] openib_post_rr: 10.1.4.56:39990 bh 5 wr_id 683c60. [D 20:00:56.508615] openib_check_cq: completed recv wr_id 683c90. [D 20:00:56.508623] openib_post_rr: 10.1.4.56:39990 bh 6 wr_id 683c90. [D 20:00:56.508630] openib_check_cq: completed recv wr_id 683cc0. [D 20:00:56.508638] openib_post_rr: 10.1.4.56:39990 bh 7 wr_id 683cc0. [D 20:00:56.508645] openib_check_cq: completed recv wr_id 683cf0. [D 20:00:56.508652] openib_post_rr: 10.1.4.56:39990 bh 8 wr_id 683cf0. [D 20:00:56.508660] openib_check_cq: completed recv wr_id 683d20. [D 20:00:56.508667] openib_post_rr: 10.1.4.56:39990 bh 9 wr_id 683d20. [D 20:00:56.508675] openib_check_cq: completed recv wr_id 683d50. [D 20:00:56.508682] openib_post_rr: 10.1.4.56:39990 bh 10 wr_id 683d50. [D 20:00:56.508689] openib_check_cq: completed recv wr_id 683d80. [D 20:00:56.508697] openib_post_rr: 10.1.4.56:39990 bh 11 wr_id 683d80. [D 20:00:56.508704] openib_check_cq: completed send wr_id 6832c0. [D 20:00:56.508711] openib_check_cq: completed send wr_id 6832f0. [D 20:00:56.508719] openib_check_cq: completed send wr_id 683320. [D 20:00:56.508734] post_sr: 10.1.4.56:39990 bh 57 len 48 wr_id 683350. [D 20:00:56.508745] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:00:56.508754] get_data_sync_mode returning 0 [D 20:00:56.508766] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.508774] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf2d1010, mem_size: 2097152 [D 20:00:56.508781] dbpf_bstream_rw_list: stream_offset: 270532608, stream_size: 2097152 [D 20:00:56.508788] dbpf_open_cache_get: called [D 20:00:56.508795] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.508802] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.508834] DBPF I/O ops in progress: 5 [D 20:00:56.508843] lio_listio called with the following aiocbs: [D 20:00:56.508849] aiocb_ptr_array[0]: fd: 13, off: 270532608, bytes: 2097152, buf: 0x2aaaaf2d1010, type: 1 [D 20:00:56.508861] issue_or_delay_io_operation: lio_listio posted 0x6ed050 (handle 2863311531, ret 0) [D 20:00:56.511471] ========================================================= [D 20:00:56.511558] PINT_process_request [D 20:00:56.511568] tiling 10 copies [D 20:00:56.511576] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.511584] lvl 0 el 0 blk 0 by 272629760 [D 20:00:56.511591] to 272629760 ta 268435456 fi 301989888 [D 20:00:56.511598] basic type or contiguous data [D 20:00:56.511605] Request disp [D 20:00:56.511611] contig_offset = 272629760 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.511618] calling distribute [D 20:00:56.511624] PINT_distribute [D 20:00:56.511631] of 272629760 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 272629760 exfl 1 [D 20:00:56.511640] begin iteration loff: 272629760 [D 20:00:56.511648] segment exceeds byte limit [D 20:00:56.511654] file being extended [D 20:00:56.511660] process a segment [D 20:00:56.511666] of 272629760 sz 2097152 [D 20:00:56.511673] add a segment 0 [D 20:00:56.511679] end iteration [D 20:00:56.511685] done with segments or bytes [D 20:00:56.511691] of 274726912 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.511697] next loff: 274726912 next poff: 274726912 [D 20:00:56.511704] return value 2097152 [D 20:00:56.511711] finished [D 20:00:56.511717] retval = 2097152 [D 20:00:56.511723] exiting distribute returned less than expected [D 20:00:56.511729] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 274726912 fo 301989888 eof 0 [D 20:00:56.511736] ========================================================= [D 20:00:56.511744] BMI_post_recv: addr: 6843072, offset: 0x2aaab0edf010, size: 2097152, tag: 16 [D 20:00:56.511762] openib_check_cq: completed send wr_id 683350. [D 20:00:56.511773] openib_check_cq: completed recv wr_id 683db0. [D 20:00:56.511789] openib_post_rr: 10.1.4.56:39990 bh 12 wr_id 683db0. [D 20:00:56.511799] openib_check_cq: completed recv wr_id 683de0. [D 20:00:56.511808] openib_post_rr: 10.1.4.56:39990 bh 13 wr_id 683de0. [D 20:00:56.511815] openib_check_cq: completed recv wr_id 683e10. [D 20:00:56.511822] openib_post_rr: 10.1.4.56:39990 bh 14 wr_id 683e10. [D 20:00:56.511843] post_sr: 10.1.4.56:39990 bh 58 len 48 wr_id 683380. [D 20:00:56.511856] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:00:56.511867] get_data_sync_mode returning 0 [D 20:00:56.511881] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.511888] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf4d2010, mem_size: 2097152 [D 20:00:56.511895] dbpf_bstream_rw_list: stream_offset: 304087040, stream_size: 2097152 [D 20:00:56.511902] dbpf_open_cache_get: called [D 20:00:56.511909] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.511915] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.511951] DBPF I/O ops in progress: 6 [D 20:00:56.511959] lio_listio called with the following aiocbs: [D 20:00:56.511965] aiocb_ptr_array[0]: fd: 13, off: 304087040, bytes: 2097152, buf: 0x2aaaaf4d2010, type: 1 [D 20:00:56.511980] issue_or_delay_io_operation: lio_listio posted 0x6ed440 (handle 2863311531, ret 0) [D 20:00:56.512938] --- aio_progress_notification called with handle 2863311531 (0x6ec440) [D 20:00:56.512991] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.513006] dbpf_open_cache_put: cached entry. [D 20:00:56.513013] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.513021] op_queue add: 0x6ec440 [D 20:00:56.513040] --- aio_progress_notification called with handle 2863311531 (0x6ec240) [D 20:00:56.513054] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.513067] dbpf_open_cache_put: cached entry. [D 20:00:56.513073] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.513086] --- aio_progress_notification called with handle 2863311531 (0x6ebf50) [D 20:00:56.513098] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.513111] dbpf_open_cache_put: cached entry. [D 20:00:56.513119] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.513131] --- aio_progress_notification called with handle 2863311531 (0x6ec710) [D 20:00:56.513141] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.513153] dbpf_open_cache_put: cached entry. [D 20:00:56.513160] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.515216] ========================================================= [D 20:00:56.515285] PINT_process_request [D 20:00:56.515306] tiling 11 copies [D 20:00:56.515317] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.515325] lvl 0 el 0 blk 0 by 306184192 [D 20:00:56.515332] to 306184192 ta 301989888 fi 335544320 [D 20:00:56.515341] basic type or contiguous data [D 20:00:56.515348] Request disp [D 20:00:56.515355] contig_offset = 306184192 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.515362] calling distribute [D 20:00:56.515369] PINT_distribute [D 20:00:56.515376] of 306184192 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 306184192 exfl 1 [D 20:00:56.515385] begin iteration loff: 306184192 [D 20:00:56.515392] segment exceeds byte limit [D 20:00:56.515398] file being extended [D 20:00:56.515405] process a segment [D 20:00:56.515411] of 306184192 sz 2097152 [D 20:00:56.515418] add a segment 0 [D 20:00:56.515425] end iteration [D 20:00:56.515431] done with segments or bytes [D 20:00:56.515437] of 308281344 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.515445] next loff: 308281344 next poff: 308281344 [D 20:00:56.515452] return value 2097152 [D 20:00:56.515461] finished [D 20:00:56.515467] retval = 2097152 [D 20:00:56.515473] exiting distribute returned less than expected [D 20:00:56.515480] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 308281344 fo 335544320 eof 0 [D 20:00:56.515488] ========================================================= [D 20:00:56.515496] BMI_post_recv: addr: 6843072, offset: 0x2aaab10e0010, size: 2097152, tag: 17 [D 20:00:56.515524] openib_check_cq: completed send wr_id 683380. [D 20:00:56.515536] openib_check_cq: completed recv wr_id 683e40. [D 20:00:56.515555] openib_post_rr: 10.1.4.56:39990 bh 15 wr_id 683e40. [D 20:00:56.515581] post_sr: 10.1.4.56:39990 bh 59 len 48 wr_id 6833b0. [D 20:00:56.515605] openib_check_cq: completed send wr_id 6833b0. [D 20:00:56.515626] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:00:56.515637] get_data_sync_mode returning 0 [D 20:00:56.515654] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.515662] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf6d3010, mem_size: 2097152 [D 20:00:56.515669] dbpf_bstream_rw_list: stream_offset: 337641472, stream_size: 2097152 [D 20:00:56.515678] dbpf_open_cache_get: called [D 20:00:56.515685] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.515692] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.515733] DBPF I/O ops in progress: 7 [D 20:00:56.515741] lio_listio called with the following aiocbs: [D 20:00:56.515748] aiocb_ptr_array[0]: fd: 13, off: 337641472, bytes: 2097152, buf: 0x2aaaaf6d3010, type: 1 [D 20:00:56.515763] issue_or_delay_io_operation: lio_listio posted 0x6ed730 (handle 2863311531, ret 0) [D 20:00:56.517465] DBPF I/O ops in progress: 6 [D 20:00:56.519287] ========================================================= [D 20:00:56.519304] PINT_process_request [D 20:00:56.519314] tiling 12 copies [D 20:00:56.519322] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.519330] lvl 0 el 0 blk 0 by 339738624 [D 20:00:56.519337] to 339738624 ta 335544320 fi 369098752 [D 20:00:56.519345] basic type or contiguous data [D 20:00:56.519351] Request disp [D 20:00:56.519359] contig_offset = 339738624 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.519366] calling distribute [D 20:00:56.519372] PINT_distribute [D 20:00:56.519380] of 339738624 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 339738624 exfl 1 [D 20:00:56.519389] begin iteration loff: 339738624 [D 20:00:56.519397] segment exceeds byte limit [D 20:00:56.519404] file being extended [D 20:00:56.519411] process a segment [D 20:00:56.519436] of 339738624 sz 2097152 [D 20:00:56.519443] add a segment 0 [D 20:00:56.519449] end iteration [D 20:00:56.519455] done with segments or bytes [D 20:00:56.519462] of 341835776 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.519469] next loff: 341835776 next poff: 341835776 [D 20:00:56.519477] return value 2097152 [D 20:00:56.519485] finished [D 20:00:56.519492] retval = 2097152 [D 20:00:56.519498] exiting distribute returned less than expected [D 20:00:56.519512] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 341835776 fo 369098752 eof 0 [D 20:00:56.519520] ========================================================= [D 20:00:56.519529] BMI_post_recv: addr: 6843072, offset: 0x2aaab12e1010, size: 2097152, tag: 18 [D 20:00:56.519569] post_sr: 10.1.4.56:39990 bh 60 len 48 wr_id 6833e0. [D 20:00:56.519586] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:00:56.519598] get_data_sync_mode returning 0 [D 20:00:56.519614] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.519622] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf8d4010, mem_size: 2097152 [D 20:00:56.519629] dbpf_bstream_rw_list: stream_offset: 371195904, stream_size: 2097152 [D 20:00:56.519638] dbpf_open_cache_get: called [D 20:00:56.519647] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.519654] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.519880] DBPF I/O ops in progress: 7 [D 20:00:56.519894] lio_listio called with the following aiocbs: [D 20:00:56.519902] aiocb_ptr_array[0]: fd: 13, off: 371195904, bytes: 2097152, buf: 0x2aaaaf8d4010, type: 1 [D 20:00:56.519925] issue_or_delay_io_operation: lio_listio posted 0x6ed880 (handle 2863311531, ret 0) [D 20:00:56.523240] ========================================================= [D 20:00:56.523341] PINT_process_request [D 20:00:56.523351] tiling 13 copies [D 20:00:56.523361] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.523369] lvl 0 el 0 blk 0 by 373293056 [D 20:00:56.523375] to 373293056 ta 369098752 fi 402653184 [D 20:00:56.523383] basic type or contiguous data [D 20:00:56.523389] Request disp [D 20:00:56.523396] contig_offset = 373293056 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.523403] calling distribute [D 20:00:56.523409] PINT_distribute [D 20:00:56.523416] of 373293056 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 373293056 exfl 1 [D 20:00:56.523425] begin iteration loff: 373293056 [D 20:00:56.523433] segment exceeds byte limit [D 20:00:56.523439] file being extended [D 20:00:56.523447] process a segment [D 20:00:56.523453] of 373293056 sz 2097152 [D 20:00:56.523460] add a segment 0 [D 20:00:56.523468] end iteration [D 20:00:56.523474] done with segments or bytes [D 20:00:56.523481] of 375390208 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.523488] next loff: 375390208 next poff: 375390208 [D 20:00:56.523495] return value 2097152 [D 20:00:56.523510] finished [D 20:00:56.523517] retval = 2097152 [D 20:00:56.523524] exiting distribute returned less than expected [D 20:00:56.523530] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 375390208 fo 402653184 eof 0 [D 20:00:56.523537] ========================================================= [D 20:00:56.523545] BMI_post_recv: addr: 6843072, offset: 0x2aaab1600010, size: 2097152, tag: 19 [D 20:00:56.523568] openib_check_cq: completed send wr_id 6833e0. [D 20:00:56.523603] post_sr: 10.1.4.56:39990 bh 61 len 48 wr_id 683410. [D 20:00:56.523619] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:00:56.523632] get_data_sync_mode returning 0 [D 20:00:56.523649] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.523657] dbpf_bstream_rw_list: mem_offset: 0x2aaaafad5010, mem_size: 2097152 [D 20:00:56.523664] dbpf_bstream_rw_list: stream_offset: 404750336, stream_size: 2097152 [D 20:00:56.523672] dbpf_open_cache_get: called [D 20:00:56.523680] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.523688] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.523733] DBPF I/O ops in progress: 8 [D 20:00:56.523741] lio_listio called with the following aiocbs: [D 20:00:56.523748] aiocb_ptr_array[0]: fd: 13, off: 404750336, bytes: 2097152, buf: 0x2aaaafad5010, type: 1 [D 20:00:56.523764] issue_or_delay_io_operation: lio_listio posted 0x6ed9d0 (handle 2863311531, ret 0) [D 20:00:56.516407] op_queue add: 0x6ec710 [D 20:00:56.525873] ========================================================= [D 20:00:56.525894] PINT_process_request [D 20:00:56.525964] tiling 14 copies [D 20:00:56.525977] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.526013] lvl 0 el 0 blk 0 by 406847488 [D 20:00:56.526022] to 406847488 ta 402653184 fi 436207616 [D 20:00:56.526066] basic type or contiguous data [D 20:00:56.526114] Request disp [D 20:00:56.526125] contig_offset = 406847488 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.526191] calling distribute [D 20:00:56.526202] PINT_distribute [D 20:00:56.526237] of 406847488 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 406847488 exfl 1 [D 20:00:56.526277] begin iteration loff: 406847488 [D 20:00:56.526320] segment exceeds byte limit [D 20:00:56.526358] file being extended [D 20:00:56.526394] process a segment [D 20:00:56.526401] of 406847488 sz 2097152 [D 20:00:56.526442] add a segment 0 [D 20:00:56.526479] end iteration [D 20:00:56.526488] done with segments or bytes [D 20:00:56.526522] of 408944640 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.526582] next loff: 408944640 next poff: 408944640 [D 20:00:56.526591] return value 2097152 [D 20:00:56.526642] finished [D 20:00:56.526683] retval = 2097152 [D 20:00:56.526718] exiting distribute returned less than expected [D 20:00:56.526754] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 408944640 fo 436207616 eof 0 [D 20:00:56.526797] ========================================================= [D 20:00:56.526832] BMI_post_recv: addr: 6843072, offset: 0x2aaab1801010, size: 2097152, tag: 20 [D 20:00:56.526850] openib_check_cq: completed send wr_id 683410. [D 20:00:56.526888] openib_check_cq: completed recv wr_id 683e70. [D 20:00:56.526933] openib_post_rr: 10.1.4.56:39990 bh 16 wr_id 683e70. [D 20:00:56.526984] post_sr: 10.1.4.56:39990 bh 62 len 48 wr_id 683440. [D 20:00:56.527003] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:00:56.527048] get_data_sync_mode returning 0 [D 20:00:56.527090] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.527125] dbpf_bstream_rw_list: mem_offset: 0x2aaaafcd6010, mem_size: 2097152 [D 20:00:56.527187] dbpf_bstream_rw_list: stream_offset: 438304768, stream_size: 2097152 [D 20:00:56.527223] dbpf_open_cache_get: called [D 20:00:56.527234] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.527269] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.525802] DBPF I/O ops in progress: 7 [D 20:00:56.527359] DBPF I/O ops in progress: 8 [D 20:00:56.527394] lio_listio called with the following aiocbs: [D 20:00:56.527435] aiocb_ptr_array[0]: fd: 13, off: 438304768, bytes: 2097152, buf: 0x2aaaafcd6010, type: 1 [D 20:00:56.527480] issue_or_delay_io_operation: lio_listio posted 0x6edb20 (handle 2863311531, ret 0) [D 20:00:56.530661] ========================================================= [D 20:00:56.530750] PINT_process_request [D 20:00:56.530761] tiling 15 copies [D 20:00:56.530770] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.530778] lvl 0 el 0 blk 0 by 440401920 [D 20:00:56.530785] to 440401920 ta 436207616 fi 469762048 [D 20:00:56.530793] basic type or contiguous data [D 20:00:56.530799] Request disp [D 20:00:56.530806] contig_offset = 440401920 contig_size = 62914560 lvl_flag = 1 [D 20:00:56.530813] calling distribute [D 20:00:56.530820] PINT_distribute [D 20:00:56.530829] of 440401920 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 1 [D 20:00:56.530839] begin iteration loff: 440401920 [D 20:00:56.530847] segment exceeds byte limit [D 20:00:56.530854] file being extended [D 20:00:56.530861] process a segment [D 20:00:56.530867] of 440401920 sz 2097152 [D 20:00:56.530875] add a segment 0 [D 20:00:56.530882] end iteration [D 20:00:56.530889] done with segments or bytes [D 20:00:56.530895] of 442499072 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.530902] next loff: 442499072 next poff: 442499072 [D 20:00:56.530909] return value 2097152 [D 20:00:56.530917] finished [D 20:00:56.530923] retval = 2097152 [D 20:00:56.530929] exiting distribute returned less than expected [D 20:00:56.530935] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 442499072 fo 469762048 eof 0 [D 20:00:56.530943] ========================================================= [D 20:00:56.530951] BMI_post_recv: addr: 6843072, offset: 0x2aaab1a02010, size: 2097152, tag: 21 [D 20:00:56.530974] openib_check_cq: completed send wr_id 683440. [D 20:00:56.530986] openib_check_cq: completed recv wr_id 683ea0. [D 20:00:56.531002] openib_post_rr: 10.1.4.56:39990 bh 17 wr_id 683ea0. [D 20:00:56.531011] openib_check_cq: completed recv wr_id 683ed0. [D 20:00:56.531019] openib_post_rr: 10.1.4.56:39990 bh 18 wr_id 683ed0. [D 20:00:56.531026] openib_check_cq: completed recv wr_id 683f00. [D 20:00:56.531034] openib_post_rr: 10.1.4.56:39990 bh 19 wr_id 683f00. [D 20:00:56.531062] post_sr: 10.1.4.56:39990 bh 63 len 48 wr_id 683470. [D 20:00:56.531076] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:00:56.531089] get_data_sync_mode returning 0 [D 20:00:56.531106] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.531114] dbpf_bstream_rw_list: mem_offset: 0x2aaaafed7010, mem_size: 2097152 [D 20:00:56.531121] dbpf_bstream_rw_list: stream_offset: 4194304, stream_size: 2097152 [D 20:00:56.531130] dbpf_open_cache_get: called [D 20:00:56.531138] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.531146] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.531188] DBPF I/O ops in progress: 9 [D 20:00:56.531198] lio_listio called with the following aiocbs: [D 20:00:56.531204] aiocb_ptr_array[0]: fd: 13, off: 4194304, bytes: 2097152, buf: 0x2aaaafed7010, type: 1 [D 20:00:56.531218] issue_or_delay_io_operation: lio_listio posted 0x6edc70 (handle 2863311531, ret 0) [D 20:00:56.534610] ========================================================= [D 20:00:56.534733] PINT_process_request [D 20:00:56.534742] tiling 2 copies [D 20:00:56.534750] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.534758] lvl 0 el 0 blk 0 by 6291456 [D 20:00:56.534765] to 6291456 ta 0 fi 33554432 [D 20:00:56.534774] basic type or contiguous data [D 20:00:56.534781] Request disp [D 20:00:56.534788] contig_offset = 6291456 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.534796] calling distribute [D 20:00:56.534803] PINT_distribute [D 20:00:56.534811] of 6291456 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 6291456 exfl 1 [D 20:00:56.534819] begin iteration loff: 6291456 [D 20:00:56.534827] segment exceeds byte limit [D 20:00:56.534833] file being extended [D 20:00:56.534839] process a segment [D 20:00:56.534845] of 6291456 sz 2097152 [D 20:00:56.534852] add a segment 0 [D 20:00:56.534858] end iteration [D 20:00:56.534864] done with segments or bytes [D 20:00:56.534870] of 8388608 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.534878] next loff: 8388608 next poff: 8388608 [D 20:00:56.534884] return value 2097152 [D 20:00:56.534892] finished [D 20:00:56.534898] retval = 2097152 [D 20:00:56.534904] exiting distribute returned less than expected [D 20:00:56.534910] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 8388608 fo 33554432 eof 0 [D 20:00:56.534918] ========================================================= [D 20:00:56.534925] BMI_post_recv: addr: 6843072, offset: 0x2aaab1c03010, size: 2097152, tag: 8 [D 20:00:56.534945] openib_check_cq: completed send wr_id 683470. [D 20:00:56.534958] openib_check_cq: completed recv wr_id 683f30. [D 20:00:56.534976] openib_post_rr: 10.1.4.56:39990 bh 20 wr_id 683f30. [D 20:00:56.534985] openib_check_cq: completed recv wr_id 683f60. [D 20:00:56.534996] openib_post_rr: 10.1.4.56:39990 bh 21 wr_id 683f60. [D 20:00:56.535004] openib_check_cq: completed recv wr_id 683f90. [D 20:00:56.535012] openib_post_rr: 10.1.4.56:39990 bh 22 wr_id 683f90. [D 20:00:56.535019] openib_check_cq: completed recv wr_id 683fc0. [D 20:00:56.535027] openib_post_rr: 10.1.4.56:39990 bh 23 wr_id 683fc0. [D 20:00:56.535035] openib_check_cq: completed recv wr_id 683ff0. [D 20:00:56.535043] openib_post_rr: 10.1.4.56:39990 bh 24 wr_id 683ff0. [D 20:00:56.535050] openib_check_cq: completed recv wr_id 684020. [D 20:00:56.535058] openib_post_rr: 10.1.4.56:39990 bh 25 wr_id 684020. [D 20:00:56.535065] openib_check_cq: completed recv wr_id 684050. [D 20:00:56.535073] openib_post_rr: 10.1.4.56:39990 bh 26 wr_id 684050. [D 20:00:56.535094] post_sr: 10.1.4.56:39990 bh 64 len 48 wr_id 6834a0. [D 20:00:56.535124] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:00:56.535137] get_data_sync_mode returning 0 [D 20:00:56.535152] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.535160] dbpf_bstream_rw_list: mem_offset: 0x2aaab00d8010, mem_size: 2097152 [D 20:00:56.535166] dbpf_bstream_rw_list: stream_offset: 37748736, stream_size: 2097152 [D 20:00:56.535173] dbpf_open_cache_get: called [D 20:00:56.535180] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.535187] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.535228] DBPF I/O ops in progress: 10 [D 20:00:56.535237] lio_listio called with the following aiocbs: [D 20:00:56.535244] aiocb_ptr_array[0]: fd: 13, off: 37748736, bytes: 2097152, buf: 0x2aaab00d8010, type: 1 [D 20:00:56.535260] issue_or_delay_io_operation: lio_listio posted 0x6ee0b0 (handle 2863311531, ret 0) [D 20:00:56.536047] op_queue add: 0x6ec240 [D 20:00:56.536122] --- aio_progress_notification called with handle 2863311531 (0x6ed730) [D 20:00:56.536136] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.536150] dbpf_open_cache_put: cached entry. [D 20:00:56.536158] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.536164] op_queue add: 0x6ed730 [D 20:00:56.536173] DBPF I/O ops in progress: 9 [D 20:00:56.536198] --- aio_progress_notification called with handle 2863311531 (0x6ed880) [D 20:00:56.536209] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.536223] dbpf_open_cache_put: cached entry. [D 20:00:56.536229] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.536236] op_queue add: 0x6ed880 [D 20:00:56.536245] DBPF I/O ops in progress: 8 [D 20:00:56.537866] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:56.537981] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:00:56.537989] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:00:56.537998] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:00:56.538006] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:00:56.538026] DBPF I/O ops in progress: 7 [D 20:00:56.538256] ========================================================= [D 20:00:56.538274] PINT_process_request [D 20:00:56.538339] tiling 3 copies [D 20:00:56.538347] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.538355] lvl 0 el 0 blk 0 by 39845888 [D 20:00:56.538362] to 39845888 ta 33554432 fi 67108864 [D 20:00:56.538370] basic type or contiguous data [D 20:00:56.538377] Request disp [D 20:00:56.538384] contig_offset = 39845888 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.538391] calling distribute [D 20:00:56.538398] PINT_distribute [D 20:00:56.538418] of 39845888 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 39845888 exfl 1 [D 20:00:56.538427] begin iteration loff: 39845888 [D 20:00:56.538434] segment exceeds byte limit [D 20:00:56.538440] file being extended [D 20:00:56.538447] process a segment [D 20:00:56.538453] of 39845888 sz 2097152 [D 20:00:56.538459] add a segment 0 [D 20:00:56.538465] end iteration [D 20:00:56.538472] done with segments or bytes [D 20:00:56.538478] of 41943040 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.538485] next loff: 41943040 next poff: 41943040 [D 20:00:56.538492] return value 2097152 [D 20:00:56.538500] finished [D 20:00:56.538506] retval = 2097152 [D 20:00:56.538512] exiting distribute returned less than expected [D 20:00:56.538518] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 41943040 fo 67108864 eof 0 [D 20:00:56.538526] ========================================================= [D 20:00:56.538535] BMI_post_recv: addr: 6843072, offset: 0x2aaab1e04010, size: 2097152, tag: 9 [D 20:00:56.538554] openib_check_cq: completed send wr_id 6834a0. [D 20:00:56.538566] openib_check_cq: completed recv wr_id 684080. [D 20:00:56.538579] openib_post_rr: 10.1.4.56:39990 bh 27 wr_id 684080. [D 20:00:56.538590] openib_check_cq: completed recv wr_id 6840b0. [D 20:00:56.538601] openib_post_rr: 10.1.4.56:39990 bh 28 wr_id 6840b0. [D 20:00:56.538609] openib_check_cq: completed recv wr_id 6840e0. [D 20:00:56.538617] openib_post_rr: 10.1.4.56:39990 bh 29 wr_id 6840e0. [D 20:00:56.538624] openib_check_cq: completed recv wr_id 684110. [D 20:00:56.538633] openib_post_rr: 10.1.4.56:39990 bh 30 wr_id 684110. [D 20:00:56.538641] openib_check_cq: completed recv wr_id 684140. [D 20:00:56.538651] openib_post_rr: 10.1.4.56:39990 bh 31 wr_id 684140. [D 20:00:56.538673] post_sr: 10.1.4.56:39990 bh 65 len 48 wr_id 6834d0. [D 20:00:56.538686] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:00:56.538696] get_data_sync_mode returning 0 [D 20:00:56.538707] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.538715] dbpf_bstream_rw_list: mem_offset: 0x2aaab02d9010, mem_size: 2097152 [D 20:00:56.538722] dbpf_bstream_rw_list: stream_offset: 71303168, stream_size: 2097152 [D 20:00:56.538730] dbpf_open_cache_get: called [D 20:00:56.538738] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.538745] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.538774] DBPF I/O ops in progress: 8 [D 20:00:56.538781] lio_listio called with the following aiocbs: [D 20:00:56.538788] aiocb_ptr_array[0]: fd: 13, off: 71303168, bytes: 2097152, buf: 0x2aaab02d9010, type: 1 [D 20:00:56.538802] issue_or_delay_io_operation: lio_listio posted 0x6ec440 (handle 2863311531, ret 0) [D 20:00:56.536095] --- aio_progress_notification called with handle 2863311531 (0x6ed050) [D 20:00:56.539077] op_queue add: 0x6ebf50 [D 20:00:56.546645] ========================================================= [D 20:00:56.546688] PINT_process_request [D 20:00:56.546697] tiling 4 copies [D 20:00:56.546705] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.546713] lvl 0 el 0 blk 0 by 73400320 [D 20:00:56.546720] to 73400320 ta 67108864 fi 100663296 [D 20:00:56.546727] basic type or contiguous data [D 20:00:56.546733] Request disp [D 20:00:56.546740] contig_offset = 73400320 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.546747] calling distribute [D 20:00:56.546753] PINT_distribute [D 20:00:56.546760] of 73400320 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 73400320 exfl 1 [D 20:00:56.546768] begin iteration loff: 73400320 [D 20:00:56.546789] segment exceeds byte limit [D 20:00:56.546796] file being extended [D 20:00:56.546802] process a segment [D 20:00:56.546808] of 73400320 sz 2097152 [D 20:00:56.546814] add a segment 0 [D 20:00:56.546821] end iteration [D 20:00:56.546827] done with segments or bytes [D 20:00:56.546833] of 75497472 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.546840] next loff: 75497472 next poff: 75497472 [D 20:00:56.546847] return value 2097152 [D 20:00:56.546854] finished [D 20:00:56.546859] retval = 2097152 [D 20:00:56.546866] exiting distribute returned less than expected [D 20:00:56.546873] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 75497472 fo 100663296 eof 0 [D 20:00:56.546880] ========================================================= [D 20:00:56.546888] BMI_post_recv: addr: 6843072, offset: 0x2aaab2005010, size: 2097152, tag: 10 [D 20:00:56.546905] openib_check_cq: completed send wr_id 6834d0. [D 20:00:56.546917] openib_check_cq: completed recv wr_id 684170. [D 20:00:56.546930] openib_post_rr: 10.1.4.56:39990 bh 32 wr_id 684170. [D 20:00:56.546940] openib_check_cq: completed recv wr_id 6841a0. [D 20:00:56.546953] openib_post_rr: 10.1.4.56:39990 bh 33 wr_id 6841a0. [D 20:00:56.546961] openib_check_cq: completed recv wr_id 6841d0. [D 20:00:56.546970] openib_post_rr: 10.1.4.56:39990 bh 34 wr_id 6841d0. [D 20:00:56.546977] openib_check_cq: completed recv wr_id 684200. [D 20:00:56.546984] openib_post_rr: 10.1.4.56:39990 bh 35 wr_id 684200. [D 20:00:56.546992] openib_check_cq: completed recv wr_id 684230. [D 20:00:56.547000] openib_post_rr: 10.1.4.56:39990 bh 36 wr_id 684230. [D 20:00:56.547007] openib_check_cq: completed recv wr_id 684260. [D 20:00:56.547015] openib_post_rr: 10.1.4.56:39990 bh 37 wr_id 684260. [D 20:00:56.547022] openib_check_cq: completed recv wr_id 684290. [D 20:00:56.547030] openib_post_rr: 10.1.4.56:39990 bh 38 wr_id 684290. [D 20:00:56.547038] openib_check_cq: completed recv wr_id 6842c0. [D 20:00:56.547045] openib_post_rr: 10.1.4.56:39990 bh 39 wr_id 6842c0. [D 20:00:56.547066] post_sr: 10.1.4.56:39990 bh 66 len 48 wr_id 683500. [D 20:00:56.547078] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:00:56.547088] get_data_sync_mode returning 0 [D 20:00:56.547102] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.547110] dbpf_bstream_rw_list: mem_offset: 0x2aaab04da010, mem_size: 2097152 [D 20:00:56.547117] dbpf_bstream_rw_list: stream_offset: 104857600, stream_size: 2097152 [D 20:00:56.547124] dbpf_open_cache_get: called [D 20:00:56.547131] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.547138] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.547164] DBPF I/O ops in progress: 9 [D 20:00:56.547172] lio_listio called with the following aiocbs: [D 20:00:56.547179] aiocb_ptr_array[0]: fd: 13, off: 104857600, bytes: 2097152, buf: 0x2aaab04da010, type: 1 [D 20:00:56.547192] issue_or_delay_io_operation: lio_listio posted 0x6ed730 (handle 2863311531, ret 0) [D 20:00:56.550017] DBPF I/O ops in progress: 8 [D 20:00:56.550256] ========================================================= [D 20:00:56.550287] PINT_process_request [D 20:00:56.550297] tiling 5 copies [D 20:00:56.550304] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.550312] lvl 0 el 0 blk 0 by 106954752 [D 20:00:56.550319] to 106954752 ta 100663296 fi 134217728 [D 20:00:56.550326] basic type or contiguous data [D 20:00:56.550332] Request disp [D 20:00:56.550339] contig_offset = 106954752 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.550346] calling distribute [D 20:00:56.550353] PINT_distribute [D 20:00:56.550359] of 106954752 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 106954752 exfl 1 [D 20:00:56.550369] begin iteration loff: 106954752 [D 20:00:56.550376] segment exceeds byte limit [D 20:00:56.550382] file being extended [D 20:00:56.550388] process a segment [D 20:00:56.550394] of 106954752 sz 2097152 [D 20:00:56.550415] add a segment 0 [D 20:00:56.550421] end iteration [D 20:00:56.550427] done with segments or bytes [D 20:00:56.550433] of 109051904 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.550440] next loff: 109051904 next poff: 109051904 [D 20:00:56.550447] return value 2097152 [D 20:00:56.550454] finished [D 20:00:56.550460] retval = 2097152 [D 20:00:56.550466] exiting distribute returned less than expected [D 20:00:56.550473] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 109051904 fo 134217728 eof 0 [D 20:00:56.550480] ========================================================= [D 20:00:56.550488] BMI_post_recv: addr: 6843072, offset: 0x2aaab2206010, size: 2097152, tag: 11 [D 20:00:56.550504] openib_check_cq: completed send wr_id 683500. [D 20:00:56.550515] openib_check_cq: completed recv wr_id 6842f0. [D 20:00:56.550527] openib_post_rr: 10.1.4.56:39990 bh 40 wr_id 6842f0. [D 20:00:56.550551] post_sr: 10.1.4.56:39990 bh 67 len 48 wr_id 683530. [D 20:00:56.550563] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:00:56.550573] get_data_sync_mode returning 0 [D 20:00:56.550586] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.550594] dbpf_bstream_rw_list: mem_offset: 0x2aaab06db010, mem_size: 2097152 [D 20:00:56.550601] dbpf_bstream_rw_list: stream_offset: 138412032, stream_size: 2097152 [D 20:00:56.550608] dbpf_open_cache_get: called [D 20:00:56.550615] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.550622] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.550835] DBPF I/O ops in progress: 9 [D 20:00:56.550848] lio_listio called with the following aiocbs: [D 20:00:56.550856] aiocb_ptr_array[0]: fd: 13, off: 138412032, bytes: 2097152, buf: 0x2aaab06db010, type: 1 [D 20:00:56.550871] issue_or_delay_io_operation: lio_listio posted 0x6ed880 (handle 2863311531, ret 0) [D 20:00:56.551848] --- aio_progress_notification called with handle 2863311531 (0x6edb20) [D 20:00:56.551884] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.551898] dbpf_open_cache_put: cached entry. [D 20:00:56.551906] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.551914] op_queue add: 0x6edb20 [D 20:00:56.551924] DBPF I/O ops in progress: 8 [D 20:00:56.551951] --- aio_progress_notification called with handle 2863311531 (0x6edc70) [D 20:00:56.551963] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.551975] dbpf_open_cache_put: cached entry. [D 20:00:56.551982] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.551988] op_queue add: 0x6edc70 [D 20:00:56.551996] DBPF I/O ops in progress: 7 [D 20:00:56.540587] --- aio_progress_notification called with handle 2863311531 (0x6ed9d0) [D 20:00:56.550022] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:00:56.552917] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.552932] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:00:56.552933] dbpf_open_cache_put: cached entry. [D 20:00:56.552954] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.553039] ========================================================= [D 20:00:56.553053] PINT_process_request [D 20:00:56.553061] tiling 6 copies [D 20:00:56.553067] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.553075] lvl 0 el 0 blk 0 by 140509184 [D 20:00:56.553081] to 140509184 ta 134217728 fi 167772160 [D 20:00:56.553020] op_queue add: 0x6ed9d0 [D 20:00:56.552946] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:00:56.553094] DBPF I/O ops in progress: 6 [D 20:00:56.553105] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:00:56.553088] basic type or contiguous data [D 20:00:56.553128] Request disp [D 20:00:56.553135] contig_offset = 140509184 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.553143] calling distribute [D 20:00:56.540424] --- aio_progress_notification called with handle 2863311531 (0x6ed440) [D 20:00:56.553252] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.553275] dbpf_open_cache_put: cached entry. [D 20:00:56.553291] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.553305] op_queue add: 0x6ed440 [D 20:00:56.553329] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:00:56.553220] PINT_distribute [D 20:00:56.540752] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.553391] dbpf_open_cache_put: cached entry. [D 20:00:56.553406] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.553420] op_queue add: 0x6ed050 [D 20:00:56.553444] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:00:56.553354] DBPF I/O ops in progress: 5 [D 20:00:56.553456] of 140509184 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 140509184 exfl 1 [D 20:00:56.553508] begin iteration loff: 140509184 [D 20:00:56.553555] segment exceeds byte limit [D 20:00:56.553563] file being extended [D 20:00:56.553614] process a segment [D 20:00:56.553621] of 140509184 sz 2097152 [D 20:00:56.553678] add a segment 0 [D 20:00:56.553714] end iteration [D 20:00:56.553760] done with segments or bytes [D 20:00:56.553775] of 142606336 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.553812] next loff: 142606336 next poff: 142606336 [D 20:00:56.553821] return value 2097152 [D 20:00:56.553860] finished [D 20:00:56.553871] retval = 2097152 [D 20:00:56.553924] exiting distribute returned less than expected [D 20:00:56.553958] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 142606336 fo 167772160 eof 0 [D 20:00:56.553992] ========================================================= [D 20:00:56.554032] BMI_post_recv: addr: 6843072, offset: 0x2aaab2407010, size: 2097152, tag: 12 [D 20:00:56.554078] openib_check_cq: completed send wr_id 683530. [D 20:00:56.554119] openib_check_cq: completed recv wr_id 684320. [D 20:00:56.554179] openib_post_rr: 10.1.4.56:39990 bh 41 wr_id 684320. [D 20:00:56.554215] openib_check_cq: completed recv wr_id 684350. [D 20:00:56.554251] openib_post_rr: 10.1.4.56:39990 bh 42 wr_id 684350. [D 20:00:56.554325] post_sr: 10.1.4.56:39990 bh 68 len 48 wr_id 683560. [D 20:00:56.554340] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:00:56.554379] get_data_sync_mode returning 0 [D 20:00:56.554424] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.554459] dbpf_bstream_rw_list: mem_offset: 0x2aaab08dc010, mem_size: 2097152 [D 20:00:56.554495] dbpf_bstream_rw_list: stream_offset: 171966464, stream_size: 2097152 [D 20:00:56.554536] dbpf_open_cache_get: called [D 20:00:56.554571] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.554605] dbpf_open_cache_get: moving to (or reordering in) used list. [A 20:00:56.555606] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65f550: io: start [D 20:00:56.555624] (0x65f550) io (prelude sm) state: getattr_if_needed [D 20:00:56.555632] About to retrieve attributes for handle 2863311531 [D 20:00:56.555645] op_queue add: 0x2aaab1500d10 [D 20:00:56.555657] [SM Exiting]: (0x65f450) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:56.555665] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:56.555683] dbpf_open_cache_get: called [D 20:00:56.555694] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.555668] [SM Entering]: (0x660710) unexpected_sm:map_request (status: 0) [D 20:00:56.555717] [SM frame get]: (0x660710) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.555725] server_post_unexpected_recv [D 20:00:56.555740] [SM Frame PUSH]: (0x2aaab1500e60) frame: 0x2aaab1501d90 [D 20:00:56.555705] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.555748] [SM Locating]: (0x2aaab1500e60) op-id: 999 [D 20:00:56.555774] server_op_state_get_machine 999 [D 20:00:56.555785] [SM Locating]: (0x2aaab1500e60) located: unexpected_sm:post_unexpected [D 20:00:56.555792] [SM frame get]: (0x2aaab1500e60) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.555799] [SM Entering]: (0x2aaab1500e60) unexpected_sm:post_unexpected (status: 0) [D 20:00:56.555773] dbpf_open_cache_put: cached entry. [D 20:00:56.555809] [SM frame get]: (0x2aaab1500e60) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.555836] [SM Exiting]: (0x2aaab1500e60) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:56.555844] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:56.555848] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 440401920 [D 20:00:56.555865] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 440401920) [D 20:00:56.555851] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:56.555886] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 17 [D 20:00:56.555893] [SM frame get]: (0x660710) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.555900] server_state_machine_start 0x660710 [D 20:00:56.555907] PINT_decode [D 20:00:56.555914] lebf_decode_req [D 20:00:56.555876] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:56.555927] [SM Locating]: (0x660710) op-id: 3 [D 20:00:56.555936] op_queue add: 0x2aaab1500d10 [D 20:00:56.555944] server_op_state_get_machine 3 [D 20:00:56.555972] [SM push_state]: (0x660710) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:56.555979] [SM Locating]: (0x660710) located: prelude_sm:setup [D 20:00:56.555986] [SM Entering]: (0x660710) prelude_sm:setup (status: 0) [D 20:00:56.555993] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.556001] [SM Exiting]: (0x660710) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:56.556008] [SM Exiting]: (0x660710) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:56.556015] [SM push_state]: (0x660710) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.556022] [SM Entering]: (0x660710) prelude_work_sm:req_sched (status: 0) [D 20:00:56.556028] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.556035] (0x660810) io (prelude sm) state: req_sched [D 20:00:56.553525] DBPF I/O ops in progress: 4 [D 20:00:56.556230] DBPF I/O ops in progress: 5 [D 20:00:56.556245] lio_listio called with the following aiocbs: [D 20:00:56.556287] aiocb_ptr_array[0]: fd: 13, off: 171966464, bytes: 2097152, buf: 0x2aaab08dc010, type: 1 [D 20:00:56.556327] issue_or_delay_io_operation: lio_listio posted 0x6ed440 (handle 2863311531, ret 0) [D 20:00:56.558468] ========================================================= [D 20:00:56.558529] PINT_process_request [D 20:00:56.558538] tiling 7 copies [D 20:00:56.558546] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.558554] lvl 0 el 0 blk 0 by 174063616 [D 20:00:56.558561] to 174063616 ta 167772160 fi 201326592 [D 20:00:56.558569] basic type or contiguous data [D 20:00:56.558577] Request disp [D 20:00:56.558583] contig_offset = 174063616 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.558590] calling distribute [D 20:00:56.558596] PINT_distribute [D 20:00:56.558602] of 174063616 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 174063616 exfl 1 [D 20:00:56.558611] begin iteration loff: 174063616 [D 20:00:56.558618] segment exceeds byte limit [D 20:00:56.558624] file being extended [D 20:00:56.558631] process a segment [D 20:00:56.558637] of 174063616 sz 2097152 [D 20:00:56.558643] add a segment 0 [D 20:00:56.558650] end iteration [D 20:00:56.558656] done with segments or bytes [D 20:00:56.558662] of 176160768 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.558669] next loff: 176160768 next poff: 176160768 [D 20:00:56.558676] return value 2097152 [D 20:00:56.558683] finished [D 20:00:56.558690] retval = 2097152 [D 20:00:56.558696] exiting distribute returned less than expected [D 20:00:56.558702] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 176160768 fo 201326592 eof 0 [D 20:00:56.558710] ========================================================= [D 20:00:56.558717] BMI_post_recv: addr: 6843072, offset: 0x2aaab2608010, size: 2097152, tag: 13 [D 20:00:56.558733] openib_check_cq: completed send wr_id 683560. [D 20:00:56.558743] openib_check_cq: completed recv wr_id 684380. [D 20:00:56.558760] openib_post_rr: 10.1.4.56:39990 bh 43 wr_id 684380. [D 20:00:56.558771] openib_check_cq: completed recv wr_id 6843b0. [D 20:00:56.558779] openib_post_rr: 10.1.4.56:39990 bh 44 wr_id 6843b0. [D 20:00:56.558786] openib_check_cq: completed recv wr_id 6843e0. [D 20:00:56.558794] openib_post_rr: 10.1.4.56:39990 bh 45 wr_id 6843e0. [D 20:00:56.558814] post_sr: 10.1.4.56:39990 bh 69 len 48 wr_id 683590. [D 20:00:56.558832] openib_check_cq: completed send wr_id 683590. [D 20:00:56.558849] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:00:56.558859] get_data_sync_mode returning 0 [D 20:00:56.558873] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.558881] dbpf_bstream_rw_list: mem_offset: 0x2aaab0add010, mem_size: 2097152 [D 20:00:56.558888] dbpf_bstream_rw_list: stream_offset: 205520896, stream_size: 2097152 [D 20:00:56.558896] dbpf_open_cache_get: called [D 20:00:56.558903] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.558911] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.558920] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:56.558949] DBPF I/O ops in progress: 6 [D 20:00:56.558957] lio_listio called with the following aiocbs: [D 20:00:56.558963] aiocb_ptr_array[0]: fd: 13, off: 205520896, bytes: 2097152, buf: 0x2aaab0add010, type: 1 [D 20:00:56.558976] issue_or_delay_io_operation: lio_listio posted 0x6ec860 (handle 2863311531, ret 0) [D 20:00:56.562009] ========================================================= [D 20:00:56.562043] PINT_process_request [D 20:00:56.562050] tiling 8 copies [D 20:00:56.562058] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.562066] lvl 0 el 0 blk 0 by 207618048 [D 20:00:56.562073] to 207618048 ta 201326592 fi 234881024 [D 20:00:56.562080] basic type or contiguous data [D 20:00:56.562086] Request disp [D 20:00:56.562092] contig_offset = 207618048 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.562100] calling distribute [D 20:00:56.562106] PINT_distribute [D 20:00:56.562112] of 207618048 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 207618048 exfl 1 [D 20:00:56.562120] begin iteration loff: 207618048 [D 20:00:56.562127] segment exceeds byte limit [D 20:00:56.562133] file being extended [D 20:00:56.562176] process a segment [D 20:00:56.562183] of 207618048 sz 2097152 [D 20:00:56.562190] add a segment 0 [D 20:00:56.562197] end iteration [D 20:00:56.562203] done with segments or bytes [D 20:00:56.562209] of 209715200 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.562216] next loff: 209715200 next poff: 209715200 [D 20:00:56.562223] return value 2097152 [D 20:00:56.562230] finished [D 20:00:56.562236] retval = 2097152 [D 20:00:56.562242] exiting distribute returned less than expected [D 20:00:56.562248] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 209715200 fo 234881024 eof 0 [D 20:00:56.562255] ========================================================= [D 20:00:56.562263] BMI_post_recv: addr: 6843072, offset: 0x2aaab2809010, size: 2097152, tag: 14 [D 20:00:56.562280] openib_check_cq: completed recv wr_id 684410. [D 20:00:56.562295] openib_post_rr: 10.1.4.56:39990 bh 46 wr_id 684410. [D 20:00:56.562320] post_sr: 10.1.4.56:39990 bh 70 len 48 wr_id 6835c0. [D 20:00:56.562333] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:00:56.562343] get_data_sync_mode returning 0 [D 20:00:56.562355] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.562362] dbpf_bstream_rw_list: mem_offset: 0x2aaab0cde010, mem_size: 2097152 [D 20:00:56.562370] dbpf_bstream_rw_list: stream_offset: 239075328, stream_size: 2097152 [D 20:00:56.562377] dbpf_open_cache_get: called [D 20:00:56.562384] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.562390] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.562414] DBPF I/O ops in progress: 7 [D 20:00:56.562421] lio_listio called with the following aiocbs: [D 20:00:56.562428] aiocb_ptr_array[0]: fd: 13, off: 239075328, bytes: 2097152, buf: 0x2aaab0cde010, type: 1 [D 20:00:56.562442] issue_or_delay_io_operation: lio_listio posted 0x6ecfa0 (handle 2863311531, ret 0) [A 20:00:56.564299] troy.scl@10.1.4.56:39990 H=2863311531 S=0x660810: io: request [D 20:00:56.564377] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:56.564404] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab1501060 [D 20:00:56.564412] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab1501060 [D 20:00:56.564421] [SM Exiting]: (0x660710) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:56.564432] [SM Entering]: (0x660710) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:56.564439] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.565739] ========================================================= [D 20:00:56.565771] PINT_process_request [D 20:00:56.565780] tiling 9 copies [D 20:00:56.565788] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.565796] lvl 0 el 0 blk 0 by 241172480 [D 20:00:56.565802] to 241172480 ta 234881024 fi 268435456 [D 20:00:56.565810] basic type or contiguous data [D 20:00:56.565816] Request disp [D 20:00:56.565823] contig_offset = 241172480 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.565830] calling distribute [D 20:00:56.565837] PINT_distribute [D 20:00:56.565843] of 241172480 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 241172480 exfl 1 [D 20:00:56.565853] begin iteration loff: 241172480 [D 20:00:56.565861] segment exceeds byte limit [D 20:00:56.565867] file being extended [D 20:00:56.565874] process a segment [D 20:00:56.565880] of 241172480 sz 2097152 [D 20:00:56.565916] add a segment 0 [D 20:00:56.565924] end iteration [D 20:00:56.565930] done with segments or bytes [D 20:00:56.565937] of 243269632 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.565944] next loff: 243269632 next poff: 243269632 [D 20:00:56.565950] return value 2097152 [D 20:00:56.565957] finished [D 20:00:56.565963] retval = 2097152 [D 20:00:56.565969] exiting distribute returned less than expected [D 20:00:56.565975] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 243269632 fo 268435456 eof 0 [D 20:00:56.565982] ========================================================= [D 20:00:56.565990] BMI_post_recv: addr: 6843072, offset: 0x2aaab2a0a010, size: 2097152, tag: 15 [D 20:00:56.566006] openib_check_cq: completed send wr_id 6835c0. [D 20:00:56.566016] openib_check_cq: completed recv wr_id 684440. [D 20:00:56.566032] openib_post_rr: 10.1.4.56:39990 bh 47 wr_id 684440. [D 20:00:56.566042] openib_check_cq: completed recv wr_id 684470. [D 20:00:56.566050] openib_post_rr: 10.1.4.56:39990 bh 48 wr_id 684470. [D 20:00:56.566069] post_sr: 10.1.4.56:39990 bh 71 len 48 wr_id 6835f0. [D 20:00:56.566083] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:00:56.566093] get_data_sync_mode returning 0 [D 20:00:56.566106] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.566114] dbpf_bstream_rw_list: mem_offset: 0x2aaab0edf010, mem_size: 2097152 [D 20:00:56.566121] dbpf_bstream_rw_list: stream_offset: 272629760, stream_size: 2097152 [D 20:00:56.566129] dbpf_open_cache_get: called [D 20:00:56.566137] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.566144] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.566165] DBPF I/O ops in progress: 8 [D 20:00:56.566173] lio_listio called with the following aiocbs: [D 20:00:56.566179] aiocb_ptr_array[0]: fd: 13, off: 272629760, bytes: 2097152, buf: 0x2aaab0edf010, type: 1 [D 20:00:56.566193] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [D 20:00:56.569286] ========================================================= [D 20:00:56.569320] PINT_process_request [D 20:00:56.569329] tiling 10 copies [D 20:00:56.569337] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.569345] lvl 0 el 0 blk 0 by 274726912 [D 20:00:56.569352] to 274726912 ta 268435456 fi 301989888 [D 20:00:56.569360] basic type or contiguous data [D 20:00:56.569366] Request disp [D 20:00:56.569373] contig_offset = 274726912 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.569380] calling distribute [D 20:00:56.569393] PINT_distribute [D 20:00:56.569401] of 274726912 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 274726912 exfl 1 [D 20:00:56.569410] begin iteration loff: 274726912 [D 20:00:56.569417] segment exceeds byte limit [D 20:00:56.569423] file being extended [D 20:00:56.569429] process a segment [D 20:00:56.569435] of 274726912 sz 2097152 [D 20:00:56.569442] add a segment 0 [D 20:00:56.569448] end iteration [D 20:00:56.569454] done with segments or bytes [D 20:00:56.569460] of 276824064 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.569468] next loff: 276824064 next poff: 276824064 [D 20:00:56.569474] return value 2097152 [D 20:00:56.569481] finished [D 20:00:56.569487] retval = 2097152 [D 20:00:56.569494] exiting distribute returned less than expected [D 20:00:56.569500] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 276824064 fo 301989888 eof 0 [D 20:00:56.569507] ========================================================= [D 20:00:56.569514] BMI_post_recv: addr: 6843072, offset: 0x2aaab2c0b010, size: 2097152, tag: 16 [D 20:00:56.569530] openib_check_cq: completed send wr_id 6835f0. [D 20:00:56.569540] openib_check_cq: completed recv wr_id 6844a0. [D 20:00:56.569558] openib_post_rr: 10.1.4.56:39990 bh 49 wr_id 6844a0. [D 20:00:56.569567] openib_check_cq: completed recv wr_id 6844d0. [D 20:00:56.569575] openib_post_rr: 10.1.4.56:39990 bh 50 wr_id 6844d0. [D 20:00:56.569595] post_sr: 10.1.4.56:39990 bh 72 len 48 wr_id 683620. [D 20:00:56.569607] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:00:56.569618] get_data_sync_mode returning 0 [D 20:00:56.569630] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.569637] dbpf_bstream_rw_list: mem_offset: 0x2aaab10e0010, mem_size: 2097152 [D 20:00:56.569644] dbpf_bstream_rw_list: stream_offset: 306184192, stream_size: 2097152 [D 20:00:56.569651] dbpf_open_cache_get: called [D 20:00:56.569658] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.569665] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.569685] DBPF I/O ops in progress: 9 [D 20:00:56.569693] lio_listio called with the following aiocbs: [D 20:00:56.569699] aiocb_ptr_array[0]: fd: 13, off: 306184192, bytes: 2097152, buf: 0x2aaab10e0010, type: 1 [D 20:00:56.569711] issue_or_delay_io_operation: lio_listio posted 0x6ee200 (handle 2863311531, ret 0) [D 20:00:56.571671] --- aio_progress_notification called with handle 2863311531 (0x6ec440) [D 20:00:56.571744] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.571760] dbpf_open_cache_put: cached entry. [D 20:00:56.571767] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.571774] op_queue add: 0x6ec440 [D 20:00:56.571788] --- aio_progress_notification called with handle 2863311531 (0x6ed730) [D 20:00:56.571799] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.571810] dbpf_open_cache_put: cached entry. [D 20:00:56.571816] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.571832] --- aio_progress_notification called with handle 2863311531 (0x6ed440) [D 20:00:56.571843] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.571856] dbpf_open_cache_put: cached entry. [D 20:00:56.571862] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [A 20:00:56.572353] troy.scl@10.1.4.56:39990 H=2863311531 S=0x660810: io: start [D 20:00:56.572377] (0x660810) io (prelude sm) state: getattr_if_needed [D 20:00:56.572409] About to retrieve attributes for handle 2863311531 [D 20:00:56.572432] op_queue add: 0x2aaab1501330 [D 20:00:56.572454] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:56.572469] dbpf_open_cache_get: called [D 20:00:56.572476] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.572483] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.572492] dbpf_open_cache_put: cached entry. [D 20:00:56.572554] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 440401920 [D 20:00:56.572567] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 440401920) [D 20:00:56.572575] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:56.572586] [SM Exiting]: (0x660710) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:56.572599] [SM Entering]: (0x6619d0) unexpected_sm:map_request (status: 0) [D 20:00:56.572607] [SM frame get]: (0x6619d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.572616] server_post_unexpected_recv [D 20:00:56.572626] [SM Frame PUSH]: (0x2aaab15016e0) frame: 0x2aaab1502df0 [D 20:00:56.572633] [SM Locating]: (0x2aaab15016e0) op-id: 999 [D 20:00:56.572640] server_op_state_get_machine 999 [D 20:00:56.572647] [SM Locating]: (0x2aaab15016e0) located: unexpected_sm:post_unexpected [D 20:00:56.572654] [SM frame get]: (0x2aaab15016e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.572662] [SM Entering]: (0x2aaab15016e0) unexpected_sm:post_unexpected (status: 0) [D 20:00:56.572670] [SM frame get]: (0x2aaab15016e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.572681] [SM Exiting]: (0x2aaab15016e0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:56.572689] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:56.572696] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:56.572702] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 18 [D 20:00:56.572710] [SM frame get]: (0x6619d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.572717] server_state_machine_start 0x6619d0 [D 20:00:56.572723] PINT_decode [D 20:00:56.572733] lebf_decode_req [D 20:00:56.572746] [SM Locating]: (0x6619d0) op-id: 3 [D 20:00:56.572753] server_op_state_get_machine 3 [D 20:00:56.572759] [SM push_state]: (0x6619d0) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:56.572766] [SM Locating]: (0x6619d0) located: prelude_sm:setup [D 20:00:56.572774] [SM Entering]: (0x6619d0) prelude_sm:setup (status: 0) [D 20:00:56.572781] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.572789] [SM Exiting]: (0x6619d0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:56.572797] [SM Exiting]: (0x6619d0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:56.572804] [SM push_state]: (0x6619d0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.572811] [SM Entering]: (0x6619d0) prelude_work_sm:req_sched (status: 0) [D 20:00:56.572818] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.572825] (0x661ad0) io (prelude sm) state: req_sched [D 20:00:56.573135] ========================================================= [D 20:00:56.573149] PINT_process_request [D 20:00:56.573157] tiling 11 copies [D 20:00:56.573165] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.573173] lvl 0 el 0 blk 0 by 308281344 [D 20:00:56.573180] to 308281344 ta 301989888 fi 335544320 [D 20:00:56.573187] basic type or contiguous data [D 20:00:56.573194] Request disp [D 20:00:56.573200] contig_offset = 308281344 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.573207] calling distribute [D 20:00:56.573214] PINT_distribute [D 20:00:56.573220] of 308281344 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 308281344 exfl 1 [D 20:00:56.573229] begin iteration loff: 308281344 [D 20:00:56.573236] segment exceeds byte limit [D 20:00:56.573243] file being extended [D 20:00:56.573250] process a segment [D 20:00:56.573255] of 308281344 sz 2097152 [D 20:00:56.573272] add a segment 0 [D 20:00:56.573279] end iteration [D 20:00:56.573285] done with segments or bytes [D 20:00:56.573291] of 310378496 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.573298] next loff: 310378496 next poff: 310378496 [D 20:00:56.573305] return value 2097152 [D 20:00:56.573311] finished [D 20:00:56.573317] retval = 2097152 [D 20:00:56.573323] exiting distribute returned less than expected [D 20:00:56.573329] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 310378496 fo 335544320 eof 0 [D 20:00:56.573337] ========================================================= [D 20:00:56.573344] BMI_post_recv: addr: 6843072, offset: 0x2aaab2e0c010, size: 2097152, tag: 17 [D 20:00:56.573360] openib_check_cq: completed send wr_id 683620. [D 20:00:56.573370] openib_check_cq: completed recv wr_id 684500. [D 20:00:56.573384] openib_post_rr: 10.1.4.56:39990 bh 51 wr_id 684500. [D 20:00:56.573395] openib_check_cq: completed recv wr_id 684530. [D 20:00:56.573403] openib_post_rr: 10.1.4.56:39990 bh 52 wr_id 684530. [D 20:00:56.573426] post_sr: 10.1.4.56:39990 bh 73 len 48 wr_id 683650. [D 20:00:56.573439] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:00:56.573448] get_data_sync_mode returning 0 [D 20:00:56.573459] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.573466] dbpf_bstream_rw_list: mem_offset: 0x2aaab12e1010, mem_size: 2097152 [D 20:00:56.573473] dbpf_bstream_rw_list: stream_offset: 339738624, stream_size: 2097152 [D 20:00:56.573481] dbpf_open_cache_get: called [D 20:00:56.573488] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.573494] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.573503] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:56.573524] DBPF I/O ops in progress: 10 [D 20:00:56.573531] lio_listio called with the following aiocbs: [D 20:00:56.573537] aiocb_ptr_array[0]: fd: 13, off: 339738624, bytes: 2097152, buf: 0x2aaab12e1010, type: 1 [D 20:00:56.573550] issue_or_delay_io_operation: lio_listio posted 0x6ee400 (handle 2863311531, ret 0) [D 20:00:56.578364] op_queue add: 0x2aaab1501330 [D 20:00:56.578393] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:00:56.578479] --- aio_progress_notification called with handle 2863311531 (0x6ec860) [D 20:00:56.579844] ========================================================= [D 20:00:56.579859] PINT_process_request [D 20:00:56.579866] tiling 12 copies [D 20:00:56.579874] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.579883] lvl 0 el 0 blk 0 by 341835776 [D 20:00:56.579889] to 341835776 ta 335544320 fi 369098752 [D 20:00:56.579897] basic type or contiguous data [D 20:00:56.579905] Request disp [D 20:00:56.579911] contig_offset = 341835776 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.579919] calling distribute [D 20:00:56.579926] PINT_distribute [D 20:00:56.579934] of 341835776 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 341835776 exfl 1 [D 20:00:56.579942] begin iteration loff: 341835776 [D 20:00:56.579949] segment exceeds byte limit [D 20:00:56.579955] file being extended [D 20:00:56.579961] process a segment [D 20:00:56.579967] of 341835776 sz 2097152 [D 20:00:56.579974] add a segment 0 [D 20:00:56.579981] end iteration [D 20:00:56.579987] done with segments or bytes [D 20:00:56.579994] of 343932928 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.580001] next loff: 343932928 next poff: 343932928 [D 20:00:56.580007] return value 2097152 [D 20:00:56.580015] finished [D 20:00:56.580021] retval = 2097152 [D 20:00:56.580027] exiting distribute returned less than expected [D 20:00:56.580034] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 343932928 fo 369098752 eof 0 [D 20:00:56.580041] ========================================================= [D 20:00:56.580050] BMI_post_recv: addr: 6843072, offset: 0x2aaab300e010, size: 2097152, tag: 18 [D 20:00:56.580068] openib_check_cq: completed send wr_id 683650. [D 20:00:56.580078] openib_check_cq: completed recv wr_id 684560. [D 20:00:56.580096] openib_post_rr: 10.1.4.56:39990 bh 53 wr_id 684560. [D 20:00:56.580108] openib_check_cq: completed recv wr_id 684590. [D 20:00:56.580116] openib_post_rr: 10.1.4.56:39990 bh 54 wr_id 684590. [D 20:00:56.580123] openib_check_cq: completed recv wr_id 6845c0. [D 20:00:56.580133] openib_post_rr: 10.1.4.56:39990 bh 55 wr_id 6845c0. [D 20:00:56.580151] post_sr: 10.1.4.56:39990 bh 74 len 48 wr_id 683680. [D 20:00:56.580170] openib_check_cq: completed send wr_id 683680. [D 20:00:56.580186] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:00:56.580196] get_data_sync_mode returning 0 [D 20:00:56.580208] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.580215] dbpf_bstream_rw_list: mem_offset: 0x2aaab1600010, mem_size: 2097152 [D 20:00:56.580222] dbpf_bstream_rw_list: stream_offset: 373293056, stream_size: 2097152 [D 20:00:56.580230] dbpf_open_cache_get: called [D 20:00:56.580236] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.580244] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.579824] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.580292] dbpf_open_cache_put: cached entry. [D 20:00:56.580302] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.580309] op_queue add: 0x6ec860 [D 20:00:56.580329] op_queue add: 0x6ed440 [D 20:00:56.580350] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:00:56.580360] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:00:56.580378] op_queue add: 0x6ed730 [D 20:00:56.580412] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:56.578364] DBPF I/O ops in progress: 9 [D 20:00:56.580440] DBPF I/O ops in progress: 10 [D 20:00:56.580448] lio_listio called with the following aiocbs: [D 20:00:56.580455] aiocb_ptr_array[0]: fd: 13, off: 373293056, bytes: 2097152, buf: 0x2aaab1600010, type: 1 [D 20:00:56.580470] issue_or_delay_io_operation: lio_listio posted 0x6ecdf0 (handle 2863311531, ret 0) [D 20:00:56.553532] --- aio_progress_notification called with handle 2863311531 (0x6ee0b0) [D 20:00:56.580495] DBPF I/O ops in progress: 9 [D 20:00:56.584427] ========================================================= [D 20:00:56.584440] PINT_process_request [D 20:00:56.584448] tiling 13 copies [D 20:00:56.584456] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.584464] lvl 0 el 0 blk 0 by 375390208 [D 20:00:56.584471] to 375390208 ta 369098752 fi 402653184 [D 20:00:56.584479] basic type or contiguous data [D 20:00:56.584486] Request disp [D 20:00:56.584493] contig_offset = 375390208 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.584500] calling distribute [D 20:00:56.584506] PINT_distribute [D 20:00:56.584512] of 375390208 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 375390208 exfl 1 [D 20:00:56.584520] begin iteration loff: 375390208 [D 20:00:56.584529] segment exceeds byte limit [D 20:00:56.584534] file being extended [D 20:00:56.584541] process a segment [D 20:00:56.584548] of 375390208 sz 2097152 [D 20:00:56.584555] add a segment 0 [D 20:00:56.584562] end iteration [D 20:00:56.584568] done with segments or bytes [D 20:00:56.584574] of 377487360 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.584581] next loff: 377487360 next poff: 377487360 [D 20:00:56.584588] return value 2097152 [D 20:00:56.584594] finished [D 20:00:56.584600] retval = 2097152 [D 20:00:56.584606] exiting distribute returned less than expected [D 20:00:56.584613] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 377487360 fo 402653184 eof 0 [D 20:00:56.584620] ========================================================= [D 20:00:56.584629] BMI_post_recv: addr: 6843072, offset: 0x2aaab320f010, size: 2097152, tag: 19 [D 20:00:56.584647] openib_check_cq: completed recv wr_id 6845f0. [D 20:00:56.584662] openib_post_rr: 10.1.4.56:39990 bh 56 wr_id 6845f0. [D 20:00:56.584672] openib_check_cq: completed recv wr_id 684620. [D 20:00:56.584685] openib_post_rr: 10.1.4.56:39990 bh 57 wr_id 684620. [D 20:00:56.584704] post_sr: 10.1.4.56:39990 bh 75 len 48 wr_id 6836b0. [D 20:00:56.584718] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:00:56.584730] get_data_sync_mode returning 0 [D 20:00:56.584743] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.584750] dbpf_bstream_rw_list: mem_offset: 0x2aaab1801010, mem_size: 2097152 [D 20:00:56.584758] dbpf_bstream_rw_list: stream_offset: 406847488, stream_size: 2097152 [D 20:00:56.584765] dbpf_open_cache_get: called [D 20:00:56.584772] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.584779] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.584405] DBPF I/O ops in progress: 8 [D 20:00:56.584836] DBPF I/O ops in progress: 7 [D 20:00:56.584851] DBPF I/O ops in progress: 8 [D 20:00:56.584859] lio_listio called with the following aiocbs: [D 20:00:56.584866] aiocb_ptr_array[0]: fd: 13, off: 406847488, bytes: 2097152, buf: 0x2aaab1801010, type: 1 [D 20:00:56.584880] issue_or_delay_io_operation: lio_listio posted 0x6ec860 (handle 2863311531, ret 0) [A 20:00:56.585766] troy.scl@10.1.4.56:39990 H=2863311531 S=0x661ad0: io: request [D 20:00:56.585802] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:56.585810] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab1501c60 [D 20:00:56.585817] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab1501c60 [D 20:00:56.585825] [SM Exiting]: (0x6619d0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:56.585836] [SM Entering]: (0x6619d0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:56.585844] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.587968] ========================================================= [D 20:00:56.588015] PINT_process_request [D 20:00:56.588023] tiling 14 copies [D 20:00:56.588032] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.588040] lvl 0 el 0 blk 0 by 408944640 [D 20:00:56.588046] to 408944640 ta 402653184 fi 436207616 [D 20:00:56.588054] basic type or contiguous data [D 20:00:56.588062] Request disp [D 20:00:56.588068] contig_offset = 408944640 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.588075] calling distribute [D 20:00:56.588081] PINT_distribute [D 20:00:56.588088] of 408944640 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 408944640 exfl 1 [D 20:00:56.588097] begin iteration loff: 408944640 [D 20:00:56.588105] segment exceeds byte limit [D 20:00:56.588111] file being extended [D 20:00:56.588117] process a segment [D 20:00:56.588123] of 408944640 sz 2097152 [D 20:00:56.588130] add a segment 0 [D 20:00:56.588136] end iteration [D 20:00:56.588142] done with segments or bytes [D 20:00:56.588148] of 411041792 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.588155] next loff: 411041792 next poff: 411041792 [D 20:00:56.588161] return value 2097152 [D 20:00:56.588168] finished [D 20:00:56.588174] retval = 2097152 [D 20:00:56.588180] exiting distribute returned less than expected [D 20:00:56.588186] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 411041792 fo 436207616 eof 0 [D 20:00:56.588194] ========================================================= [D 20:00:56.588202] BMI_post_recv: addr: 6843072, offset: 0x2aaab3410010, size: 2097152, tag: 20 [D 20:00:56.588220] openib_check_cq: completed send wr_id 6836b0. [D 20:00:56.588231] openib_check_cq: completed recv wr_id 684650. [D 20:00:56.588243] openib_post_rr: 10.1.4.56:39990 bh 58 wr_id 684650. [D 20:00:56.588267] post_sr: 10.1.4.56:39990 bh 76 len 48 wr_id 6836e0. [D 20:00:56.588280] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:00:56.588294] get_data_sync_mode returning 0 [D 20:00:56.588307] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.588315] dbpf_bstream_rw_list: mem_offset: 0x2aaab1a02010, mem_size: 2097152 [D 20:00:56.588322] dbpf_bstream_rw_list: stream_offset: 440401920, stream_size: 2097152 [D 20:00:56.588329] dbpf_open_cache_get: called [D 20:00:56.588336] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.588343] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.588365] DBPF I/O ops in progress: 9 [D 20:00:56.588372] lio_listio called with the following aiocbs: [D 20:00:56.588379] aiocb_ptr_array[0]: fd: 13, off: 440401920, bytes: 2097152, buf: 0x2aaab1a02010, type: 1 [D 20:00:56.583095] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.583049] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:00:56.588423] dbpf_open_cache_put: cached entry. [D 20:00:56.588400] issue_or_delay_io_operation: lio_listio posted 0x6ed440 (handle 2863311531, ret 0) [D 20:00:56.588425] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.591208] dbpf_open_cache_put: cached entry. [D 20:00:56.591232] ========================================================= [D 20:00:56.591246] PINT_process_request [D 20:00:56.591558] tiling 15 copies [D 20:00:56.591569] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.591577] lvl 0 el 0 blk 0 by 442499072 [D 20:00:56.591583] to 442499072 ta 436207616 fi 469762048 [D 20:00:56.591592] basic type or contiguous data [D 20:00:56.591599] Request disp [D 20:00:56.591605] contig_offset = 442499072 contig_size = 60817408 lvl_flag = 1 [D 20:00:56.591613] calling distribute [D 20:00:56.591619] PINT_distribute [D 20:00:56.591631] of 442499072 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 1 [D 20:00:56.591641] begin iteration loff: 442499072 [D 20:00:56.591649] segment exceeds byte limit [D 20:00:56.591655] file being extended [D 20:00:56.591661] process a segment [D 20:00:56.591667] of 442499072 sz 2097152 [D 20:00:56.591674] add a segment 0 [D 20:00:56.591681] end iteration [D 20:00:56.591687] done with segments or bytes [D 20:00:56.591693] of 444596224 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.591700] next loff: 444596224 next poff: 444596224 [D 20:00:56.591707] return value 2097152 [D 20:00:56.591713] finished [D 20:00:56.591719] retval = 2097152 [D 20:00:56.591725] exiting distribute returned less than expected [D 20:00:56.591732] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 444596224 fo 469762048 eof 0 [D 20:00:56.591739] ========================================================= [D 20:00:56.591747] BMI_post_recv: addr: 6843072, offset: 0x2aaab3611010, size: 2097152, tag: 21 [D 20:00:56.591764] openib_check_cq: completed send wr_id 6836e0. [D 20:00:56.591774] openib_check_cq: completed recv wr_id 684680. [D 20:00:56.591790] openib_post_rr: 10.1.4.56:39990 bh 59 wr_id 684680. [D 20:00:56.591800] openib_check_cq: completed recv wr_id 6846b0. [D 20:00:56.591809] openib_post_rr: 10.1.4.56:39990 bh 60 wr_id 6846b0. [D 20:00:56.591829] post_sr: 10.1.4.56:39990 bh 77 len 48 wr_id 683710. [D 20:00:56.591842] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:00:56.591852] get_data_sync_mode returning 0 [D 20:00:56.591866] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.591873] dbpf_bstream_rw_list: mem_offset: 0x2aaab1c03010, mem_size: 2097152 [D 20:00:56.591880] dbpf_bstream_rw_list: stream_offset: 6291456, stream_size: 2097152 [D 20:00:56.591888] dbpf_open_cache_get: called [D 20:00:56.591902] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.591910] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.591933] DBPF I/O ops in progress: 10 [D 20:00:56.591941] lio_listio called with the following aiocbs: [D 20:00:56.591948] aiocb_ptr_array[0]: fd: 13, off: 6291456, bytes: 2097152, buf: 0x2aaab1c03010, type: 1 [D 20:00:56.591961] issue_or_delay_io_operation: lio_listio posted 0x6f0f60 (handle 2863311531, ret 0) [A 20:00:56.594157] troy.scl@10.1.4.56:39990 H=2863311531 S=0x661ad0: io: start [D 20:00:56.594223] (0x661ad0) io (prelude sm) state: getattr_if_needed [D 20:00:56.594230] About to retrieve attributes for handle 2863311531 [D 20:00:56.594253] op_queue add: 0x2aaab1501a60 [D 20:00:56.594872] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:56.594891] dbpf_open_cache_get: called [D 20:00:56.594900] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.594907] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.594916] dbpf_open_cache_put: cached entry. [D 20:00:56.594971] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 440401920 [D 20:00:56.594984] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 440401920) [D 20:00:56.594992] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:56.594999] op_queue add: 0x2aaab1501a60 [D 20:00:56.595029] [SM Exiting]: (0x6619d0) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:56.595043] [SM Entering]: (0x662c90) unexpected_sm:map_request (status: 0) [D 20:00:56.595051] [SM frame get]: (0x662c90) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.595059] server_post_unexpected_recv [D 20:00:56.595070] [SM Frame PUSH]: (0x2aaab1501a60) frame: 0x2aaab1503e50 [D 20:00:56.595077] [SM Locating]: (0x2aaab1501a60) op-id: 999 [D 20:00:56.595084] server_op_state_get_machine 999 [D 20:00:56.595091] [SM Locating]: (0x2aaab1501a60) located: unexpected_sm:post_unexpected [D 20:00:56.595098] [SM frame get]: (0x2aaab1501a60) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.595105] [SM Entering]: (0x2aaab1501a60) unexpected_sm:post_unexpected (status: 0) [D 20:00:56.595113] [SM frame get]: (0x2aaab1501a60) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.595123] [SM Exiting]: (0x2aaab1501a60) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:56.595131] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:56.595139] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:56.595145] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 19 [D 20:00:56.595153] [SM frame get]: (0x662c90) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.595160] server_state_machine_start 0x662c90 [D 20:00:56.595167] PINT_decode [D 20:00:56.595175] lebf_decode_req [D 20:00:56.595188] [SM Locating]: (0x662c90) op-id: 3 [D 20:00:56.595195] server_op_state_get_machine 3 [D 20:00:56.595203] [SM push_state]: (0x662c90) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:56.595210] [SM Locating]: (0x662c90) located: prelude_sm:setup [D 20:00:56.595218] [SM Entering]: (0x662c90) prelude_sm:setup (status: 0) [D 20:00:56.595225] [SM frame get]: (0x662c90) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.595232] [SM Exiting]: (0x662c90) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:56.595240] [SM Exiting]: (0x662c90) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:56.595247] [SM push_state]: (0x662c90) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.595255] [SM Entering]: (0x662c90) prelude_work_sm:req_sched (status: 0) [D 20:00:56.595262] [SM frame get]: (0x662c90) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.595269] (0x662d90) io (prelude sm) state: req_sched [D 20:00:56.595523] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.595550] op_queue add: 0x6eda80 [D 20:00:56.595594] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:00:56.595613] DBPF I/O ops in progress: 9 [D 20:00:56.595570] ========================================================= [D 20:00:56.595675] PINT_process_request [D 20:00:56.595683] tiling 2 copies [D 20:00:56.595691] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.595699] lvl 0 el 0 blk 0 by 8388608 [D 20:00:56.595706] to 8388608 ta 0 fi 33554432 [D 20:00:56.595714] basic type or contiguous data [D 20:00:56.595720] Request disp [D 20:00:56.595727] contig_offset = 8388608 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.595734] calling distribute [D 20:00:56.595740] PINT_distribute [D 20:00:56.595747] of 8388608 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 8388608 exfl 1 [D 20:00:56.595755] begin iteration loff: 8388608 [D 20:00:56.595762] segment exceeds byte limit [D 20:00:56.595768] file being extended [D 20:00:56.595775] process a segment [D 20:00:56.595781] of 8388608 sz 2097152 [D 20:00:56.595787] add a segment 0 [D 20:00:56.595793] end iteration [D 20:00:56.595799] done with segments or bytes [D 20:00:56.595805] of 10485760 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.595812] next loff: 10485760 next poff: 10485760 [D 20:00:56.595818] return value 2097152 [D 20:00:56.595825] finished [D 20:00:56.595831] retval = 2097152 [D 20:00:56.595837] exiting distribute returned less than expected [D 20:00:56.595843] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 10485760 fo 33554432 eof 0 [D 20:00:56.595850] ========================================================= [D 20:00:56.595857] BMI_post_recv: addr: 6843072, offset: 0x2aaab3812010, size: 2097152, tag: 8 [D 20:00:56.595874] openib_check_cq: completed send wr_id 683710. [D 20:00:56.595884] openib_check_cq: completed recv wr_id 6846e0. [D 20:00:56.595900] openib_post_rr: 10.1.4.56:39990 bh 61 wr_id 6846e0. [D 20:00:56.595910] openib_check_cq: completed recv wr_id 684710. [D 20:00:56.595919] openib_post_rr: 10.1.4.56:39990 bh 62 wr_id 684710. [D 20:00:56.595939] post_sr: 10.1.4.56:39990 bh 78 len 48 wr_id 683740. [D 20:00:56.595951] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:00:56.595960] get_data_sync_mode returning 0 [D 20:00:56.595970] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.595978] dbpf_bstream_rw_list: mem_offset: 0x2aaab1e04010, mem_size: 2097152 [D 20:00:56.595984] dbpf_bstream_rw_list: stream_offset: 39845888, stream_size: 2097152 [D 20:00:56.595991] dbpf_open_cache_get: called [D 20:00:56.595998] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.596004] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.596012] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:56.596031] DBPF I/O ops in progress: 10 [D 20:00:56.596039] lio_listio called with the following aiocbs: [D 20:00:56.596045] aiocb_ptr_array[0]: fd: 13, off: 39845888, bytes: 2097152, buf: 0x2aaab1e04010, type: 1 [D 20:00:56.596058] issue_or_delay_io_operation: lio_listio posted 0x6f11f0 (handle 2863311531, ret 0) [D 20:00:56.599139] ========================================================= [D 20:00:56.599164] PINT_process_request [D 20:00:56.599173] tiling 3 copies [D 20:00:56.599180] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.599188] lvl 0 el 0 blk 0 by 41943040 [D 20:00:56.599195] to 41943040 ta 33554432 fi 67108864 [D 20:00:56.599202] basic type or contiguous data [D 20:00:56.599208] Request disp [D 20:00:56.599215] contig_offset = 41943040 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.599223] calling distribute [D 20:00:56.599230] PINT_distribute [D 20:00:56.599237] of 41943040 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 41943040 exfl 1 [D 20:00:56.599253] begin iteration loff: 41943040 [D 20:00:56.599261] segment exceeds byte limit [D 20:00:56.599267] file being extended [D 20:00:56.599274] process a segment [D 20:00:56.599280] of 41943040 sz 2097152 [D 20:00:56.599287] add a segment 0 [D 20:00:56.599293] end iteration [D 20:00:56.599299] done with segments or bytes [D 20:00:56.599305] of 44040192 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.599312] next loff: 44040192 next poff: 44040192 [D 20:00:56.599319] return value 2097152 [D 20:00:56.599326] finished [D 20:00:56.599333] retval = 2097152 [D 20:00:56.599339] exiting distribute returned less than expected [D 20:00:56.599346] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 44040192 fo 67108864 eof 0 [D 20:00:56.599353] ========================================================= [D 20:00:56.599360] BMI_post_recv: addr: 6843072, offset: 0x2aaab3a13010, size: 2097152, tag: 9 [D 20:00:56.599378] openib_check_cq: completed send wr_id 683740. [D 20:00:56.599387] openib_check_cq: completed recv wr_id 684740. [D 20:00:56.599404] openib_post_rr: 10.1.4.56:39990 bh 63 wr_id 684740. [D 20:00:56.599414] openib_check_cq: completed recv wr_id 684770. [D 20:00:56.599423] openib_post_rr: 10.1.4.56:39990 bh 64 wr_id 684770. [D 20:00:56.599446] post_sr: 10.1.4.56:39990 bh 79 len 48 wr_id 683770. [D 20:00:56.599465] openib_check_cq: completed send wr_id 683770. [D 20:00:56.599481] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:00:56.599491] get_data_sync_mode returning 0 [D 20:00:56.599504] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.599519] dbpf_bstream_rw_list: mem_offset: 0x2aaab2005010, mem_size: 2097152 [D 20:00:56.599527] dbpf_bstream_rw_list: stream_offset: 73400320, stream_size: 2097152 [D 20:00:56.599534] dbpf_open_cache_get: called [D 20:00:56.599541] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.599548] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.599571] DBPF I/O ops in progress: 11 [D 20:00:56.599578] lio_listio called with the following aiocbs: [D 20:00:56.599584] aiocb_ptr_array[0]: fd: 13, off: 73400320, bytes: 2097152, buf: 0x2aaab2005010, type: 1 [D 20:00:56.599598] issue_or_delay_io_operation: lio_listio posted 0x6eddc0 (handle 2863311531, ret 0) [D 20:00:56.601979] --- aio_progress_notification called with handle 2863311531 (0x6ee200) [D 20:00:56.602062] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.602081] dbpf_open_cache_put: cached entry. [D 20:00:56.602089] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.602096] op_queue add: 0x6ee200 [D 20:00:56.602122] --- aio_progress_notification called with handle 2863311531 (0x6ee400) [D 20:00:56.602136] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.602148] dbpf_open_cache_put: cached entry. [D 20:00:56.602155] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.575332] --- aio_progress_notification called with handle 2863311531 (0x6ed880) [D 20:00:56.602180] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.583113] --- aio_progress_notification called with handle 2863311531 (0x6ecfa0) [D 20:00:56.602194] --- aio_progress_notification called with handle 2863311531 (0x6ecdf0) [D 20:00:56.602197] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.602213] --- aio_progress_notification called with handle 2863311531 (0x6ec860) [D 20:00:56.602225] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.602228] dbpf_open_cache_put: cached entry. [D 20:00:56.602208] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.602249] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.602284] op_queue add: 0x6ecfa0 [D 20:00:56.602253] DBPF I/O ops in progress: 10 [D 20:00:56.602315] DBPF I/O ops in progress: 9 [D 20:00:56.602344] ========================================================= [D 20:00:56.602356] PINT_process_request [D 20:00:56.602366] tiling 4 copies [D 20:00:56.602374] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.602382] lvl 0 el 0 blk 0 by 75497472 [D 20:00:56.602388] to 75497472 ta 67108864 fi 100663296 [D 20:00:56.602395] basic type or contiguous data [D 20:00:56.602402] Request disp [D 20:00:56.602409] contig_offset = 75497472 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.602418] calling distribute [D 20:00:56.602424] PINT_distribute [D 20:00:56.602431] of 75497472 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 75497472 exfl 1 [D 20:00:56.602440] begin iteration loff: 75497472 [D 20:00:56.602448] segment exceeds byte limit [D 20:00:56.602454] file being extended [D 20:00:56.602461] process a segment [D 20:00:56.602467] of 75497472 sz 2097152 [D 20:00:56.602473] add a segment 0 [D 20:00:56.602480] end iteration [D 20:00:56.602485] done with segments or bytes [D 20:00:56.602491] of 77594624 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.602498] next loff: 77594624 next poff: 77594624 [D 20:00:56.602504] return value 2097152 [D 20:00:56.602510] finished [D 20:00:56.602517] retval = 2097152 [D 20:00:56.602523] exiting distribute returned less than expected [D 20:00:56.602529] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 77594624 fo 100663296 eof 0 [D 20:00:56.602536] ========================================================= [D 20:00:56.602543] BMI_post_recv: addr: 6843072, offset: 0x2aaab3c14010, size: 2097152, tag: 10 [D 20:00:56.602559] openib_check_cq: completed recv wr_id 6847a0. [D 20:00:56.602574] openib_post_rr: 10.1.4.56:39990 bh 65 wr_id 6847a0. [D 20:00:56.602584] openib_check_cq: completed recv wr_id 6847d0. [D 20:00:56.602592] openib_post_rr: 10.1.4.56:39990 bh 66 wr_id 6847d0. [D 20:00:56.602613] post_sr: 10.1.4.56:39990 bh 80 len 48 wr_id 6837a0. [D 20:00:56.602626] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:00:56.602635] get_data_sync_mode returning 0 [D 20:00:56.602647] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.602654] dbpf_bstream_rw_list: mem_offset: 0x2aaab2206010, mem_size: 2097152 [D 20:00:56.602661] dbpf_bstream_rw_list: stream_offset: 106954752, stream_size: 2097152 [D 20:00:56.602669] dbpf_open_cache_get: called [D 20:00:56.588435] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.602726] op_queue add: 0x6ee0b0 [D 20:00:56.602735] DBPF I/O ops in progress: 8 [D 20:00:56.602751] op_queue add: 0x6ee400 [D 20:00:56.602771] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:00:56.602786] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:00:56.602794] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:00:56.602803] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:00:56.602818] DBPF I/O ops in progress: 7 [D 20:00:56.602261] dbpf_open_cache_put: cached entry. [D 20:00:56.604452] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.604464] op_queue add: 0x6ecdf0 [D 20:00:56.604493] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:00:56.604516] DBPF I/O ops in progress: 6 [D 20:00:56.608441] --- aio_progress_notification called with handle 2863311531 (0x6ed440) [D 20:00:56.608493] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.608505] dbpf_open_cache_put: cached entry. [D 20:00:56.608513] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.608520] op_queue add: 0x6ed440 [D 20:00:56.608547] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:00:56.608568] DBPF I/O ops in progress: 5 [D 20:00:56.608603] --- aio_progress_notification called with handle 2863311531 (0x6f0f60) [D 20:00:56.608613] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.608625] dbpf_open_cache_put: cached entry. [D 20:00:56.608632] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.608639] op_queue add: 0x6f0f60 [D 20:00:56.608654] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:00:56.608668] DBPF I/O ops in progress: 4 [D 20:00:56.608686] dbpf_open_cache_put: cached entry. [D 20:00:56.608699] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.608708] op_queue add: 0x6ed880 [D 20:00:56.608724] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:56.608737] DBPF I/O ops in progress: 3 [D 20:00:56.608780] dbpf_open_cache_put: cached entry. [D 20:00:56.608796] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.608807] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.608831] DBPF I/O ops in progress: 4 [D 20:00:56.608839] lio_listio called with the following aiocbs: [D 20:00:56.608845] aiocb_ptr_array[0]: fd: 13, off: 106954752, bytes: 2097152, buf: 0x2aaab2206010, type: 1 [D 20:00:56.608856] issue_or_delay_io_operation: lio_listio posted 0x6f13f0 (handle 2863311531, ret 0) [D 20:00:56.610843] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.610938] op_queue add: 0x6ec860 [D 20:00:56.610966] ========================================================= [D 20:00:56.610975] PINT_process_request [D 20:00:56.610983] tiling 5 copies [D 20:00:56.610990] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.610999] lvl 0 el 0 blk 0 by 109051904 [D 20:00:56.611005] to 109051904 ta 100663296 fi 134217728 [D 20:00:56.611013] basic type or contiguous data [D 20:00:56.611019] Request disp [D 20:00:56.611025] contig_offset = 109051904 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.611033] calling distribute [D 20:00:56.611040] PINT_distribute [D 20:00:56.611047] of 109051904 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 109051904 exfl 1 [D 20:00:56.611055] begin iteration loff: 109051904 [D 20:00:56.611062] segment exceeds byte limit [D 20:00:56.611068] file being extended [D 20:00:56.611074] process a segment [D 20:00:56.611080] of 109051904 sz 2097152 [D 20:00:56.611086] add a segment 0 [D 20:00:56.611092] end iteration [D 20:00:56.611098] done with segments or bytes [D 20:00:56.611104] of 111149056 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.611111] next loff: 111149056 next poff: 111149056 [D 20:00:56.611118] return value 2097152 [D 20:00:56.611124] finished [D 20:00:56.611130] retval = 2097152 [D 20:00:56.611136] exiting distribute returned less than expected [D 20:00:56.611143] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 111149056 fo 134217728 eof 0 [D 20:00:56.611150] ========================================================= [D 20:00:56.611158] BMI_post_recv: addr: 6843072, offset: 0x2aaab3e15010, size: 2097152, tag: 11 [D 20:00:56.611175] openib_check_cq: completed send wr_id 6837a0. [D 20:00:56.611186] openib_check_cq: completed recv wr_id 684800. [D 20:00:56.611203] openib_post_rr: 10.1.4.56:39990 bh 67 wr_id 684800. [D 20:00:56.611213] openib_check_cq: completed recv wr_id 684830. [D 20:00:56.611222] openib_post_rr: 10.1.4.56:39990 bh 68 wr_id 684830. [D 20:00:56.611230] openib_check_cq: completed recv wr_id 684860. [D 20:00:56.611238] openib_post_rr: 10.1.4.56:39990 bh 69 wr_id 684860. [D 20:00:56.611259] post_sr: 10.1.4.56:39990 bh 81 len 48 wr_id 6837d0. [D 20:00:56.611272] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:00:56.611282] get_data_sync_mode returning 0 [D 20:00:56.611294] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.611301] dbpf_bstream_rw_list: mem_offset: 0x2aaab2407010, mem_size: 2097152 [D 20:00:56.611308] dbpf_bstream_rw_list: stream_offset: 140509184, stream_size: 2097152 [D 20:00:56.611315] dbpf_open_cache_get: called [D 20:00:56.611321] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.611329] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.611359] DBPF I/O ops in progress: 5 [D 20:00:56.611367] lio_listio called with the following aiocbs: [D 20:00:56.611374] aiocb_ptr_array[0]: fd: 13, off: 140509184, bytes: 2097152, buf: 0x2aaab2407010, type: 1 [D 20:00:56.611390] issue_or_delay_io_operation: lio_listio posted 0x6ecf80 (handle 2863311531, ret 0) [D 20:00:56.614087] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:00:56.614186] DBPF I/O ops in progress: 4 [D 20:00:56.614250] ========================================================= [D 20:00:56.614262] PINT_process_request [D 20:00:56.614270] tiling 6 copies [D 20:00:56.614278] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.614286] lvl 0 el 0 blk 0 by 142606336 [D 20:00:56.614292] to 142606336 ta 134217728 fi 167772160 [D 20:00:56.614299] basic type or contiguous data [D 20:00:56.614305] Request disp [D 20:00:56.614312] contig_offset = 142606336 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.614320] calling distribute [D 20:00:56.614327] PINT_distribute [D 20:00:56.614333] of 142606336 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 142606336 exfl 1 [D 20:00:56.614342] begin iteration loff: 142606336 [D 20:00:56.614349] segment exceeds byte limit [D 20:00:56.614355] file being extended [D 20:00:56.614361] process a segment [D 20:00:56.614368] of 142606336 sz 2097152 [D 20:00:56.614375] add a segment 0 [D 20:00:56.614382] end iteration [D 20:00:56.614388] done with segments or bytes [D 20:00:56.614394] of 144703488 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.614401] next loff: 144703488 next poff: 144703488 [D 20:00:56.614408] return value 2097152 [D 20:00:56.614416] finished [D 20:00:56.614422] retval = 2097152 [D 20:00:56.614428] exiting distribute returned less than expected [D 20:00:56.614435] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 144703488 fo 167772160 eof 0 [D 20:00:56.614442] ========================================================= [D 20:00:56.614449] BMI_post_recv: addr: 6843072, offset: 0x2aaab4016010, size: 2097152, tag: 12 [D 20:00:56.614465] openib_check_cq: completed send wr_id 6837d0. [D 20:00:56.614476] openib_check_cq: completed recv wr_id 684890. [D 20:00:56.614488] openib_post_rr: 10.1.4.56:39990 bh 70 wr_id 684890. [D 20:00:56.614513] post_sr: 10.1.4.56:39990 bh 82 len 48 wr_id 683800. [D 20:00:56.614525] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:00:56.614534] get_data_sync_mode returning 0 [D 20:00:56.614546] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.614554] dbpf_bstream_rw_list: mem_offset: 0x2aaab2608010, mem_size: 2097152 [D 20:00:56.614561] dbpf_bstream_rw_list: stream_offset: 174063616, stream_size: 2097152 [D 20:00:56.614569] dbpf_open_cache_get: called [D 20:00:56.614576] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.614583] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.614605] DBPF I/O ops in progress: 5 [D 20:00:56.614612] lio_listio called with the following aiocbs: [D 20:00:56.614619] aiocb_ptr_array[0]: fd: 13, off: 174063616, bytes: 2097152, buf: 0x2aaab2608010, type: 1 [D 20:00:56.614633] issue_or_delay_io_operation: lio_listio posted 0x6ed8e0 (handle 2863311531, ret 0) [D 20:00:56.617322] --- aio_progress_notification called with handle 2863311531 (0x6f11f0) [D 20:00:56.617374] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.617385] dbpf_open_cache_put: cached entry. [D 20:00:56.617392] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.617399] op_queue add: 0x6f11f0 [D 20:00:56.617412] DBPF I/O ops in progress: 4 [D 20:00:56.617423] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:00:56.617443] --- aio_progress_notification called with handle 2863311531 (0x6eddc0) [D 20:00:56.617456] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.617458] ========================================================= [D 20:00:56.617477] PINT_process_request [D 20:00:56.617485] tiling 7 copies [D 20:00:56.617492] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.617500] lvl 0 el 0 blk 0 by 176160768 [D 20:00:56.617507] to 176160768 ta 167772160 fi 201326592 [D 20:00:56.617515] basic type or contiguous data [D 20:00:56.617524] Request disp [D 20:00:56.617531] contig_offset = 176160768 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.617538] calling distribute [D 20:00:56.617544] PINT_distribute [D 20:00:56.617551] of 176160768 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 176160768 exfl 1 [D 20:00:56.617559] begin iteration loff: 176160768 [D 20:00:56.617566] segment exceeds byte limit [D 20:00:56.617572] file being extended [D 20:00:56.617579] process a segment [D 20:00:56.617584] of 176160768 sz 2097152 [D 20:00:56.617591] add a segment 0 [D 20:00:56.617597] end iteration [D 20:00:56.617603] done with segments or bytes [D 20:00:56.617609] of 178257920 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.617617] next loff: 178257920 next poff: 178257920 [D 20:00:56.617624] return value 2097152 [D 20:00:56.617631] finished [D 20:00:56.617636] retval = 2097152 [D 20:00:56.617642] exiting distribute returned less than expected [D 20:00:56.617649] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 178257920 fo 201326592 eof 0 [D 20:00:56.617656] ========================================================= [D 20:00:56.617663] BMI_post_recv: addr: 6843072, offset: 0x2aaab4217010, size: 2097152, tag: 13 [D 20:00:56.617677] openib_check_cq: completed send wr_id 683800. [D 20:00:56.617687] openib_check_cq: completed recv wr_id 6848c0. [D 20:00:56.617704] openib_post_rr: 10.1.4.56:39990 bh 71 wr_id 6848c0. [D 20:00:56.617713] openib_check_cq: completed recv wr_id 6848f0. [D 20:00:56.617723] openib_post_rr: 10.1.4.56:39990 bh 72 wr_id 6848f0. [D 20:00:56.617742] post_sr: 10.1.4.56:39990 bh 83 len 48 wr_id 683830. [D 20:00:56.617753] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:00:56.617762] get_data_sync_mode returning 0 [D 20:00:56.617772] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.617780] dbpf_bstream_rw_list: mem_offset: 0x2aaab2809010, mem_size: 2097152 [D 20:00:56.617786] dbpf_bstream_rw_list: stream_offset: 207618048, stream_size: 2097152 [D 20:00:56.617794] dbpf_open_cache_get: called [D 20:00:56.617468] dbpf_open_cache_put: cached entry. [D 20:00:56.620471] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.620485] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.620513] DBPF I/O ops in progress: 5 [D 20:00:56.620526] lio_listio called with the following aiocbs: [D 20:00:56.620532] aiocb_ptr_array[0]: fd: 13, off: 207618048, bytes: 2097152, buf: 0x2aaab2809010, type: 1 [D 20:00:56.620548] issue_or_delay_io_operation: lio_listio posted 0x6ed300 (handle 2863311531, ret 0) [D 20:00:56.622667] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.622735] op_queue add: 0x6eddc0 [D 20:00:56.622770] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:00:56.622794] DBPF I/O ops in progress: 4 [D 20:00:56.622849] --- aio_progress_notification called with handle 2863311531 (0x6f13f0) [D 20:00:56.622865] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.622878] dbpf_open_cache_put: cached entry. [D 20:00:56.622886] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.622892] op_queue add: 0x6f13f0 [D 20:00:56.622910] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:56.622922] DBPF I/O ops in progress: 3 [D 20:00:56.623219] ========================================================= [D 20:00:56.623233] PINT_process_request [D 20:00:56.623240] tiling 8 copies [D 20:00:56.623248] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.623256] lvl 0 el 0 blk 0 by 209715200 [D 20:00:56.623263] to 209715200 ta 201326592 fi 234881024 [D 20:00:56.623270] basic type or contiguous data [D 20:00:56.623277] Request disp [D 20:00:56.623283] contig_offset = 209715200 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.623291] calling distribute [D 20:00:56.623298] PINT_distribute [D 20:00:56.623305] of 209715200 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 209715200 exfl 1 [D 20:00:56.623314] begin iteration loff: 209715200 [D 20:00:56.623322] segment exceeds byte limit [D 20:00:56.623328] file being extended [D 20:00:56.623335] process a segment [D 20:00:56.623340] of 209715200 sz 2097152 [D 20:00:56.623348] add a segment 0 [D 20:00:56.623355] end iteration [D 20:00:56.623361] done with segments or bytes [D 20:00:56.623367] of 211812352 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.623374] next loff: 211812352 next poff: 211812352 [D 20:00:56.623381] return value 2097152 [D 20:00:56.623387] finished [D 20:00:56.623393] retval = 2097152 [D 20:00:56.623399] exiting distribute returned less than expected [D 20:00:56.623405] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 211812352 fo 234881024 eof 0 [D 20:00:56.623412] ========================================================= [D 20:00:56.623419] BMI_post_recv: addr: 6843072, offset: 0x2aaab4418010, size: 2097152, tag: 14 [D 20:00:56.623437] openib_check_cq: completed send wr_id 683830. [D 20:00:56.623445] openib_check_cq: completed recv wr_id 684920. [D 20:00:56.623466] openib_post_rr: 10.1.4.56:39990 bh 73 wr_id 684920. [D 20:00:56.623475] openib_check_cq: completed recv wr_id 684950. [D 20:00:56.623484] openib_post_rr: 10.1.4.56:39990 bh 74 wr_id 684950. [D 20:00:56.623491] openib_check_cq: completed recv wr_id 684980. [D 20:00:56.623500] openib_post_rr: 10.1.4.56:39990 bh 75 wr_id 684980. [D 20:00:56.623529] post_sr: 10.1.4.56:39990 bh 84 len 48 wr_id 683860. [D 20:00:56.623548] openib_check_cq: completed send wr_id 683860. [D 20:00:56.623565] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:00:56.623575] get_data_sync_mode returning 0 [D 20:00:56.623588] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.623596] dbpf_bstream_rw_list: mem_offset: 0x2aaab2a0a010, mem_size: 2097152 [D 20:00:56.623603] dbpf_bstream_rw_list: stream_offset: 241172480, stream_size: 2097152 [D 20:00:56.623610] dbpf_open_cache_get: called [D 20:00:56.623617] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.623624] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.623650] DBPF I/O ops in progress: 4 [D 20:00:56.623657] lio_listio called with the following aiocbs: [D 20:00:56.623663] aiocb_ptr_array[0]: fd: 13, off: 241172480, bytes: 2097152, buf: 0x2aaab2a0a010, type: 1 [D 20:00:56.623678] issue_or_delay_io_operation: lio_listio posted 0x6ea020 (handle 2863311531, ret 0) [D 20:00:56.626381] ========================================================= [D 20:00:56.626442] PINT_process_request [D 20:00:56.626450] tiling 9 copies [D 20:00:56.626458] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.626466] lvl 0 el 0 blk 0 by 243269632 [D 20:00:56.626473] to 243269632 ta 234881024 fi 268435456 [D 20:00:56.626480] basic type or contiguous data [D 20:00:56.626486] Request disp [D 20:00:56.626492] contig_offset = 243269632 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.626500] calling distribute [D 20:00:56.626506] PINT_distribute [D 20:00:56.626513] of 243269632 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 243269632 exfl 1 [D 20:00:56.626522] begin iteration loff: 243269632 [D 20:00:56.626529] segment exceeds byte limit [D 20:00:56.626535] file being extended [D 20:00:56.626541] process a segment [D 20:00:56.626547] of 243269632 sz 2097152 [D 20:00:56.626554] add a segment 0 [D 20:00:56.626560] end iteration [D 20:00:56.626566] done with segments or bytes [D 20:00:56.626572] of 245366784 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.626579] next loff: 245366784 next poff: 245366784 [D 20:00:56.626585] return value 2097152 [D 20:00:56.626592] finished [D 20:00:56.626598] retval = 2097152 [D 20:00:56.626604] exiting distribute returned less than expected [D 20:00:56.626610] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 245366784 fo 268435456 eof 0 [D 20:00:56.626617] ========================================================= [D 20:00:56.626628] BMI_post_recv: addr: 6843072, offset: 0x2aaab4619010, size: 2097152, tag: 15 [D 20:00:56.626644] openib_check_cq: completed recv wr_id 6849b0. [D 20:00:56.626658] openib_post_rr: 10.1.4.56:39990 bh 76 wr_id 6849b0. [D 20:00:56.626684] post_sr: 10.1.4.56:39990 bh 85 len 48 wr_id 683890. [D 20:00:56.626696] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:00:56.626707] get_data_sync_mode returning 0 [D 20:00:56.626720] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.626728] dbpf_bstream_rw_list: mem_offset: 0x2aaab2c0b010, mem_size: 2097152 [D 20:00:56.626735] dbpf_bstream_rw_list: stream_offset: 274726912, stream_size: 2097152 [D 20:00:56.626742] dbpf_open_cache_get: called [D 20:00:56.626748] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.626755] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.626777] DBPF I/O ops in progress: 5 [D 20:00:56.626785] lio_listio called with the following aiocbs: [D 20:00:56.626791] aiocb_ptr_array[0]: fd: 13, off: 274726912, bytes: 2097152, buf: 0x2aaab2c0b010, type: 1 [D 20:00:56.626804] issue_or_delay_io_operation: lio_listio posted 0x6f0a90 (handle 2863311531, ret 0) [D 20:00:56.629559] ========================================================= [D 20:00:56.629631] PINT_process_request [D 20:00:56.629639] tiling 10 copies [D 20:00:56.629646] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.629655] lvl 0 el 0 blk 0 by 276824064 [D 20:00:56.629661] to 276824064 ta 268435456 fi 301989888 [D 20:00:56.629668] basic type or contiguous data [D 20:00:56.629675] Request disp [D 20:00:56.629681] contig_offset = 276824064 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.629688] calling distribute [D 20:00:56.629695] PINT_distribute [D 20:00:56.629703] of 276824064 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 276824064 exfl 1 [D 20:00:56.629711] begin iteration loff: 276824064 [D 20:00:56.629718] segment exceeds byte limit [D 20:00:56.629724] file being extended [D 20:00:56.629731] process a segment [D 20:00:56.629737] of 276824064 sz 2097152 [D 20:00:56.629743] add a segment 0 [D 20:00:56.629749] end iteration [D 20:00:56.629756] done with segments or bytes [D 20:00:56.629761] of 278921216 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.629769] next loff: 278921216 next poff: 278921216 [D 20:00:56.629776] return value 2097152 [D 20:00:56.629784] finished [D 20:00:56.629790] retval = 2097152 [D 20:00:56.629797] exiting distribute returned less than expected [D 20:00:56.629804] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 278921216 fo 301989888 eof 0 [D 20:00:56.629811] ========================================================= [D 20:00:56.629820] BMI_post_recv: addr: 6843072, offset: 0x2aaab481a010, size: 2097152, tag: 16 [D 20:00:56.629836] openib_check_cq: completed send wr_id 683890. [D 20:00:56.629847] openib_check_cq: completed recv wr_id 6849e0. [D 20:00:56.629863] openib_post_rr: 10.1.4.56:39990 bh 77 wr_id 6849e0. [D 20:00:56.629888] post_sr: 10.1.4.56:39990 bh 86 len 48 wr_id 6838c0. [D 20:00:56.629902] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:00:56.629912] get_data_sync_mode returning 0 [D 20:00:56.629923] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.629930] dbpf_bstream_rw_list: mem_offset: 0x2aaab2e0c010, mem_size: 2097152 [D 20:00:56.629938] dbpf_bstream_rw_list: stream_offset: 308281344, stream_size: 2097152 [D 20:00:56.629945] dbpf_open_cache_get: called [D 20:00:56.629952] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.629959] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.629983] DBPF I/O ops in progress: 6 [D 20:00:56.629991] lio_listio called with the following aiocbs: [D 20:00:56.629998] aiocb_ptr_array[0]: fd: 13, off: 308281344, bytes: 2097152, buf: 0x2aaab2e0c010, type: 1 [D 20:00:56.630011] issue_or_delay_io_operation: lio_listio posted 0x6f0be0 (handle 2863311531, ret 0) [D 20:00:56.632667] --- aio_progress_notification called with handle 2863311531 (0x6ed8e0) [D 20:00:56.632770] ========================================================= [D 20:00:56.632779] PINT_process_request [D 20:00:56.632786] tiling 11 copies [D 20:00:56.632794] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.632801] lvl 0 el 0 blk 0 by 310378496 [D 20:00:56.632808] to 310378496 ta 301989888 fi 335544320 [D 20:00:56.632815] basic type or contiguous data [D 20:00:56.632821] Request disp [D 20:00:56.632827] contig_offset = 310378496 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.632835] calling distribute [D 20:00:56.632840] PINT_distribute [D 20:00:56.632847] of 310378496 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 310378496 exfl 1 [D 20:00:56.632855] begin iteration loff: 310378496 [D 20:00:56.632863] segment exceeds byte limit [D 20:00:56.632869] file being extended [D 20:00:56.632875] process a segment [D 20:00:56.632881] of 310378496 sz 2097152 [D 20:00:56.632887] add a segment 0 [D 20:00:56.632894] end iteration [D 20:00:56.632899] done with segments or bytes [D 20:00:56.632905] of 312475648 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.632912] next loff: 312475648 next poff: 312475648 [D 20:00:56.632919] return value 2097152 [D 20:00:56.632926] finished [D 20:00:56.632932] retval = 2097152 [D 20:00:56.632938] exiting distribute returned less than expected [D 20:00:56.632944] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 312475648 fo 335544320 eof 0 [D 20:00:56.632952] ========================================================= [D 20:00:56.632959] BMI_post_recv: addr: 6843072, offset: 0x2aaab4a1b010, size: 2097152, tag: 17 [D 20:00:56.632974] openib_check_cq: completed recv wr_id 684a10. [D 20:00:56.632989] openib_post_rr: 10.1.4.56:39990 bh 78 wr_id 684a10. [D 20:00:56.632999] openib_check_cq: completed send wr_id 6838c0. [D 20:00:56.633007] openib_check_cq: completed recv wr_id 684a40. [D 20:00:56.633019] openib_post_rr: 10.1.4.56:39990 bh 79 wr_id 684a40. [D 20:00:56.633026] openib_check_cq: completed recv wr_id 684a70. [D 20:00:56.633034] openib_post_rr: 10.1.4.56:39990 bh 80 wr_id 684a70. [D 20:00:56.633056] post_sr: 10.1.4.56:39990 bh 87 len 48 wr_id 6838f0. [D 20:00:56.633069] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:00:56.633078] get_data_sync_mode returning 0 [D 20:00:56.633090] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.633097] dbpf_bstream_rw_list: mem_offset: 0x2aaab300e010, mem_size: 2097152 [D 20:00:56.633104] dbpf_bstream_rw_list: stream_offset: 341835776, stream_size: 2097152 [D 20:00:56.633112] dbpf_open_cache_get: called [D 20:00:56.633118] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.633125] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.633150] DBPF I/O ops in progress: 7 [D 20:00:56.633157] lio_listio called with the following aiocbs: [D 20:00:56.633163] aiocb_ptr_array[0]: fd: 13, off: 341835776, bytes: 2097152, buf: 0x2aaab300e010, type: 1 [D 20:00:56.633174] issue_or_delay_io_operation: lio_listio posted 0x6f0d30 (handle 2863311531, ret 0) [D 20:00:56.634102] --- aio_progress_notification called with handle 2863311531 (0x6ecf80) [D 20:00:56.634161] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.634176] dbpf_open_cache_put: cached entry. [D 20:00:56.634184] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.634192] op_queue add: 0x6ecf80 [D 20:00:56.634210] --- aio_progress_notification called with handle 2863311531 (0x6ed300) [D 20:00:56.634222] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.634234] dbpf_open_cache_put: cached entry. [D 20:00:56.634240] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.635604] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.635764] ========================================================= [D 20:00:56.932407] PINT_process_request [D 20:00:56.932418] tiling 12 copies [D 20:00:56.932425] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.932432] lvl 0 el 0 blk 0 by 343932928 [D 20:00:56.932439] to 343932928 ta 335544320 fi 369098752 [D 20:00:56.932446] basic type or contiguous data [D 20:00:56.932452] Request disp [D 20:00:56.932458] contig_offset = 343932928 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.932465] calling distribute [D 20:00:56.932471] PINT_distribute [D 20:00:56.932478] of 343932928 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 343932928 exfl 1 [D 20:00:56.932486] begin iteration loff: 343932928 [D 20:00:56.932510] segment exceeds byte limit [D 20:00:56.932516] file being extended [D 20:00:56.932522] process a segment [D 20:00:56.932590] of 343932928 sz 2097152 [D 20:00:56.932597] add a segment 0 [D 20:00:56.932603] end iteration [D 20:00:56.932609] done with segments or bytes [D 20:00:56.932615] of 346030080 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.932622] next loff: 346030080 next poff: 346030080 [D 20:00:56.932628] return value 2097152 [D 20:00:56.932634] finished [D 20:00:56.932640] retval = 2097152 [D 20:00:56.932645] exiting distribute returned less than expected [D 20:00:56.932651] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 346030080 fo 369098752 eof 0 [D 20:00:56.932659] ========================================================= [D 20:00:56.932666] BMI_post_recv: addr: 6843072, offset: 0x2aaab4c1c010, size: 2097152, tag: 18 [D 20:00:56.932779] openib_check_cq: completed send wr_id 6838f0. [D 20:00:56.932787] openib_check_cq: completed recv wr_id 684aa0. [D 20:00:56.932800] openib_post_rr: 10.1.4.56:39990 bh 81 wr_id 684aa0. [D 20:00:56.932809] openib_check_cq: completed recv wr_id 684ad0. [D 20:00:56.932817] openib_post_rr: 10.1.4.56:39990 bh 82 wr_id 684ad0. [D 20:00:56.932824] openib_check_cq: completed recv wr_id 684b00. [D 20:00:56.932832] openib_post_rr: 10.1.4.56:39990 bh 83 wr_id 684b00. [D 20:00:56.932850] post_sr: 10.1.4.56:39990 bh 88 len 48 wr_id 683920. [D 20:00:56.932861] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:00:56.932899] get_data_sync_mode returning 0 [D 20:00:56.932916] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.932986] dbpf_bstream_rw_list: mem_offset: 0x2aaab320f010, mem_size: 2097152 [D 20:00:56.932992] dbpf_bstream_rw_list: stream_offset: 375390208, stream_size: 2097152 [D 20:00:56.933000] dbpf_open_cache_get: called [D 20:00:56.933006] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.933013] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.636422] op_queue add: 0x6ed300 [D 20:00:56.933046] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:56.933058] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:00:56.640452] --- aio_progress_notification called with handle 2863311531 (0x6ea020) [D 20:00:56.933078] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.933156] dbpf_open_cache_put: cached entry. [D 20:00:56.933162] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.933169] op_queue add: 0x6ea020 [D 20:00:56.933184] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:00:56.644418] --- aio_progress_notification called with handle 2863311531 (0x6f0be0) [D 20:00:56.933202] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.933216] dbpf_open_cache_put: cached entry. [D 20:00:56.933223] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.933230] op_queue add: 0x6f0be0 [D 20:00:56.933266] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:00:56.644488] --- aio_progress_notification called with handle 2863311531 (0x6f0a90) [D 20:00:56.933347] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.933362] dbpf_open_cache_put: cached entry. [D 20:00:56.933369] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.933375] op_queue add: 0x6f0a90 [D 20:00:56.933390] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:00:56.646798] --- aio_progress_notification called with handle 2863311531 (0x6f0d30) [A 20:00:56.861002] troy.scl@10.1.4.56:39990 H=2863311531 S=0x662d90: io: request [D 20:00:56.933414] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:56.933432] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab15050a0 [D 20:00:56.933446] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab15050a0 [D 20:00:56.933457] [SM Exiting]: (0x662c90) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:56.933466] [SM Entering]: (0x662c90) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:56.933473] [SM frame get]: (0x662c90) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.636410] DBPF I/O ops in progress: 6 [D 20:00:56.933613] DBPF I/O ops in progress: 7 [D 20:00:56.933625] lio_listio called with the following aiocbs: [D 20:00:56.933633] aiocb_ptr_array[0]: fd: 13, off: 375390208, bytes: 2097152, buf: 0x2aaab320f010, type: 1 [D 20:00:56.933645] issue_or_delay_io_operation: lio_listio posted 0x6f0f30 (handle 2863311531, ret 0) [D 20:00:56.935025] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.935066] dbpf_open_cache_put: cached entry. [D 20:00:56.935076] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.935083] op_queue add: 0x6f0d30 [D 20:00:56.935106] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:00:56.935128] DBPF I/O ops in progress: 6 [D 20:00:56.935166] dbpf_open_cache_put: cached entry. [D 20:00:56.935184] ========================================================= [D 20:00:56.935195] PINT_process_request [D 20:00:56.935203] tiling 13 copies [D 20:00:56.935211] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.935218] lvl 0 el 0 blk 0 by 377487360 [D 20:00:56.935225] to 377487360 ta 369098752 fi 402653184 [D 20:00:56.935232] basic type or contiguous data [D 20:00:56.935238] Request disp [D 20:00:56.935244] contig_offset = 377487360 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.935259] calling distribute [D 20:00:56.935265] PINT_distribute [D 20:00:56.935271] of 377487360 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 377487360 exfl 1 [D 20:00:56.935279] begin iteration loff: 377487360 [D 20:00:56.935287] segment exceeds byte limit [D 20:00:56.935293] file being extended [D 20:00:56.935300] process a segment [D 20:00:56.935306] of 377487360 sz 2097152 [D 20:00:56.935316] add a segment 0 [D 20:00:56.935322] end iteration [D 20:00:56.935328] done with segments or bytes [D 20:00:56.935338] of 379584512 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.935349] next loff: 379584512 next poff: 379584512 [D 20:00:56.935355] return value 2097152 [D 20:00:56.935361] finished [D 20:00:56.935377] retval = 2097152 [D 20:00:56.935383] exiting distribute returned less than expected [D 20:00:56.935389] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 379584512 fo 402653184 eof 0 [D 20:00:56.935397] ========================================================= [D 20:00:56.935407] BMI_post_recv: addr: 6843072, offset: 0x2aaab4e1d010, size: 2097152, tag: 19 [D 20:00:56.935419] openib_check_cq: completed send wr_id 683920. [D 20:00:56.935440] post_sr: 10.1.4.56:39990 bh 89 len 48 wr_id 683950. [D 20:00:56.935460] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:00:56.935469] get_data_sync_mode returning 0 [D 20:00:56.935479] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.935486] dbpf_bstream_rw_list: mem_offset: 0x2aaab3410010, mem_size: 2097152 [D 20:00:56.935493] dbpf_bstream_rw_list: stream_offset: 408944640, stream_size: 2097152 [D 20:00:56.935503] dbpf_open_cache_get: called [D 20:00:56.935512] DBPF I/O ops in progress: 5 [D 20:00:56.936407] DBPF I/O ops in progress: 4 [D 20:00:56.936513] DBPF I/O ops in progress: 3 [D 20:00:56.936658] DBPF I/O ops in progress: 2 [D 20:00:56.938871] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.938902] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.938922] DBPF I/O ops in progress: 3 [D 20:00:56.938930] lio_listio called with the following aiocbs: [D 20:00:56.938936] aiocb_ptr_array[0]: fd: 13, off: 408944640, bytes: 2097152, buf: 0x2aaab3410010, type: 1 [D 20:00:56.938948] issue_or_delay_io_operation: lio_listio posted 0x6ec210 (handle 2863311531, ret 0) [A 20:00:56.939540] troy.scl@10.1.4.56:39990 H=2863311531 S=0x662d90: io: start [D 20:00:56.939563] (0x662d90) io (prelude sm) state: getattr_if_needed [D 20:00:56.939571] About to retrieve attributes for handle 2863311531 [D 20:00:56.939589] op_queue add: 0x2aaab1505220 [D 20:00:56.940148] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:56.940158] --- aio_progress_notification called with handle 2863311531 (0x6f0f30) [D 20:00:56.940170] [SM Exiting]: (0x662c90) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:56.940150] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.940167] dbpf_open_cache_get: called [D 20:00:56.940205] op_queue add: 0x6ed8e0 [D 20:00:56.940212] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.940234] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.940241] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:00:56.940245] dbpf_open_cache_put: cached entry. [D 20:00:56.940269] DBPF I/O ops in progress: 2 [D 20:00:56.940290] ========================================================= [D 20:00:56.940301] PINT_process_request [D 20:00:56.940308] [SM Entering]: (0x65e190) io_sm:start_flow (status: 0) [D 20:00:56.940322] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.940309] tiling 14 copies [D 20:00:56.940340] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.940362] lvl 0 el 0 blk 0 by 411041792 [D 20:00:56.940368] to 411041792 ta 402653184 fi 436207616 [D 20:00:56.940375] basic type or contiguous data [D 20:00:56.940382] Request disp [D 20:00:56.940388] contig_offset = 411041792 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.940403] calling distribute [D 20:00:56.940409] PINT_distribute [D 20:00:56.940415] of 411041792 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 411041792 exfl 1 [D 20:00:56.940433] begin iteration loff: 411041792 [D 20:00:56.940440] segment exceeds byte limit [D 20:00:56.940446] file being extended [D 20:00:56.940452] process a segment [D 20:00:56.940467] of 411041792 sz 2097152 [D 20:00:56.940473] add a segment 0 [D 20:00:56.940479] end iteration [D 20:00:56.940486] done with segments or bytes [D 20:00:56.940495] of 413138944 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.940503] next loff: 413138944 next poff: 413138944 [D 20:00:56.940509] return value 2097152 [D 20:00:56.940179] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.940293] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 442499072 [D 20:00:56.940520] finished [D 20:00:56.940536] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 442499072) [D 20:00:56.940540] retval = 2097152 [D 20:00:56.940558] exiting distribute returned less than expected [D 20:00:56.940566] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 413138944 fo 436207616 eof 0 [D 20:00:56.940550] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:56.940585] op_queue add: 0x2aaab1505220 [D 20:00:56.940332] io_start_flow() issuing flow to write data. [D 20:00:56.940605] flow: fsize: 0, server_nr: 0, server_ct: 1 [D 20:00:56.940574] ========================================================= [D 20:00:56.940612] file_req_offset: 469762048, aggregate_size: 33554432, handle: 2863311531 [D 20:00:56.940640] PINT_new_request_states n=1 [D 20:00:56.940628] BMI_post_recv: addr: 6843072, offset: 0x2aaab501e010, size: 2097152, tag: 20 [D 20:00:56.940651] flowproto posting 0x2aaab1505370 [D 20:00:56.940672] openib_check_cq: completed send wr_id 683950. [D 20:00:56.940696] flowproto-multiqueue forcing trove_write_callback_fn. [D 20:00:56.940718] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:00:56.940709] openib_check_cq: completed recv wr_id 684b30. [D 20:00:56.942995] ========================================================= [D 20:00:56.942996] openib_post_rr: 10.1.4.56:39990 bh 84 wr_id 684b30. [D 20:00:56.943046] PINT_process_request [D 20:00:56.943057] openib_check_cq: completed recv wr_id 684b60. [D 20:00:56.943057] tiling 16 copies [D 20:00:56.943079] openib_post_rr: 10.1.4.56:39990 bh 85 wr_id 684b60. [D 20:00:56.943081] skipping ahead to target_offset [D 20:00:56.943102] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.943091] openib_check_cq: completed recv wr_id 684b90. [D 20:00:56.943111] lvl 0 el 0 blk 0 by 0 [D 20:00:56.943136] to 0 ta 469762048 fi 503316480 [D 20:00:56.943147] basic type or contiguous data [D 20:00:56.943156] Request disp [D 20:00:56.943166] contig_offset = 0 contig_size = 536870912 lvl_flag = 1 [D 20:00:56.943128] openib_post_rr: 10.1.4.56:39990 bh 86 wr_id 684b90. [D 20:00:56.943176] process logical skip [D 20:00:56.943201] retval = 469762048 [D 20:00:56.943193] openib_check_cq: completed recv wr_id 684bc0. [D 20:00:56.943207] exiting logical skip because distribute indicates done [D 20:00:56.943229] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.943239] lvl 0 el 0 blk 0 by 469762048 [D 20:00:56.943248] to 469762048 ta 469762048 fi 503316480 [D 20:00:56.943257] basic type or contiguous data [D 20:00:56.943266] Request disp [D 20:00:56.943274] contig_offset = 469762048 contig_size = 67108864 lvl_flag = 1 [D 20:00:56.943219] openib_post_rr: 10.1.4.56:39990 bh 87 wr_id 684bc0. [D 20:00:56.943282] calling distribute [D 20:00:56.943312] PINT_distribute [D 20:00:56.943321] of 469762048 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 0 exfl 1 [D 20:00:56.943302] post_sr: 10.1.4.56:39990 bh 90 len 48 wr_id 683980. [D 20:00:56.943332] begin iteration loff: 469762048 [D 20:00:56.943352] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:00:56.943353] segment exceeds byte limit [D 20:00:56.943376] file being extended [D 20:00:56.943365] get_data_sync_mode returning 0 [D 20:00:56.943382] process a segment [D 20:00:56.943401] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.943404] of 469762048 sz 2097152 [D 20:00:56.943413] dbpf_bstream_rw_list: mem_offset: 0x2aaab3611010, mem_size: 2097152 [D 20:00:56.943425] add a segment 0 [D 20:00:56.943450] end iteration [D 20:00:56.943459] done with segments or bytes [D 20:00:56.943468] of 471859200 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.943477] next loff: 471859200 next poff: 471859200 [D 20:00:56.943488] return value 2097152 [D 20:00:56.943497] finished [D 20:00:56.943503] retval = 2097152 [D 20:00:56.943512] exiting distribute returned less than expected [D 20:00:56.943531] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 471859200 fo 503316480 eof 0 [D 20:00:56.943440] dbpf_bstream_rw_list: stream_offset: 442499072, stream_size: 2097152 [D 20:00:56.943540] ========================================================= [D 20:00:56.943559] BMI_post_recv: addr: 6843072, offset: 0x2aaab521f010, size: 2097152, tag: 22 [D 20:00:56.943573] openib_check_cq: completed send wr_id 683980. [D 20:00:56.943548] dbpf_open_cache_get: called [D 20:00:56.943587] post_sr: 10.1.4.56:39990 bh 91 len 48 wr_id 6839b0. [D 20:00:56.943606] flowproto posted 0x2aaab1505370 [D 20:00:56.943614] Job flows in progress (post time): 15 [D 20:00:56.943629] [SM Exiting]: (0x65e190) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:00:56.943642] [SM Entering]: (0x663f50) unexpected_sm:map_request (status: 0) [D 20:00:56.943651] [SM frame get]: (0x663f50) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.943660] server_post_unexpected_recv [D 20:00:56.943682] [SM Frame PUSH]: (0x2aaab1500d10) frame: 0x2aaab1507cc0 [D 20:00:56.943690] [SM Locating]: (0x2aaab1500d10) op-id: 999 [D 20:00:56.943697] server_op_state_get_machine 999 [D 20:00:56.943704] [SM Locating]: (0x2aaab1500d10) located: unexpected_sm:post_unexpected [D 20:00:56.943711] [SM frame get]: (0x2aaab1500d10) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.943718] [SM Entering]: (0x2aaab1500d10) unexpected_sm:post_unexpected (status: 0) [D 20:00:56.943726] [SM frame get]: (0x2aaab1500d10) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.943738] [SM Exiting]: (0x2aaab1500d10) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:56.943747] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:56.943754] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:56.943761] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 20 [D 20:00:56.943769] [SM frame get]: (0x663f50) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.943776] server_state_machine_start 0x663f50 [D 20:00:56.943784] PINT_decode [D 20:00:56.943794] lebf_decode_req [D 20:00:56.943811] [SM Locating]: (0x663f50) op-id: 3 [D 20:00:56.943818] server_op_state_get_machine 3 [D 20:00:56.943825] [SM push_state]: (0x663f50) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:56.943832] [SM Locating]: (0x663f50) located: prelude_sm:setup [D 20:00:56.943840] [SM Entering]: (0x663f50) prelude_sm:setup (status: 0) [D 20:00:56.943847] [SM frame get]: (0x663f50) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.943856] [SM Exiting]: (0x663f50) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:56.943863] [SM Exiting]: (0x663f50) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:56.943870] [SM push_state]: (0x663f50) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.943877] [SM Entering]: (0x663f50) prelude_work_sm:req_sched (status: 0) [D 20:00:56.943884] [SM frame get]: (0x663f50) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.943892] (0x664050) io (prelude sm) state: req_sched [D 20:00:56.940526] dbpf_open_cache_put: cached entry. [D 20:00:56.944432] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.944445] op_queue add: 0x6f0f30 [D 20:00:56.944436] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.944471] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.944481] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:00:56.944499] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:56.944486] DBPF I/O ops in progress: 1 [D 20:00:56.944540] DBPF I/O ops in progress: 2 [D 20:00:56.944551] lio_listio called with the following aiocbs: [D 20:00:56.944559] aiocb_ptr_array[0]: fd: 13, off: 442499072, bytes: 2097152, buf: 0x2aaab3611010, type: 1 [D 20:00:56.944573] issue_or_delay_io_operation: lio_listio posted 0x6ecf80 (handle 2863311531, ret 0) [D 20:00:56.946140] ========================================================= [D 20:00:56.946198] PINT_process_request [D 20:00:56.946206] tiling 15 copies [D 20:00:56.946215] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.946223] lvl 0 el 0 blk 0 by 444596224 [D 20:00:56.946229] to 444596224 ta 436207616 fi 469762048 [D 20:00:56.946237] basic type or contiguous data [D 20:00:56.946243] Request disp [D 20:00:56.946250] contig_offset = 444596224 contig_size = 58720256 lvl_flag = 1 [D 20:00:56.946257] calling distribute [D 20:00:56.946263] PINT_distribute [D 20:00:56.946270] of 444596224 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 444596224 exfl 1 [D 20:00:56.946280] begin iteration loff: 444596224 [D 20:00:56.946288] segment exceeds byte limit [D 20:00:56.946294] file being extended [D 20:00:56.946302] process a segment [D 20:00:56.946308] of 444596224 sz 2097152 [D 20:00:56.946315] add a segment 0 [D 20:00:56.946322] end iteration [D 20:00:56.946328] done with segments or bytes [D 20:00:56.946334] of 446693376 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.946342] next loff: 446693376 next poff: 446693376 [D 20:00:56.946349] return value 2097152 [D 20:00:56.946356] finished [D 20:00:56.946362] retval = 2097152 [D 20:00:56.946368] exiting distribute returned less than expected [D 20:00:56.946374] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 446693376 fo 469762048 eof 0 [D 20:00:56.946382] ========================================================= [D 20:00:56.946389] BMI_post_recv: addr: 6843072, offset: 0x2aaab5420010, size: 2097152, tag: 21 [D 20:00:56.946411] openib_check_cq: completed send wr_id 6839b0. [D 20:00:56.946443] post_sr: 10.1.4.56:39990 bh 92 len 48 wr_id 6839e0. [D 20:00:56.946458] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:00:56.946470] get_data_sync_mode returning 0 [D 20:00:56.946484] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.946492] dbpf_bstream_rw_list: mem_offset: 0x2aaab3812010, mem_size: 2097152 [D 20:00:56.946500] dbpf_bstream_rw_list: stream_offset: 8388608, stream_size: 2097152 [D 20:00:56.946508] dbpf_open_cache_get: called [D 20:00:56.946515] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.946522] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.946549] DBPF I/O ops in progress: 3 [D 20:00:56.946556] lio_listio called with the following aiocbs: [D 20:00:56.946562] aiocb_ptr_array[0]: fd: 13, off: 8388608, bytes: 2097152, buf: 0x2aaab3812010, type: 1 [D 20:00:56.946579] issue_or_delay_io_operation: lio_listio posted 0x6ed8e0 (handle 2863311531, ret 0) [D 20:00:56.950071] ========================================================= [D 20:00:56.950119] PINT_process_request [D 20:00:56.950127] tiling 2 copies [D 20:00:56.950136] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.950145] lvl 0 el 0 blk 0 by 10485760 [D 20:00:56.950152] to 10485760 ta 0 fi 33554432 [D 20:00:56.950159] basic type or contiguous data [D 20:00:56.950165] Request disp [D 20:00:56.950172] contig_offset = 10485760 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.950181] calling distribute [D 20:00:56.950188] PINT_distribute [D 20:00:56.950195] of 10485760 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 10485760 exfl 1 [D 20:00:56.950204] begin iteration loff: 10485760 [D 20:00:56.950211] segment exceeds byte limit [D 20:00:56.950217] file being extended [D 20:00:56.950223] process a segment [D 20:00:56.950230] of 10485760 sz 2097152 [D 20:00:56.950237] add a segment 0 [D 20:00:56.950244] end iteration [D 20:00:56.950250] done with segments or bytes [D 20:00:56.950257] of 12582912 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.950267] next loff: 12582912 next poff: 12582912 [D 20:00:56.950274] return value 2097152 [D 20:00:56.950281] finished [D 20:00:56.950287] retval = 2097152 [D 20:00:56.950293] exiting distribute returned less than expected [D 20:00:56.950299] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 12582912 fo 33554432 eof 0 [D 20:00:56.950306] ========================================================= [D 20:00:56.950313] BMI_post_recv: addr: 6843072, offset: 0x2aaab5621010, size: 2097152, tag: 8 [D 20:00:56.950333] openib_check_cq: completed send wr_id 6839e0. [D 20:00:56.950344] openib_check_cq: completed recv wr_id 684bf0. [D 20:00:56.950361] openib_post_rr: 10.1.4.56:39990 bh 88 wr_id 684bf0. [D 20:00:56.950394] post_sr: 10.1.4.56:39990 bh 93 len 48 wr_id 683a10. [D 20:00:56.950409] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:00:56.950421] get_data_sync_mode returning 0 [D 20:00:56.950436] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.950444] dbpf_bstream_rw_list: mem_offset: 0x2aaab3a13010, mem_size: 2097152 [D 20:00:56.950452] dbpf_bstream_rw_list: stream_offset: 41943040, stream_size: 2097152 [D 20:00:56.950459] dbpf_open_cache_get: called [D 20:00:56.950466] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.950473] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.950498] DBPF I/O ops in progress: 4 [D 20:00:56.950506] lio_listio called with the following aiocbs: [D 20:00:56.950512] aiocb_ptr_array[0]: fd: 13, off: 41943040, bytes: 2097152, buf: 0x2aaab3a13010, type: 1 [D 20:00:56.950527] issue_or_delay_io_operation: lio_listio posted 0x6f0f30 (handle 2863311531, ret 0) [A 20:00:56.952591] troy.scl@10.1.4.56:39990 H=2863311531 S=0x664050: io: request [D 20:00:56.952667] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:56.952677] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab1508f10 [D 20:00:56.952684] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab1508f10 [D 20:00:56.952692] [SM Exiting]: (0x663f50) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:56.952705] [SM Entering]: (0x663f50) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:56.952712] [SM frame get]: (0x663f50) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.953738] ========================================================= [D 20:00:56.953763] PINT_process_request [D 20:00:56.953770] tiling 3 copies [D 20:00:56.953778] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.953786] lvl 0 el 0 blk 0 by 44040192 [D 20:00:56.953792] to 44040192 ta 33554432 fi 67108864 [D 20:00:56.953800] basic type or contiguous data [D 20:00:56.953819] Request disp [D 20:00:56.953826] contig_offset = 44040192 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.953833] calling distribute [D 20:00:56.953839] PINT_distribute [D 20:00:56.953845] of 44040192 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 44040192 exfl 1 [D 20:00:56.953854] begin iteration loff: 44040192 [D 20:00:56.953861] segment exceeds byte limit [D 20:00:56.953867] file being extended [D 20:00:56.953874] process a segment [D 20:00:56.953880] of 44040192 sz 2097152 [D 20:00:56.953886] add a segment 0 [D 20:00:56.953893] end iteration [D 20:00:56.953899] done with segments or bytes [D 20:00:56.953905] of 46137344 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.953912] next loff: 46137344 next poff: 46137344 [D 20:00:56.953918] return value 2097152 [D 20:00:56.953925] finished [D 20:00:56.953931] retval = 2097152 [D 20:00:56.953937] exiting distribute returned less than expected [D 20:00:56.953943] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 46137344 fo 67108864 eof 0 [D 20:00:56.953950] ========================================================= [D 20:00:56.953957] BMI_post_recv: addr: 6843072, offset: 0x2aaab5822010, size: 2097152, tag: 9 [D 20:00:56.953974] openib_check_cq: completed send wr_id 683a10. [D 20:00:56.953988] openib_check_cq: completed recv wr_id 684c20. [D 20:00:56.953998] openib_post_rr: 10.1.4.56:39990 bh 89 wr_id 684c20. [D 20:00:56.954008] openib_check_cq: completed recv wr_id 684c50. [D 20:00:56.954022] openib_post_rr: 10.1.4.56:39990 bh 90 wr_id 684c50. [D 20:00:56.954029] openib_check_cq: completed recv wr_id 684c80. [D 20:00:56.954038] openib_post_rr: 10.1.4.56:39990 bh 91 wr_id 684c80. [D 20:00:56.954046] openib_check_cq: completed recv wr_id 684cb0. [D 20:00:56.954054] openib_post_rr: 10.1.4.56:39990 bh 92 wr_id 684cb0. [D 20:00:56.954061] openib_check_cq: completed recv wr_id 684ce0. [D 20:00:56.954069] openib_post_rr: 10.1.4.56:39990 bh 93 wr_id 684ce0. [D 20:00:56.954092] post_sr: 10.1.4.56:39990 bh 94 len 48 wr_id 683a40. [D 20:00:56.954105] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:00:56.954115] get_data_sync_mode returning 0 [D 20:00:56.954128] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.954136] dbpf_bstream_rw_list: mem_offset: 0x2aaab3c14010, mem_size: 2097152 [D 20:00:56.954143] dbpf_bstream_rw_list: stream_offset: 75497472, stream_size: 2097152 [D 20:00:56.954151] dbpf_open_cache_get: called [D 20:00:56.954159] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.954166] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.954189] DBPF I/O ops in progress: 5 [D 20:00:56.954196] lio_listio called with the following aiocbs: [D 20:00:56.954203] aiocb_ptr_array[0]: fd: 13, off: 75497472, bytes: 2097152, buf: 0x2aaab3c14010, type: 1 [D 20:00:56.954216] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [D 20:00:56.956122] --- aio_progress_notification called with handle 2863311531 (0x6ec210) [D 20:00:56.956224] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.956241] dbpf_open_cache_put: cached entry. [D 20:00:56.956249] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.956256] op_queue add: 0x6ec210 [D 20:00:56.956275] --- aio_progress_notification called with handle 2863311531 (0x6ecf80) [D 20:00:56.956288] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.956301] dbpf_open_cache_put: cached entry. [D 20:00:56.956307] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.956319] --- aio_progress_notification called with handle 2863311531 (0x6ed8e0) [D 20:00:56.956330] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.956341] dbpf_open_cache_put: cached entry. [D 20:00:56.956347] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.956577] ========================================================= [D 20:00:56.956589] PINT_process_request [D 20:00:56.956599] tiling 4 copies [D 20:00:56.956607] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.956615] lvl 0 el 0 blk 0 by 77594624 [D 20:00:56.956621] to 77594624 ta 67108864 fi 100663296 [D 20:00:56.956629] basic type or contiguous data [D 20:00:56.956635] Request disp [D 20:00:56.956642] contig_offset = 77594624 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.956650] calling distribute [D 20:00:56.956657] PINT_distribute [D 20:00:56.956664] of 77594624 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 77594624 exfl 1 [D 20:00:56.956673] begin iteration loff: 77594624 [D 20:00:56.956813] segment exceeds byte limit [D 20:00:56.956819] file being extended [D 20:00:56.956826] process a segment [D 20:00:56.956832] of 77594624 sz 2097152 [D 20:00:56.956839] add a segment 0 [D 20:00:56.956846] end iteration [D 20:00:56.956851] done with segments or bytes [D 20:00:56.956857] of 79691776 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.956864] next loff: 79691776 next poff: 79691776 [D 20:00:56.956871] return value 2097152 [D 20:00:56.956878] finished [D 20:00:56.956884] retval = 2097152 [D 20:00:56.956892] exiting distribute returned less than expected [D 20:00:56.956898] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 79691776 fo 100663296 eof 0 [D 20:00:56.956906] ========================================================= [D 20:00:56.956913] BMI_post_recv: addr: 6843072, offset: 0x2aaab5a23010, size: 2097152, tag: 10 [D 20:00:56.957097] openib_check_cq: completed send wr_id 683a40. [D 20:00:56.957110] openib_check_cq: completed recv wr_id 684d10. [D 20:00:56.957130] openib_post_rr: 10.1.4.56:39990 bh 94 wr_id 684d10. [D 20:00:56.957140] openib_check_cq: completed recv wr_id 684d40. [D 20:00:56.957149] openib_post_rr: 10.1.4.56:39990 bh 95 wr_id 684d40. [D 20:00:56.957180] post_sr: 10.1.4.56:39990 bh 95 len 48 wr_id 683a70. [D 20:00:56.957201] openib_check_cq: completed send wr_id 683a70. [D 20:00:56.957221] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:00:56.957232] get_data_sync_mode returning 0 [D 20:00:56.957247] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.957254] dbpf_bstream_rw_list: mem_offset: 0x2aaab521f010, mem_size: 2097152 [D 20:00:56.957262] dbpf_bstream_rw_list: stream_offset: 469762048, stream_size: 2097152 [D 20:00:56.957270] dbpf_open_cache_get: called [D 20:00:56.957277] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.957284] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.957310] DBPF I/O ops in progress: 6 [D 20:00:56.957317] lio_listio called with the following aiocbs: [D 20:00:56.957324] aiocb_ptr_array[0]: fd: 13, off: 469762048, bytes: 2097152, buf: 0x2aaab521f010, type: 1 [D 20:00:56.957340] issue_or_delay_io_operation: lio_listio posted 0x6f0a90 (handle 2863311531, ret 0) [D 20:00:56.962437] ========================================================= [D 20:00:56.962465] PINT_process_request [D 20:00:56.962473] tiling 16 copies [D 20:00:56.962480] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.962488] lvl 0 el 0 blk 0 by 471859200 [D 20:00:56.962495] to 471859200 ta 469762048 fi 503316480 [D 20:00:56.962502] basic type or contiguous data [D 20:00:56.962508] Request disp [D 20:00:56.962515] contig_offset = 471859200 contig_size = 65011712 lvl_flag = 1 [D 20:00:56.962522] calling distribute [D 20:00:56.962529] PINT_distribute [D 20:00:56.962535] of 471859200 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 1 [D 20:00:56.962543] begin iteration loff: 471859200 [D 20:00:56.962551] segment exceeds byte limit [D 20:00:56.962557] file being extended [D 20:00:56.962563] process a segment [D 20:00:56.962569] of 471859200 sz 2097152 [D 20:00:56.962576] add a segment 0 [D 20:00:56.962582] end iteration [D 20:00:56.962588] done with segments or bytes [D 20:00:56.962594] of 473956352 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.962602] next loff: 473956352 next poff: 473956352 [D 20:00:56.962608] return value 2097152 [D 20:00:56.962615] finished [D 20:00:56.962621] retval = 2097152 [D 20:00:56.962627] exiting distribute returned less than expected [D 20:00:56.962634] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 473956352 fo 503316480 eof 0 [D 20:00:56.962641] ========================================================= [D 20:00:56.962648] BMI_post_recv: addr: 6843072, offset: 0x2aaab5c25010, size: 2097152, tag: 22 [D 20:00:56.962667] openib_check_cq: completed recv wr_id 684d70. [D 20:00:56.962683] openib_post_rr: 10.1.4.56:39990 bh 96 wr_id 684d70. [D 20:00:56.962693] openib_check_cq: completed recv wr_id 684da0. [D 20:00:56.962702] openib_post_rr: 10.1.4.56:39990 bh 97 wr_id 684da0. [D 20:00:56.962710] openib_check_cq: completed recv wr_id 684dd0. [D 20:00:56.962720] openib_post_rr: 10.1.4.56:39990 bh 98 wr_id 684dd0. [D 20:00:56.962744] post_sr: 10.1.4.56:39990 bh 96 len 48 wr_id 683aa0. [D 20:00:56.962758] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:00:56.962770] get_data_sync_mode returning 0 [D 20:00:56.962783] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.962794] dbpf_bstream_rw_list: mem_offset: 0x2aaab3e15010, mem_size: 2097152 [D 20:00:56.962801] dbpf_bstream_rw_list: stream_offset: 109051904, stream_size: 2097152 [D 20:00:56.962810] dbpf_open_cache_get: called [D 20:00:56.962818] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.962825] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.959936] --- aio_progress_notification called with handle 2863311531 (0x6f0f30) [D 20:00:56.962854] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.962867] dbpf_open_cache_put: cached entry. [D 20:00:56.962873] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.962881] op_queue add: 0x6f0f30 [D 20:00:56.962895] op_queue add: 0x6ecf80 [D 20:00:56.964534] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:00:56.964596] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.964616] dbpf_open_cache_put: cached entry. [D 20:00:56.964624] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.964632] op_queue add: 0x6ec5d0 [D 20:00:56.962365] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:00:56.964682] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:00:56.964691] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:00:56.964700] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:00:56.959968] DBPF I/O ops in progress: 5 [D 20:00:56.964732] DBPF I/O ops in progress: 6 [D 20:00:56.964742] lio_listio called with the following aiocbs: [D 20:00:56.964751] aiocb_ptr_array[0]: fd: 13, off: 109051904, bytes: 2097152, buf: 0x2aaab3e15010, type: 1 [D 20:00:56.964766] issue_or_delay_io_operation: lio_listio posted 0x6ec210 (handle 2863311531, ret 0) [A 20:00:56.967124] troy.scl@10.1.4.56:39990 H=2863311531 S=0x664050: io: start [D 20:00:56.967184] (0x664050) io (prelude sm) state: getattr_if_needed [D 20:00:56.967191] About to retrieve attributes for handle 2863311531 [D 20:00:56.967211] op_queue add: 0x2aaab1508f90 [D 20:00:56.967229] [SM Exiting]: (0x663f50) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:56.967241] [SM Entering]: (0x665210) unexpected_sm:map_request (status: 0) [D 20:00:56.967249] [SM frame get]: (0x665210) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.967257] server_post_unexpected_recv [D 20:00:56.967284] [SM Frame PUSH]: (0x2aaab15090e0) frame: 0x2aaab150a1c0 [D 20:00:56.967293] [SM Locating]: (0x2aaab15090e0) op-id: 999 [D 20:00:56.967300] server_op_state_get_machine 999 [D 20:00:56.967306] [SM Locating]: (0x2aaab15090e0) located: unexpected_sm:post_unexpected [D 20:00:56.967314] [SM frame get]: (0x2aaab15090e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.967321] [SM Entering]: (0x2aaab15090e0) unexpected_sm:post_unexpected (status: 0) [D 20:00:56.967328] [SM frame get]: (0x2aaab15090e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.967338] [SM Exiting]: (0x2aaab15090e0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:56.967346] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:56.967354] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:56.967361] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 21 [D 20:00:56.967368] [SM frame get]: (0x665210) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.967374] server_state_machine_start 0x665210 [D 20:00:56.967381] PINT_decode [D 20:00:56.967388] lebf_decode_req [D 20:00:56.967402] [SM Locating]: (0x665210) op-id: 3 [D 20:00:56.967409] server_op_state_get_machine 3 [D 20:00:56.967415] [SM push_state]: (0x665210) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:56.967422] [SM Locating]: (0x665210) located: prelude_sm:setup [D 20:00:56.967430] [SM Entering]: (0x665210) prelude_sm:setup (status: 0) [D 20:00:56.967437] [SM frame get]: (0x665210) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.967445] [SM Exiting]: (0x665210) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:56.967453] [SM Exiting]: (0x665210) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:56.967460] [SM push_state]: (0x665210) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.967467] [SM Entering]: (0x665210) prelude_work_sm:req_sched (status: 0) [D 20:00:56.967474] [SM frame get]: (0x665210) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.967481] (0x665310) io (prelude sm) state: req_sched [D 20:00:56.967715] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:56.967730] dbpf_open_cache_get: called [D 20:00:56.967739] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.967746] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.967755] dbpf_open_cache_put: cached entry. [D 20:00:56.967808] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 471859200 [D 20:00:56.967823] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 471859200) [D 20:00:56.967831] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:56.967838] op_queue add: 0x2aaab1508f90 [D 20:00:56.967940] ========================================================= [D 20:00:56.967950] PINT_process_request [D 20:00:56.967957] tiling 5 copies [D 20:00:56.967964] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.967972] lvl 0 el 0 blk 0 by 111149056 [D 20:00:56.967978] to 111149056 ta 100663296 fi 134217728 [D 20:00:56.967987] basic type or contiguous data [D 20:00:56.967993] Request disp [D 20:00:56.968000] contig_offset = 111149056 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.968007] calling distribute [D 20:00:56.968014] PINT_distribute [D 20:00:56.968021] of 111149056 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 111149056 exfl 1 [D 20:00:56.968030] begin iteration loff: 111149056 [D 20:00:56.968037] segment exceeds byte limit [D 20:00:56.968043] file being extended [D 20:00:56.968050] process a segment [D 20:00:56.968056] of 111149056 sz 2097152 [D 20:00:56.968063] add a segment 0 [D 20:00:56.968069] end iteration [D 20:00:56.968075] done with segments or bytes [D 20:00:56.968081] of 113246208 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.968088] next loff: 113246208 next poff: 113246208 [D 20:00:56.968095] return value 2097152 [D 20:00:56.968101] finished [D 20:00:56.968107] retval = 2097152 [D 20:00:56.968113] exiting distribute returned less than expected [D 20:00:56.968120] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 113246208 fo 134217728 eof 0 [D 20:00:56.968127] ========================================================= [D 20:00:56.968135] BMI_post_recv: addr: 6843072, offset: 0x2aaab5e26010, size: 2097152, tag: 11 [D 20:00:56.968151] openib_check_cq: completed send wr_id 683aa0. [D 20:00:56.968160] openib_check_cq: completed recv wr_id 684e00. [D 20:00:56.968177] openib_post_rr: 10.1.4.56:39990 bh 99 wr_id 684e00. [D 20:00:56.968186] openib_check_cq: completed recv wr_id 683b70. [D 20:00:56.968197] openib_post_rr: 10.1.4.56:39990 bh 0 wr_id 683b70. [D 20:00:56.968205] openib_check_cq: completed recv wr_id 683ba0. [D 20:00:56.968213] openib_post_rr: 10.1.4.56:39990 bh 1 wr_id 683ba0. [D 20:00:56.968236] post_sr: 10.1.4.56:39990 bh 97 len 48 wr_id 683ad0. [D 20:00:56.968248] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:00:56.968257] get_data_sync_mode returning 0 [D 20:00:56.968267] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.968274] dbpf_bstream_rw_list: mem_offset: 0x2aaab4016010, mem_size: 2097152 [D 20:00:56.968281] dbpf_bstream_rw_list: stream_offset: 142606336, stream_size: 2097152 [D 20:00:56.968289] dbpf_open_cache_get: called [D 20:00:56.968295] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.968302] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.968310] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:56.967918] DBPF I/O ops in progress: 5 [D 20:00:56.968358] DBPF I/O ops in progress: 4 [D 20:00:56.968383] DBPF I/O ops in progress: 3 [D 20:00:56.968403] DBPF I/O ops in progress: 4 [D 20:00:56.968411] lio_listio called with the following aiocbs: [D 20:00:56.968417] aiocb_ptr_array[0]: fd: 13, off: 142606336, bytes: 2097152, buf: 0x2aaab4016010, type: 1 [D 20:00:56.968431] issue_or_delay_io_operation: lio_listio posted 0x6f0f30 (handle 2863311531, ret 0) [D 20:00:56.971551] ========================================================= [D 20:00:56.971586] PINT_process_request [D 20:00:56.971595] tiling 6 copies [D 20:00:56.971603] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.971611] lvl 0 el 0 blk 0 by 144703488 [D 20:00:56.971618] to 144703488 ta 134217728 fi 167772160 [D 20:00:56.971626] basic type or contiguous data [D 20:00:56.971632] Request disp [D 20:00:56.971638] contig_offset = 144703488 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.971646] calling distribute [D 20:00:56.971652] PINT_distribute [D 20:00:56.971660] of 144703488 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 144703488 exfl 1 [D 20:00:56.971668] begin iteration loff: 144703488 [D 20:00:56.971676] segment exceeds byte limit [D 20:00:56.971683] file being extended [D 20:00:56.971689] process a segment [D 20:00:56.971695] of 144703488 sz 2097152 [D 20:00:56.971702] add a segment 0 [D 20:00:56.971708] end iteration [D 20:00:56.971714] done with segments or bytes [D 20:00:56.971720] of 146800640 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.971727] next loff: 146800640 next poff: 146800640 [D 20:00:56.971734] return value 2097152 [D 20:00:56.971742] finished [D 20:00:56.971748] retval = 2097152 [D 20:00:56.971754] exiting distribute returned less than expected [D 20:00:56.971760] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 146800640 fo 167772160 eof 0 [D 20:00:56.971767] ========================================================= [D 20:00:56.971776] BMI_post_recv: addr: 6843072, offset: 0x2aaab6027010, size: 2097152, tag: 12 [D 20:00:56.971795] openib_check_cq: completed send wr_id 683ad0. [D 20:00:56.971805] openib_check_cq: completed recv wr_id 683bd0. [D 20:00:56.971818] openib_post_rr: 10.1.4.56:39990 bh 2 wr_id 683bd0. [D 20:00:56.971845] post_sr: 10.1.4.56:39990 bh 98 len 48 wr_id 683b00. [D 20:00:56.971860] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:00:56.971871] get_data_sync_mode returning 0 [D 20:00:56.971884] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.971891] dbpf_bstream_rw_list: mem_offset: 0x2aaab4217010, mem_size: 2097152 [D 20:00:56.971898] dbpf_bstream_rw_list: stream_offset: 176160768, stream_size: 2097152 [D 20:00:56.971907] dbpf_open_cache_get: called [D 20:00:56.971914] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.971921] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.971943] DBPF I/O ops in progress: 5 [D 20:00:56.971950] lio_listio called with the following aiocbs: [D 20:00:56.971957] aiocb_ptr_array[0]: fd: 13, off: 176160768, bytes: 2097152, buf: 0x2aaab4217010, type: 1 [D 20:00:56.971970] issue_or_delay_io_operation: lio_listio posted 0x6ecf80 (handle 2863311531, ret 0) [D 20:00:56.972020] op_queue add: 0x6ed8e0 [D 20:00:56.972045] --- aio_progress_notification called with handle 2863311531 (0x6ec210) [D 20:00:56.972058] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.972069] dbpf_open_cache_put: cached entry. [D 20:00:56.972076] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [A 20:00:56.973948] troy.scl@10.1.4.56:39990 H=2863311531 S=0x665310: io: request [D 20:00:56.974025] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:56.974035] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab1509030 [D 20:00:56.974044] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab1509030 [D 20:00:56.974053] [SM Exiting]: (0x665210) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:56.974064] [SM Entering]: (0x665210) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:56.974071] [SM frame get]: (0x665210) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.975061] ========================================================= [D 20:00:56.975093] PINT_process_request [D 20:00:56.975100] tiling 7 copies [D 20:00:56.975107] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.975116] lvl 0 el 0 blk 0 by 178257920 [D 20:00:56.975123] to 178257920 ta 167772160 fi 201326592 [D 20:00:56.975130] basic type or contiguous data [D 20:00:56.975137] Request disp [D 20:00:56.975144] contig_offset = 178257920 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.975151] calling distribute [D 20:00:56.975157] PINT_distribute [D 20:00:56.975164] of 178257920 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 178257920 exfl 1 [D 20:00:56.975173] begin iteration loff: 178257920 [D 20:00:56.975180] segment exceeds byte limit [D 20:00:56.975186] file being extended [D 20:00:56.975192] process a segment [D 20:00:56.975198] of 178257920 sz 2097152 [D 20:00:56.975204] add a segment 0 [D 20:00:56.975210] end iteration [D 20:00:56.975216] done with segments or bytes [D 20:00:56.975222] of 180355072 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.975229] next loff: 180355072 next poff: 180355072 [D 20:00:56.975236] return value 2097152 [D 20:00:56.975243] finished [D 20:00:56.975249] retval = 2097152 [D 20:00:56.975255] exiting distribute returned less than expected [D 20:00:56.975262] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 180355072 fo 201326592 eof 0 [D 20:00:56.975270] ========================================================= [D 20:00:56.975277] BMI_post_recv: addr: 6843072, offset: 0x2aaab6228010, size: 2097152, tag: 13 [D 20:00:56.975293] openib_check_cq: completed send wr_id 683b00. [D 20:00:56.975304] openib_check_cq: completed recv wr_id 683c00. [D 20:00:56.975322] openib_post_rr: 10.1.4.56:39990 bh 3 wr_id 683c00. [D 20:00:56.975331] openib_check_cq: completed recv wr_id 683c30. [D 20:00:56.975341] openib_post_rr: 10.1.4.56:39990 bh 4 wr_id 683c30. [D 20:00:56.975366] post_sr: 10.1.4.56:39990 bh 99 len 48 wr_id 683b30. [D 20:00:56.975378] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:00:56.975389] get_data_sync_mode returning 0 [D 20:00:56.975402] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.975409] dbpf_bstream_rw_list: mem_offset: 0x2aaab4418010, mem_size: 2097152 [D 20:00:56.975416] dbpf_bstream_rw_list: stream_offset: 209715200, stream_size: 2097152 [D 20:00:56.975424] dbpf_open_cache_get: called [D 20:00:56.975431] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.975438] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.975460] DBPF I/O ops in progress: 6 [D 20:00:56.975468] lio_listio called with the following aiocbs: [D 20:00:56.975474] aiocb_ptr_array[0]: fd: 13, off: 209715200, bytes: 2097152, buf: 0x2aaab4418010, type: 1 [D 20:00:56.975488] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [D 20:00:56.976267] --- aio_progress_notification called with handle 2863311531 (0x6f0a90) [D 20:00:56.980489] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.980514] dbpf_open_cache_put: cached entry. [D 20:00:56.980523] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.976304] DBPF I/O ops in progress: 5 [D 20:00:56.979041] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:00:56.979101] ========================================================= [D 20:00:56.980619] PINT_process_request [D 20:00:56.980626] tiling 8 copies [D 20:00:56.980635] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.980643] lvl 0 el 0 blk 0 by 211812352 [D 20:00:56.980650] to 211812352 ta 201326592 fi 234881024 [D 20:00:56.980657] basic type or contiguous data [D 20:00:56.980664] Request disp [D 20:00:56.980671] contig_offset = 211812352 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.980678] calling distribute [D 20:00:56.980684] PINT_distribute [D 20:00:56.980690] of 211812352 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 211812352 exfl 1 [D 20:00:56.980699] begin iteration loff: 211812352 [D 20:00:56.980706] segment exceeds byte limit [D 20:00:56.980712] file being extended [D 20:00:56.980719] process a segment [D 20:00:56.980725] of 211812352 sz 2097152 [D 20:00:56.980732] add a segment 0 [D 20:00:56.980738] end iteration [D 20:00:56.980744] done with segments or bytes [D 20:00:56.980750] of 213909504 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.980757] next loff: 213909504 next poff: 213909504 [D 20:00:56.980763] return value 2097152 [D 20:00:56.980770] finished [D 20:00:56.980775] retval = 2097152 [D 20:00:56.980782] exiting distribute returned less than expected [D 20:00:56.980788] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 213909504 fo 234881024 eof 0 [D 20:00:56.980795] ========================================================= [D 20:00:56.980803] BMI_post_recv: addr: 6843072, offset: 0x2aaab642a010, size: 2097152, tag: 14 [D 20:00:56.980818] openib_check_cq: completed send wr_id 683b30. [D 20:00:56.980829] openib_check_cq: completed recv wr_id 683c60. [D 20:00:56.980846] openib_post_rr: 10.1.4.56:39990 bh 5 wr_id 683c60. [D 20:00:56.980855] openib_check_cq: completed recv wr_id 683c90. [D 20:00:56.980865] openib_post_rr: 10.1.4.56:39990 bh 6 wr_id 683c90. [D 20:00:56.980873] openib_check_cq: completed recv wr_id 683cc0. [D 20:00:56.980881] openib_post_rr: 10.1.4.56:39990 bh 7 wr_id 683cc0. [D 20:00:56.980906] post_sr: 10.1.4.56:39990 bh 0 len 48 wr_id 6828a0. [D 20:00:56.980932] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:00:56.980942] get_data_sync_mode returning 0 [D 20:00:56.980952] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.980960] dbpf_bstream_rw_list: mem_offset: 0x2aaab4619010, mem_size: 2097152 [D 20:00:56.980967] dbpf_bstream_rw_list: stream_offset: 243269632, stream_size: 2097152 [D 20:00:56.980974] dbpf_open_cache_get: called [D 20:00:56.980980] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.980987] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.981012] DBPF I/O ops in progress: 6 [D 20:00:56.981020] lio_listio called with the following aiocbs: [D 20:00:56.981026] aiocb_ptr_array[0]: fd: 13, off: 243269632, bytes: 2097152, buf: 0x2aaab4619010, type: 1 [D 20:00:56.981038] issue_or_delay_io_operation: lio_listio posted 0x6ed8e0 (handle 2863311531, ret 0) [D 20:00:56.984165] ========================================================= [D 20:00:56.984191] PINT_process_request [D 20:00:56.984200] tiling 9 copies [D 20:00:56.984207] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.984215] lvl 0 el 0 blk 0 by 245366784 [D 20:00:56.984221] to 245366784 ta 234881024 fi 268435456 [D 20:00:56.984230] basic type or contiguous data [D 20:00:56.984236] Request disp [D 20:00:56.984243] contig_offset = 245366784 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.984250] calling distribute [D 20:00:56.984257] PINT_distribute [D 20:00:56.984265] of 245366784 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 245366784 exfl 1 [D 20:00:56.984273] begin iteration loff: 245366784 [D 20:00:56.984281] segment exceeds byte limit [D 20:00:56.984298] file being extended [D 20:00:56.984305] process a segment [D 20:00:56.984311] of 245366784 sz 2097152 [D 20:00:56.984318] add a segment 0 [D 20:00:56.984325] end iteration [D 20:00:56.984331] done with segments or bytes [D 20:00:56.984337] of 247463936 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.984346] next loff: 247463936 next poff: 247463936 [D 20:00:56.984353] return value 2097152 [D 20:00:56.984360] finished [D 20:00:56.984366] retval = 2097152 [D 20:00:56.984372] exiting distribute returned less than expected [D 20:00:56.984378] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 247463936 fo 268435456 eof 0 [D 20:00:56.984385] ========================================================= [D 20:00:56.984416] BMI_post_recv: addr: 6843072, offset: 0x2aaab662b010, size: 2097152, tag: 15 [D 20:00:56.984435] openib_check_cq: completed send wr_id 6828a0. [D 20:00:56.984445] openib_check_cq: completed recv wr_id 683cf0. [D 20:00:56.984458] openib_post_rr: 10.1.4.56:39990 bh 8 wr_id 683cf0. [D 20:00:56.984487] post_sr: 10.1.4.56:39990 bh 1 len 48 wr_id 6828d0. [D 20:00:56.984501] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:00:56.984511] get_data_sync_mode returning 0 [D 20:00:56.984524] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.984531] dbpf_bstream_rw_list: mem_offset: 0x2aaab481a010, mem_size: 2097152 [D 20:00:56.984538] dbpf_bstream_rw_list: stream_offset: 276824064, stream_size: 2097152 [D 20:00:56.984545] dbpf_open_cache_get: called [D 20:00:56.984553] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.984560] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.984581] DBPF I/O ops in progress: 7 [D 20:00:56.984589] lio_listio called with the following aiocbs: [D 20:00:56.984595] aiocb_ptr_array[0]: fd: 13, off: 276824064, bytes: 2097152, buf: 0x2aaab481a010, type: 1 [D 20:00:56.984608] issue_or_delay_io_operation: lio_listio posted 0x6f0be0 (handle 2863311531, ret 0) [A 20:00:56.986562] troy.scl@10.1.4.56:39990 H=2863311531 S=0x665310: io: start [D 20:00:56.986627] (0x665310) io (prelude sm) state: getattr_if_needed [D 20:00:56.986635] About to retrieve attributes for handle 2863311531 [D 20:00:56.986656] op_queue add: 0x2aaab1509760 [D 20:00:56.987539] [SM Exiting]: (0x665210) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:56.987572] [SM Entering]: (0x6664d0) unexpected_sm:map_request (status: 0) [D 20:00:56.987588] [SM frame get]: (0x6664d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.987596] server_post_unexpected_recv [D 20:00:56.987611] [SM Frame PUSH]: (0x2aaab15099f0) frame: 0x2aaab150b220 [D 20:00:56.987619] [SM Locating]: (0x2aaab15099f0) op-id: 999 [D 20:00:56.987626] server_op_state_get_machine 999 [D 20:00:56.987632] [SM Locating]: (0x2aaab15099f0) located: unexpected_sm:post_unexpected [D 20:00:56.987639] [SM frame get]: (0x2aaab15099f0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.987647] [SM Entering]: (0x2aaab15099f0) unexpected_sm:post_unexpected (status: 0) [D 20:00:56.987654] [SM frame get]: (0x2aaab15099f0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.987664] [SM Exiting]: (0x2aaab15099f0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:56.987672] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:56.987679] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:56.987686] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 22 [D 20:00:56.987693] [SM frame get]: (0x6664d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:56.987700] server_state_machine_start 0x6664d0 [D 20:00:56.987707] PINT_decode [D 20:00:56.987715] lebf_decode_req [D 20:00:56.987729] [SM Locating]: (0x6664d0) op-id: 3 [D 20:00:56.987736] server_op_state_get_machine 3 [D 20:00:56.987743] [SM push_state]: (0x6664d0) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:56.987750] [SM Locating]: (0x6664d0) located: prelude_sm:setup [D 20:00:56.987758] [SM Entering]: (0x6664d0) prelude_sm:setup (status: 0) [D 20:00:56.987765] [SM frame get]: (0x6664d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.987773] [SM Exiting]: (0x6664d0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:56.987780] [SM Exiting]: (0x6664d0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:56.987791] [SM push_state]: (0x6664d0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:56.987798] [SM Entering]: (0x6664d0) prelude_work_sm:req_sched (status: 0) [D 20:00:56.987805] [SM frame get]: (0x6664d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.987812] (0x6665d0) io (prelude sm) state: req_sched [D 20:00:56.988064] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:56.988081] dbpf_open_cache_get: called [D 20:00:56.988090] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.988097] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.988106] dbpf_open_cache_put: cached entry. [D 20:00:56.988155] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 471859200 [D 20:00:56.988181] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 471859200) [D 20:00:56.988189] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:56.976278] --- aio_progress_notification called with handle 2863311531 (0x6f0f30) [D 20:00:56.988216] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.988253] ========================================================= [D 20:00:56.988262] PINT_process_request [D 20:00:56.988270] tiling 10 copies [D 20:00:56.988277] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.988285] lvl 0 el 0 blk 0 by 278921216 [D 20:00:56.988291] to 278921216 ta 268435456 fi 301989888 [D 20:00:56.988299] basic type or contiguous data [D 20:00:56.988306] Request disp [D 20:00:56.988312] contig_offset = 278921216 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.988320] calling distribute [D 20:00:56.988326] PINT_distribute [D 20:00:56.988332] of 278921216 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 278921216 exfl 1 [D 20:00:56.988342] begin iteration loff: 278921216 [D 20:00:56.988349] segment exceeds byte limit [D 20:00:56.988355] file being extended [D 20:00:56.988361] process a segment [D 20:00:56.988367] of 278921216 sz 2097152 [D 20:00:56.988373] add a segment 0 [D 20:00:56.988380] end iteration [D 20:00:56.988386] done with segments or bytes [D 20:00:56.988398] of 281018368 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.988406] next loff: 281018368 next poff: 281018368 [D 20:00:56.988410] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:00:56.988412] return value 2097152 [D 20:00:56.988435] finished [D 20:00:56.988441] retval = 2097152 [D 20:00:56.988447] exiting distribute returned less than expected [D 20:00:56.988453] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 281018368 fo 301989888 eof 0 [D 20:00:56.988461] ========================================================= [D 20:00:56.988469] BMI_post_recv: addr: 6843072, offset: 0x2aaab682c010, size: 2097152, tag: 16 [D 20:00:56.988486] openib_check_cq: completed send wr_id 6828d0. [D 20:00:56.988494] openib_check_cq: completed recv wr_id 683d20. [D 20:00:56.988511] openib_post_rr: 10.1.4.56:39990 bh 9 wr_id 683d20. [D 20:00:56.988520] openib_check_cq: completed recv wr_id 683d50. [D 20:00:56.988530] openib_post_rr: 10.1.4.56:39990 bh 10 wr_id 683d50. [D 20:00:56.988537] openib_check_cq: completed recv wr_id 683d80. [D 20:00:56.988546] openib_post_rr: 10.1.4.56:39990 bh 11 wr_id 683d80. [D 20:00:56.988572] post_sr: 10.1.4.56:39990 bh 2 len 48 wr_id 682900. [D 20:00:56.988585] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:00:56.988594] get_data_sync_mode returning 0 [D 20:00:56.988607] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.988614] dbpf_bstream_rw_list: mem_offset: 0x2aaab4a1b010, mem_size: 2097152 [D 20:00:56.988621] dbpf_bstream_rw_list: stream_offset: 310378496, stream_size: 2097152 [D 20:00:56.988629] dbpf_open_cache_get: called [D 20:00:56.980410] --- aio_progress_notification called with handle 2863311531 (0x6ecf80) [D 20:00:56.988648] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.980418] op_queue add: 0x6ec210 [D 20:00:56.988681] DBPF I/O ops in progress: 6 [D 20:00:56.988417] dbpf_open_cache_put: cached entry. [D 20:00:56.988739] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.988746] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.988755] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:56.988778] DBPF I/O ops in progress: 7 [D 20:00:56.988785] lio_listio called with the following aiocbs: [D 20:00:56.988792] aiocb_ptr_array[0]: fd: 13, off: 310378496, bytes: 2097152, buf: 0x2aaab4a1b010, type: 1 [D 20:00:56.988807] issue_or_delay_io_operation: lio_listio posted 0x6f0d30 (handle 2863311531, ret 0) [D 20:00:56.991859] ========================================================= [D 20:00:56.991891] PINT_process_request [D 20:00:56.991899] tiling 11 copies [D 20:00:56.991913] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.991922] lvl 0 el 0 blk 0 by 312475648 [D 20:00:56.991928] to 312475648 ta 301989888 fi 335544320 [D 20:00:56.991935] basic type or contiguous data [D 20:00:56.991942] Request disp [D 20:00:56.991949] contig_offset = 312475648 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.991956] calling distribute [D 20:00:56.991962] PINT_distribute [D 20:00:56.991969] of 312475648 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 312475648 exfl 1 [D 20:00:56.991979] begin iteration loff: 312475648 [D 20:00:56.991986] segment exceeds byte limit [D 20:00:56.991992] file being extended [D 20:00:56.991999] process a segment [D 20:00:56.992004] of 312475648 sz 2097152 [D 20:00:56.992011] add a segment 0 [D 20:00:56.992017] end iteration [D 20:00:56.992023] done with segments or bytes [D 20:00:56.992029] of 314572800 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.992036] next loff: 314572800 next poff: 314572800 [D 20:00:56.992043] return value 2097152 [D 20:00:56.992051] finished [D 20:00:56.992057] retval = 2097152 [D 20:00:56.992063] exiting distribute returned less than expected [D 20:00:56.992069] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 314572800 fo 335544320 eof 0 [D 20:00:56.992077] ========================================================= [D 20:00:56.992085] BMI_post_recv: addr: 6843072, offset: 0x2aaab6a2d010, size: 2097152, tag: 17 [D 20:00:56.992102] openib_check_cq: completed send wr_id 682900. [D 20:00:56.992111] openib_check_cq: completed recv wr_id 683db0. [D 20:00:56.992124] openib_post_rr: 10.1.4.56:39990 bh 12 wr_id 683db0. [D 20:00:56.992156] post_sr: 10.1.4.56:39990 bh 3 len 48 wr_id 682930. [D 20:00:56.992170] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:00:56.992181] get_data_sync_mode returning 0 [D 20:00:56.992196] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.992203] dbpf_bstream_rw_list: mem_offset: 0x2aaab4c1c010, mem_size: 2097152 [D 20:00:56.992210] dbpf_bstream_rw_list: stream_offset: 343932928, stream_size: 2097152 [D 20:00:56.992218] dbpf_open_cache_get: called [D 20:00:56.992226] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.992233] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.992254] DBPF I/O ops in progress: 8 [D 20:00:56.992261] lio_listio called with the following aiocbs: [D 20:00:56.992268] aiocb_ptr_array[0]: fd: 13, off: 343932928, bytes: 2097152, buf: 0x2aaab4c1c010, type: 1 [D 20:00:56.992281] issue_or_delay_io_operation: lio_listio posted 0x6ea020 (handle 2863311531, ret 0) [A 20:00:56.994323] troy.scl@10.1.4.56:39990 H=2863311531 S=0x6665d0: io: request [D 20:00:56.994400] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:56.994408] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab1509f40 [D 20:00:56.994415] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab1509f40 [D 20:00:56.994423] [SM Exiting]: (0x6664d0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:56.994436] [SM Entering]: (0x6664d0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:56.994444] [SM frame get]: (0x6664d0) op-id: 3 index: 0 base-frm: 0 [D 20:00:56.991817] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.995482] ========================================================= [D 20:00:56.995505] PINT_process_request [D 20:00:56.995513] tiling 12 copies [D 20:00:56.995539] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:56.995548] lvl 0 el 0 blk 0 by 346030080 [D 20:00:56.995555] to 346030080 ta 335544320 fi 369098752 [D 20:00:56.995563] basic type or contiguous data [D 20:00:56.995569] Request disp [D 20:00:56.995575] contig_offset = 346030080 contig_size = 56623104 lvl_flag = 1 [D 20:00:56.995583] calling distribute [D 20:00:56.995589] PINT_distribute [D 20:00:56.995596] of 346030080 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 346030080 exfl 1 [D 20:00:56.995604] begin iteration loff: 346030080 [D 20:00:56.995613] segment exceeds byte limit [D 20:00:56.995619] file being extended [D 20:00:56.995625] process a segment [D 20:00:56.995631] of 346030080 sz 2097152 [D 20:00:56.995638] add a segment 0 [D 20:00:56.995644] end iteration [D 20:00:56.995650] done with segments or bytes [D 20:00:56.995656] of 348127232 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:56.995663] next loff: 348127232 next poff: 348127232 [D 20:00:56.995670] return value 2097152 [D 20:00:56.995676] finished [D 20:00:56.995682] retval = 2097152 [D 20:00:56.995688] exiting distribute returned less than expected [D 20:00:56.995694] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 348127232 fo 369098752 eof 0 [D 20:00:56.995702] ========================================================= [D 20:00:56.995710] BMI_post_recv: addr: 6843072, offset: 0x2aaab6c2e010, size: 2097152, tag: 18 [D 20:00:56.995727] openib_check_cq: completed send wr_id 682930. [D 20:00:56.995735] openib_check_cq: completed recv wr_id 683de0. [D 20:00:56.995754] openib_post_rr: 10.1.4.56:39990 bh 13 wr_id 683de0. [D 20:00:56.995764] openib_check_cq: completed recv wr_id 683e10. [D 20:00:56.995773] openib_post_rr: 10.1.4.56:39990 bh 14 wr_id 683e10. [D 20:00:56.995800] post_sr: 10.1.4.56:39990 bh 4 len 48 wr_id 682960. [D 20:00:56.995814] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:00:56.995826] get_data_sync_mode returning 0 [D 20:00:56.995838] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:56.995846] dbpf_bstream_rw_list: mem_offset: 0x2aaab4e1d010, mem_size: 2097152 [D 20:00:56.995853] dbpf_bstream_rw_list: stream_offset: 377487360, stream_size: 2097152 [D 20:00:56.995860] dbpf_open_cache_get: called [D 20:00:56.995867] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:56.995874] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:56.995895] DBPF I/O ops in progress: 9 [D 20:00:56.995902] lio_listio called with the following aiocbs: [D 20:00:56.995909] aiocb_ptr_array[0]: fd: 13, off: 377487360, bytes: 2097152, buf: 0x2aaab4e1d010, type: 1 [D 20:00:56.995921] issue_or_delay_io_operation: lio_listio posted 0x6ea170 (handle 2863311531, ret 0) [D 20:00:56.998174] --- aio_progress_notification called with handle 2863311531 (0x6ed8e0) [D 20:00:56.998252] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.998268] dbpf_open_cache_put: cached entry. [D 20:00:56.998276] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.998294] --- aio_progress_notification called with handle 2863311531 (0x6f0be0) [D 20:00:56.998306] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.998318] dbpf_open_cache_put: cached entry. [D 20:00:56.998324] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.988427] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:56.998452] dbpf_open_cache_put: cached entry. [D 20:00:56.998459] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:56.995464] op_queue add: 0x6f0f30 [D 20:00:56.998553] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:56.998580] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:56.998602] op_queue add: 0x2aaab1509760 [D 20:00:56.998636] DBPF I/O ops in progress: 8 [D 20:00:56.998821] op_queue add: 0x6f0a90 [D 20:00:56.998835] dbpf_open_cache_put: cached entry. [D 20:00:56.998806] ========================================================= [D 20:00:57.001674] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.001749] PINT_process_request [D 20:00:57.001759] tiling 13 copies [D 20:00:57.001767] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.001775] lvl 0 el 0 blk 0 by 379584512 [D 20:00:57.001781] to 379584512 ta 369098752 fi 402653184 [D 20:00:57.001788] basic type or contiguous data [D 20:00:57.001795] Request disp [D 20:00:57.001801] contig_offset = 379584512 contig_size = 56623104 lvl_flag = 1 [D 20:00:57.001808] calling distribute [D 20:00:57.001814] PINT_distribute [D 20:00:57.001821] of 379584512 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 379584512 exfl 1 [D 20:00:57.001829] begin iteration loff: 379584512 [D 20:00:57.001836] segment exceeds byte limit [D 20:00:57.001842] file being extended [D 20:00:57.001848] process a segment [D 20:00:57.001854] of 379584512 sz 2097152 [D 20:00:57.001860] add a segment 0 [D 20:00:57.001867] end iteration [D 20:00:57.001872] done with segments or bytes [D 20:00:57.001878] of 381681664 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.001885] next loff: 381681664 next poff: 381681664 [D 20:00:57.001892] return value 2097152 [D 20:00:57.001898] finished [D 20:00:57.001904] retval = 2097152 [D 20:00:57.001910] exiting distribute returned less than expected [D 20:00:57.001916] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 381681664 fo 402653184 eof 0 [D 20:00:57.001924] ========================================================= [D 20:00:57.001931] BMI_post_recv: addr: 6843072, offset: 0x2aaab6e2f010, size: 2097152, tag: 19 [D 20:00:57.001944] openib_check_cq: completed send wr_id 682960. [D 20:00:57.001953] openib_check_cq: completed recv wr_id 683e40. [D 20:00:57.001966] openib_post_rr: 10.1.4.56:39990 bh 15 wr_id 683e40. [D 20:00:57.001975] openib_check_cq: completed recv wr_id 683e70. [D 20:00:57.001984] openib_post_rr: 10.1.4.56:39990 bh 16 wr_id 683e70. [D 20:00:57.001991] openib_check_cq: completed recv wr_id 683ea0. [D 20:00:57.001999] openib_post_rr: 10.1.4.56:39990 bh 17 wr_id 683ea0. [D 20:00:57.002019] post_sr: 10.1.4.56:39990 bh 5 len 48 wr_id 682990. [D 20:00:57.002039] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:00:57.002048] get_data_sync_mode returning 0 [D 20:00:57.002058] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.002066] dbpf_bstream_rw_list: mem_offset: 0x2aaab501e010, mem_size: 2097152 [D 20:00:57.002072] dbpf_bstream_rw_list: stream_offset: 411041792, stream_size: 2097152 [D 20:00:57.002079] dbpf_open_cache_get: called [D 20:00:57.002086] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.002093] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.002109] DBPF I/O ops in progress: 9 [D 20:00:57.002116] lio_listio called with the following aiocbs: [D 20:00:57.002123] aiocb_ptr_array[0]: fd: 13, off: 411041792, bytes: 2097152, buf: 0x2aaab501e010, type: 1 [D 20:00:57.001738] --- aio_progress_notification called with handle 2863311531 (0x6f0d30) [D 20:00:57.002186] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.002204] dbpf_open_cache_put: cached entry. [D 20:00:57.002223] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.002214] --- aio_progress_notification called with handle 2863311531 (0x6ea020) [D 20:00:57.002253] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.002264] dbpf_open_cache_put: cached entry. [D 20:00:57.002270] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.002289] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [D 20:00:57.002138] op_queue add: 0x6ecf80 [D 20:00:57.002384] DBPF I/O ops in progress: 8 [A 20:00:57.004787] troy.scl@10.1.4.56:39990 H=2863311531 S=0x6665d0: io: start [D 20:00:57.004840] (0x6665d0) io (prelude sm) state: getattr_if_needed [D 20:00:57.004848] About to retrieve attributes for handle 2863311531 [D 20:00:57.004869] op_queue add: 0x2aaab1509c30 [D 20:00:57.005257] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:00:57.005276] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:00:57.005309] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:57.005331] dbpf_open_cache_get: called [D 20:00:57.005339] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.005346] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.005356] dbpf_open_cache_put: cached entry. [D 20:00:57.005404] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 471859200 [D 20:00:57.005417] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 471859200) [D 20:00:57.005425] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:57.005432] op_queue add: 0x2aaab1509c30 [D 20:00:57.005451] [SM Exiting]: (0x6664d0) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:57.005465] [SM Entering]: (0x667790) unexpected_sm:map_request (status: 0) [D 20:00:57.005474] [SM frame get]: (0x667790) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.005482] server_post_unexpected_recv [D 20:00:57.005491] [SM Frame PUSH]: (0x2aaab1509c30) frame: 0x2aaab150c280 [D 20:00:57.005499] [SM Locating]: (0x2aaab1509c30) op-id: 999 [D 20:00:57.005506] server_op_state_get_machine 999 [D 20:00:57.005513] [SM Locating]: (0x2aaab1509c30) located: unexpected_sm:post_unexpected [D 20:00:57.005520] [SM frame get]: (0x2aaab1509c30) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.005527] [SM Entering]: (0x2aaab1509c30) unexpected_sm:post_unexpected (status: 0) [D 20:00:57.005535] [SM frame get]: (0x2aaab1509c30) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.005544] [SM Exiting]: (0x2aaab1509c30) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:57.005552] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:57.005560] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:57.005567] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 23 [D 20:00:57.005575] [SM frame get]: (0x667790) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.005582] server_state_machine_start 0x667790 [D 20:00:57.005589] PINT_decode [D 20:00:57.005597] lebf_decode_req [D 20:00:57.005610] [SM Locating]: (0x667790) op-id: 3 [D 20:00:57.005617] server_op_state_get_machine 3 [D 20:00:57.005625] [SM push_state]: (0x667790) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:57.005632] [SM Locating]: (0x667790) located: prelude_sm:setup [D 20:00:57.005639] [SM Entering]: (0x667790) prelude_sm:setup (status: 0) [D 20:00:57.005646] [SM frame get]: (0x667790) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.005654] [SM Exiting]: (0x667790) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:57.005661] [SM Exiting]: (0x667790) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:57.005669] [SM push_state]: (0x667790) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:57.005676] [SM Entering]: (0x667790) prelude_work_sm:req_sched (status: 0) [D 20:00:57.005683] [SM frame get]: (0x667790) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.309381] (0x667890) io (prelude sm) state: req_sched [D 20:00:57.005702] DBPF I/O ops in progress: 7 [D 20:00:57.005747] ========================================================= [D 20:00:57.310727] PINT_process_request [D 20:00:57.310739] tiling 14 copies [D 20:00:57.310747] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.310755] lvl 0 el 0 blk 0 by 413138944 [D 20:00:57.310761] to 413138944 ta 402653184 fi 436207616 [D 20:00:57.310769] basic type or contiguous data [D 20:00:57.310775] Request disp [D 20:00:57.310781] contig_offset = 413138944 contig_size = 56623104 lvl_flag = 1 [D 20:00:57.310788] calling distribute [D 20:00:57.310794] PINT_distribute [D 20:00:57.310801] of 413138944 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 413138944 exfl 1 [D 20:00:57.310809] begin iteration loff: 413138944 [D 20:00:57.310816] segment exceeds byte limit [D 20:00:57.310822] file being extended [D 20:00:57.310828] process a segment [D 20:00:57.310834] of 413138944 sz 2097152 [D 20:00:57.310840] add a segment 0 [D 20:00:57.310847] end iteration [D 20:00:57.310853] done with segments or bytes [D 20:00:57.310863] of 415236096 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.310871] next loff: 415236096 next poff: 415236096 [D 20:00:57.310877] return value 2097152 [D 20:00:57.310883] finished [D 20:00:57.310889] retval = 2097152 [D 20:00:57.310895] exiting distribute returned less than expected [D 20:00:57.310902] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 415236096 fo 436207616 eof 0 [D 20:00:57.310909] ========================================================= [D 20:00:57.310916] BMI_post_recv: addr: 6843072, offset: 0x2aaab7030010, size: 2097152, tag: 20 [D 20:00:57.310930] openib_check_cq: completed send wr_id 682990. [D 20:00:57.310939] openib_check_cq: completed recv wr_id 683ed0. [D 20:00:57.310949] openib_post_rr: 10.1.4.56:39990 bh 18 wr_id 683ed0. [D 20:00:57.310958] openib_check_cq: completed recv wr_id 683f00. [D 20:00:57.310968] openib_post_rr: 10.1.4.56:39990 bh 19 wr_id 683f00. [D 20:00:57.310988] post_sr: 10.1.4.56:39990 bh 6 len 48 wr_id 6829c0. [D 20:00:57.310999] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:00:57.311008] get_data_sync_mode returning 0 [D 20:00:57.311019] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.311026] dbpf_bstream_rw_list: mem_offset: 0x2aaab5420010, mem_size: 2097152 [D 20:00:57.311033] dbpf_bstream_rw_list: stream_offset: 444596224, stream_size: 2097152 [D 20:00:57.311040] dbpf_open_cache_get: called [D 20:00:57.311047] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.311053] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.311061] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:57.311082] DBPF I/O ops in progress: 8 [D 20:00:57.311090] lio_listio called with the following aiocbs: [D 20:00:57.311096] aiocb_ptr_array[0]: fd: 13, off: 444596224, bytes: 2097152, buf: 0x2aaab5420010, type: 1 [D 20:00:57.311109] issue_or_delay_io_operation: lio_listio posted 0x6f0f00 (handle 2863311531, ret 0) [D 20:00:57.005726] op_queue add: 0x6f0be0 [D 20:00:57.314208] ========================================================= [D 20:00:57.314247] PINT_process_request [D 20:00:57.314257] tiling 15 copies [D 20:00:57.314265] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.314273] lvl 0 el 0 blk 0 by 446693376 [D 20:00:57.314280] to 446693376 ta 436207616 fi 469762048 [D 20:00:57.314287] basic type or contiguous data [D 20:00:57.314294] Request disp [D 20:00:57.314300] contig_offset = 446693376 contig_size = 56623104 lvl_flag = 1 [D 20:00:57.314307] calling distribute [D 20:00:57.314314] PINT_distribute [D 20:00:57.314321] of 446693376 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 446693376 exfl 1 [D 20:00:57.314330] begin iteration loff: 446693376 [D 20:00:57.314338] segment exceeds byte limit [D 20:00:57.314345] file being extended [D 20:00:57.314352] process a segment [D 20:00:57.314358] of 446693376 sz 2097152 [D 20:00:57.314364] add a segment 0 [D 20:00:57.314370] end iteration [D 20:00:57.314377] done with segments or bytes [D 20:00:57.314383] of 448790528 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.314390] next loff: 448790528 next poff: 448790528 [D 20:00:57.314397] return value 2097152 [D 20:00:57.314405] finished [D 20:00:57.314411] retval = 2097152 [D 20:00:57.314417] exiting distribute returned less than expected [D 20:00:57.314423] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 448790528 fo 469762048 eof 0 [D 20:00:57.314431] ========================================================= [D 20:00:57.314438] BMI_post_recv: addr: 6843072, offset: 0x2aaab7231010, size: 2097152, tag: 21 [D 20:00:57.314455] openib_check_cq: completed send wr_id 6829c0. [D 20:00:57.314466] openib_check_cq: completed recv wr_id 683f30. [D 20:00:57.314478] openib_post_rr: 10.1.4.56:39990 bh 20 wr_id 683f30. [D 20:00:57.314508] post_sr: 10.1.4.56:39990 bh 7 len 48 wr_id 6829f0. [D 20:00:57.314520] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:00:57.314530] get_data_sync_mode returning 0 [D 20:00:57.314544] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.314552] dbpf_bstream_rw_list: mem_offset: 0x2aaab5621010, mem_size: 2097152 [D 20:00:57.314559] dbpf_bstream_rw_list: stream_offset: 10485760, stream_size: 2097152 [D 20:00:57.314567] dbpf_open_cache_get: called [D 20:00:57.314575] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.314582] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.314603] DBPF I/O ops in progress: 9 [D 20:00:57.314611] lio_listio called with the following aiocbs: [D 20:00:57.314617] aiocb_ptr_array[0]: fd: 13, off: 10485760, bytes: 2097152, buf: 0x2aaab5621010, type: 1 [D 20:00:57.314632] issue_or_delay_io_operation: lio_listio posted 0x6ecf80 (handle 2863311531, ret 0) [A 20:00:57.316760] troy.scl@10.1.4.56:39990 H=2863311531 S=0x667890: io: request [D 20:00:57.316832] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:57.316841] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab150d510 [D 20:00:57.316847] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab150d510 [D 20:00:57.316856] [SM Exiting]: (0x667790) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:57.316869] [SM Entering]: (0x667790) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:57.316876] [SM frame get]: (0x667790) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.318831] ========================================================= [D 20:00:57.318867] PINT_process_request [D 20:00:57.318877] tiling 2 copies [D 20:00:57.318885] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.318893] lvl 0 el 0 blk 0 by 12582912 [D 20:00:57.318900] to 12582912 ta 0 fi 33554432 [D 20:00:57.318907] basic type or contiguous data [D 20:00:57.318914] Request disp [D 20:00:57.318921] contig_offset = 12582912 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.318929] calling distribute [D 20:00:57.318935] PINT_distribute [D 20:00:57.318941] of 12582912 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 12582912 exfl 1 [D 20:00:57.318951] begin iteration loff: 12582912 [D 20:00:57.318958] segment exceeds byte limit [D 20:00:57.318964] file being extended [D 20:00:57.318971] process a segment [D 20:00:57.318977] of 12582912 sz 2097152 [D 20:00:57.318983] add a segment 0 [D 20:00:57.318990] end iteration [D 20:00:57.318996] done with segments or bytes [D 20:00:57.319002] of 14680064 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.319009] next loff: 14680064 next poff: 14680064 [D 20:00:57.319015] return value 2097152 [D 20:00:57.319023] finished [D 20:00:57.319029] retval = 2097152 [D 20:00:57.319035] exiting distribute returned less than expected [D 20:00:57.319041] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 14680064 fo 33554432 eof 0 [D 20:00:57.319050] ========================================================= [D 20:00:57.319057] BMI_post_recv: addr: 6843072, offset: 0x2aaab7432010, size: 2097152, tag: 8 [D 20:00:57.319076] openib_check_cq: completed send wr_id 6829f0. [D 20:00:57.319085] openib_check_cq: completed recv wr_id 683f60. [D 20:00:57.319103] openib_post_rr: 10.1.4.56:39990 bh 21 wr_id 683f60. [D 20:00:57.319113] openib_check_cq: completed recv wr_id 683f90. [D 20:00:57.319121] openib_post_rr: 10.1.4.56:39990 bh 22 wr_id 683f90. [D 20:00:57.319129] openib_check_cq: completed recv wr_id 683fc0. [D 20:00:57.319137] openib_post_rr: 10.1.4.56:39990 bh 23 wr_id 683fc0. [D 20:00:57.319164] post_sr: 10.1.4.56:39990 bh 8 len 48 wr_id 682a20. [D 20:00:57.319176] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:00:57.319187] get_data_sync_mode returning 0 [D 20:00:57.319200] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.319208] dbpf_bstream_rw_list: mem_offset: 0x2aaab5822010, mem_size: 2097152 [D 20:00:57.319215] dbpf_bstream_rw_list: stream_offset: 44040192, stream_size: 2097152 [D 20:00:57.319224] dbpf_open_cache_get: called [D 20:00:57.319231] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.319237] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.319264] DBPF I/O ops in progress: 10 [D 20:00:57.319271] lio_listio called with the following aiocbs: [D 20:00:57.319278] aiocb_ptr_array[0]: fd: 13, off: 44040192, bytes: 2097152, buf: 0x2aaab5822010, type: 1 [D 20:00:57.319294] issue_or_delay_io_operation: lio_listio posted 0x6ec210 (handle 2863311531, ret 0) [D 20:00:57.319620] --- aio_progress_notification called with handle 2863311531 (0x6f0f00) [D 20:00:57.319642] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.319655] dbpf_open_cache_put: cached entry. [D 20:00:57.319663] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.008421] --- aio_progress_notification called with handle 2863311531 (0x6ea170) [D 20:00:57.321014] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.321421] dbpf_open_cache_put: cached entry. [D 20:00:57.321428] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.321435] op_queue add: 0x6ea170 [D 20:00:57.321444] DBPF I/O ops in progress: 9 [D 20:00:57.321321] ========================================================= [D 20:00:57.009379] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:00:57.321490] PINT_process_request [D 20:00:57.321504] op_queue add: 0x6ea020 [D 20:00:57.321505] tiling 3 copies [D 20:00:57.321493] DBPF I/O ops in progress: 8 [D 20:00:57.321527] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.321544] lvl 0 el 0 blk 0 by 46137344 [D 20:00:57.321548] DBPF I/O ops in progress: 7 [D 20:00:57.321553] to 46137344 ta 33554432 fi 67108864 [D 20:00:57.321546] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.321570] basic type or contiguous data [D 20:00:57.321583] dbpf_open_cache_put: cached entry. [D 20:00:57.321555] op_queue add: 0x6ed8e0 [D 20:00:57.321611] DBPF I/O ops in progress: 6 [D 20:00:57.321620] op_queue add: 0x6f0f00 [D 20:00:57.321588] Request disp [D 20:00:57.321656] contig_offset = 46137344 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.321664] calling distribute [D 20:00:57.321646] op_queue add: 0x6f0d30 [D 20:00:57.321670] PINT_distribute [D 20:00:57.321595] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.321691] of 46137344 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 46137344 exfl 1 [D 20:00:57.321685] DBPF I/O ops in progress: 5 [D 20:00:57.321708] begin iteration loff: 46137344 [D 20:00:57.321698] op_queue add: 0x6eda80 [D 20:00:57.321749] DBPF I/O ops in progress: 4 [D 20:00:57.321761] op_queue add: 0x6ec5d0 [D 20:00:57.321771] DBPF I/O ops in progress: 3 [D 20:00:57.321717] segment exceeds byte limit [D 20:00:57.321791] file being extended [D 20:00:57.321801] process a segment [D 20:00:57.321807] of 46137344 sz 2097152 [D 20:00:57.321813] add a segment 0 [D 20:00:57.321820] end iteration [D 20:00:57.321826] done with segments or bytes [D 20:00:57.321831] of 48234496 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.321838] next loff: 48234496 next poff: 48234496 [D 20:00:57.321845] return value 2097152 [D 20:00:57.321851] finished [D 20:00:57.321857] retval = 2097152 [D 20:00:57.321863] exiting distribute returned less than expected [D 20:00:57.321870] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 48234496 fo 67108864 eof 0 [D 20:00:57.321877] ========================================================= [D 20:00:57.321885] BMI_post_recv: addr: 6843072, offset: 0x2aaab7633010, size: 2097152, tag: 9 [D 20:00:57.321898] openib_check_cq: completed send wr_id 682a20. [D 20:00:57.321906] openib_check_cq: completed recv wr_id 683ff0. [D 20:00:57.321918] openib_post_rr: 10.1.4.56:39990 bh 24 wr_id 683ff0. [D 20:00:57.321942] post_sr: 10.1.4.56:39990 bh 9 len 48 wr_id 682a50. [D 20:00:57.321954] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:00:57.321964] get_data_sync_mode returning 0 [D 20:00:57.321973] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.321980] dbpf_bstream_rw_list: mem_offset: 0x2aaab5a23010, mem_size: 2097152 [D 20:00:57.321987] dbpf_bstream_rw_list: stream_offset: 77594624, stream_size: 2097152 [D 20:00:57.321994] dbpf_open_cache_get: called [D 20:00:57.322001] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.322007] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.322028] DBPF I/O ops in progress: 4 [D 20:00:57.322036] lio_listio called with the following aiocbs: [D 20:00:57.322042] aiocb_ptr_array[0]: fd: 13, off: 77594624, bytes: 2097152, buf: 0x2aaab5a23010, type: 1 [D 20:00:57.322052] issue_or_delay_io_operation: lio_listio posted 0x6f0a90 (handle 2863311531, ret 0) [D 20:00:57.322085] DBPF I/O ops in progress: 3 [D 20:00:57.325033] --- aio_progress_notification called with handle 2863311531 (0x6ecf80) [D 20:00:57.326361] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.326386] dbpf_open_cache_put: cached entry. [D 20:00:57.326393] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.326402] op_queue add: 0x6ecf80 [D 20:00:57.326409] DBPF I/O ops in progress: 2 [D 20:00:57.326324] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:00:57.326471] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:00:57.326481] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:00:57.326489] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:00:57.326496] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:00:57.326510] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:00:57.326518] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:00:57.326526] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:00:57.326533] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:00:57.326595] ========================================================= [D 20:00:57.326615] PINT_process_request [D 20:00:57.326622] tiling 4 copies [D 20:00:57.326630] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.326638] lvl 0 el 0 blk 0 by 79691776 [D 20:00:57.326644] to 79691776 ta 67108864 fi 100663296 [D 20:00:57.326652] basic type or contiguous data [D 20:00:57.326658] Request disp [D 20:00:57.326664] contig_offset = 79691776 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.326671] calling distribute [D 20:00:57.326678] PINT_distribute [D 20:00:57.326684] of 79691776 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 79691776 exfl 1 [D 20:00:57.326692] begin iteration loff: 79691776 [D 20:00:57.326699] segment exceeds byte limit [D 20:00:57.326705] file being extended [D 20:00:57.326712] process a segment [D 20:00:57.326718] of 79691776 sz 2097152 [D 20:00:57.326724] add a segment 0 [D 20:00:57.326731] end iteration [D 20:00:57.326736] done with segments or bytes [D 20:00:57.326742] of 81788928 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.326750] next loff: 81788928 next poff: 81788928 [D 20:00:57.326756] return value 2097152 [D 20:00:57.326763] finished [D 20:00:57.326769] retval = 2097152 [D 20:00:57.326775] exiting distribute returned less than expected [D 20:00:57.326782] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 81788928 fo 100663296 eof 0 [D 20:00:57.326789] ========================================================= [D 20:00:57.326796] BMI_post_recv: addr: 6843072, offset: 0x2aaab7834010, size: 2097152, tag: 10 [D 20:00:57.326814] openib_check_cq: completed send wr_id 682a50. [D 20:00:57.326823] openib_check_cq: completed recv wr_id 684020. [D 20:00:57.326841] openib_post_rr: 10.1.4.56:39990 bh 25 wr_id 684020. [D 20:00:57.326851] openib_check_cq: completed recv wr_id 684050. [D 20:00:57.326861] openib_post_rr: 10.1.4.56:39990 bh 26 wr_id 684050. [D 20:00:57.326869] openib_check_cq: completed recv wr_id 684080. [D 20:00:57.326877] openib_post_rr: 10.1.4.56:39990 bh 27 wr_id 684080. [D 20:00:57.326897] post_sr: 10.1.4.56:39990 bh 10 len 48 wr_id 682a80. [D 20:00:57.326914] openib_check_cq: completed send wr_id 682a80. [D 20:00:57.326930] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:00:57.326939] get_data_sync_mode returning 0 [D 20:00:57.326951] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.326958] dbpf_bstream_rw_list: mem_offset: 0x2aaab5e26010, mem_size: 2097152 [D 20:00:57.326966] dbpf_bstream_rw_list: stream_offset: 111149056, stream_size: 2097152 [D 20:00:57.326973] dbpf_open_cache_get: called [D 20:00:57.326981] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.326987] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.327016] DBPF I/O ops in progress: 3 [D 20:00:57.327023] lio_listio called with the following aiocbs: [D 20:00:57.327029] aiocb_ptr_array[0]: fd: 13, off: 111149056, bytes: 2097152, buf: 0x2aaab5e26010, type: 1 [D 20:00:57.327042] issue_or_delay_io_operation: lio_listio posted 0x6ed8e0 (handle 2863311531, ret 0) [A 20:00:57.328999] troy.scl@10.1.4.56:39990 H=2863311531 S=0x667890: io: start [D 20:00:57.329052] (0x667890) io (prelude sm) state: getattr_if_needed [D 20:00:57.329062] About to retrieve attributes for handle 2863311531 [D 20:00:57.329083] op_queue add: 0x2aaab150d6d0 [D 20:00:57.329910] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:57.329949] dbpf_open_cache_get: called [D 20:00:57.329968] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.329975] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.329986] dbpf_open_cache_put: cached entry. [D 20:00:57.330033] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 471859200 [D 20:00:57.330046] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 471859200) [D 20:00:57.330053] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:57.330061] op_queue add: 0x2aaab150d6d0 [D 20:00:57.330092] [SM Exiting]: (0x667790) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:57.330106] [SM Entering]: (0x668a50) unexpected_sm:map_request (status: 0) [D 20:00:57.330114] [SM frame get]: (0x668a50) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.330122] server_post_unexpected_recv [D 20:00:57.330135] [SM Frame PUSH]: (0x2aaab150d6d0) frame: 0x2aaab150d7d0 [D 20:00:57.330143] [SM Locating]: (0x2aaab150d6d0) op-id: 999 [D 20:00:57.330150] server_op_state_get_machine 999 [D 20:00:57.330157] [SM Locating]: (0x2aaab150d6d0) located: unexpected_sm:post_unexpected [D 20:00:57.330164] [SM frame get]: (0x2aaab150d6d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.330172] [SM Entering]: (0x2aaab150d6d0) unexpected_sm:post_unexpected (status: 0) [D 20:00:57.330179] [SM frame get]: (0x2aaab150d6d0) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.330189] [SM Exiting]: (0x2aaab150d6d0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:57.330197] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:57.330204] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:57.330211] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 24 [D 20:00:57.330218] [SM frame get]: (0x668a50) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.330225] server_state_machine_start 0x668a50 [D 20:00:57.330232] PINT_decode [D 20:00:57.330241] lebf_decode_req [D 20:00:57.330255] [SM Locating]: (0x668a50) op-id: 3 [D 20:00:57.330263] server_op_state_get_machine 3 [D 20:00:57.330270] [SM push_state]: (0x668a50) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:57.330277] [SM Locating]: (0x668a50) located: prelude_sm:setup [D 20:00:57.330284] [SM Entering]: (0x668a50) prelude_sm:setup (status: 0) [D 20:00:57.330291] [SM frame get]: (0x668a50) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.330300] [SM Exiting]: (0x668a50) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:57.330307] [SM Exiting]: (0x668a50) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:57.330315] [SM push_state]: (0x668a50) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:57.330322] [SM Entering]: (0x668a50) prelude_work_sm:req_sched (status: 0) [D 20:00:57.330329] [SM frame get]: (0x668a50) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.330336] (0x668b50) io (prelude sm) state: req_sched [D 20:00:57.330628] ========================================================= [D 20:00:57.330642] PINT_process_request [D 20:00:57.330650] tiling 5 copies [D 20:00:57.330657] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.330665] lvl 0 el 0 blk 0 by 113246208 [D 20:00:57.330672] to 113246208 ta 100663296 fi 134217728 [D 20:00:57.330679] basic type or contiguous data [D 20:00:57.330686] Request disp [D 20:00:57.330693] contig_offset = 113246208 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.330700] calling distribute [D 20:00:57.330706] PINT_distribute [D 20:00:57.330713] of 113246208 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 113246208 exfl 1 [D 20:00:57.330733] begin iteration loff: 113246208 [D 20:00:57.330741] segment exceeds byte limit [D 20:00:57.330747] file being extended [D 20:00:57.330754] process a segment [D 20:00:57.330760] of 113246208 sz 2097152 [D 20:00:57.330767] add a segment 0 [D 20:00:57.330774] end iteration [D 20:00:57.330780] done with segments or bytes [D 20:00:57.330785] of 115343360 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.330792] next loff: 115343360 next poff: 115343360 [D 20:00:57.330799] return value 2097152 [D 20:00:57.330805] finished [D 20:00:57.330811] retval = 2097152 [D 20:00:57.330817] exiting distribute returned less than expected [D 20:00:57.330824] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 115343360 fo 134217728 eof 0 [D 20:00:57.330831] ========================================================= [D 20:00:57.330839] BMI_post_recv: addr: 6843072, offset: 0x2aaab7a35010, size: 2097152, tag: 11 [D 20:00:57.330855] openib_check_cq: completed recv wr_id 6840b0. [D 20:00:57.330871] openib_post_rr: 10.1.4.56:39990 bh 28 wr_id 6840b0. [D 20:00:57.330881] openib_check_cq: completed recv wr_id 6840e0. [D 20:00:57.330892] openib_post_rr: 10.1.4.56:39990 bh 29 wr_id 6840e0. [D 20:00:57.330916] post_sr: 10.1.4.56:39990 bh 11 len 48 wr_id 682ab0. [D 20:00:57.330929] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:00:57.330940] get_data_sync_mode returning 0 [D 20:00:57.330951] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.330959] dbpf_bstream_rw_list: mem_offset: 0x2aaab6027010, mem_size: 2097152 [D 20:00:57.330966] dbpf_bstream_rw_list: stream_offset: 144703488, stream_size: 2097152 [D 20:00:57.330974] dbpf_open_cache_get: called [D 20:00:57.330980] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.330986] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.330996] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:57.331019] DBPF I/O ops in progress: 4 [D 20:00:57.331027] lio_listio called with the following aiocbs: [D 20:00:57.331033] aiocb_ptr_array[0]: fd: 13, off: 144703488, bytes: 2097152, buf: 0x2aaab6027010, type: 1 [D 20:00:57.331047] issue_or_delay_io_operation: lio_listio posted 0x6f0f00 (handle 2863311531, ret 0) [D 20:00:57.333820] ========================================================= [D 20:00:57.333880] PINT_process_request [D 20:00:57.333889] tiling 6 copies [D 20:00:57.333896] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.333905] lvl 0 el 0 blk 0 by 146800640 [D 20:00:57.333912] to 146800640 ta 134217728 fi 167772160 [D 20:00:57.333919] basic type or contiguous data [D 20:00:57.333925] Request disp [D 20:00:57.333931] contig_offset = 146800640 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.333939] calling distribute [D 20:00:57.333945] PINT_distribute [D 20:00:57.333953] of 146800640 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 146800640 exfl 1 [D 20:00:57.333961] begin iteration loff: 146800640 [D 20:00:57.334262] segment exceeds byte limit [D 20:00:57.334275] file being extended [D 20:00:57.334282] process a segment [D 20:00:57.334288] of 146800640 sz 2097152 [D 20:00:57.334295] add a segment 0 [D 20:00:57.334302] end iteration [D 20:00:57.334308] done with segments or bytes [D 20:00:57.334314] of 148897792 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.334322] next loff: 148897792 next poff: 148897792 [D 20:00:57.334329] return value 2097152 [D 20:00:57.334335] finished [D 20:00:57.334341] retval = 2097152 [D 20:00:57.334355] exiting distribute returned less than expected [D 20:00:57.334362] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 148897792 fo 167772160 eof 0 [D 20:00:57.334369] ========================================================= [D 20:00:57.334377] BMI_post_recv: addr: 6843072, offset: 0x2aaab7c36010, size: 2097152, tag: 12 [D 20:00:57.334395] openib_check_cq: completed send wr_id 682ab0. [D 20:00:57.334406] openib_check_cq: completed recv wr_id 684110. [D 20:00:57.334419] openib_post_rr: 10.1.4.56:39990 bh 30 wr_id 684110. [D 20:00:57.334450] post_sr: 10.1.4.56:39990 bh 12 len 48 wr_id 682ae0. [D 20:00:57.334465] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:00:57.334476] get_data_sync_mode returning 0 [D 20:00:57.334489] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.334496] dbpf_bstream_rw_list: mem_offset: 0x2aaab6228010, mem_size: 2097152 [D 20:00:57.334504] dbpf_bstream_rw_list: stream_offset: 178257920, stream_size: 2097152 [D 20:00:57.334512] dbpf_open_cache_get: called [D 20:00:57.334518] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.334525] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.334549] DBPF I/O ops in progress: 5 [D 20:00:57.334556] lio_listio called with the following aiocbs: [D 20:00:57.334563] aiocb_ptr_array[0]: fd: 13, off: 178257920, bytes: 2097152, buf: 0x2aaab6228010, type: 1 [D 20:00:57.334576] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [A 20:00:57.336756] troy.scl@10.1.4.56:39990 H=2863311531 S=0x668b50: io: request [D 20:00:57.336840] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:57.336848] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab1509e80 [D 20:00:57.336855] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab1509e80 [D 20:00:57.336862] [SM Exiting]: (0x668a50) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:57.336875] [SM Entering]: (0x668a50) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:57.336882] [SM frame get]: (0x668a50) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.337762] ========================================================= [D 20:00:57.337783] PINT_process_request [D 20:00:57.337792] tiling 7 copies [D 20:00:57.337799] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.337807] lvl 0 el 0 blk 0 by 180355072 [D 20:00:57.337814] to 180355072 ta 167772160 fi 201326592 [D 20:00:57.337821] basic type or contiguous data [D 20:00:57.337828] Request disp [D 20:00:57.337834] contig_offset = 180355072 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.337851] calling distribute [D 20:00:57.337858] PINT_distribute [D 20:00:57.337864] of 180355072 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 180355072 exfl 1 [D 20:00:57.337874] begin iteration loff: 180355072 [D 20:00:57.337881] segment exceeds byte limit [D 20:00:57.337887] file being extended [D 20:00:57.337894] process a segment [D 20:00:57.337900] of 180355072 sz 2097152 [D 20:00:57.337906] add a segment 0 [D 20:00:57.337913] end iteration [D 20:00:57.337919] done with segments or bytes [D 20:00:57.337925] of 182452224 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.337932] next loff: 182452224 next poff: 182452224 [D 20:00:57.337939] return value 2097152 [D 20:00:57.337946] finished [D 20:00:57.337952] retval = 2097152 [D 20:00:57.337958] exiting distribute returned less than expected [D 20:00:57.337965] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 182452224 fo 201326592 eof 0 [D 20:00:57.337973] ========================================================= [D 20:00:57.337980] BMI_post_recv: addr: 6843072, offset: 0x2aaab7e37010, size: 2097152, tag: 13 [D 20:00:57.337996] openib_check_cq: completed send wr_id 682ae0. [D 20:00:57.338006] openib_check_cq: completed recv wr_id 684140. [D 20:00:57.338024] openib_post_rr: 10.1.4.56:39990 bh 31 wr_id 684140. [D 20:00:57.338033] openib_check_cq: completed recv wr_id 684170. [D 20:00:57.338043] openib_post_rr: 10.1.4.56:39990 bh 32 wr_id 684170. [D 20:00:57.338071] post_sr: 10.1.4.56:39990 bh 13 len 48 wr_id 682b10. [D 20:00:57.338085] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:00:57.338096] get_data_sync_mode returning 0 [D 20:00:57.338108] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.338116] dbpf_bstream_rw_list: mem_offset: 0x2aaab642a010, mem_size: 2097152 [D 20:00:57.338123] dbpf_bstream_rw_list: stream_offset: 211812352, stream_size: 2097152 [D 20:00:57.338131] dbpf_open_cache_get: called [D 20:00:57.338140] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.338148] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.338169] DBPF I/O ops in progress: 6 [D 20:00:57.338176] lio_listio called with the following aiocbs: [D 20:00:57.338183] aiocb_ptr_array[0]: fd: 13, off: 211812352, bytes: 2097152, buf: 0x2aaab642a010, type: 1 [D 20:00:57.338195] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [D 20:00:57.339674] --- aio_progress_notification called with handle 2863311531 (0x6ec210) [D 20:00:57.339752] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.339767] dbpf_open_cache_put: cached entry. [D 20:00:57.339775] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.339782] op_queue add: 0x6ec210 [D 20:00:57.339800] --- aio_progress_notification called with handle 2863311531 (0x6f0a90) [D 20:00:57.339813] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.339827] dbpf_open_cache_put: cached entry. [D 20:00:57.339833] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.339843] --- aio_progress_notification called with handle 2863311531 (0x6f0f00) [D 20:00:57.339855] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.339867] dbpf_open_cache_put: cached entry. [D 20:00:57.339874] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.337734] --- aio_progress_notification called with handle 2863311531 (0x6ed8e0) [D 20:00:57.340710] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.340915] ========================================================= [D 20:00:57.340933] PINT_process_request [D 20:00:57.340940] tiling 8 copies [D 20:00:57.340947] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.340955] lvl 0 el 0 blk 0 by 213909504 [D 20:00:57.340961] to 213909504 ta 201326592 fi 234881024 [D 20:00:57.340970] basic type or contiguous data [D 20:00:57.340976] Request disp [D 20:00:57.340982] contig_offset = 213909504 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.340990] calling distribute [D 20:00:57.340996] PINT_distribute [D 20:00:57.341003] of 213909504 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 213909504 exfl 1 [D 20:00:57.341012] begin iteration loff: 213909504 [D 20:00:57.341019] segment exceeds byte limit [D 20:00:57.341025] file being extended [D 20:00:57.341031] process a segment [D 20:00:57.341037] of 213909504 sz 2097152 [D 20:00:57.341043] add a segment 0 [D 20:00:57.341050] end iteration [D 20:00:57.341056] done with segments or bytes [D 20:00:57.341062] of 216006656 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.341069] next loff: 216006656 next poff: 216006656 [D 20:00:57.341077] return value 2097152 [D 20:00:57.341083] finished [D 20:00:57.341089] retval = 2097152 [D 20:00:57.341095] exiting distribute returned less than expected [D 20:00:57.341102] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 216006656 fo 234881024 eof 0 [D 20:00:57.341109] ========================================================= [D 20:00:57.341116] BMI_post_recv: addr: 6843072, offset: 0x2aaab8038010, size: 2097152, tag: 14 [D 20:00:57.341131] openib_check_cq: completed send wr_id 682b10. [D 20:00:57.341141] openib_check_cq: completed recv wr_id 6841a0. [D 20:00:57.341157] openib_post_rr: 10.1.4.56:39990 bh 33 wr_id 6841a0. [D 20:00:57.341167] openib_check_cq: completed recv wr_id 6841d0. [D 20:00:57.341175] openib_post_rr: 10.1.4.56:39990 bh 34 wr_id 6841d0. [D 20:00:57.341200] post_sr: 10.1.4.56:39990 bh 14 len 48 wr_id 682b40. [D 20:00:57.341213] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:00:57.341222] get_data_sync_mode returning 0 [D 20:00:57.341234] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.341241] dbpf_bstream_rw_list: mem_offset: 0x2aaab662b010, mem_size: 2097152 [D 20:00:57.341250] dbpf_bstream_rw_list: stream_offset: 245366784, stream_size: 2097152 [D 20:00:57.341258] dbpf_open_cache_get: called [D 20:00:57.340893] dbpf_open_cache_put: cached entry. [D 20:00:57.341273] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.341280] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.341303] DBPF I/O ops in progress: 7 [D 20:00:57.341310] lio_listio called with the following aiocbs: [D 20:00:57.341317] aiocb_ptr_array[0]: fd: 13, off: 245366784, bytes: 2097152, buf: 0x2aaab662b010, type: 1 [D 20:00:57.341330] issue_or_delay_io_operation: lio_listio posted 0x6ecf80 (handle 2863311531, ret 0) [D 20:00:57.341343] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.345937] ========================================================= [D 20:00:57.345962] PINT_process_request [D 20:00:57.345971] tiling 9 copies [D 20:00:57.345978] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.345986] lvl 0 el 0 blk 0 by 247463936 [D 20:00:57.345993] to 247463936 ta 234881024 fi 268435456 [D 20:00:57.346001] basic type or contiguous data [D 20:00:57.346008] Request disp [D 20:00:57.346014] contig_offset = 247463936 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.346021] calling distribute [D 20:00:57.346029] PINT_distribute [D 20:00:57.346036] of 247463936 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 247463936 exfl 1 [D 20:00:57.346044] begin iteration loff: 247463936 [D 20:00:57.346051] segment exceeds byte limit [D 20:00:57.346058] file being extended [D 20:00:57.346064] process a segment [D 20:00:57.346070] of 247463936 sz 2097152 [D 20:00:57.346076] add a segment 0 [D 20:00:57.346082] end iteration [D 20:00:57.346088] done with segments or bytes [D 20:00:57.346094] of 249561088 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.346102] next loff: 249561088 next poff: 249561088 [D 20:00:57.346109] return value 2097152 [D 20:00:57.346116] finished [D 20:00:57.346122] retval = 2097152 [D 20:00:57.346129] exiting distribute returned less than expected [D 20:00:57.346135] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 249561088 fo 268435456 eof 0 [D 20:00:57.346143] ========================================================= [D 20:00:57.346151] BMI_post_recv: addr: 6843072, offset: 0x2aaab8239010, size: 2097152, tag: 15 [D 20:00:57.346166] openib_check_cq: completed send wr_id 682b40. [D 20:00:57.346176] openib_check_cq: completed recv wr_id 684200. [D 20:00:57.346194] openib_post_rr: 10.1.4.56:39990 bh 35 wr_id 684200. [D 20:00:57.346203] openib_check_cq: completed recv wr_id 684230. [D 20:00:57.346211] openib_post_rr: 10.1.4.56:39990 bh 36 wr_id 684230. [D 20:00:57.346219] openib_check_cq: completed recv wr_id 684260. [D 20:00:57.346227] openib_post_rr: 10.1.4.56:39990 bh 37 wr_id 684260. [D 20:00:57.346256] post_sr: 10.1.4.56:39990 bh 15 len 48 wr_id 682b70. [D 20:00:57.346273] openib_check_cq: completed send wr_id 682b70. [D 20:00:57.346291] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:00:57.346301] get_data_sync_mode returning 0 [D 20:00:57.346313] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.346321] dbpf_bstream_rw_list: mem_offset: 0x2aaab682c010, mem_size: 2097152 [D 20:00:57.346328] dbpf_bstream_rw_list: stream_offset: 278921216, stream_size: 2097152 [D 20:00:57.346335] dbpf_open_cache_get: called [D 20:00:57.346343] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.346350] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.346386] op_queue add: 0x6ed8e0 [D 20:00:57.348427] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:00:57.348473] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.348488] dbpf_open_cache_put: cached entry. [D 20:00:57.348496] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.348507] op_queue add: 0x6ec5d0 [D 20:00:57.343512] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:00:57.348544] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.348557] dbpf_open_cache_put: cached entry. [D 20:00:57.348563] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.348570] op_queue add: 0x6eda80 [D 20:00:57.345821] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:00:57.348614] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:57.348625] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:00:57.348633] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:00:57.343525] DBPF I/O ops in progress: 6 [D 20:00:57.348666] DBPF I/O ops in progress: 7 [D 20:00:57.348676] lio_listio called with the following aiocbs: [D 20:00:57.348684] aiocb_ptr_array[0]: fd: 13, off: 278921216, bytes: 2097152, buf: 0x2aaab682c010, type: 1 [D 20:00:57.348696] issue_or_delay_io_operation: lio_listio posted 0x6f0be0 (handle 2863311531, ret 0) [D 20:00:57.350720] DBPF I/O ops in progress: 6 [D 20:00:57.350800] DBPF I/O ops in progress: 5 [D 20:00:57.350823] DBPF I/O ops in progress: 4 [D 20:00:57.350853] op_queue add: 0x6f0a90 [D 20:00:57.350881] DBPF I/O ops in progress: 3 [D 20:00:57.350937] op_queue add: 0x6f0f00 [D 20:00:57.350964] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:00:57.350980] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:57.350996] DBPF I/O ops in progress: 2 [D 20:00:57.351228] ========================================================= [D 20:00:57.351245] PINT_process_request [D 20:00:57.351253] tiling 10 copies [D 20:00:57.351261] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.351269] lvl 0 el 0 blk 0 by 281018368 [D 20:00:57.351275] to 281018368 ta 268435456 fi 301989888 [D 20:00:57.351284] basic type or contiguous data [D 20:00:57.351290] Request disp [D 20:00:57.351296] contig_offset = 281018368 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.351303] calling distribute [D 20:00:57.351310] PINT_distribute [D 20:00:57.351317] of 281018368 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 281018368 exfl 1 [D 20:00:57.351325] begin iteration loff: 281018368 [D 20:00:57.351332] segment exceeds byte limit [D 20:00:57.351338] file being extended [D 20:00:57.351345] process a segment [D 20:00:57.351351] of 281018368 sz 2097152 [D 20:00:57.351358] add a segment 0 [D 20:00:57.351365] end iteration [D 20:00:57.351371] done with segments or bytes [D 20:00:57.351377] of 283115520 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.351384] next loff: 283115520 next poff: 283115520 [D 20:00:57.351391] return value 2097152 [D 20:00:57.351398] finished [D 20:00:57.351403] retval = 2097152 [D 20:00:57.351410] exiting distribute returned less than expected [D 20:00:57.351417] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 283115520 fo 301989888 eof 0 [D 20:00:57.351424] ========================================================= [D 20:00:57.351431] BMI_post_recv: addr: 6843072, offset: 0x2aaab843a010, size: 2097152, tag: 16 [D 20:00:57.351450] openib_check_cq: completed recv wr_id 684290. [D 20:00:57.351466] openib_post_rr: 10.1.4.56:39990 bh 38 wr_id 684290. [D 20:00:57.351475] openib_check_cq: completed recv wr_id 6842c0. [D 20:00:57.351488] openib_post_rr: 10.1.4.56:39990 bh 39 wr_id 6842c0. [D 20:00:57.351515] post_sr: 10.1.4.56:39990 bh 16 len 48 wr_id 682ba0. [D 20:00:57.351527] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:00:57.351545] get_data_sync_mode returning 0 [D 20:00:57.351556] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.351563] dbpf_bstream_rw_list: mem_offset: 0x2aaab6a2d010, mem_size: 2097152 [D 20:00:57.351571] dbpf_bstream_rw_list: stream_offset: 312475648, stream_size: 2097152 [D 20:00:57.351579] dbpf_open_cache_get: called [D 20:00:57.351586] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.351592] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.351620] DBPF I/O ops in progress: 3 [D 20:00:57.351627] lio_listio called with the following aiocbs: [D 20:00:57.351634] aiocb_ptr_array[0]: fd: 13, off: 312475648, bytes: 2097152, buf: 0x2aaab6a2d010, type: 1 [D 20:00:57.351645] issue_or_delay_io_operation: lio_listio posted 0x6f0d30 (handle 2863311531, ret 0) [D 20:00:57.351207] --- aio_progress_notification called with handle 2863311531 (0x6ecf80) [D 20:00:57.354266] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.354333] dbpf_open_cache_put: cached entry. [D 20:00:57.354369] ========================================================= [D 20:00:57.354379] PINT_process_request [D 20:00:57.354386] tiling 11 copies [D 20:00:57.354393] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.354401] lvl 0 el 0 blk 0 by 314572800 [D 20:00:57.354407] to 314572800 ta 301989888 fi 335544320 [D 20:00:57.354416] basic type or contiguous data [D 20:00:57.354422] Request disp [D 20:00:57.354429] contig_offset = 314572800 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.354437] calling distribute [D 20:00:57.354443] PINT_distribute [D 20:00:57.354449] of 314572800 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 314572800 exfl 1 [D 20:00:57.354458] begin iteration loff: 314572800 [D 20:00:57.354465] segment exceeds byte limit [D 20:00:57.354471] file being extended [D 20:00:57.354477] process a segment [D 20:00:57.354483] of 314572800 sz 2097152 [D 20:00:57.354490] add a segment 0 [D 20:00:57.354496] end iteration [D 20:00:57.354502] done with segments or bytes [D 20:00:57.354508] of 316669952 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.354515] next loff: 316669952 next poff: 316669952 [D 20:00:57.354521] return value 2097152 [D 20:00:57.354528] finished [D 20:00:57.354533] retval = 2097152 [D 20:00:57.354539] exiting distribute returned less than expected [D 20:00:57.354546] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 316669952 fo 335544320 eof 0 [D 20:00:57.354553] ========================================================= [D 20:00:57.354560] BMI_post_recv: addr: 6843072, offset: 0x2aaab863b010, size: 2097152, tag: 17 [D 20:00:57.354575] openib_check_cq: completed send wr_id 682ba0. [D 20:00:57.354586] openib_check_cq: completed recv wr_id 6842f0. [D 20:00:57.354599] openib_post_rr: 10.1.4.56:39990 bh 40 wr_id 6842f0. [D 20:00:57.354628] post_sr: 10.1.4.56:39990 bh 17 len 48 wr_id 682bd0. [D 20:00:57.354642] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:00:57.354650] get_data_sync_mode returning 0 [D 20:00:57.354662] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.354669] dbpf_bstream_rw_list: mem_offset: 0x2aaab6c2e010, mem_size: 2097152 [D 20:00:57.354677] dbpf_bstream_rw_list: stream_offset: 346030080, stream_size: 2097152 [D 20:00:57.354685] dbpf_open_cache_get: called [D 20:00:57.354691] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.354698] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.354721] DBPF I/O ops in progress: 4 [D 20:00:57.354728] lio_listio called with the following aiocbs: [D 20:00:57.354735] aiocb_ptr_array[0]: fd: 13, off: 346030080, bytes: 2097152, buf: 0x2aaab6c2e010, type: 1 [D 20:00:57.354749] issue_or_delay_io_operation: lio_listio posted 0x6ed8e0 (handle 2863311531, ret 0) [D 20:00:57.355703] --- aio_progress_notification called with handle 2863311531 (0x6f0be0) [D 20:00:57.355759] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.355775] dbpf_open_cache_put: cached entry. [D 20:00:57.355782] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.355793] op_queue add: 0x6f0be0 [D 20:00:57.354352] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.357423] ========================================================= [D 20:00:57.357470] PINT_process_request [D 20:00:57.357478] tiling 12 copies [D 20:00:57.357486] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.357494] lvl 0 el 0 blk 0 by 348127232 [D 20:00:57.357501] to 348127232 ta 335544320 fi 369098752 [D 20:00:57.357509] basic type or contiguous data [D 20:00:57.357516] Request disp [D 20:00:57.357524] contig_offset = 348127232 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.357533] calling distribute [D 20:00:57.357539] PINT_distribute [D 20:00:57.357547] of 348127232 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 348127232 exfl 1 [D 20:00:57.357556] begin iteration loff: 348127232 [D 20:00:57.357564] segment exceeds byte limit [D 20:00:57.357570] file being extended [D 20:00:57.357577] process a segment [D 20:00:57.357582] of 348127232 sz 2097152 [D 20:00:57.357589] add a segment 0 [D 20:00:57.357595] end iteration [D 20:00:57.357602] done with segments or bytes [D 20:00:57.357608] of 350224384 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.357615] next loff: 350224384 next poff: 350224384 [D 20:00:57.357622] return value 2097152 [D 20:00:57.357629] finished [D 20:00:57.357635] retval = 2097152 [D 20:00:57.357641] exiting distribute returned less than expected [D 20:00:57.357647] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 350224384 fo 369098752 eof 0 [D 20:00:57.357654] ========================================================= [D 20:00:57.357661] BMI_post_recv: addr: 6843072, offset: 0x2aaab883c010, size: 2097152, tag: 18 [D 20:00:57.357679] openib_check_cq: completed send wr_id 682bd0. [D 20:00:57.357691] openib_check_cq: completed recv wr_id 684320. [D 20:00:57.357710] openib_post_rr: 10.1.4.56:39990 bh 41 wr_id 684320. [D 20:00:57.357745] post_sr: 10.1.4.56:39990 bh 18 len 48 wr_id 682c00. [D 20:00:57.357760] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:00:57.357770] get_data_sync_mode returning 0 [D 20:00:57.357783] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.357790] dbpf_bstream_rw_list: mem_offset: 0x2aaab6e2f010, mem_size: 2097152 [D 20:00:57.357797] dbpf_bstream_rw_list: stream_offset: 379584512, stream_size: 2097152 [D 20:00:57.357804] dbpf_open_cache_get: called [D 20:00:57.357811] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.357818] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.357846] DBPF I/O ops in progress: 5 [D 20:00:57.357854] lio_listio called with the following aiocbs: [D 20:00:57.357860] aiocb_ptr_array[0]: fd: 13, off: 379584512, bytes: 2097152, buf: 0x2aaab6e2f010, type: 1 [D 20:00:57.357875] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [D 20:00:57.359856] op_queue add: 0x6ecf80 [D 20:00:57.359812] --- aio_progress_notification called with handle 2863311531 (0x6f0d30) [D 20:00:57.361087] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.361105] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:00:57.361132] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:00:57.361218] ========================================================= [D 20:00:57.361228] PINT_process_request [D 20:00:57.361235] tiling 13 copies [D 20:00:57.361243] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.361251] lvl 0 el 0 blk 0 by 381681664 [D 20:00:57.361257] to 381681664 ta 369098752 fi 402653184 [D 20:00:57.361265] basic type or contiguous data [D 20:00:57.361272] Request disp [D 20:00:57.361279] contig_offset = 381681664 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.361286] calling distribute [D 20:00:57.361292] PINT_distribute [D 20:00:57.361300] of 381681664 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 381681664 exfl 1 [D 20:00:57.361309] begin iteration loff: 381681664 [D 20:00:57.361317] segment exceeds byte limit [D 20:00:57.361323] file being extended [D 20:00:57.361330] process a segment [D 20:00:57.361336] of 381681664 sz 2097152 [D 20:00:57.361342] add a segment 0 [D 20:00:57.361349] end iteration [D 20:00:57.361355] done with segments or bytes [D 20:00:57.361361] of 383778816 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.361368] next loff: 383778816 next poff: 383778816 [D 20:00:57.361375] return value 2097152 [D 20:00:57.361381] finished [D 20:00:57.361387] retval = 2097152 [D 20:00:57.361393] exiting distribute returned less than expected [D 20:00:57.361399] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 383778816 fo 402653184 eof 0 [D 20:00:57.361407] ========================================================= [D 20:00:57.361414] BMI_post_recv: addr: 6843072, offset: 0x2aaab8a3d010, size: 2097152, tag: 19 [D 20:00:57.361429] openib_check_cq: completed send wr_id 682c00. [D 20:00:57.361439] openib_check_cq: completed recv wr_id 684350. [D 20:00:57.361454] openib_post_rr: 10.1.4.56:39990 bh 42 wr_id 684350. [D 20:00:57.361463] openib_check_cq: completed recv wr_id 684380. [D 20:00:57.361471] openib_post_rr: 10.1.4.56:39990 bh 43 wr_id 684380. [D 20:00:57.361494] post_sr: 10.1.4.56:39990 bh 19 len 48 wr_id 682c30. [D 20:00:57.361506] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:00:57.361516] get_data_sync_mode returning 0 [D 20:00:57.361525] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.361533] dbpf_bstream_rw_list: mem_offset: 0x2aaab7030010, mem_size: 2097152 [D 20:00:57.361540] dbpf_bstream_rw_list: stream_offset: 413138944, stream_size: 2097152 [D 20:00:57.361547] dbpf_open_cache_get: called [D 20:00:57.364426] --- aio_progress_notification called with handle 2863311531 (0x6ed8e0) [D 20:00:57.364484] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.361110] dbpf_open_cache_put: cached entry. [D 20:00:57.364532] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.364543] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.364575] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.364589] op_queue add: 0x6f0d30 [D 20:00:57.364617] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:00:57.364643] dbpf_open_cache_put: cached entry. [D 20:00:57.364652] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.364659] op_queue add: 0x6ed8e0 [D 20:00:57.364674] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:00:57.359830] DBPF I/O ops in progress: 4 [D 20:00:57.364743] DBPF I/O ops in progress: 3 [D 20:00:57.364758] DBPF I/O ops in progress: 4 [D 20:00:57.364767] lio_listio called with the following aiocbs: [D 20:00:57.364774] aiocb_ptr_array[0]: fd: 13, off: 413138944, bytes: 2097152, buf: 0x2aaab7030010, type: 1 [D 20:00:57.364787] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [D 20:00:57.366881] DBPF I/O ops in progress: 3 [D 20:00:57.367002] ========================================================= [D 20:00:57.367012] PINT_process_request [D 20:00:57.367019] tiling 14 copies [D 20:00:57.367027] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.367035] lvl 0 el 0 blk 0 by 415236096 [D 20:00:57.367042] to 415236096 ta 402653184 fi 436207616 [D 20:00:57.367050] basic type or contiguous data [D 20:00:57.367057] Request disp [D 20:00:57.367063] contig_offset = 415236096 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.367070] calling distribute [D 20:00:57.367076] PINT_distribute [D 20:00:57.367083] of 415236096 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 415236096 exfl 1 [D 20:00:57.367093] begin iteration loff: 415236096 [D 20:00:57.367101] segment exceeds byte limit [D 20:00:57.367109] file being extended [D 20:00:57.367116] process a segment [D 20:00:57.367122] of 415236096 sz 2097152 [D 20:00:57.367129] add a segment 0 [D 20:00:57.367135] end iteration [D 20:00:57.367142] done with segments or bytes [D 20:00:57.367148] of 417333248 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.367155] next loff: 417333248 next poff: 417333248 [D 20:00:57.367161] return value 2097152 [D 20:00:57.367169] finished [D 20:00:57.367175] retval = 2097152 [D 20:00:57.367182] exiting distribute returned less than expected [D 20:00:57.367188] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 417333248 fo 436207616 eof 0 [D 20:00:57.367195] ========================================================= [D 20:00:57.367202] BMI_post_recv: addr: 6843072, offset: 0x2aaab8c3e010, size: 2097152, tag: 20 [D 20:00:57.367218] openib_check_cq: completed send wr_id 682c30. [D 20:00:57.367229] openib_check_cq: completed recv wr_id 6843b0. [D 20:00:57.367247] openib_post_rr: 10.1.4.56:39990 bh 44 wr_id 6843b0. [D 20:00:57.367257] openib_check_cq: completed recv wr_id 6843e0. [D 20:00:57.367265] openib_post_rr: 10.1.4.56:39990 bh 45 wr_id 6843e0. [D 20:00:57.367274] openib_check_cq: completed recv wr_id 684410. [D 20:00:57.367284] openib_post_rr: 10.1.4.56:39990 bh 46 wr_id 684410. [D 20:00:57.367291] openib_check_cq: completed recv wr_id 684440. [D 20:00:57.367299] openib_post_rr: 10.1.4.56:39990 bh 47 wr_id 684440. [D 20:00:57.367325] post_sr: 10.1.4.56:39990 bh 20 len 48 wr_id 682c60. [D 20:00:57.367352] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:00:57.367363] get_data_sync_mode returning 0 [D 20:00:57.367375] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.367382] dbpf_bstream_rw_list: mem_offset: 0x2aaab5c25010, mem_size: 2097152 [D 20:00:57.367389] dbpf_bstream_rw_list: stream_offset: 471859200, stream_size: 2097152 [D 20:00:57.367397] dbpf_open_cache_get: called [D 20:00:57.367403] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.367410] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.367436] DBPF I/O ops in progress: 4 [D 20:00:57.367444] lio_listio called with the following aiocbs: [D 20:00:57.367450] aiocb_ptr_array[0]: fd: 13, off: 471859200, bytes: 2097152, buf: 0x2aaab5c25010, type: 1 [D 20:00:57.367462] issue_or_delay_io_operation: lio_listio posted 0x6ecf80 (handle 2863311531, ret 0) [D 20:00:57.370091] DBPF I/O ops in progress: 3 [D 20:00:57.370237] ========================================================= [D 20:00:57.370249] PINT_process_request [D 20:00:57.370256] tiling 16 copies [D 20:00:57.370263] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.370271] lvl 0 el 0 blk 0 by 473956352 [D 20:00:57.370277] to 473956352 ta 469762048 fi 503316480 [D 20:00:57.370284] basic type or contiguous data [D 20:00:57.370291] Request disp [D 20:00:57.370297] contig_offset = 473956352 contig_size = 62914560 lvl_flag = 1 [D 20:00:57.370304] calling distribute [D 20:00:57.370310] PINT_distribute [D 20:00:57.370317] of 473956352 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 473956352 exfl 1 [D 20:00:57.370327] begin iteration loff: 473956352 [D 20:00:57.370334] segment exceeds byte limit [D 20:00:57.370340] file being extended [D 20:00:57.370347] process a segment [D 20:00:57.370353] of 473956352 sz 2097152 [D 20:00:57.370360] add a segment 0 [D 20:00:57.370367] end iteration [D 20:00:57.370373] done with segments or bytes [D 20:00:57.370379] of 476053504 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.370386] next loff: 476053504 next poff: 476053504 [D 20:00:57.370393] return value 2097152 [D 20:00:57.370401] finished [D 20:00:57.370408] retval = 2097152 [D 20:00:57.370414] exiting distribute returned less than expected [D 20:00:57.370420] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 476053504 fo 503316480 eof 0 [D 20:00:57.370428] ========================================================= [D 20:00:57.370437] BMI_post_recv: addr: 6843072, offset: 0x2aaab8e3f010, size: 2097152, tag: 22 [D 20:00:57.370453] openib_check_cq: completed send wr_id 682c60. [D 20:00:57.370463] openib_check_cq: completed recv wr_id 684470. [D 20:00:57.370476] openib_post_rr: 10.1.4.56:39990 bh 48 wr_id 684470. [D 20:00:57.370505] post_sr: 10.1.4.56:39990 bh 21 len 48 wr_id 682c90. [D 20:00:57.370518] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:00:57.370528] get_data_sync_mode returning 0 [D 20:00:57.370541] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.370548] dbpf_bstream_rw_list: mem_offset: 0x2aaab7231010, mem_size: 2097152 [D 20:00:57.370555] dbpf_bstream_rw_list: stream_offset: 446693376, stream_size: 2097152 [D 20:00:57.370562] dbpf_open_cache_get: called [D 20:00:57.370569] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.370576] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.370599] DBPF I/O ops in progress: 4 [D 20:00:57.370606] lio_listio called with the following aiocbs: [D 20:00:57.370612] aiocb_ptr_array[0]: fd: 13, off: 446693376, bytes: 2097152, buf: 0x2aaab7231010, type: 1 [D 20:00:57.370625] issue_or_delay_io_operation: lio_listio posted 0x6f0a90 (handle 2863311531, ret 0) [D 20:00:57.373304] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:00:57.373355] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.373367] dbpf_open_cache_put: cached entry. [D 20:00:57.373374] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.373382] op_queue add: 0x6ec5d0 [D 20:00:57.373395] DBPF I/O ops in progress: 3 [D 20:00:57.373408] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:00:57.373427] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:00:57.373440] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.373442] ========================================================= [D 20:00:57.373463] PINT_process_request [D 20:00:57.373472] tiling 15 copies [D 20:00:57.373479] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.373487] lvl 0 el 0 blk 0 by 448790528 [D 20:00:57.373494] to 448790528 ta 436207616 fi 469762048 [D 20:00:57.373501] basic type or contiguous data [D 20:00:57.373508] Request disp [D 20:00:57.373515] contig_offset = 448790528 contig_size = 54525952 lvl_flag = 1 [D 20:00:57.373522] calling distribute [D 20:00:57.373528] PINT_distribute [D 20:00:57.373535] of 448790528 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 448790528 exfl 1 [D 20:00:57.373544] begin iteration loff: 448790528 [D 20:00:57.373553] segment exceeds byte limit [D 20:00:57.373559] file being extended [D 20:00:57.373566] process a segment [D 20:00:57.373572] of 448790528 sz 2097152 [D 20:00:57.373579] add a segment 0 [D 20:00:57.373586] end iteration [D 20:00:57.373592] done with segments or bytes [D 20:00:57.373598] of 450887680 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.373606] next loff: 450887680 next poff: 450887680 [D 20:00:57.373613] return value 2097152 [D 20:00:57.373620] finished [D 20:00:57.373625] retval = 2097152 [D 20:00:57.373632] exiting distribute returned less than expected [D 20:00:57.373638] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 450887680 fo 469762048 eof 0 [D 20:00:57.373645] ========================================================= [D 20:00:57.373653] BMI_post_recv: addr: 6843072, offset: 0x2aaab9040010, size: 2097152, tag: 21 [D 20:00:57.373669] openib_check_cq: completed send wr_id 682c90. [D 20:00:57.373678] openib_check_cq: completed recv wr_id 6844a0. [D 20:00:57.373694] openib_post_rr: 10.1.4.56:39990 bh 49 wr_id 6844a0. [D 20:00:57.373704] openib_check_cq: completed recv wr_id 6844d0. [D 20:00:57.373712] openib_post_rr: 10.1.4.56:39990 bh 50 wr_id 6844d0. [D 20:00:57.373740] post_sr: 10.1.4.56:39990 bh 22 len 48 wr_id 682cc0. [D 20:00:57.373753] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:00:57.373761] get_data_sync_mode returning 0 [D 20:00:57.373772] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.373780] dbpf_bstream_rw_list: mem_offset: 0x2aaab7432010, mem_size: 2097152 [D 20:00:57.373787] dbpf_bstream_rw_list: stream_offset: 12582912, stream_size: 2097152 [D 20:00:57.373795] dbpf_open_cache_get: called [D 20:00:57.373453] dbpf_open_cache_put: cached entry. [D 20:00:57.376507] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.376517] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.376547] DBPF I/O ops in progress: 4 [D 20:00:57.376556] lio_listio called with the following aiocbs: [D 20:00:57.376563] aiocb_ptr_array[0]: fd: 13, off: 12582912, bytes: 2097152, buf: 0x2aaab7432010, type: 1 [D 20:00:57.376578] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [D 20:00:57.378742] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.378827] op_queue add: 0x6eda80 [D 20:00:57.378873] ========================================================= [D 20:00:57.378885] PINT_process_request [D 20:00:57.378892] tiling 2 copies [D 20:00:57.378899] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.378908] lvl 0 el 0 blk 0 by 14680064 [D 20:00:57.378915] to 14680064 ta 0 fi 33554432 [D 20:00:57.378922] basic type or contiguous data [D 20:00:57.378929] Request disp [D 20:00:57.378936] contig_offset = 14680064 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.378943] calling distribute [D 20:00:57.378949] PINT_distribute [D 20:00:57.378956] of 14680064 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 14680064 exfl 1 [D 20:00:57.378964] begin iteration loff: 14680064 [D 20:00:57.378972] segment exceeds byte limit [D 20:00:57.378978] file being extended [D 20:00:57.378985] process a segment [D 20:00:57.378990] of 14680064 sz 2097152 [D 20:00:57.378997] add a segment 0 [D 20:00:57.379004] end iteration [D 20:00:57.379010] done with segments or bytes [D 20:00:57.379016] of 16777216 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.379023] next loff: 16777216 next poff: 16777216 [D 20:00:57.379030] return value 2097152 [D 20:00:57.379036] finished [D 20:00:57.379043] retval = 2097152 [D 20:00:57.379049] exiting distribute returned less than expected [D 20:00:57.379055] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 16777216 fo 33554432 eof 0 [D 20:00:57.379063] ========================================================= [D 20:00:57.379070] BMI_post_recv: addr: 6843072, offset: 0x2aaab9241010, size: 2097152, tag: 8 [D 20:00:57.379086] openib_check_cq: completed send wr_id 682cc0. [D 20:00:57.379098] openib_check_cq: completed recv wr_id 684500. [D 20:00:57.379114] openib_post_rr: 10.1.4.56:39990 bh 51 wr_id 684500. [D 20:00:57.379124] openib_check_cq: completed recv wr_id 684530. [D 20:00:57.379133] openib_post_rr: 10.1.4.56:39990 bh 52 wr_id 684530. [D 20:00:57.379141] openib_check_cq: completed recv wr_id 684560. [D 20:00:57.379150] openib_post_rr: 10.1.4.56:39990 bh 53 wr_id 684560. [D 20:00:57.379178] post_sr: 10.1.4.56:39990 bh 23 len 48 wr_id 682cf0. [D 20:00:57.379193] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:00:57.379203] get_data_sync_mode returning 0 [D 20:00:57.379214] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.379221] dbpf_bstream_rw_list: mem_offset: 0x2aaab7633010, mem_size: 2097152 [D 20:00:57.379229] dbpf_bstream_rw_list: stream_offset: 46137344, stream_size: 2097152 [D 20:00:57.379236] dbpf_open_cache_get: called [D 20:00:57.379243] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.379251] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.379279] DBPF I/O ops in progress: 5 [D 20:00:57.379286] lio_listio called with the following aiocbs: [D 20:00:57.379295] aiocb_ptr_array[0]: fd: 13, off: 46137344, bytes: 2097152, buf: 0x2aaab7633010, type: 1 [D 20:00:57.379310] issue_or_delay_io_operation: lio_listio posted 0x6f0be0 (handle 2863311531, ret 0) [D 20:00:57.382028] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:00:57.382118] DBPF I/O ops in progress: 4 [D 20:00:57.382180] ========================================================= [D 20:00:57.382190] PINT_process_request [D 20:00:57.382198] tiling 3 copies [D 20:00:57.382206] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.382214] lvl 0 el 0 blk 0 by 48234496 [D 20:00:57.382221] to 48234496 ta 33554432 fi 67108864 [D 20:00:57.382228] basic type or contiguous data [D 20:00:57.382234] Request disp [D 20:00:57.382240] contig_offset = 48234496 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.382248] calling distribute [D 20:00:57.382254] PINT_distribute [D 20:00:57.382261] of 48234496 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 48234496 exfl 1 [D 20:00:57.382270] begin iteration loff: 48234496 [D 20:00:57.382277] segment exceeds byte limit [D 20:00:57.382283] file being extended [D 20:00:57.382290] process a segment [D 20:00:57.382295] of 48234496 sz 2097152 [D 20:00:57.382302] add a segment 0 [D 20:00:57.382309] end iteration [D 20:00:57.382315] done with segments or bytes [D 20:00:57.382320] of 50331648 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.382328] next loff: 50331648 next poff: 50331648 [D 20:00:57.382335] return value 2097152 [D 20:00:57.382343] finished [D 20:00:57.382349] retval = 2097152 [D 20:00:57.382355] exiting distribute returned less than expected [D 20:00:57.382361] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 50331648 fo 67108864 eof 0 [D 20:00:57.382369] ========================================================= [D 20:00:57.382377] BMI_post_recv: addr: 6843072, offset: 0x2aaab9442010, size: 2097152, tag: 9 [D 20:00:57.382393] openib_check_cq: completed send wr_id 682cf0. [D 20:00:57.382403] openib_check_cq: completed recv wr_id 684590. [D 20:00:57.382417] openib_post_rr: 10.1.4.56:39990 bh 54 wr_id 684590. [D 20:00:57.382449] post_sr: 10.1.4.56:39990 bh 24 len 48 wr_id 682d20. [D 20:00:57.382461] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:00:57.382470] get_data_sync_mode returning 0 [D 20:00:57.382480] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.382488] dbpf_bstream_rw_list: mem_offset: 0x2aaab7834010, mem_size: 2097152 [D 20:00:57.382495] dbpf_bstream_rw_list: stream_offset: 79691776, stream_size: 2097152 [D 20:00:57.382502] dbpf_open_cache_get: called [D 20:00:57.382508] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.382515] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.382538] DBPF I/O ops in progress: 5 [D 20:00:57.382545] lio_listio called with the following aiocbs: [D 20:00:57.382552] aiocb_ptr_array[0]: fd: 13, off: 79691776, bytes: 2097152, buf: 0x2aaab7834010, type: 1 [D 20:00:57.382563] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [D 20:00:57.385356] ========================================================= [D 20:00:57.385433] PINT_process_request [D 20:00:57.385441] tiling 4 copies [D 20:00:57.385450] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.385458] lvl 0 el 0 blk 0 by 81788928 [D 20:00:57.385465] to 81788928 ta 67108864 fi 100663296 [D 20:00:57.385473] basic type or contiguous data [D 20:00:57.385479] Request disp [D 20:00:57.385486] contig_offset = 81788928 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.385493] calling distribute [D 20:00:57.385499] PINT_distribute [D 20:00:57.385507] of 81788928 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 81788928 exfl 1 [D 20:00:57.385515] begin iteration loff: 81788928 [D 20:00:57.385522] segment exceeds byte limit [D 20:00:57.385528] file being extended [D 20:00:57.385534] process a segment [D 20:00:57.385542] of 81788928 sz 2097152 [D 20:00:57.385550] add a segment 0 [D 20:00:57.385557] end iteration [D 20:00:57.385563] done with segments or bytes [D 20:00:57.385569] of 83886080 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.385576] next loff: 83886080 next poff: 83886080 [D 20:00:57.385582] return value 2097152 [D 20:00:57.385589] finished [D 20:00:57.385595] retval = 2097152 [D 20:00:57.385601] exiting distribute returned less than expected [D 20:00:57.385607] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 83886080 fo 100663296 eof 0 [D 20:00:57.385614] ========================================================= [D 20:00:57.385622] BMI_post_recv: addr: 6843072, offset: 0x2aaab9643010, size: 2097152, tag: 10 [D 20:00:57.385637] openib_check_cq: completed send wr_id 682d20. [D 20:00:57.385649] openib_check_cq: completed recv wr_id 6845c0. [D 20:00:57.385668] openib_post_rr: 10.1.4.56:39990 bh 55 wr_id 6845c0. [D 20:00:57.385677] openib_check_cq: completed recv wr_id 6845f0. [D 20:00:57.385687] openib_post_rr: 10.1.4.56:39990 bh 56 wr_id 6845f0. [D 20:00:57.385715] post_sr: 10.1.4.56:39990 bh 25 len 48 wr_id 682d50. [D 20:00:57.385731] openib_check_cq: completed send wr_id 682d50. [D 20:00:57.385750] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:00:57.385760] get_data_sync_mode returning 0 [D 20:00:57.385771] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.385779] dbpf_bstream_rw_list: mem_offset: 0x2aaab8e3f010, mem_size: 2097152 [D 20:00:57.385786] dbpf_bstream_rw_list: stream_offset: 473956352, stream_size: 2097152 [D 20:00:57.385793] dbpf_open_cache_get: called [D 20:00:57.385801] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.385808] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.385830] DBPF I/O ops in progress: 6 [D 20:00:57.385837] lio_listio called with the following aiocbs: [D 20:00:57.385844] aiocb_ptr_array[0]: fd: 13, off: 473956352, bytes: 2097152, buf: 0x2aaab8e3f010, type: 1 [D 20:00:57.385855] issue_or_delay_io_operation: lio_listio posted 0x6ea160 (handle 2863311531, ret 0) [D 20:00:57.388501] ========================================================= [D 20:00:57.388582] PINT_process_request [D 20:00:57.388591] tiling 16 copies [D 20:00:57.388599] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.388607] lvl 0 el 0 blk 0 by 476053504 [D 20:00:57.388614] to 476053504 ta 469762048 fi 503316480 [D 20:00:57.388622] basic type or contiguous data [D 20:00:57.388629] Request disp [D 20:00:57.388635] contig_offset = 476053504 contig_size = 60817408 lvl_flag = 1 [D 20:00:57.388643] calling distribute [D 20:00:57.388649] PINT_distribute [D 20:00:57.388655] of 476053504 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 1 [D 20:00:57.388663] begin iteration loff: 476053504 [D 20:00:57.388670] segment exceeds byte limit [D 20:00:57.388676] file being extended [D 20:00:57.388683] process a segment [D 20:00:57.388688] of 476053504 sz 2097152 [D 20:00:57.388696] add a segment 0 [D 20:00:57.388702] end iteration [D 20:00:57.388708] done with segments or bytes [D 20:00:57.388714] of 478150656 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.388721] next loff: 478150656 next poff: 478150656 [D 20:00:57.388728] return value 2097152 [D 20:00:57.388735] finished [D 20:00:57.388741] retval = 2097152 [D 20:00:57.388748] exiting distribute returned less than expected [D 20:00:57.388754] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 478150656 fo 503316480 eof 0 [D 20:00:57.388762] ========================================================= [D 20:00:57.388770] BMI_post_recv: addr: 6843072, offset: 0x2aaab9844010, size: 2097152, tag: 22 [D 20:00:57.388792] openib_check_cq: completed recv wr_id 684620. [D 20:00:57.388809] openib_post_rr: 10.1.4.56:39990 bh 57 wr_id 684620. [D 20:00:57.388849] post_sr: 10.1.4.56:39990 bh 26 len 48 wr_id 682d80. [D 20:00:57.388864] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:00:57.596293] get_data_sync_mode returning 0 [D 20:00:57.596310] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.596318] dbpf_bstream_rw_list: mem_offset: 0x2aaab7a35010, mem_size: 2097152 [D 20:00:57.596325] dbpf_bstream_rw_list: stream_offset: 113246208, stream_size: 2097152 [D 20:00:57.596332] dbpf_open_cache_get: called [D 20:00:57.596338] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.596345] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.596361] DBPF I/O ops in progress: 7 [D 20:00:57.596368] lio_listio called with the following aiocbs: [D 20:00:57.596374] aiocb_ptr_array[0]: fd: 13, off: 113246208, bytes: 2097152, buf: 0x2aaab7a35010, type: 1 [D 20:00:57.596386] issue_or_delay_io_operation: lio_listio posted 0x6ea2b0 (handle 2863311531, ret 0) [D 20:00:57.599557] ========================================================= [D 20:00:57.599625] PINT_process_request [D 20:00:57.599633] tiling 5 copies [D 20:00:57.599642] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.599650] lvl 0 el 0 blk 0 by 115343360 [D 20:00:57.599657] to 115343360 ta 100663296 fi 134217728 [D 20:00:57.599664] basic type or contiguous data [D 20:00:57.599671] Request disp [D 20:00:57.599677] contig_offset = 115343360 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.599685] calling distribute [D 20:00:57.599693] PINT_distribute [D 20:00:57.599700] of 115343360 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 115343360 exfl 1 [D 20:00:57.599709] begin iteration loff: 115343360 [D 20:00:57.599716] segment exceeds byte limit [D 20:00:57.599723] file being extended [D 20:00:57.599729] process a segment [D 20:00:57.599735] of 115343360 sz 2097152 [D 20:00:57.599742] add a segment 0 [D 20:00:57.599748] end iteration [D 20:00:57.599754] done with segments or bytes [D 20:00:57.599760] of 117440512 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.599767] next loff: 117440512 next poff: 117440512 [D 20:00:57.599774] return value 2097152 [D 20:00:57.599782] finished [D 20:00:57.599788] retval = 2097152 [D 20:00:57.599794] exiting distribute returned less than expected [D 20:00:57.599800] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 117440512 fo 134217728 eof 0 [D 20:00:57.599808] ========================================================= [D 20:00:57.599816] BMI_post_recv: addr: 6843072, offset: 0x2aaab9a45010, size: 2097152, tag: 11 [D 20:00:57.599835] openib_check_cq: completed send wr_id 682d80. [D 20:00:57.599845] openib_check_cq: completed recv wr_id 684650. [D 20:00:57.599859] openib_post_rr: 10.1.4.56:39990 bh 58 wr_id 684650. [D 20:00:57.599870] openib_check_cq: completed recv wr_id 684680. [D 20:00:57.599880] openib_post_rr: 10.1.4.56:39990 bh 59 wr_id 684680. [D 20:00:57.599888] openib_check_cq: completed recv wr_id 6846b0. [D 20:00:57.599898] openib_post_rr: 10.1.4.56:39990 bh 60 wr_id 6846b0. [D 20:00:57.599905] openib_check_cq: completed recv wr_id 6846e0. [D 20:00:57.599914] openib_post_rr: 10.1.4.56:39990 bh 61 wr_id 6846e0. [D 20:00:57.599941] post_sr: 10.1.4.56:39990 bh 27 len 48 wr_id 682db0. [D 20:00:57.599955] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:00:57.599967] get_data_sync_mode returning 0 [D 20:00:57.599981] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.599988] dbpf_bstream_rw_list: mem_offset: 0x2aaab7c36010, mem_size: 2097152 [D 20:00:57.599995] dbpf_bstream_rw_list: stream_offset: 146800640, stream_size: 2097152 [D 20:00:57.600002] dbpf_open_cache_get: called [D 20:00:57.600009] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.600016] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.600039] DBPF I/O ops in progress: 8 [D 20:00:57.600046] lio_listio called with the following aiocbs: [D 20:00:57.600053] aiocb_ptr_array[0]: fd: 13, off: 146800640, bytes: 2097152, buf: 0x2aaab7c36010, type: 1 [D 20:00:57.600060] --- aio_progress_notification called with handle 2863311531 (0x6ea2b0) [D 20:00:57.600089] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.600102] dbpf_open_cache_put: cached entry. [D 20:00:57.600110] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.600117] op_queue add: 0x6ea2b0 [D 20:00:57.600138] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:57.600153] DBPF I/O ops in progress: 7 [D 20:00:57.600068] issue_or_delay_io_operation: lio_listio posted 0x6ea400 (handle 2863311531, ret 0) [D 20:00:57.604116] ========================================================= [D 20:00:57.604150] PINT_process_request [D 20:00:57.604160] tiling 6 copies [D 20:00:57.604168] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.604176] lvl 0 el 0 blk 0 by 148897792 [D 20:00:57.604183] to 148897792 ta 134217728 fi 167772160 [D 20:00:57.604190] basic type or contiguous data [D 20:00:57.604203] Request disp [D 20:00:57.604210] contig_offset = 148897792 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.604217] calling distribute [D 20:00:57.604224] PINT_distribute [D 20:00:57.604232] of 148897792 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 148897792 exfl 1 [D 20:00:57.604241] begin iteration loff: 148897792 [D 20:00:57.604248] segment exceeds byte limit [D 20:00:57.604255] file being extended [D 20:00:57.604262] process a segment [D 20:00:57.604268] of 148897792 sz 2097152 [D 20:00:57.604274] add a segment 0 [D 20:00:57.604280] end iteration [D 20:00:57.604286] done with segments or bytes [D 20:00:57.604292] of 150994944 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.604299] next loff: 150994944 next poff: 150994944 [D 20:00:57.604306] return value 2097152 [D 20:00:57.604313] finished [D 20:00:57.604319] retval = 2097152 [D 20:00:57.604325] exiting distribute returned less than expected [D 20:00:57.604331] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 150994944 fo 167772160 eof 0 [D 20:00:57.604339] ========================================================= [D 20:00:57.604346] BMI_post_recv: addr: 6843072, offset: 0x2aaab9c46010, size: 2097152, tag: 12 [D 20:00:57.604361] openib_check_cq: completed send wr_id 682db0. [D 20:00:57.604371] openib_check_cq: completed recv wr_id 684710. [D 20:00:57.604383] openib_post_rr: 10.1.4.56:39990 bh 62 wr_id 684710. [D 20:00:57.604384] --- aio_progress_notification called with handle 2863311531 (0x6ea400) [D 20:00:57.604404] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.604427] dbpf_open_cache_put: cached entry. [D 20:00:57.604435] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.604441] op_queue add: 0x6ea400 [D 20:00:57.604463] DBPF I/O ops in progress: 6 [D 20:00:57.604393] openib_check_cq: completed recv wr_id 684740. [D 20:00:57.604615] openib_post_rr: 10.1.4.56:39990 bh 63 wr_id 684740. [D 20:00:57.604637] post_sr: 10.1.4.56:39990 bh 28 len 48 wr_id 682de0. [D 20:00:57.604650] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:00:57.604651] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:57.604661] get_data_sync_mode returning 0 [D 20:00:57.604863] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.604870] dbpf_bstream_rw_list: mem_offset: 0x2aaab7e37010, mem_size: 2097152 [D 20:00:57.604877] dbpf_bstream_rw_list: stream_offset: 180355072, stream_size: 2097152 [D 20:00:57.604884] dbpf_open_cache_get: called [D 20:00:57.604891] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.604897] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.604914] DBPF I/O ops in progress: 7 [D 20:00:57.604921] lio_listio called with the following aiocbs: [D 20:00:57.604927] aiocb_ptr_array[0]: fd: 13, off: 180355072, bytes: 2097152, buf: 0x2aaab7e37010, type: 1 [D 20:00:57.604939] issue_or_delay_io_operation: lio_listio posted 0x6ea2b0 (handle 2863311531, ret 0) [A 20:00:57.606220] troy.scl@10.1.4.56:39990 H=2863311531 S=0x668b50: io: start [D 20:00:57.606268] (0x668b50) io (prelude sm) state: getattr_if_needed [D 20:00:57.606276] About to retrieve attributes for handle 2863311531 [D 20:00:57.606296] op_queue add: 0x2aaab150fd20 [D 20:00:57.608061] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:57.608122] dbpf_open_cache_get: called [D 20:00:57.608130] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.608137] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.608148] dbpf_open_cache_put: cached entry. [D 20:00:57.608195] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 476053504 [D 20:00:57.608212] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 476053504) [D 20:00:57.608220] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:57.608229] op_queue add: 0x2aaab150fd20 [D 20:00:57.608250] [SM Exiting]: (0x668a50) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:57.608262] [SM Entering]: (0x669d10) unexpected_sm:map_request (status: 0) [D 20:00:57.608270] [SM frame get]: (0x669d10) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.608278] server_post_unexpected_recv [D 20:00:57.608301] [SM Frame PUSH]: (0x2aaab150fd20) frame: 0x2aaab150ffe0 [D 20:00:57.608310] [SM Locating]: (0x2aaab150fd20) op-id: 999 [D 20:00:57.608317] server_op_state_get_machine 999 [D 20:00:57.608324] [SM Locating]: (0x2aaab150fd20) located: unexpected_sm:post_unexpected [D 20:00:57.608331] [SM frame get]: (0x2aaab150fd20) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.608339] [SM Entering]: (0x2aaab150fd20) unexpected_sm:post_unexpected (status: 0) [D 20:00:57.608346] [SM frame get]: (0x2aaab150fd20) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.608357] [SM Exiting]: (0x2aaab150fd20) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:57.608365] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:57.608373] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:57.608380] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 25 [D 20:00:57.608387] [SM frame get]: (0x669d10) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.608394] server_state_machine_start 0x669d10 [D 20:00:57.608401] PINT_decode [D 20:00:57.608418] lebf_decode_req [D 20:00:57.608432] [SM Locating]: (0x669d10) op-id: 3 [D 20:00:57.608439] server_op_state_get_machine 3 [D 20:00:57.608446] [SM push_state]: (0x669d10) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:57.608453] [SM Locating]: (0x669d10) located: prelude_sm:setup [D 20:00:57.608460] [SM Entering]: (0x669d10) prelude_sm:setup (status: 0) [D 20:00:57.608468] [SM frame get]: (0x669d10) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.608475] [SM Exiting]: (0x669d10) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:57.608483] [SM Exiting]: (0x669d10) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:57.608490] [SM push_state]: (0x669d10) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:57.608497] [SM Entering]: (0x669d10) prelude_work_sm:req_sched (status: 0) [D 20:00:57.608504] [SM frame get]: (0x669d10) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.608511] (0x669e10) io (prelude sm) state: req_sched [D 20:00:57.609082] --- aio_progress_notification called with handle 2863311531 (0x6ea2b0) [D 20:00:57.609158] ========================================================= [D 20:00:57.609172] PINT_process_request [D 20:00:57.609179] tiling 7 copies [D 20:00:57.609185] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.609193] lvl 0 el 0 blk 0 by 182452224 [D 20:00:57.609199] to 182452224 ta 167772160 fi 201326592 [D 20:00:57.609206] basic type or contiguous data [D 20:00:57.609212] Request disp [D 20:00:57.609218] contig_offset = 182452224 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.609225] calling distribute [D 20:00:57.609232] PINT_distribute [D 20:00:57.609238] of 182452224 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 182452224 exfl 1 [D 20:00:57.609346] begin iteration loff: 182452224 [D 20:00:57.609353] segment exceeds byte limit [D 20:00:57.609359] file being extended [D 20:00:57.609365] process a segment [D 20:00:57.609371] of 182452224 sz 2097152 [D 20:00:57.609377] add a segment 0 [D 20:00:57.609383] end iteration [D 20:00:57.609389] done with segments or bytes [D 20:00:57.609395] of 184549376 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.609402] next loff: 184549376 next poff: 184549376 [D 20:00:57.609408] return value 2097152 [D 20:00:57.609414] finished [D 20:00:57.609440] retval = 2097152 [D 20:00:57.609446] exiting distribute returned less than expected [D 20:00:57.609452] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 184549376 fo 201326592 eof 0 [D 20:00:57.609524] ========================================================= [D 20:00:57.609531] BMI_post_recv: addr: 6843072, offset: 0x2aaab9e47010, size: 2097152, tag: 13 [D 20:00:57.609543] openib_check_cq: completed send wr_id 682de0. [D 20:00:57.609552] openib_check_cq: completed recv wr_id 684770. [D 20:00:57.609562] openib_post_rr: 10.1.4.56:39990 bh 64 wr_id 684770. [D 20:00:57.609571] openib_check_cq: completed recv wr_id 6847a0. [D 20:00:57.609581] openib_post_rr: 10.1.4.56:39990 bh 65 wr_id 6847a0. [D 20:00:57.609603] post_sr: 10.1.4.56:39990 bh 29 len 48 wr_id 682e10. [D 20:00:57.609686] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:00:57.609695] get_data_sync_mode returning 0 [D 20:00:57.609704] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.609711] dbpf_bstream_rw_list: mem_offset: 0x2aaab8038010, mem_size: 2097152 [D 20:00:57.609717] dbpf_bstream_rw_list: stream_offset: 213909504, stream_size: 2097152 [D 20:00:57.609724] dbpf_open_cache_get: called [D 20:00:57.609731] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.609737] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.609745] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:57.609759] DBPF I/O ops in progress: 8 [D 20:00:57.609778] lio_listio called with the following aiocbs: [D 20:00:57.609784] aiocb_ptr_array[0]: fd: 13, off: 213909504, bytes: 2097152, buf: 0x2aaab8038010, type: 1 [D 20:00:57.609838] issue_or_delay_io_operation: lio_listio posted 0x6ea680 (handle 2863311531, ret 0) [D 20:00:57.388470] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:00:57.609156] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.611212] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.388893] --- aio_progress_notification called with handle 2863311531 (0x6f0a90) [D 20:00:57.611302] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.388909] --- aio_progress_notification called with handle 2863311531 (0x6ecf80) [D 20:00:57.390030] --- aio_progress_notification called with handle 2863311531 (0x6f0be0) [D 20:00:57.611354] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.611385] ========================================================= [D 20:00:57.611393] PINT_process_request [D 20:00:57.611400] tiling 8 copies [D 20:00:57.611406] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.611414] lvl 0 el 0 blk 0 by 216006656 [D 20:00:57.611421] to 216006656 ta 201326592 fi 234881024 [D 20:00:57.611427] basic type or contiguous data [D 20:00:57.611434] Request disp [D 20:00:57.611440] contig_offset = 216006656 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.611446] calling distribute [D 20:00:57.611452] PINT_distribute [D 20:00:57.611458] of 216006656 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 216006656 exfl 1 [D 20:00:57.611466] begin iteration loff: 216006656 [D 20:00:57.611472] segment exceeds byte limit [D 20:00:57.611478] file being extended [D 20:00:57.611484] process a segment [D 20:00:57.611490] of 216006656 sz 2097152 [D 20:00:57.611496] add a segment 0 [D 20:00:57.611502] end iteration [D 20:00:57.611508] done with segments or bytes [D 20:00:57.611513] of 218103808 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.611520] next loff: 218103808 next poff: 218103808 [D 20:00:57.611527] return value 2097152 [D 20:00:57.611533] finished [D 20:00:57.611538] retval = 2097152 [D 20:00:57.611545] exiting distribute returned less than expected [D 20:00:57.611551] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 218103808 fo 234881024 eof 0 [D 20:00:57.611558] ========================================================= [D 20:00:57.611564] BMI_post_recv: addr: 6843072, offset: 0x2aaaba048010, size: 2097152, tag: 14 [D 20:00:57.611575] openib_check_cq: completed send wr_id 682e10. [D 20:00:57.611595] post_sr: 10.1.4.56:39990 bh 30 len 48 wr_id 682e40. [D 20:00:57.611616] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:00:57.611625] get_data_sync_mode returning 0 [D 20:00:57.611634] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.611641] dbpf_bstream_rw_list: mem_offset: 0x2aaab8239010, mem_size: 2097152 [D 20:00:57.611648] dbpf_bstream_rw_list: stream_offset: 247463936, stream_size: 2097152 [D 20:00:57.611655] dbpf_open_cache_get: called [D 20:00:57.611366] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.396077] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:00:57.611675] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.396122] --- aio_progress_notification called with handle 2863311531 (0x6ea160) [D 20:00:57.611692] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.611245] dbpf_open_cache_put: cached entry. [D 20:00:57.612440] dbpf_open_cache_put: cached entry. [D 20:00:57.612450] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.612459] op_queue add: 0x6ec5d0 [D 20:00:57.612472] DBPF I/O ops in progress: 7 [D 20:00:57.612482] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:00:57.612503] dbpf_open_cache_put: cached entry. [D 20:00:57.612516] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.612523] op_queue add: 0x6f0a90 [D 20:00:57.612534] DBPF I/O ops in progress: 6 [D 20:00:57.612540] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:00:57.612549] dbpf_open_cache_put: cached entry. [D 20:00:57.612562] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.612566] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.612574] op_queue add: 0x6ecf80 [D 20:00:57.612589] DBPF I/O ops in progress: 5 [D 20:00:57.612594] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:00:57.612604] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.612617] op_queue add: 0x6ea2b0 [D 20:00:57.612577] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.612639] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:00:57.612653] DBPF I/O ops in progress: 6 [D 20:00:57.612664] lio_listio called with the following aiocbs: [D 20:00:57.612666] DBPF I/O ops in progress: 5 [D 20:00:57.612673] aiocb_ptr_array[0]: fd: 13, off: 247463936, bytes: 2097152, buf: 0x2aaab8239010, type: 1 [D 20:00:57.612703] issue_or_delay_io_operation: lio_listio posted 0x6ea7d0 (handle 2863311531, ret 0) [D 20:00:57.612676] dbpf_open_cache_put: cached entry. [D 20:00:57.614072] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.614080] op_queue add: 0x6f0be0 [D 20:00:57.614092] DBPF I/O ops in progress: 4 [D 20:00:57.614097] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:00:57.614144] dbpf_open_cache_put: cached entry. [D 20:00:57.614157] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.614164] op_queue add: 0x6eda80 [D 20:00:57.614173] DBPF I/O ops in progress: 3 [D 20:00:57.614177] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:00:57.614201] ========================================================= [D 20:00:57.614213] PINT_process_request [D 20:00:57.614221] tiling 9 copies [D 20:00:57.614203] dbpf_open_cache_put: cached entry. [D 20:00:57.614228] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.614240] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.614250] lvl 0 el 0 blk 0 by 249561088 [D 20:00:57.614274] to 249561088 ta 234881024 fi 268435456 [D 20:00:57.614285] basic type or contiguous data [D 20:00:57.614265] op_queue add: 0x6ea160 [D 20:00:57.614295] Request disp [D 20:00:57.614324] contig_offset = 249561088 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.614310] DBPF I/O ops in progress: 2 [D 20:00:57.614314] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:00:57.614337] calling distribute [D 20:00:57.614368] PINT_distribute [D 20:00:57.614376] of 249561088 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 249561088 exfl 1 [D 20:00:57.614385] begin iteration loff: 249561088 [D 20:00:57.614393] segment exceeds byte limit [D 20:00:57.614399] file being extended [D 20:00:57.614406] process a segment [D 20:00:57.614412] of 249561088 sz 2097152 [D 20:00:57.614418] add a segment 0 [D 20:00:57.614425] end iteration [D 20:00:57.614431] done with segments or bytes [D 20:00:57.614438] of 251658240 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.614532] next loff: 251658240 next poff: 251658240 [D 20:00:57.614541] return value 2097152 [D 20:00:57.614547] finished [D 20:00:57.614553] retval = 2097152 [D 20:00:57.614560] exiting distribute returned less than expected [D 20:00:57.614566] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 251658240 fo 268435456 eof 0 [D 20:00:57.614573] ========================================================= [D 20:00:57.614582] BMI_post_recv: addr: 6843072, offset: 0x2aaaba249010, size: 2097152, tag: 15 [D 20:00:57.614596] openib_check_cq: completed send wr_id 682e40. [D 20:00:57.614626] post_sr: 10.1.4.56:39990 bh 31 len 48 wr_id 682e70. [D 20:00:57.614640] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:00:57.614649] get_data_sync_mode returning 0 [D 20:00:57.614661] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.614668] dbpf_bstream_rw_list: mem_offset: 0x2aaab843a010, mem_size: 2097152 [D 20:00:57.614675] dbpf_bstream_rw_list: stream_offset: 281018368, stream_size: 2097152 [D 20:00:57.614682] dbpf_open_cache_get: called [D 20:00:57.614689] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.614828] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.614855] DBPF I/O ops in progress: 3 [D 20:00:57.614862] lio_listio called with the following aiocbs: [D 20:00:57.614869] aiocb_ptr_array[0]: fd: 13, off: 281018368, bytes: 2097152, buf: 0x2aaab843a010, type: 1 [D 20:00:57.614884] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [A 20:00:57.614819] troy.scl@10.1.4.56:39990 H=2863311531 S=0x669e10: io: request [D 20:00:57.617416] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:57.617427] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab150eea0 [D 20:00:57.617434] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab150eea0 [D 20:00:57.617444] [SM Exiting]: (0x669d10) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:57.617455] [SM Entering]: (0x669d10) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:57.617462] [SM frame get]: (0x669d10) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.617755] ========================================================= [D 20:00:57.617768] PINT_process_request [D 20:00:57.617775] tiling 10 copies [D 20:00:57.617783] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.617791] lvl 0 el 0 blk 0 by 283115520 [D 20:00:57.617798] to 283115520 ta 268435456 fi 301989888 [D 20:00:57.617805] basic type or contiguous data [D 20:00:57.617811] Request disp [D 20:00:57.617831] contig_offset = 283115520 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.617839] calling distribute [D 20:00:57.617845] PINT_distribute [D 20:00:57.617851] of 283115520 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 283115520 exfl 1 [D 20:00:57.617861] begin iteration loff: 283115520 [D 20:00:57.617868] segment exceeds byte limit [D 20:00:57.617874] file being extended [D 20:00:57.617881] process a segment [D 20:00:57.617886] of 283115520 sz 2097152 [D 20:00:57.617893] add a segment 0 [D 20:00:57.617899] end iteration [D 20:00:57.617905] done with segments or bytes [D 20:00:57.617911] of 285212672 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.617918] next loff: 285212672 next poff: 285212672 [D 20:00:57.617925] return value 2097152 [D 20:00:57.617932] finished [D 20:00:57.617938] retval = 2097152 [D 20:00:57.617943] exiting distribute returned less than expected [D 20:00:57.617950] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 285212672 fo 301989888 eof 0 [D 20:00:57.617958] ========================================================= [D 20:00:57.617965] BMI_post_recv: addr: 6843072, offset: 0x2aaaba44a010, size: 2097152, tag: 16 [D 20:00:57.617981] openib_check_cq: completed send wr_id 682e70. [D 20:00:57.617991] openib_check_cq: completed recv wr_id 6847d0. [D 20:00:57.618004] openib_post_rr: 10.1.4.56:39990 bh 66 wr_id 6847d0. [D 20:00:57.618013] openib_check_cq: completed recv wr_id 684800. [D 20:00:57.618021] openib_post_rr: 10.1.4.56:39990 bh 67 wr_id 684800. [D 20:00:57.618029] openib_check_cq: completed recv wr_id 684830. [D 20:00:57.618041] openib_post_rr: 10.1.4.56:39990 bh 68 wr_id 684830. [D 20:00:57.618048] openib_check_cq: completed recv wr_id 684860. [D 20:00:57.618056] openib_post_rr: 10.1.4.56:39990 bh 69 wr_id 684860. [D 20:00:57.618083] post_sr: 10.1.4.56:39990 bh 32 len 48 wr_id 682ea0. [D 20:00:57.618097] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:00:57.618107] get_data_sync_mode returning 0 [D 20:00:57.618119] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.618127] dbpf_bstream_rw_list: mem_offset: 0x2aaab863b010, mem_size: 2097152 [D 20:00:57.618134] dbpf_bstream_rw_list: stream_offset: 314572800, stream_size: 2097152 [D 20:00:57.618141] dbpf_open_cache_get: called [D 20:00:57.618148] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.618155] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.618176] DBPF I/O ops in progress: 4 [D 20:00:57.618183] lio_listio called with the following aiocbs: [D 20:00:57.618189] aiocb_ptr_array[0]: fd: 13, off: 314572800, bytes: 2097152, buf: 0x2aaab863b010, type: 1 [D 20:00:57.618201] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [D 20:00:57.620637] ========================================================= [D 20:00:57.621195] PINT_process_request [D 20:00:57.621208] tiling 11 copies [D 20:00:57.621217] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.621227] lvl 0 el 0 blk 0 by 316669952 [D 20:00:57.621233] to 316669952 ta 301989888 fi 335544320 [D 20:00:57.621241] basic type or contiguous data [D 20:00:57.621248] Request disp [D 20:00:57.621254] contig_offset = 316669952 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.621262] calling distribute [D 20:00:57.621268] PINT_distribute [D 20:00:57.621276] of 316669952 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 316669952 exfl 1 [D 20:00:57.621284] begin iteration loff: 316669952 [D 20:00:57.621292] segment exceeds byte limit [D 20:00:57.621299] file being extended [D 20:00:57.621306] process a segment [D 20:00:57.621312] of 316669952 sz 2097152 [D 20:00:57.621327] add a segment 0 [D 20:00:57.621334] end iteration [D 20:00:57.621340] done with segments or bytes [D 20:00:57.621346] of 318767104 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.621353] next loff: 318767104 next poff: 318767104 [D 20:00:57.621360] return value 2097152 [D 20:00:57.621367] finished [D 20:00:57.621373] retval = 2097152 [D 20:00:57.621379] exiting distribute returned less than expected [D 20:00:57.621385] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 318767104 fo 335544320 eof 0 [D 20:00:57.621393] ========================================================= [D 20:00:57.621400] BMI_post_recv: addr: 6843072, offset: 0x2aaaba64b010, size: 2097152, tag: 17 [D 20:00:57.621421] openib_check_cq: completed send wr_id 682ea0. [D 20:00:57.621432] openib_check_cq: completed recv wr_id 684890. [D 20:00:57.621451] openib_post_rr: 10.1.4.56:39990 bh 70 wr_id 684890. [D 20:00:57.621461] openib_check_cq: completed recv wr_id 6848c0. [D 20:00:57.621469] openib_post_rr: 10.1.4.56:39990 bh 71 wr_id 6848c0. [D 20:00:57.621508] post_sr: 10.1.4.56:39990 bh 33 len 48 wr_id 682ed0. [D 20:00:57.621523] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:00:57.621534] get_data_sync_mode returning 0 [D 20:00:57.621549] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.621557] dbpf_bstream_rw_list: mem_offset: 0x2aaab883c010, mem_size: 2097152 [D 20:00:57.621564] dbpf_bstream_rw_list: stream_offset: 348127232, stream_size: 2097152 [D 20:00:57.621571] dbpf_open_cache_get: called [D 20:00:57.621578] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.621585] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.621611] DBPF I/O ops in progress: 5 [D 20:00:57.621619] lio_listio called with the following aiocbs: [D 20:00:57.621625] aiocb_ptr_array[0]: fd: 13, off: 348127232, bytes: 2097152, buf: 0x2aaab883c010, type: 1 [D 20:00:57.621641] issue_or_delay_io_operation: lio_listio posted 0x6f0a90 (handle 2863311531, ret 0) [A 20:00:57.623601] troy.scl@10.1.4.56:39990 H=2863311531 S=0x669e10: io: start [D 20:00:57.623667] (0x669e10) io (prelude sm) state: getattr_if_needed [D 20:00:57.623675] About to retrieve attributes for handle 2863311531 [D 20:00:57.623696] op_queue add: 0x2aaab150f290 [D 20:00:57.624677] [SM Exiting]: (0x669d10) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:57.624713] [SM Entering]: (0x66afd0) unexpected_sm:map_request (status: 0) [D 20:00:57.624722] [SM frame get]: (0x66afd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.624731] server_post_unexpected_recv [D 20:00:57.624753] [SM Frame PUSH]: (0x2aaab150f3e0) frame: 0x2aaab1511040 [D 20:00:57.624760] [SM Locating]: (0x2aaab150f3e0) op-id: 999 [D 20:00:57.624767] server_op_state_get_machine 999 [D 20:00:57.624774] [SM Locating]: (0x2aaab150f3e0) located: unexpected_sm:post_unexpected [D 20:00:57.624781] [SM frame get]: (0x2aaab150f3e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.624789] [SM Entering]: (0x2aaab150f3e0) unexpected_sm:post_unexpected (status: 0) [D 20:00:57.624796] [SM frame get]: (0x2aaab150f3e0) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.624807] [SM Exiting]: (0x2aaab150f3e0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:57.624818] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:57.624826] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:57.624833] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 26 [D 20:00:57.624840] [SM frame get]: (0x66afd0) op-id: 999 index: 0 base-frm: 0 [D 20:00:57.624847] server_state_machine_start 0x66afd0 [D 20:00:57.624854] PINT_decode [D 20:00:57.624862] lebf_decode_req [D 20:00:57.624875] [SM Locating]: (0x66afd0) op-id: 3 [D 20:00:57.624882] server_op_state_get_machine 3 [D 20:00:57.624889] [SM push_state]: (0x66afd0) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:57.624897] [SM Locating]: (0x66afd0) located: prelude_sm:setup [D 20:00:57.624904] [SM Entering]: (0x66afd0) prelude_sm:setup (status: 0) [D 20:00:57.624912] [SM frame get]: (0x66afd0) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.624920] [SM Exiting]: (0x66afd0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:57.624927] [SM Exiting]: (0x66afd0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:57.624935] [SM push_state]: (0x66afd0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:57.624942] [SM Entering]: (0x66afd0) prelude_work_sm:req_sched (status: 0) [D 20:00:57.624949] [SM frame get]: (0x66afd0) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.624955] (0x66b0d0) io (prelude sm) state: req_sched [D 20:00:57.625225] ========================================================= [D 20:00:57.625237] PINT_process_request [D 20:00:57.625244] tiling 12 copies [D 20:00:57.625252] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.625260] lvl 0 el 0 blk 0 by 350224384 [D 20:00:57.625267] to 350224384 ta 335544320 fi 369098752 [D 20:00:57.625274] basic type or contiguous data [D 20:00:57.625281] Request disp [D 20:00:57.625288] contig_offset = 350224384 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.625295] calling distribute [D 20:00:57.625302] PINT_distribute [D 20:00:57.625309] of 350224384 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 350224384 exfl 1 [D 20:00:57.625330] begin iteration loff: 350224384 [D 20:00:57.625338] segment exceeds byte limit [D 20:00:57.625344] file being extended [D 20:00:57.625350] process a segment [D 20:00:57.625356] of 350224384 sz 2097152 [D 20:00:57.625363] add a segment 0 [D 20:00:57.625370] end iteration [D 20:00:57.625376] done with segments or bytes [D 20:00:57.625382] of 352321536 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.625389] next loff: 352321536 next poff: 352321536 [D 20:00:57.625395] return value 2097152 [D 20:00:57.625402] finished [D 20:00:57.625408] retval = 2097152 [D 20:00:57.625414] exiting distribute returned less than expected [D 20:00:57.625420] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 352321536 fo 369098752 eof 0 [D 20:00:57.625428] ========================================================= [D 20:00:57.625435] BMI_post_recv: addr: 6843072, offset: 0x2aaaba84c010, size: 2097152, tag: 18 [D 20:00:57.625451] openib_check_cq: completed send wr_id 682ed0. [D 20:00:57.625460] openib_check_cq: completed recv wr_id 6848f0. [D 20:00:57.625474] openib_post_rr: 10.1.4.56:39990 bh 72 wr_id 6848f0. [D 20:00:57.625483] openib_check_cq: completed recv wr_id 684920. [D 20:00:57.625491] openib_post_rr: 10.1.4.56:39990 bh 73 wr_id 684920. [D 20:00:57.625523] post_sr: 10.1.4.56:39990 bh 34 len 48 wr_id 682f00. [D 20:00:57.625537] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:00:57.625546] get_data_sync_mode returning 0 [D 20:00:57.625558] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.625565] dbpf_bstream_rw_list: mem_offset: 0x2aaab8a3d010, mem_size: 2097152 [D 20:00:57.625572] dbpf_bstream_rw_list: stream_offset: 381681664, stream_size: 2097152 [D 20:00:57.625580] dbpf_open_cache_get: called [D 20:00:57.625587] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.625594] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.625616] DBPF I/O ops in progress: 6 [D 20:00:57.625624] lio_listio called with the following aiocbs: [D 20:00:57.625631] aiocb_ptr_array[0]: fd: 13, off: 381681664, bytes: 2097152, buf: 0x2aaab8a3d010, type: 1 [D 20:00:57.625644] issue_or_delay_io_operation: lio_listio posted 0x6ea160 (handle 2863311531, ret 0) [D 20:00:57.625664] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:57.627985] dbpf_open_cache_get: called [D 20:00:57.628010] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.628019] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.628031] dbpf_open_cache_put: cached entry. [D 20:00:57.628093] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 476053504 [D 20:00:57.628112] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 476053504) [D 20:00:57.628122] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:57.628131] op_queue add: 0x2aaab150f290 [D 20:00:57.626316] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:00:57.628254] ========================================================= [D 20:00:57.628268] PINT_process_request [D 20:00:57.628277] tiling 13 copies [D 20:00:57.628285] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.628292] lvl 0 el 0 blk 0 by 383778816 [D 20:00:57.628299] to 383778816 ta 369098752 fi 402653184 [D 20:00:57.628307] basic type or contiguous data [D 20:00:57.628460] Request disp [D 20:00:57.628467] contig_offset = 383778816 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.628475] calling distribute [D 20:00:57.628483] PINT_distribute [D 20:00:57.628490] of 383778816 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 383778816 exfl 1 [D 20:00:57.628501] begin iteration loff: 383778816 [D 20:00:57.628509] segment exceeds byte limit [D 20:00:57.628516] file being extended [D 20:00:57.628522] process a segment [D 20:00:57.628528] of 383778816 sz 2097152 [D 20:00:57.628536] add a segment 0 [D 20:00:57.628543] end iteration [D 20:00:57.628549] done with segments or bytes [D 20:00:57.628555] of 385875968 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.628744] next loff: 385875968 next poff: 385875968 [D 20:00:57.628753] return value 2097152 [D 20:00:57.628760] finished [D 20:00:57.628767] retval = 2097152 [D 20:00:57.628773] exiting distribute returned less than expected [D 20:00:57.628779] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 385875968 fo 402653184 eof 0 [D 20:00:57.628787] ========================================================= [D 20:00:57.628796] BMI_post_recv: addr: 6843072, offset: 0x2aaabaa4d010, size: 2097152, tag: 19 [D 20:00:57.628814] openib_check_cq: completed send wr_id 682f00. [D 20:00:57.628827] openib_check_cq: completed recv wr_id 684950. [D 20:00:57.628845] openib_post_rr: 10.1.4.56:39990 bh 74 wr_id 684950. [D 20:00:57.628857] openib_check_cq: completed recv wr_id 684980. [D 20:00:57.628865] openib_post_rr: 10.1.4.56:39990 bh 75 wr_id 684980. [D 20:00:57.628902] post_sr: 10.1.4.56:39990 bh 35 len 48 wr_id 682f30. [D 20:00:57.628937] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:00:57.628948] get_data_sync_mode returning 0 [D 20:00:57.628961] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.628968] dbpf_bstream_rw_list: mem_offset: 0x2aaab8c3e010, mem_size: 2097152 [D 20:00:57.628975] dbpf_bstream_rw_list: stream_offset: 415236096, stream_size: 2097152 [D 20:00:57.628982] dbpf_open_cache_get: called [D 20:00:57.628989] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.628996] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.629005] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:57.629029] DBPF I/O ops in progress: 7 [D 20:00:57.629037] lio_listio called with the following aiocbs: [D 20:00:57.629044] aiocb_ptr_array[0]: fd: 13, off: 415236096, bytes: 2097152, buf: 0x2aaab8c3e010, type: 1 [D 20:00:57.629062] issue_or_delay_io_operation: lio_listio posted 0x6eaaf0 (handle 2863311531, ret 0) [D 20:00:57.628230] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.633036] ========================================================= [D 20:00:57.633076] PINT_process_request [D 20:00:57.633084] tiling 14 copies [D 20:00:57.633093] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.633101] lvl 0 el 0 blk 0 by 417333248 [D 20:00:57.633108] to 417333248 ta 402653184 fi 436207616 [D 20:00:57.633115] basic type or contiguous data [D 20:00:57.633121] Request disp [D 20:00:57.633128] contig_offset = 417333248 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.633135] calling distribute [D 20:00:57.633141] PINT_distribute [D 20:00:57.633147] of 417333248 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 417333248 exfl 1 [D 20:00:57.633157] begin iteration loff: 417333248 [D 20:00:57.633165] segment exceeds byte limit [D 20:00:57.633171] file being extended [D 20:00:57.633178] process a segment [D 20:00:57.633183] of 417333248 sz 2097152 [D 20:00:57.633190] add a segment 0 [D 20:00:57.633196] end iteration [D 20:00:57.633202] done with segments or bytes [D 20:00:57.633208] of 419430400 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.633215] next loff: 419430400 next poff: 419430400 [D 20:00:57.633222] return value 2097152 [D 20:00:57.633229] finished [D 20:00:57.633235] retval = 2097152 [D 20:00:57.633241] exiting distribute returned less than expected [D 20:00:57.633247] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 419430400 fo 436207616 eof 0 [D 20:00:57.633255] ========================================================= [D 20:00:57.633262] BMI_post_recv: addr: 6843072, offset: 0x2aaabac4f010, size: 2097152, tag: 20 [D 20:00:57.633279] openib_check_cq: completed send wr_id 682f30. [D 20:00:57.633291] openib_check_cq: completed recv wr_id 6849b0. [D 20:00:57.633303] openib_post_rr: 10.1.4.56:39990 bh 76 wr_id 6849b0. [D 20:00:57.633313] openib_check_cq: completed recv wr_id 6849e0. [D 20:00:57.633320] openib_post_rr: 10.1.4.56:39990 bh 77 wr_id 6849e0. [D 20:00:57.633351] post_sr: 10.1.4.56:39990 bh 36 len 48 wr_id 682f60. [D 20:00:57.633365] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:00:57.633375] get_data_sync_mode returning 0 [D 20:00:57.633389] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.633396] dbpf_bstream_rw_list: mem_offset: 0x2aaab9040010, mem_size: 2097152 [D 20:00:57.633403] dbpf_bstream_rw_list: stream_offset: 448790528, stream_size: 2097152 [D 20:00:57.633411] dbpf_open_cache_get: called [D 20:00:57.629085] dbpf_open_cache_put: cached entry. [D 20:00:57.633428] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.633436] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.633459] DBPF I/O ops in progress: 8 [D 20:00:57.633467] lio_listio called with the following aiocbs: [D 20:00:57.633473] aiocb_ptr_array[0]: fd: 13, off: 448790528, bytes: 2097152, buf: 0x2aaab9040010, type: 1 [D 20:00:57.633487] issue_or_delay_io_operation: lio_listio posted 0x6ea2b0 (handle 2863311531, ret 0) [A 20:00:57.635408] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66b0d0: io: request [D 20:00:57.635451] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:57.635458] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab150f110 [D 20:00:57.635465] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab150f110 [D 20:00:57.635473] [SM Exiting]: (0x66afd0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:57.635482] [SM Entering]: (0x66afd0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:57.635489] [SM frame get]: (0x66afd0) op-id: 3 index: 0 base-frm: 0 [D 20:00:57.626311] --- aio_progress_notification called with handle 2863311531 (0x6ea7d0) [D 20:00:57.635671] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.635684] dbpf_open_cache_put: cached entry. [D 20:00:57.635693] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.635694] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.635864] ========================================================= [D 20:00:57.635877] PINT_process_request [D 20:00:57.635885] tiling 15 copies [D 20:00:57.635892] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.635900] lvl 0 el 0 blk 0 by 450887680 [D 20:00:57.635907] to 450887680 ta 436207616 fi 469762048 [D 20:00:57.635914] basic type or contiguous data [D 20:00:57.635920] Request disp [D 20:00:57.635927] contig_offset = 450887680 contig_size = 52428800 lvl_flag = 1 [D 20:00:57.635942] calling distribute [D 20:00:57.635949] PINT_distribute [D 20:00:57.635957] of 450887680 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 450887680 exfl 1 [D 20:00:57.635966] begin iteration loff: 450887680 [D 20:00:57.635973] segment exceeds byte limit [D 20:00:57.635979] file being extended [D 20:00:57.635985] process a segment [D 20:00:57.635991] of 450887680 sz 2097152 [D 20:00:57.635999] add a segment 0 [D 20:00:57.636007] end iteration [D 20:00:57.636013] done with segments or bytes [D 20:00:57.636019] of 452984832 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.636026] next loff: 452984832 next poff: 452984832 [D 20:00:57.636032] return value 2097152 [D 20:00:57.636038] finished [D 20:00:57.636044] retval = 2097152 [D 20:00:57.636050] exiting distribute returned less than expected [D 20:00:57.636057] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 452984832 fo 469762048 eof 0 [D 20:00:57.636064] ========================================================= [D 20:00:57.636071] BMI_post_recv: addr: 6843072, offset: 0x2aaabae50010, size: 2097152, tag: 21 [D 20:00:57.636091] openib_check_cq: completed recv wr_id 684a10. [D 20:00:57.636111] openib_post_rr: 10.1.4.56:39990 bh 78 wr_id 684a10. [D 20:00:57.636123] openib_check_cq: completed send wr_id 682f60. [D 20:00:57.636130] openib_check_cq: completed recv wr_id 684a40. [D 20:00:57.636138] openib_post_rr: 10.1.4.56:39990 bh 79 wr_id 684a40. [D 20:00:57.636172] post_sr: 10.1.4.56:39990 bh 37 len 48 wr_id 682f90. [D 20:00:57.636186] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:00:57.636198] get_data_sync_mode returning 0 [D 20:00:57.636213] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.636220] dbpf_bstream_rw_list: mem_offset: 0x2aaab9241010, mem_size: 2097152 [D 20:00:57.636228] dbpf_bstream_rw_list: stream_offset: 14680064, stream_size: 2097152 [D 20:00:57.636236] dbpf_open_cache_get: called [D 20:00:57.636244] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.636251] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.636277] DBPF I/O ops in progress: 9 [D 20:00:57.636284] lio_listio called with the following aiocbs: [D 20:00:57.636291] aiocb_ptr_array[0]: fd: 13, off: 14680064, bytes: 2097152, buf: 0x2aaab9241010, type: 1 [D 20:00:57.636302] issue_or_delay_io_operation: lio_listio posted 0x6eac40 (handle 2863311531, ret 0) [D 20:00:57.636311] ========================================================= [D 20:00:57.636318] PINT_process_request [D 20:00:57.636325] tiling 2 copies [D 20:00:57.636331] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.636339] lvl 0 el 0 blk 0 by 16777216 [D 20:00:57.636345] to 16777216 ta 0 fi 33554432 [D 20:00:57.636352] basic type or contiguous data [D 20:00:57.636358] Request disp [D 20:00:57.636364] contig_offset = 16777216 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.636370] calling distribute [D 20:00:57.636376] PINT_distribute [D 20:00:57.636382] of 16777216 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 16777216 exfl 1 [D 20:00:57.636390] begin iteration loff: 16777216 [D 20:00:57.636396] segment exceeds byte limit [D 20:00:57.636403] file being extended [D 20:00:57.636420] process a segment [D 20:00:57.636427] of 16777216 sz 2097152 [D 20:00:57.636433] add a segment 0 [D 20:00:57.636439] end iteration [D 20:00:57.636445] done with segments or bytes [D 20:00:57.636451] of 18874368 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.636458] next loff: 18874368 next poff: 18874368 [D 20:00:57.636464] return value 2097152 [D 20:00:57.636470] finished [D 20:00:57.636476] retval = 2097152 [D 20:00:57.636482] exiting distribute returned less than expected [D 20:00:57.636488] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 18874368 fo 33554432 eof 0 [D 20:00:57.636495] ========================================================= [D 20:00:57.636502] BMI_post_recv: addr: 6843072, offset: 0x2aaaac6bb010, size: 2097152, tag: 8 [D 20:00:57.636509] openib_check_cq: completed send wr_id 682f90. [D 20:00:57.636520] post_sr: 10.1.4.56:39990 bh 38 len 48 wr_id 682fc0. [D 20:00:57.636531] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:00:57.636538] get_data_sync_mode returning 0 [D 20:00:57.636546] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.636552] dbpf_bstream_rw_list: mem_offset: 0x2aaab9442010, mem_size: 2097152 [D 20:00:57.636559] dbpf_bstream_rw_list: stream_offset: 48234496, stream_size: 2097152 [D 20:00:57.636566] dbpf_open_cache_get: called [D 20:00:57.636572] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.636578] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.636595] DBPF I/O ops in progress: 10 [D 20:00:57.636602] lio_listio called with the following aiocbs: [D 20:00:57.636608] aiocb_ptr_array[0]: fd: 13, off: 48234496, bytes: 2097152, buf: 0x2aaab9442010, type: 1 [D 20:00:57.636617] issue_or_delay_io_operation: lio_listio posted 0x6ead90 (handle 2863311531, ret 0) [D 20:00:57.636626] ========================================================= [D 20:00:57.636632] PINT_process_request [D 20:00:57.636638] tiling 3 copies [D 20:00:57.636644] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.636651] lvl 0 el 0 blk 0 by 50331648 [D 20:00:57.636658] to 50331648 ta 33554432 fi 67108864 [D 20:00:57.636665] basic type or contiguous data [D 20:00:57.636671] Request disp [D 20:00:57.636677] contig_offset = 50331648 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.636684] calling distribute [D 20:00:57.636690] PINT_distribute [D 20:00:57.636695] of 50331648 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 50331648 exfl 1 [D 20:00:57.636703] begin iteration loff: 50331648 [D 20:00:57.636710] segment exceeds byte limit [D 20:00:57.636715] file being extended [D 20:00:57.636721] process a segment [D 20:00:57.636727] of 50331648 sz 2097152 [D 20:00:57.636733] add a segment 0 [D 20:00:57.636740] end iteration [D 20:00:57.636745] done with segments or bytes [D 20:00:57.636751] of 52428800 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.636758] next loff: 52428800 next poff: 52428800 [D 20:00:57.636764] return value 2097152 [D 20:00:57.636771] finished [D 20:00:57.636776] retval = 2097152 [D 20:00:57.636782] exiting distribute returned less than expected [D 20:00:57.636788] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 52428800 fo 67108864 eof 0 [D 20:00:57.636795] ========================================================= [D 20:00:57.636801] BMI_post_recv: addr: 6843072, offset: 0x2aaaac8bc010, size: 2097152, tag: 9 [D 20:00:57.636810] openib_check_cq: completed send wr_id 682fc0. [D 20:00:57.636820] post_sr: 10.1.4.56:39990 bh 39 len 48 wr_id 682ff0. [D 20:00:57.636830] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:00:57.636838] get_data_sync_mode returning 0 [D 20:00:57.636847] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.636854] dbpf_bstream_rw_list: mem_offset: 0x2aaab9643010, mem_size: 2097152 [D 20:00:57.636861] dbpf_bstream_rw_list: stream_offset: 81788928, stream_size: 2097152 [D 20:00:57.636868] dbpf_open_cache_get: called [D 20:00:57.636875] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.636881] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.636899] DBPF I/O ops in progress: 11 [D 20:00:57.636907] lio_listio called with the following aiocbs: [D 20:00:57.636913] aiocb_ptr_array[0]: fd: 13, off: 81788928, bytes: 2097152, buf: 0x2aaab9643010, type: 1 [D 20:00:57.636922] issue_or_delay_io_operation: lio_listio posted 0x7063c0 (handle 2863311531, ret 0) [D 20:00:57.636930] ========================================================= [D 20:00:57.636936] PINT_process_request [D 20:00:57.636943] tiling 4 copies [D 20:00:57.636949] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.636956] lvl 0 el 0 blk 0 by 83886080 [D 20:00:57.636963] to 83886080 ta 67108864 fi 100663296 [D 20:00:57.636970] basic type or contiguous data [D 20:00:57.636976] Request disp [D 20:00:57.636982] contig_offset = 83886080 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.636989] calling distribute [D 20:00:57.636995] PINT_distribute [D 20:00:57.637001] of 83886080 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 83886080 exfl 1 [D 20:00:57.637009] begin iteration loff: 83886080 [D 20:00:57.637015] segment exceeds byte limit [D 20:00:57.637021] file being extended [D 20:00:57.637027] process a segment [D 20:00:57.637033] of 83886080 sz 2097152 [D 20:00:57.637039] add a segment 0 [D 20:00:57.637045] end iteration [D 20:00:57.637051] done with segments or bytes [D 20:00:57.637057] of 85983232 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.637064] next loff: 85983232 next poff: 85983232 [D 20:00:57.637070] return value 2097152 [D 20:00:57.637076] finished [D 20:00:57.637082] retval = 2097152 [D 20:00:57.637088] exiting distribute returned less than expected [D 20:00:57.637094] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 85983232 fo 100663296 eof 0 [D 20:00:57.637101] ========================================================= [D 20:00:57.637108] BMI_post_recv: addr: 6843072, offset: 0x2aaaacabd010, size: 2097152, tag: 10 [D 20:00:57.637116] openib_check_cq: completed send wr_id 682ff0. [D 20:00:57.637128] post_sr: 10.1.4.56:39990 bh 40 len 48 wr_id 683020. [D 20:00:57.637154] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:00:57.637163] get_data_sync_mode returning 0 [D 20:00:57.637171] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.637179] dbpf_bstream_rw_list: mem_offset: 0x2aaab9a45010, mem_size: 2097152 [D 20:00:57.637186] dbpf_bstream_rw_list: stream_offset: 115343360, stream_size: 2097152 [D 20:00:57.637192] dbpf_open_cache_get: called [D 20:00:57.637199] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.637205] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.637224] DBPF I/O ops in progress: 12 [D 20:00:57.637230] lio_listio called with the following aiocbs: [D 20:00:57.637237] aiocb_ptr_array[0]: fd: 13, off: 115343360, bytes: 2097152, buf: 0x2aaab9a45010, type: 1 [D 20:00:57.637246] issue_or_delay_io_operation: lio_listio posted 0x708f20 (handle 2863311531, ret 0) [D 20:00:57.637254] ========================================================= [D 20:00:57.637260] PINT_process_request [D 20:00:57.637267] tiling 5 copies [D 20:00:57.637273] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.637280] lvl 0 el 0 blk 0 by 117440512 [D 20:00:57.637287] to 117440512 ta 100663296 fi 134217728 [D 20:00:57.637294] basic type or contiguous data [D 20:00:57.637300] Request disp [D 20:00:57.637306] contig_offset = 117440512 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.637313] calling distribute [D 20:00:57.637319] PINT_distribute [D 20:00:57.637325] of 117440512 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 117440512 exfl 1 [D 20:00:57.637333] begin iteration loff: 117440512 [D 20:00:57.637339] segment exceeds byte limit [D 20:00:57.637346] file being extended [D 20:00:57.637353] process a segment [D 20:00:57.637359] of 117440512 sz 2097152 [D 20:00:57.637365] add a segment 0 [D 20:00:57.637371] end iteration [D 20:00:57.637377] done with segments or bytes [D 20:00:57.637383] of 119537664 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.637390] next loff: 119537664 next poff: 119537664 [D 20:00:57.637397] return value 2097152 [D 20:00:57.637403] finished [D 20:00:57.637409] retval = 2097152 [D 20:00:57.637415] exiting distribute returned less than expected [D 20:00:57.637421] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 119537664 fo 134217728 eof 0 [D 20:00:57.637428] ========================================================= [D 20:00:57.637435] BMI_post_recv: addr: 6843072, offset: 0x2aaaaccbe010, size: 2097152, tag: 11 [D 20:00:57.637443] openib_check_cq: completed send wr_id 683020. [D 20:00:57.637454] post_sr: 10.1.4.56:39990 bh 41 len 48 wr_id 683050. [D 20:00:57.637464] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:00:57.637473] get_data_sync_mode returning 0 [D 20:00:57.637481] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.637487] dbpf_bstream_rw_list: mem_offset: 0x2aaab9c46010, mem_size: 2097152 [D 20:00:57.637495] dbpf_bstream_rw_list: stream_offset: 148897792, stream_size: 2097152 [D 20:00:57.637502] dbpf_open_cache_get: called [D 20:00:57.637508] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.637514] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.637545] DBPF I/O ops in progress: 13 [D 20:00:57.637554] lio_listio called with the following aiocbs: [D 20:00:57.637560] aiocb_ptr_array[0]: fd: 13, off: 148897792, bytes: 2097152, buf: 0x2aaab9c46010, type: 1 [D 20:00:57.637569] issue_or_delay_io_operation: lio_listio posted 0x70ba80 (handle 2863311531, ret 0) [D 20:00:57.637577] ========================================================= [D 20:00:57.637583] PINT_process_request [D 20:00:57.637590] tiling 6 copies [D 20:00:57.637596] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.637603] lvl 0 el 0 blk 0 by 150994944 [D 20:00:57.637610] to 150994944 ta 134217728 fi 167772160 [D 20:00:57.637617] basic type or contiguous data [D 20:00:57.637623] Request disp [D 20:00:57.637629] contig_offset = 150994944 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.637636] calling distribute [D 20:00:57.637643] PINT_distribute [D 20:00:57.637649] of 150994944 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 150994944 exfl 1 [D 20:00:57.637657] begin iteration loff: 150994944 [D 20:00:57.637663] segment exceeds byte limit [D 20:00:57.637669] file being extended [D 20:00:57.637675] process a segment [D 20:00:57.637681] of 150994944 sz 2097152 [D 20:00:57.637688] add a segment 0 [D 20:00:57.637694] end iteration [D 20:00:57.637699] done with segments or bytes [D 20:00:57.637705] of 153092096 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.637713] next loff: 153092096 next poff: 153092096 [D 20:00:57.637719] return value 2097152 [D 20:00:57.637726] finished [D 20:00:57.637732] retval = 2097152 [D 20:00:57.637738] exiting distribute returned less than expected [D 20:00:57.637744] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 153092096 fo 167772160 eof 0 [D 20:00:57.637752] ========================================================= [D 20:00:57.637758] BMI_post_recv: addr: 6843072, offset: 0x2aaaacebf010, size: 2097152, tag: 12 [D 20:00:57.637767] openib_check_cq: completed send wr_id 683050. [D 20:00:57.637777] post_sr: 10.1.4.56:39990 bh 42 len 48 wr_id 683080. [D 20:00:57.637787] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:00:57.637795] get_data_sync_mode returning 0 [D 20:00:57.637802] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.637809] dbpf_bstream_rw_list: mem_offset: 0x2aaab9e47010, mem_size: 2097152 [D 20:00:57.637816] dbpf_bstream_rw_list: stream_offset: 182452224, stream_size: 2097152 [D 20:00:57.637823] dbpf_open_cache_get: called [D 20:00:57.637830] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.637836] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.635730] op_queue add: 0x6ea7d0 [D 20:00:57.626305] --- aio_progress_notification called with handle 2863311531 (0x6ea680) [D 20:00:57.639349] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:00:57.639333] --- aio_progress_notification called with handle 2863311531 (0x6f0a90) [D 20:00:57.639383] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.639396] dbpf_open_cache_put: cached entry. [D 20:00:57.639404] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.639412] op_queue add: 0x6f0a90 [D 20:00:57.639357] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.639438] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:00:57.639446] dbpf_open_cache_put: cached entry. [D 20:00:57.639455] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.639461] op_queue add: 0x6ea680 [D 20:00:57.639326] DBPF I/O ops in progress: 12 [D 20:00:57.639477] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:00:57.630398] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:00:57.639519] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.639530] dbpf_open_cache_put: cached entry. [D 20:00:57.639536] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.639545] DBPF I/O ops in progress: 13 [D 20:00:57.639545] op_queue add: 0x6eda80 [D 20:00:57.639571] DBPF I/O ops in progress: 12 [D 20:00:57.639574] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:00:57.639585] DBPF I/O ops in progress: 11 [D 20:00:57.639561] lio_listio called with the following aiocbs: [D 20:00:57.639604] DBPF I/O ops in progress: 10 [D 20:00:57.639605] aiocb_ptr_array[0]: fd: 13, off: 182452224, bytes: 2097152, buf: 0x2aaab9e47010, type: 1 [D 20:00:57.639650] issue_or_delay_io_operation: lio_listio posted 0x70e5e0 (handle 2863311531, ret 0) [D 20:00:57.639660] ========================================================= [D 20:00:57.639667] PINT_process_request [D 20:00:57.639674] tiling 7 copies [D 20:00:57.639686] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.639694] lvl 0 el 0 blk 0 by 184549376 [D 20:00:57.639700] to 184549376 ta 167772160 fi 201326592 [D 20:00:57.639707] basic type or contiguous data [D 20:00:57.639713] Request disp [D 20:00:57.639719] contig_offset = 184549376 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.639726] calling distribute [D 20:00:57.639732] PINT_distribute [D 20:00:57.639738] of 184549376 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 184549376 exfl 1 [D 20:00:57.639746] begin iteration loff: 184549376 [D 20:00:57.639752] segment exceeds byte limit [D 20:00:57.639758] file being extended [D 20:00:57.639764] process a segment [D 20:00:57.639770] of 184549376 sz 2097152 [D 20:00:57.639776] add a segment 0 [D 20:00:57.639782] end iteration [D 20:00:57.639788] done with segments or bytes [D 20:00:57.639794] of 186646528 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.639801] next loff: 186646528 next poff: 186646528 [D 20:00:57.639807] return value 2097152 [D 20:00:57.639813] finished [D 20:00:57.639819] retval = 2097152 [D 20:00:57.639824] exiting distribute returned less than expected [D 20:00:57.639831] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 186646528 fo 201326592 eof 0 [D 20:00:57.639838] ========================================================= [D 20:00:57.639844] BMI_post_recv: addr: 6843072, offset: 0x2aaaad0c0010, size: 2097152, tag: 13 [D 20:00:57.639855] openib_check_cq: completed send wr_id 683080. [D 20:00:57.639876] post_sr: 10.1.4.56:39990 bh 43 len 48 wr_id 6830b0. [D 20:00:57.639885] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:00:57.639893] get_data_sync_mode returning 0 [D 20:00:57.639902] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.639909] dbpf_bstream_rw_list: mem_offset: 0x2aaaba048010, mem_size: 2097152 [D 20:00:57.639915] dbpf_bstream_rw_list: stream_offset: 216006656, stream_size: 2097152 [D 20:00:57.639922] dbpf_open_cache_get: called [D 20:00:57.639928] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.639935] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.639947] DBPF I/O ops in progress: 11 [D 20:00:57.639954] lio_listio called with the following aiocbs: [D 20:00:57.639960] aiocb_ptr_array[0]: fd: 13, off: 216006656, bytes: 2097152, buf: 0x2aaaba048010, type: 1 [D 20:00:57.639968] issue_or_delay_io_operation: lio_listio posted 0x6f0a90 (handle 2863311531, ret 0) [D 20:00:57.639976] ========================================================= [D 20:00:57.639982] PINT_process_request [D 20:00:57.639988] tiling 8 copies [D 20:00:57.639994] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.640002] lvl 0 el 0 blk 0 by 218103808 [D 20:00:57.640008] to 218103808 ta 201326592 fi 234881024 [D 20:00:57.640015] basic type or contiguous data [D 20:00:57.640021] Request disp [D 20:00:57.640027] contig_offset = 218103808 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.640033] calling distribute [D 20:00:57.640039] PINT_distribute [D 20:00:57.640045] of 218103808 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 218103808 exfl 1 [D 20:00:57.640052] begin iteration loff: 218103808 [D 20:00:57.640059] segment exceeds byte limit [D 20:00:57.640064] file being extended [D 20:00:57.640070] process a segment [D 20:00:57.640076] of 218103808 sz 2097152 [D 20:00:57.640082] add a segment 0 [D 20:00:57.640088] end iteration [D 20:00:57.640094] done with segments or bytes [D 20:00:57.640100] of 220200960 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.640106] next loff: 220200960 next poff: 220200960 [D 20:00:57.640113] return value 2097152 [D 20:00:57.640119] finished [D 20:00:57.640124] retval = 2097152 [D 20:00:57.640130] exiting distribute returned less than expected [D 20:00:57.640136] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 220200960 fo 234881024 eof 0 [D 20:00:57.640143] ========================================================= [D 20:00:57.640149] BMI_post_recv: addr: 6843072, offset: 0x2aaaad2c1010, size: 2097152, tag: 14 [D 20:00:57.640157] openib_check_cq: completed send wr_id 6830b0. [D 20:00:57.640167] post_sr: 10.1.4.56:39990 bh 44 len 48 wr_id 6830e0. [D 20:00:57.640175] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:00:57.640182] get_data_sync_mode returning 0 [D 20:00:57.640189] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.640196] dbpf_bstream_rw_list: mem_offset: 0x2aaaba249010, mem_size: 2097152 [D 20:00:57.640202] dbpf_bstream_rw_list: stream_offset: 249561088, stream_size: 2097152 [D 20:00:57.640209] dbpf_open_cache_get: called [D 20:00:57.640215] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.640221] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.640235] DBPF I/O ops in progress: 12 [D 20:00:57.640242] lio_listio called with the following aiocbs: [D 20:00:57.640247] aiocb_ptr_array[0]: fd: 13, off: 249561088, bytes: 2097152, buf: 0x2aaaba249010, type: 1 [D 20:00:57.640256] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [D 20:00:57.640263] ========================================================= [D 20:00:57.640269] PINT_process_request [D 20:00:57.640275] tiling 9 copies [D 20:00:57.640281] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.640289] lvl 0 el 0 blk 0 by 251658240 [D 20:00:57.640295] to 251658240 ta 234881024 fi 268435456 [D 20:00:57.640302] basic type or contiguous data [D 20:00:57.640308] Request disp [D 20:00:57.640314] contig_offset = 251658240 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.640321] calling distribute [D 20:00:57.640326] PINT_distribute [D 20:00:57.640332] of 251658240 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 251658240 exfl 1 [D 20:00:57.640340] begin iteration loff: 251658240 [D 20:00:57.640346] segment exceeds byte limit [D 20:00:57.640352] file being extended [D 20:00:57.640357] process a segment [D 20:00:57.640363] of 251658240 sz 2097152 [D 20:00:57.640369] add a segment 0 [D 20:00:57.640375] end iteration [D 20:00:57.640381] done with segments or bytes [D 20:00:57.640386] of 253755392 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.640393] next loff: 253755392 next poff: 253755392 [D 20:00:57.640399] return value 2097152 [D 20:00:57.640405] finished [D 20:00:57.640422] --- aio_progress_notification called with handle 2863311531 (0x6ea160) [D 20:00:57.640455] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.640471] dbpf_open_cache_put: cached entry. [D 20:00:57.640481] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.640434] retval = 2097152 [D 20:00:57.640425] op_queue add: 0x6ec5d0 [D 20:00:57.640503] exiting distribute returned less than expected [D 20:00:57.640522] DBPF I/O ops in progress: 11 [D 20:00:57.640527] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 253755392 fo 268435456 eof 0 [D 20:00:57.640557] op_queue add: 0x6ea160 [D 20:00:57.640566] ========================================================= [D 20:00:57.640579] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:00:57.640600] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:00:57.640582] DBPF I/O ops in progress: 10 [D 20:00:57.640591] BMI_post_recv: addr: 6843072, offset: 0x2aaaad4c2010, size: 2097152, tag: 15 [D 20:00:57.640634] openib_check_cq: completed send wr_id 6830e0. [D 20:00:57.640645] post_sr: 10.1.4.56:39990 bh 45 len 48 wr_id 683110. [D 20:00:57.640663] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:00:57.640671] get_data_sync_mode returning 0 [D 20:00:57.640678] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.640684] dbpf_bstream_rw_list: mem_offset: 0x2aaaba44a010, mem_size: 2097152 [D 20:00:57.640691] dbpf_bstream_rw_list: stream_offset: 283115520, stream_size: 2097152 [D 20:00:57.640697] dbpf_open_cache_get: called [D 20:00:57.640704] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.640710] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.640723] DBPF I/O ops in progress: 11 [D 20:00:57.640730] lio_listio called with the following aiocbs: [D 20:00:57.640736] aiocb_ptr_array[0]: fd: 13, off: 283115520, bytes: 2097152, buf: 0x2aaaba44a010, type: 1 [D 20:00:57.640745] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [D 20:00:57.640753] ========================================================= [D 20:00:57.640759] PINT_process_request [D 20:00:57.640765] tiling 10 copies [D 20:00:57.640771] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.640778] lvl 0 el 0 blk 0 by 285212672 [D 20:00:57.640784] to 285212672 ta 268435456 fi 301989888 [D 20:00:57.640791] basic type or contiguous data [D 20:00:57.640797] Request disp [D 20:00:57.640803] contig_offset = 285212672 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.640810] calling distribute [D 20:00:57.640816] PINT_distribute [D 20:00:57.640821] of 285212672 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 285212672 exfl 1 [D 20:00:57.640829] begin iteration loff: 285212672 [D 20:00:57.640835] segment exceeds byte limit [D 20:00:57.640841] file being extended [D 20:00:57.640847] process a segment [D 20:00:57.640853] of 285212672 sz 2097152 [D 20:00:57.640859] add a segment 0 [D 20:00:57.640865] end iteration [D 20:00:57.640871] done with segments or bytes [D 20:00:57.640877] of 287309824 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.640884] next loff: 287309824 next poff: 287309824 [D 20:00:57.640890] return value 2097152 [D 20:00:57.640896] finished [D 20:00:57.640901] retval = 2097152 [D 20:00:57.640907] exiting distribute returned less than expected [D 20:00:57.640913] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 287309824 fo 301989888 eof 0 [D 20:00:57.640920] ========================================================= [D 20:00:57.640927] BMI_post_recv: addr: 6843072, offset: 0x2aaaad6c3010, size: 2097152, tag: 16 [D 20:00:57.640934] openib_check_cq: completed send wr_id 683110. [D 20:00:57.640944] post_sr: 10.1.4.56:39990 bh 46 len 48 wr_id 683140. [D 20:00:57.640952] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:00:57.640959] get_data_sync_mode returning 0 [D 20:00:57.640966] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.640972] dbpf_bstream_rw_list: mem_offset: 0x2aaaba64b010, mem_size: 2097152 [D 20:00:57.640979] dbpf_bstream_rw_list: stream_offset: 316669952, stream_size: 2097152 [D 20:00:57.640985] dbpf_open_cache_get: called [D 20:00:57.640991] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.640997] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.641009] DBPF I/O ops in progress: 12 [D 20:00:57.641015] lio_listio called with the following aiocbs: [D 20:00:57.641021] aiocb_ptr_array[0]: fd: 13, off: 316669952, bytes: 2097152, buf: 0x2aaaba64b010, type: 1 [D 20:00:57.641030] issue_or_delay_io_operation: lio_listio posted 0x6f0f00 (handle 2863311531, ret 0) [D 20:00:57.641037] ========================================================= [D 20:00:57.641044] PINT_process_request [D 20:00:57.641050] tiling 11 copies [D 20:00:57.641056] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.641063] lvl 0 el 0 blk 0 by 318767104 [D 20:00:57.641069] to 318767104 ta 301989888 fi 335544320 [D 20:00:57.641076] basic type or contiguous data [D 20:00:57.641082] Request disp [D 20:00:57.641088] contig_offset = 318767104 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.641094] calling distribute [D 20:00:57.641100] PINT_distribute [D 20:00:57.641106] of 318767104 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 318767104 exfl 1 [D 20:00:57.641113] begin iteration loff: 318767104 [D 20:00:57.641119] segment exceeds byte limit [D 20:00:57.641125] file being extended [D 20:00:57.641131] process a segment [D 20:00:57.641136] of 318767104 sz 2097152 [D 20:00:57.641142] add a segment 0 [D 20:00:57.641148] end iteration [D 20:00:57.641154] done with segments or bytes [D 20:00:57.641160] of 320864256 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.641166] next loff: 320864256 next poff: 320864256 [D 20:00:57.641173] return value 2097152 [D 20:00:57.641179] finished [D 20:00:57.641184] retval = 2097152 [D 20:00:57.641190] exiting distribute returned less than expected [D 20:00:57.641196] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 320864256 fo 335544320 eof 0 [D 20:00:57.641203] ========================================================= [D 20:00:57.641209] BMI_post_recv: addr: 6843072, offset: 0x2aaaad8c4010, size: 2097152, tag: 17 [D 20:00:57.641217] openib_check_cq: completed send wr_id 683140. [D 20:00:57.641226] post_sr: 10.1.4.56:39990 bh 47 len 48 wr_id 683170. [D 20:00:57.641234] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:00:57.641241] get_data_sync_mode returning 0 [D 20:00:57.641248] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.641254] dbpf_bstream_rw_list: mem_offset: 0x2aaaba84c010, mem_size: 2097152 [D 20:00:57.641261] dbpf_bstream_rw_list: stream_offset: 350224384, stream_size: 2097152 [D 20:00:57.641267] dbpf_open_cache_get: called [D 20:00:57.641273] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.641280] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.641292] DBPF I/O ops in progress: 13 [D 20:00:57.641299] lio_listio called with the following aiocbs: [D 20:00:57.641305] aiocb_ptr_array[0]: fd: 13, off: 350224384, bytes: 2097152, buf: 0x2aaaba84c010, type: 1 [D 20:00:57.641313] issue_or_delay_io_operation: lio_listio posted 0x6ea140 (handle 2863311531, ret 0) [D 20:00:57.641321] ========================================================= [D 20:00:57.641327] PINT_process_request [D 20:00:57.641333] tiling 12 copies [D 20:00:57.641339] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.641346] lvl 0 el 0 blk 0 by 352321536 [D 20:00:57.641353] to 352321536 ta 335544320 fi 369098752 [D 20:00:57.641359] basic type or contiguous data [D 20:00:57.641365] Request disp [D 20:00:57.641371] contig_offset = 352321536 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.641377] calling distribute [D 20:00:57.641383] PINT_distribute [D 20:00:57.641389] of 352321536 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 352321536 exfl 1 [D 20:00:57.641396] begin iteration loff: 352321536 [D 20:00:57.641402] segment exceeds byte limit [D 20:00:57.641408] file being extended [D 20:00:57.641413] process a segment [D 20:00:57.641419] of 352321536 sz 2097152 [D 20:00:57.641425] add a segment 0 [D 20:00:57.641431] end iteration [D 20:00:57.641437] done with segments or bytes [D 20:00:57.641442] of 354418688 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:57.641449] next loff: 354418688 next poff: 354418688 [D 20:00:57.641455] return value 2097152 [D 20:00:57.641461] finished [D 20:00:57.641467] retval = 2097152 [D 20:00:57.641473] exiting distribute returned less than expected [D 20:00:57.641479] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 354418688 fo 369098752 eof 0 [D 20:00:57.641486] ========================================================= [D 20:00:57.641492] BMI_post_recv: addr: 6843072, offset: 0x2aaaadac5010, size: 2097152, tag: 18 [D 20:00:57.641500] openib_check_cq: completed send wr_id 683170. [D 20:00:57.641510] post_sr: 10.1.4.56:39990 bh 48 len 48 wr_id 6831a0. [D 20:00:57.641518] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:00:57.641525] get_data_sync_mode returning 0 [D 20:00:57.641532] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:57.641538] dbpf_bstream_rw_list: mem_offset: 0x2aaabaa4d010, mem_size: 2097152 [D 20:00:57.641545] dbpf_bstream_rw_list: stream_offset: 383778816, stream_size: 2097152 [D 20:00:57.641575] dbpf_open_cache_get: called [D 20:00:57.641583] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:57.641589] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:57.641601] DBPF I/O ops in progress: 14 [D 20:00:57.641608] lio_listio called with the following aiocbs: [D 20:00:57.641614] aiocb_ptr_array[0]: fd: 13, off: 383778816, bytes: 2097152, buf: 0x2aaabaa4d010, type: 1 [D 20:00:57.641622] issue_or_delay_io_operation: lio_listio posted 0x6ea5d0 (handle 2863311531, ret 0) [D 20:00:57.641630] ========================================================= [D 20:00:57.641636] PINT_process_request [D 20:00:57.641642] tiling 13 copies [D 20:00:57.641648] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:57.641655] lvl 0 el 0 blk 0 by 385875968 [D 20:00:57.641661] to 385875968 ta 369098752 fi 402653184 [D 20:00:57.641668] basic type or contiguous data [D 20:00:57.641674] Request disp [D 20:00:57.641680] contig_offset = 385875968 contig_size = 50331648 lvl_flag = 1 [D 20:00:57.641687] calling distribute [D 20:00:57.641692] PINT_distribute [D 20:00:57.641698] of 385875968 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 385875968 exfl 1 [D 20:00:57.641705] begin iteration loff: 385875968 [D 20:00:57.641711] segment exceeds byte limit [D 20:00:57.641717] file being extended [D 20:00:57.641723] process a segment [D 20:00:57.641729] of 385875968 sz 2097152 [A 20:00:57.641994] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66b0d0: io: start [D 20:00:59.922400] (0x66b0d0) io (prelude sm) state: getattr_if_needed [D 20:00:59.922414] About to retrieve attributes for handle 2863311531 [D 20:00:59.922431] op_queue add: 0x2aaab15121a0 [D 20:00:59.922447] [SM Exiting]: (0x66afd0) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:00:59.922458] [SM Entering]: (0x66c290) unexpected_sm:map_request (status: 0) [D 20:00:59.922465] [SM frame get]: (0x66c290) op-id: 999 index: 0 base-frm: 0 [D 20:00:59.922473] server_post_unexpected_recv [D 20:00:59.922484] [SM Frame PUSH]: (0x2aaab15122f0) frame: 0x2aaab15123f0 [D 20:00:59.922491] [SM Locating]: (0x2aaab15122f0) op-id: 999 [D 20:00:59.922498] server_op_state_get_machine 999 [D 20:00:59.922505] [SM Locating]: (0x2aaab15122f0) located: unexpected_sm:post_unexpected [D 20:00:59.922511] [SM frame get]: (0x2aaab15122f0) op-id: 999 index: 0 base-frm: 0 [D 20:00:59.922519] [SM Entering]: (0x2aaab15122f0) unexpected_sm:post_unexpected (status: 0) [D 20:00:59.922526] [SM frame get]: (0x2aaab15122f0) op-id: 999 index: 0 base-frm: 0 [D 20:00:59.922535] [SM Exiting]: (0x2aaab15122f0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:00:59.922543] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:00:59.922549] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:00:59.922557] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 27 [D 20:00:59.922586] [SM frame get]: (0x66c290) op-id: 999 index: 0 base-frm: 0 [D 20:00:59.922594] server_state_machine_start 0x66c290 [D 20:00:59.922600] PINT_decode [D 20:00:59.922608] lebf_decode_req [D 20:00:59.922618] [SM Locating]: (0x66c290) op-id: 3 [D 20:00:59.922625] server_op_state_get_machine 3 [D 20:00:59.922632] [SM push_state]: (0x66c290) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:00:59.922638] [SM Locating]: (0x66c290) located: prelude_sm:setup [D 20:00:59.922645] [SM Entering]: (0x66c290) prelude_sm:setup (status: 0) [D 20:00:59.922652] [SM frame get]: (0x66c290) op-id: 3 index: 0 base-frm: 0 [D 20:00:59.922660] [SM Exiting]: (0x66c290) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:00:59.922667] [SM Exiting]: (0x66c290) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:00:59.922674] [SM push_state]: (0x66c290) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:00:59.922681] [SM Entering]: (0x66c290) prelude_work_sm:req_sched (status: 0) [D 20:00:59.922688] [SM frame get]: (0x66c290) op-id: 3 index: 0 base-frm: 0 [D 20:00:59.922695] (0x66c390) io (prelude sm) state: req_sched [D 20:00:59.922845] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:00:59.922857] dbpf_open_cache_get: called [D 20:00:59.922864] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:59.922871] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:59.922879] dbpf_open_cache_put: cached entry. [D 20:00:59.922911] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 476053504 [D 20:00:59.922924] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 476053504) [D 20:00:59.922931] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:00:59.922943] op_queue add: 0x2aaab15121a0 [D 20:00:59.922957] add a segment 0 [D 20:00:59.922966] end iteration [D 20:00:59.922972] done with segments or bytes [D 20:00:59.922978] of 387973120 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:59.922986] next loff: 387973120 next poff: 387973120 [D 20:00:59.922992] return value 2097152 [D 20:00:59.922998] finished [D 20:00:59.923004] retval = 2097152 [D 20:00:59.923011] exiting distribute returned less than expected [D 20:00:59.923017] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 387973120 fo 402653184 eof 0 [D 20:00:59.923024] ========================================================= [D 20:00:59.923032] BMI_post_recv: addr: 6843072, offset: 0x2aaaadcc6010, size: 2097152, tag: 19 [D 20:00:59.923045] openib_check_cq: completed send wr_id 6831a0. [D 20:00:59.923054] openib_check_cq: completed recv wr_id 684a70. [D 20:00:59.923064] openib_post_rr: 10.1.4.56:39990 bh 80 wr_id 684a70. [D 20:00:59.923072] openib_check_cq: completed recv wr_id 684aa0. [D 20:00:59.923080] openib_post_rr: 10.1.4.56:39990 bh 81 wr_id 684aa0. [D 20:00:59.923087] openib_check_cq: completed recv wr_id 684ad0. [D 20:00:59.923094] openib_post_rr: 10.1.4.56:39990 bh 82 wr_id 684ad0. [D 20:00:59.923101] openib_check_cq: completed recv wr_id 684b00. [D 20:00:59.923109] openib_post_rr: 10.1.4.56:39990 bh 83 wr_id 684b00. [D 20:00:59.923116] openib_check_cq: completed recv wr_id 684b30. [D 20:00:59.923125] openib_post_rr: 10.1.4.56:39990 bh 84 wr_id 684b30. [D 20:00:59.923132] openib_check_cq: completed recv wr_id 684b60. [D 20:00:59.923140] openib_post_rr: 10.1.4.56:39990 bh 85 wr_id 684b60. [D 20:00:59.923147] openib_check_cq: completed recv wr_id 684b90. [D 20:00:59.923155] openib_post_rr: 10.1.4.56:39990 bh 86 wr_id 684b90. [D 20:00:59.923162] openib_check_cq: completed recv wr_id 684bc0. [D 20:00:59.923169] openib_post_rr: 10.1.4.56:39990 bh 87 wr_id 684bc0. [D 20:00:59.923176] openib_check_cq: completed recv wr_id 684bf0. [D 20:00:59.923183] openib_post_rr: 10.1.4.56:39990 bh 88 wr_id 684bf0. [D 20:00:59.923191] openib_check_cq: completed recv wr_id 684c20. [D 20:00:59.923198] openib_post_rr: 10.1.4.56:39990 bh 89 wr_id 684c20. [D 20:00:59.923205] openib_check_cq: completed recv wr_id 684c50. [D 20:00:59.923213] openib_post_rr: 10.1.4.56:39990 bh 90 wr_id 684c50. [D 20:00:59.923220] openib_check_cq: completed recv wr_id 684c80. [D 20:00:59.923227] openib_post_rr: 10.1.4.56:39990 bh 91 wr_id 684c80. [D 20:00:59.923234] openib_check_cq: completed recv wr_id 684cb0. [D 20:00:59.923241] openib_post_rr: 10.1.4.56:39990 bh 92 wr_id 684cb0. [D 20:00:59.923249] openib_check_cq: completed recv wr_id 684ce0. [D 20:00:59.923256] openib_post_rr: 10.1.4.56:39990 bh 93 wr_id 684ce0. [D 20:00:59.923263] openib_check_cq: completed recv wr_id 684d10. [D 20:00:59.923271] openib_post_rr: 10.1.4.56:39990 bh 94 wr_id 684d10. [D 20:00:59.923278] openib_check_cq: completed recv wr_id 684d40. [D 20:00:59.923285] openib_post_rr: 10.1.4.56:39990 bh 95 wr_id 684d40. [D 20:00:59.923292] openib_check_cq: completed recv wr_id 684d70. [D 20:00:59.923299] openib_post_rr: 10.1.4.56:39990 bh 96 wr_id 684d70. [D 20:00:59.923306] openib_check_cq: completed recv wr_id 684da0. [D 20:00:59.923314] openib_post_rr: 10.1.4.56:39990 bh 97 wr_id 684da0. [D 20:00:59.923321] openib_check_cq: completed recv wr_id 684dd0. [D 20:00:59.923328] openib_post_rr: 10.1.4.56:39990 bh 98 wr_id 684dd0. [D 20:00:59.923335] openib_check_cq: completed recv wr_id 684e00. [D 20:00:59.923343] openib_post_rr: 10.1.4.56:39990 bh 99 wr_id 684e00. [D 20:00:59.923350] openib_check_cq: completed recv wr_id 683b70. [D 20:00:59.923357] openib_post_rr: 10.1.4.56:39990 bh 0 wr_id 683b70. [D 20:00:59.923365] openib_check_cq: completed recv wr_id 683ba0. [D 20:00:59.923372] openib_post_rr: 10.1.4.56:39990 bh 1 wr_id 683ba0. [D 20:00:59.923379] openib_check_cq: completed recv wr_id 683bd0. [D 20:00:59.923387] openib_post_rr: 10.1.4.56:39990 bh 2 wr_id 683bd0. [D 20:00:59.923394] openib_check_cq: completed recv wr_id 683c00. [D 20:00:59.923402] openib_post_rr: 10.1.4.56:39990 bh 3 wr_id 683c00. [D 20:00:59.923409] openib_check_cq: completed recv wr_id 683c30. [D 20:00:59.923416] openib_post_rr: 10.1.4.56:39990 bh 4 wr_id 683c30. [D 20:00:59.923423] openib_check_cq: completed recv wr_id 683c60. [D 20:00:59.923431] openib_post_rr: 10.1.4.56:39990 bh 5 wr_id 683c60. [D 20:00:59.923453] post_sr: 10.1.4.56:39990 bh 49 len 48 wr_id 6831d0. [D 20:00:59.923464] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:00:59.923472] get_data_sync_mode returning 0 [D 20:00:59.923481] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:00:59.923488] dbpf_bstream_rw_list: mem_offset: 0x2aaab9844010, mem_size: 2097152 [D 20:00:59.923798] dbpf_bstream_rw_list: stream_offset: 476053504, stream_size: 2097152 [D 20:00:59.923808] dbpf_open_cache_get: called [D 20:00:59.923832] dbpf_open_cache_get: found bstream entry in used list. [D 20:00:59.923839] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:00:59.923846] dbpf_attr_cache_remove: removing 2863311531 [D 20:00:59.923865] DBPF I/O ops in progress: 15 [D 20:00:59.923873] lio_listio called with the following aiocbs: [D 20:00:59.923879] aiocb_ptr_array[0]: fd: 13, off: 476053504, bytes: 2097152, buf: 0x2aaab9844010, type: 1 [D 20:00:59.923899] issue_or_delay_io_operation: lio_listio posted 0x711280 (handle 2863311531, ret 0) [D 20:00:57.645021] --- aio_progress_notification called with handle 2863311531 (0x6eaaf0) [D 20:00:59.928541] ========================================================= [D 20:00:59.928583] PINT_process_request [D 20:00:59.928590] tiling 16 copies [D 20:00:59.928597] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:00:59.928606] lvl 0 el 0 blk 0 by 478150656 [D 20:00:59.928612] to 478150656 ta 469762048 fi 503316480 [D 20:00:59.928619] basic type or contiguous data [D 20:00:59.928625] Request disp [D 20:00:59.928631] contig_offset = 478150656 contig_size = 58720256 lvl_flag = 1 [D 20:00:59.928638] calling distribute [D 20:00:59.928645] PINT_distribute [D 20:00:59.928523] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.652436] --- aio_progress_notification called with handle 2863311531 (0x6eac40) [A 20:00:59.928651] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66c390: io: request [D 20:00:59.928678] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.652524] --- aio_progress_notification called with handle 2863311531 (0x6ea2b0) [D 20:00:59.928692] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:00:59.928651] of 478150656 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 1 [D 20:00:59.928724] begin iteration loff: 478150656 [D 20:00:59.928703] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.928704] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab15121a0 [D 20:00:59.928734] segment exceeds byte limit [D 20:00:59.928755] file being extended [D 20:00:59.928764] process a segment [D 20:00:59.928772] of 478150656 sz 2097152 [D 20:00:59.928781] add a segment 0 [D 20:00:59.928791] end iteration [D 20:00:57.656459] --- aio_progress_notification called with handle 2863311531 (0x6ead90) [D 20:00:59.928746] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab15121a0 [D 20:00:59.928800] done with segments or bytes [D 20:00:59.928818] [SM Exiting]: (0x66c290) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:00:59.928822] of 480247808 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:00:59.928839] next loff: 480247808 next poff: 480247808 [D 20:00:57.658330] --- aio_progress_notification called with handle 2863311531 (0x7063c0) [D 20:00:59.928871] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.928848] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:57.668434] --- aio_progress_notification called with handle 2863311531 (0x70e5e0) [D 20:00:59.928907] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.928670] dbpf_open_cache_put: cached entry. [D 20:00:59.928929] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.928937] op_queue add: 0x6eaaf0 [D 20:00:59.928963] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:00:59.928982] DBPF I/O ops in progress: 14 [D 20:00:57.668505] --- aio_progress_notification called with handle 2863311531 (0x70ba80) [D 20:00:59.929026] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.929049] dbpf_open_cache_put: cached entry. [D 20:00:59.929057] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929063] op_queue add: 0x70ba80 [D 20:00:59.929079] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:00:59.929091] DBPF I/O ops in progress: 13 [D 20:00:59.929107] dbpf_open_cache_put: cached entry. [D 20:00:59.929116] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929123] op_queue add: 0x6eac40 [D 20:00:57.668536] --- aio_progress_notification called with handle 2863311531 (0x708f20) [D 20:00:57.671196] --- aio_progress_notification called with handle 2863311531 (0x6f0a90) [D 20:00:59.929162] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.929169] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.929177] dbpf_open_cache_put: cached entry. [D 20:00:59.929188] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929194] op_queue add: 0x708f20 [D 20:00:59.929203] dbpf_open_cache_put: cached entry. [D 20:00:59.929211] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929218] op_queue add: 0x6ea2b0 [D 20:00:59.929143] DBPF I/O ops in progress: 12 [D 20:00:57.680436] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:00:59.929246] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.929261] dbpf_open_cache_put: cached entry. [D 20:00:59.929268] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929274] op_queue add: 0x6eda80 [D 20:00:59.929281] DBPF I/O ops in progress: 11 [D 20:00:59.929298] dbpf_open_cache_put: cached entry. [D 20:00:59.929307] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929314] op_queue add: 0x7063c0 [D 20:00:59.929320] DBPF I/O ops in progress: 10 [D 20:00:59.929333] dbpf_open_cache_put: cached entry. [D 20:00:59.929342] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.680507] --- aio_progress_notification called with handle 2863311531 (0x6f0f00) [D 20:00:59.929348] dbpf_open_cache_put: cached entry. [D 20:00:59.929359] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.929376] dbpf_open_cache_put: cached entry. [D 20:00:59.929384] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929399] dbpf_open_cache_put: cached entry. [D 20:00:59.929407] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929349] op_queue add: 0x6ead90 [D 20:00:57.680533] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:00:59.929361] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:57.683078] --- aio_progress_notification called with handle 2863311531 (0x6ea140) [D 20:00:59.928831] [SM Entering]: (0x66c290) prelude_work_sm:getattr_if_needed (status: 0) [D 20:00:59.929352] DBPF I/O ops in progress: 9 [D 20:00:59.929434] op_queue add: 0x6f0a90 [D 20:00:59.929459] DBPF I/O ops in progress: 8 [D 20:00:59.929472] op_queue add: 0x6f0f00 [D 20:00:59.929136] --- aio_progress_notification called with handle 2863311531 (0x711280) [D 20:00:59.929141] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:00:59.929487] DBPF I/O ops in progress: 7 [D 20:00:59.929492] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:00:59.929500] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:00:59.929529] dbpf_open_cache_put: cached entry. [D 20:00:59.929541] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:00:59.929546] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.929557] DBPF I/O ops in progress: 6 [D 20:00:59.929562] op_queue add: 0x711280 [D 20:01:00.008486] [SM frame get]: (0x66c290) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.008627] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.008678] dbpf_open_cache_put: cached entry. [D 20:01:00.008686] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:00:59.928846] return value 2097152 [D 20:00:57.686136] --- aio_progress_notification called with handle 2863311531 (0x6ea5d0) [D 20:00:59.929554] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:00.008747] finished [D 20:01:00.008767] retval = 2097152 [D 20:01:00.008775] exiting distribute returned less than expected [D 20:01:00.008752] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.008870] dbpf_open_cache_put: cached entry. [D 20:01:00.008883] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.008889] op_queue add: 0x6ea5d0 [D 20:01:00.008761] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:00.008967] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:00.008979] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:00.008987] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:00.008781] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 480247808 fo 503316480 eof 0 [D 20:01:00.009205] ========================================================= [D 20:01:00.009213] BMI_post_recv: addr: 6843072, offset: 0x2aaabb051010, size: 2097152, tag: 22 [D 20:01:00.009226] openib_check_cq: completed send wr_id 6831d0. [D 20:01:00.009235] openib_check_cq: completed recv wr_id 683c90. [D 20:01:00.009245] openib_post_rr: 10.1.4.56:39990 bh 6 wr_id 683c90. [D 20:01:00.009254] openib_check_cq: completed recv wr_id 683cc0. [D 20:01:00.009264] openib_post_rr: 10.1.4.56:39990 bh 7 wr_id 683cc0. [D 20:01:00.009305] post_sr: 10.1.4.56:39990 bh 50 len 48 wr_id 683200. [D 20:01:00.009319] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:00.009329] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:00.009343] openib_check_cq: completed send wr_id 683200. [D 20:01:00.009358] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:00.009366] get_data_sync_mode returning 0 [D 20:01:00.009375] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.009382] dbpf_bstream_rw_list: mem_offset: 0x2aaabac4f010, mem_size: 2097152 [D 20:01:00.009389] dbpf_bstream_rw_list: stream_offset: 417333248, stream_size: 2097152 [D 20:01:00.009396] dbpf_open_cache_get: called [D 20:01:00.009402] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.009409] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.010222] op_queue add: 0x70e5e0 [D 20:00:59.929589] DBPF I/O ops in progress: 5 [D 20:01:00.010247] op_queue add: 0x6ea140 [D 20:01:00.010273] DBPF I/O ops in progress: 4 [D 20:01:00.010285] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:00.010289] DBPF I/O ops in progress: 3 [D 20:01:00.010300] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:00.010322] DBPF I/O ops in progress: 4 [D 20:01:00.010331] lio_listio called with the following aiocbs: [D 20:01:00.010338] aiocb_ptr_array[0]: fd: 13, off: 417333248, bytes: 2097152, buf: 0x2aaabac4f010, type: 1 [D 20:01:00.010350] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [D 20:01:00.010359] ========================================================= [D 20:01:00.010366] PINT_process_request [D 20:01:00.010372] tiling 14 copies [D 20:01:00.010378] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.010386] lvl 0 el 0 blk 0 by 419430400 [D 20:01:00.010392] to 419430400 ta 402653184 fi 436207616 [D 20:01:00.010400] basic type or contiguous data [D 20:01:00.010406] Request disp [D 20:01:00.010411] contig_offset = 419430400 contig_size = 50331648 lvl_flag = 1 [D 20:01:00.010419] calling distribute [D 20:01:00.010424] PINT_distribute [D 20:01:00.010430] of 419430400 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 419430400 exfl 1 [D 20:01:00.010439] begin iteration loff: 419430400 [D 20:01:00.010445] segment exceeds byte limit [D 20:01:00.010451] file being extended [D 20:01:00.010458] process a segment [D 20:01:00.010463] of 419430400 sz 2097152 [D 20:01:00.010470] add a segment 0 [D 20:01:00.010476] end iteration [D 20:01:00.010482] done with segments or bytes [D 20:01:00.010488] of 421527552 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.010494] next loff: 421527552 next poff: 421527552 [D 20:01:00.010501] return value 2097152 [D 20:01:00.010507] finished [D 20:01:00.010513] retval = 2097152 [D 20:01:00.010519] exiting distribute returned less than expected [D 20:01:00.010525] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 421527552 fo 436207616 eof 0 [D 20:01:00.010532] ========================================================= [D 20:01:00.010539] BMI_post_recv: addr: 6843072, offset: 0x2aaaadec7010, size: 2097152, tag: 20 [D 20:01:00.010552] post_sr: 10.1.4.56:39990 bh 51 len 48 wr_id 683230. [D 20:01:00.010561] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:00.010569] get_data_sync_mode returning 0 [D 20:01:00.010576] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.010583] dbpf_bstream_rw_list: mem_offset: 0x2aaabae50010, mem_size: 2097152 [D 20:01:00.010590] dbpf_bstream_rw_list: stream_offset: 450887680, stream_size: 2097152 [D 20:01:00.010597] dbpf_open_cache_get: called [D 20:01:00.010603] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.010609] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.010625] DBPF I/O ops in progress: 5 [D 20:01:00.010632] lio_listio called with the following aiocbs: [D 20:01:00.010638] aiocb_ptr_array[0]: fd: 13, off: 450887680, bytes: 2097152, buf: 0x2aaabae50010, type: 1 [D 20:01:00.010647] issue_or_delay_io_operation: lio_listio posted 0x6f0a90 (handle 2863311531, ret 0) [D 20:01:00.010655] ========================================================= [D 20:01:00.010662] PINT_process_request [D 20:01:00.010668] tiling 15 copies [D 20:01:00.010674] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.010682] lvl 0 el 0 blk 0 by 452984832 [D 20:01:00.010688] to 452984832 ta 436207616 fi 469762048 [D 20:01:00.010695] basic type or contiguous data [D 20:01:00.010701] Request disp [D 20:01:00.010707] contig_offset = 452984832 contig_size = 50331648 lvl_flag = 1 [D 20:01:00.010714] calling distribute [D 20:01:00.010720] PINT_distribute [D 20:01:00.010726] of 452984832 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 452984832 exfl 1 [D 20:01:00.010733] begin iteration loff: 452984832 [D 20:01:00.010740] segment exceeds byte limit [D 20:01:00.010746] file being extended [D 20:01:00.010751] process a segment [D 20:01:00.010757] of 452984832 sz 2097152 [D 20:01:00.010763] add a segment 0 [D 20:01:00.010769] end iteration [D 20:01:00.010775] done with segments or bytes [D 20:01:00.010781] of 455081984 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.010788] next loff: 455081984 next poff: 455081984 [D 20:01:00.010795] return value 2097152 [D 20:01:00.010801] finished [D 20:01:00.010807] retval = 2097152 [D 20:01:00.010813] exiting distribute returned less than expected [D 20:01:00.010819] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 455081984 fo 469762048 eof 0 [D 20:01:00.010826] ========================================================= [D 20:01:00.010833] BMI_post_recv: addr: 6843072, offset: 0x2aaaae0c8010, size: 2097152, tag: 21 [D 20:01:00.010841] openib_check_cq: completed send wr_id 683230. [D 20:01:00.010851] post_sr: 10.1.4.56:39990 bh 52 len 48 wr_id 683260. [D 20:01:00.010860] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:01:00.010869] get_data_sync_mode returning 0 [D 20:01:00.010875] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.010882] dbpf_bstream_rw_list: mem_offset: 0x2aaaac6bb010, mem_size: 2097152 [D 20:01:00.010889] dbpf_bstream_rw_list: stream_offset: 16777216, stream_size: 2097152 [D 20:01:00.010896] dbpf_open_cache_get: called [D 20:01:00.010902] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.010909] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.010925] DBPF I/O ops in progress: 6 [D 20:01:00.010932] lio_listio called with the following aiocbs: [D 20:01:00.010938] aiocb_ptr_array[0]: fd: 13, off: 16777216, bytes: 2097152, buf: 0x2aaaac6bb010, type: 1 [D 20:01:00.010947] issue_or_delay_io_operation: lio_listio posted 0x6f0f00 (handle 2863311531, ret 0) [D 20:01:00.010955] ========================================================= [D 20:01:00.010961] PINT_process_request [D 20:01:00.010968] tiling 2 copies [D 20:01:00.010974] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.010981] lvl 0 el 0 blk 0 by 18874368 [D 20:01:00.010988] to 18874368 ta 0 fi 33554432 [D 20:01:00.010994] basic type or contiguous data [D 20:01:00.011001] Request disp [D 20:01:00.011006] contig_offset = 18874368 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.011013] calling distribute [D 20:01:00.011019] PINT_distribute [D 20:01:00.011025] of 18874368 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 18874368 exfl 1 [D 20:01:00.011032] begin iteration loff: 18874368 [D 20:01:00.011038] segment exceeds byte limit [D 20:01:00.011044] file being extended [D 20:01:00.011050] process a segment [D 20:01:00.011055] of 18874368 sz 2097152 [D 20:01:00.011061] add a segment 0 [D 20:01:00.011067] end iteration [D 20:01:00.011074] done with segments or bytes [D 20:01:00.011080] of 20971520 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.011086] next loff: 20971520 next poff: 20971520 [D 20:01:00.011093] return value 2097152 [D 20:01:00.011099] finished [D 20:01:00.011105] retval = 2097152 [D 20:01:00.011111] exiting distribute returned less than expected [D 20:01:00.011117] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 20971520 fo 33554432 eof 0 [D 20:01:00.011124] ========================================================= [D 20:01:00.011131] BMI_post_recv: addr: 6843072, offset: 0x2aaaae2c9010, size: 2097152, tag: 8 [D 20:01:00.011139] openib_check_cq: completed send wr_id 683260. [D 20:01:00.011149] post_sr: 10.1.4.56:39990 bh 53 len 48 wr_id 683290. [D 20:01:00.011158] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:01:00.011166] get_data_sync_mode returning 0 [D 20:01:00.011174] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.011180] dbpf_bstream_rw_list: mem_offset: 0x2aaaac8bc010, mem_size: 2097152 [D 20:01:00.011187] dbpf_bstream_rw_list: stream_offset: 50331648, stream_size: 2097152 [D 20:01:00.011193] dbpf_open_cache_get: called [D 20:01:00.011199] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.011206] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.011221] DBPF I/O ops in progress: 7 [D 20:01:00.011228] lio_listio called with the following aiocbs: [D 20:01:00.011235] aiocb_ptr_array[0]: fd: 13, off: 50331648, bytes: 2097152, buf: 0x2aaaac8bc010, type: 1 [D 20:01:00.011244] issue_or_delay_io_operation: lio_listio posted 0x6ea5d0 (handle 2863311531, ret 0) [D 20:01:00.011252] ========================================================= [D 20:01:00.011258] PINT_process_request [D 20:01:00.011264] tiling 3 copies [D 20:01:00.011270] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.011278] lvl 0 el 0 blk 0 by 52428800 [D 20:01:00.011284] to 52428800 ta 33554432 fi 67108864 [D 20:01:00.011291] basic type or contiguous data [D 20:01:00.011297] Request disp [D 20:01:00.011303] contig_offset = 52428800 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.011310] calling distribute [D 20:01:00.011316] PINT_distribute [D 20:01:00.011322] of 52428800 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 52428800 exfl 1 [D 20:01:00.011329] begin iteration loff: 52428800 [D 20:01:00.011335] segment exceeds byte limit [D 20:01:00.011341] file being extended [D 20:01:00.011347] process a segment [D 20:01:00.011352] of 52428800 sz 2097152 [D 20:01:00.011358] add a segment 0 [D 20:01:00.011364] end iteration [D 20:01:00.011370] done with segments or bytes [D 20:01:00.011376] of 54525952 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.011383] next loff: 54525952 next poff: 54525952 [D 20:01:00.011389] return value 2097152 [D 20:01:00.011395] finished [D 20:01:00.011401] retval = 2097152 [D 20:01:00.011407] exiting distribute returned less than expected [D 20:01:00.011413] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 54525952 fo 67108864 eof 0 [D 20:01:00.011420] ========================================================= [D 20:01:00.011426] BMI_post_recv: addr: 6843072, offset: 0x2aaaae4ca010, size: 2097152, tag: 9 [D 20:01:00.011434] openib_check_cq: completed send wr_id 683290. [D 20:01:00.011444] post_sr: 10.1.4.56:39990 bh 54 len 48 wr_id 6832c0. [D 20:01:00.011453] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:01:00.011461] get_data_sync_mode returning 0 [D 20:01:00.011469] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.011476] dbpf_bstream_rw_list: mem_offset: 0x2aaaacabd010, mem_size: 2097152 [D 20:01:00.011483] dbpf_bstream_rw_list: stream_offset: 83886080, stream_size: 2097152 [D 20:01:00.011490] dbpf_open_cache_get: called [D 20:01:00.011495] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.011502] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.011518] DBPF I/O ops in progress: 8 [D 20:01:00.011525] lio_listio called with the following aiocbs: [D 20:01:00.011532] aiocb_ptr_array[0]: fd: 13, off: 83886080, bytes: 2097152, buf: 0x2aaaacabd010, type: 1 [D 20:01:00.011540] issue_or_delay_io_operation: lio_listio posted 0x6ea140 (handle 2863311531, ret 0) [D 20:01:00.011548] ========================================================= [D 20:01:00.011555] PINT_process_request [D 20:01:00.011561] tiling 4 copies [D 20:01:00.011567] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.011574] lvl 0 el 0 blk 0 by 85983232 [D 20:01:00.011581] to 85983232 ta 67108864 fi 100663296 [D 20:01:00.011588] basic type or contiguous data [D 20:01:00.011594] Request disp [D 20:01:00.011599] contig_offset = 85983232 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.011606] calling distribute [D 20:01:00.011612] PINT_distribute [D 20:01:00.011617] of 85983232 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 85983232 exfl 1 [D 20:01:00.011626] begin iteration loff: 85983232 [D 20:01:00.011632] segment exceeds byte limit [D 20:01:00.011638] file being extended [D 20:01:00.011644] process a segment [D 20:01:00.011650] of 85983232 sz 2097152 [D 20:01:00.011656] add a segment 0 [D 20:01:00.011662] end iteration [D 20:01:00.011667] done with segments or bytes [D 20:01:00.011673] of 88080384 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.011680] next loff: 88080384 next poff: 88080384 [D 20:01:00.011687] return value 2097152 [D 20:01:00.011693] finished [D 20:01:00.011698] retval = 2097152 [D 20:01:00.011704] exiting distribute returned less than expected [D 20:01:00.011710] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 88080384 fo 100663296 eof 0 [D 20:01:00.011717] ========================================================= [D 20:01:00.011723] BMI_post_recv: addr: 6843072, offset: 0x2aaaae6cb010, size: 2097152, tag: 10 [D 20:01:00.011731] openib_check_cq: completed send wr_id 6832c0. [D 20:01:00.011742] post_sr: 10.1.4.56:39990 bh 55 len 48 wr_id 6832f0. [D 20:01:00.011763] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:01:00.011772] get_data_sync_mode returning 0 [D 20:01:00.011779] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.011786] dbpf_bstream_rw_list: mem_offset: 0x2aaaaccbe010, mem_size: 2097152 [D 20:01:00.011793] dbpf_bstream_rw_list: stream_offset: 117440512, stream_size: 2097152 [D 20:01:00.011800] dbpf_open_cache_get: called [D 20:01:00.011805] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.011812] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.011828] DBPF I/O ops in progress: 9 [D 20:01:00.011836] lio_listio called with the following aiocbs: [D 20:01:00.011842] aiocb_ptr_array[0]: fd: 13, off: 117440512, bytes: 2097152, buf: 0x2aaaaccbe010, type: 1 [D 20:01:00.011850] issue_or_delay_io_operation: lio_listio posted 0x6ea480 (handle 2863311531, ret 0) [D 20:01:00.011858] ========================================================= [D 20:01:00.011864] PINT_process_request [D 20:01:00.011870] tiling 5 copies [D 20:01:00.011877] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.011884] lvl 0 el 0 blk 0 by 119537664 [D 20:01:00.011891] to 119537664 ta 100663296 fi 134217728 [D 20:01:00.011898] basic type or contiguous data [D 20:01:00.011904] Request disp [D 20:01:00.011910] contig_offset = 119537664 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.011917] calling distribute [D 20:01:00.011923] PINT_distribute [D 20:01:00.011928] of 119537664 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 119537664 exfl 1 [D 20:01:00.011935] begin iteration loff: 119537664 [D 20:01:00.011941] segment exceeds byte limit [D 20:01:00.011947] file being extended [D 20:01:00.011953] process a segment [D 20:01:00.011959] of 119537664 sz 2097152 [D 20:01:00.011965] add a segment 0 [D 20:01:00.011970] end iteration [D 20:01:00.012010] done with segments or bytes [D 20:01:00.012017] of 121634816 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.012024] next loff: 121634816 next poff: 121634816 [D 20:01:00.012030] return value 2097152 [D 20:01:00.012038] finished [D 20:01:00.012046] retval = 2097152 [D 20:01:00.012061] exiting distribute returned less than expected [D 20:01:00.012071] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 121634816 fo 134217728 eof 0 [D 20:01:00.012078] ========================================================= [D 20:01:00.012085] BMI_post_recv: addr: 6843072, offset: 0x2aaaae8cc010, size: 2097152, tag: 11 [D 20:01:00.012093] openib_check_cq: completed send wr_id 6832f0. [D 20:01:00.012105] post_sr: 10.1.4.56:39990 bh 56 len 48 wr_id 683320. [D 20:01:00.012113] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:01:00.012121] get_data_sync_mode returning 0 [D 20:01:00.012128] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.012135] dbpf_bstream_rw_list: mem_offset: 0x2aaaacebf010, mem_size: 2097152 [D 20:01:00.012142] dbpf_bstream_rw_list: stream_offset: 150994944, stream_size: 2097152 [D 20:01:00.012148] dbpf_open_cache_get: called [D 20:01:00.012154] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.012160] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.012176] DBPF I/O ops in progress: 10 [D 20:01:00.012185] lio_listio called with the following aiocbs: [D 20:01:00.012191] aiocb_ptr_array[0]: fd: 13, off: 150994944, bytes: 2097152, buf: 0x2aaaacebf010, type: 1 [D 20:01:00.012199] issue_or_delay_io_operation: lio_listio posted 0x6ea290 (handle 2863311531, ret 0) [D 20:01:00.012208] ========================================================= [D 20:01:00.012215] PINT_process_request [D 20:01:00.012221] tiling 6 copies [D 20:01:00.012227] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.012234] lvl 0 el 0 blk 0 by 153092096 [D 20:01:00.012241] to 153092096 ta 134217728 fi 167772160 [D 20:01:00.012248] basic type or contiguous data [D 20:01:00.012254] Request disp [D 20:01:00.012260] contig_offset = 153092096 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.012267] calling distribute [D 20:01:00.012272] PINT_distribute [D 20:01:00.012278] of 153092096 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 153092096 exfl 1 [D 20:01:00.012286] begin iteration loff: 153092096 [D 20:01:00.012293] segment exceeds byte limit [D 20:01:00.012299] file being extended [D 20:01:00.012304] process a segment [D 20:01:00.012310] of 153092096 sz 2097152 [D 20:01:00.012316] add a segment 0 [D 20:01:00.012322] end iteration [D 20:01:00.012328] done with segments or bytes [D 20:01:00.012334] of 155189248 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.012341] next loff: 155189248 next poff: 155189248 [D 20:01:00.012347] return value 2097152 [D 20:01:00.012353] finished [D 20:01:00.012359] retval = 2097152 [D 20:01:00.012365] exiting distribute returned less than expected [D 20:01:00.012371] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 155189248 fo 167772160 eof 0 [D 20:01:00.012378] ========================================================= [D 20:01:00.012384] BMI_post_recv: addr: 6843072, offset: 0x2aaaaeacd010, size: 2097152, tag: 12 [D 20:01:00.012392] openib_check_cq: completed send wr_id 683320. [D 20:01:00.012402] post_sr: 10.1.4.56:39990 bh 57 len 48 wr_id 683350. [D 20:01:00.012411] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:01:00.012419] get_data_sync_mode returning 0 [D 20:01:00.012426] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.012433] dbpf_bstream_rw_list: mem_offset: 0x2aaaad0c0010, mem_size: 2097152 [D 20:01:00.012440] dbpf_bstream_rw_list: stream_offset: 184549376, stream_size: 2097152 [D 20:01:00.012447] dbpf_open_cache_get: called [D 20:01:00.012453] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.012459] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.012482] DBPF I/O ops in progress: 11 [D 20:01:00.012490] lio_listio called with the following aiocbs: [D 20:01:00.012497] aiocb_ptr_array[0]: fd: 13, off: 184549376, bytes: 2097152, buf: 0x2aaaad0c0010, type: 1 [D 20:01:00.012506] issue_or_delay_io_operation: lio_listio posted 0x6eaad0 (handle 2863311531, ret 0) [D 20:01:00.012514] ========================================================= [D 20:01:00.012521] PINT_process_request [D 20:01:00.012527] tiling 7 copies [D 20:01:00.012533] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.012540] lvl 0 el 0 blk 0 by 186646528 [D 20:01:00.012547] to 186646528 ta 167772160 fi 201326592 [D 20:01:00.012554] basic type or contiguous data [D 20:01:00.012560] Request disp [D 20:01:00.012566] contig_offset = 186646528 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.012572] calling distribute [D 20:01:00.012578] PINT_distribute [D 20:01:00.012584] of 186646528 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 186646528 exfl 1 [D 20:01:00.012592] begin iteration loff: 186646528 [D 20:01:00.012598] segment exceeds byte limit [D 20:01:00.012604] file being extended [D 20:01:00.012610] process a segment [D 20:01:00.012615] of 186646528 sz 2097152 [D 20:01:00.012622] add a segment 0 [D 20:01:00.012627] end iteration [D 20:01:00.012633] done with segments or bytes [D 20:01:00.012640] of 188743680 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.012648] next loff: 188743680 next poff: 188743680 [D 20:01:00.012654] return value 2097152 [D 20:01:00.012660] finished [D 20:01:00.012666] retval = 2097152 [D 20:01:00.012672] exiting distribute returned less than expected [D 20:01:00.012678] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 188743680 fo 201326592 eof 0 [D 20:01:00.012685] ========================================================= [D 20:01:00.012691] BMI_post_recv: addr: 6843072, offset: 0x2aaaaecce010, size: 2097152, tag: 13 [D 20:01:00.012699] openib_check_cq: completed send wr_id 683350. [D 20:01:00.012711] post_sr: 10.1.4.56:39990 bh 58 len 48 wr_id 683380. [D 20:01:00.012720] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:01:00.012728] get_data_sync_mode returning 0 [D 20:01:00.012737] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.012744] dbpf_bstream_rw_list: mem_offset: 0x2aaaad2c1010, mem_size: 2097152 [D 20:01:00.012751] dbpf_bstream_rw_list: stream_offset: 218103808, stream_size: 2097152 [D 20:01:00.012758] dbpf_open_cache_get: called [D 20:01:00.012764] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.012770] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.012788] DBPF I/O ops in progress: 12 [D 20:01:00.012795] lio_listio called with the following aiocbs: [D 20:01:00.012801] aiocb_ptr_array[0]: fd: 13, off: 218103808, bytes: 2097152, buf: 0x2aaaad2c1010, type: 1 [D 20:01:00.012809] issue_or_delay_io_operation: lio_listio posted 0x6eac20 (handle 2863311531, ret 0) [D 20:01:00.012817] ========================================================= [D 20:01:00.012824] PINT_process_request [D 20:01:00.012830] tiling 8 copies [D 20:01:00.012836] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.012844] lvl 0 el 0 blk 0 by 220200960 [D 20:01:00.012850] to 220200960 ta 201326592 fi 234881024 [D 20:01:00.012857] basic type or contiguous data [D 20:01:00.012863] Request disp [D 20:01:00.012868] contig_offset = 220200960 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.012875] calling distribute [D 20:01:00.012881] PINT_distribute [D 20:01:00.012886] of 220200960 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 220200960 exfl 1 [D 20:01:00.012894] begin iteration loff: 220200960 [D 20:01:00.012900] segment exceeds byte limit [D 20:01:00.012906] file being extended [D 20:01:00.012912] process a segment [D 20:01:00.012918] of 220200960 sz 2097152 [D 20:01:00.012924] add a segment 0 [D 20:01:00.012929] end iteration [D 20:01:00.012935] done with segments or bytes [D 20:01:00.012941] of 222298112 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.012948] next loff: 222298112 next poff: 222298112 [D 20:01:00.012954] return value 2097152 [D 20:01:00.012960] finished [D 20:01:00.012966] retval = 2097152 [D 20:01:00.012971] exiting distribute returned less than expected [D 20:01:00.012977] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 222298112 fo 234881024 eof 0 [D 20:01:00.012985] ========================================================= [D 20:01:00.012991] BMI_post_recv: addr: 6843072, offset: 0x2aaaaeecf010, size: 2097152, tag: 14 [D 20:01:00.012998] openib_check_cq: completed send wr_id 683380. [D 20:01:00.013009] post_sr: 10.1.4.56:39990 bh 59 len 48 wr_id 6833b0. [D 20:01:00.013018] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:01:00.013026] get_data_sync_mode returning 0 [D 20:01:00.013034] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.013041] dbpf_bstream_rw_list: mem_offset: 0x2aaaad4c2010, mem_size: 2097152 [D 20:01:00.013048] dbpf_bstream_rw_list: stream_offset: 251658240, stream_size: 2097152 [D 20:01:00.013054] dbpf_open_cache_get: called [D 20:01:00.013060] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.013066] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.013085] DBPF I/O ops in progress: 13 [D 20:01:00.013092] lio_listio called with the following aiocbs: [D 20:01:00.013098] aiocb_ptr_array[0]: fd: 13, off: 251658240, bytes: 2097152, buf: 0x2aaaad4c2010, type: 1 [D 20:01:00.013107] issue_or_delay_io_operation: lio_listio posted 0x6ead70 (handle 2863311531, ret 0) [D 20:01:00.013115] ========================================================= [D 20:01:00.013122] PINT_process_request [D 20:01:00.013128] tiling 9 copies [D 20:01:00.013134] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.013141] lvl 0 el 0 blk 0 by 253755392 [D 20:01:00.013148] to 253755392 ta 234881024 fi 268435456 [D 20:01:00.013154] basic type or contiguous data [D 20:01:00.013160] Request disp [D 20:01:00.013166] contig_offset = 253755392 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.013173] calling distribute [D 20:01:00.013179] PINT_distribute [D 20:01:00.013185] of 253755392 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 253755392 exfl 1 [D 20:01:00.013193] begin iteration loff: 253755392 [D 20:01:00.013199] segment exceeds byte limit [D 20:01:00.013205] file being extended [D 20:01:00.013212] process a segment [D 20:01:00.013218] of 253755392 sz 2097152 [D 20:01:00.013224] add a segment 0 [D 20:01:00.013230] end iteration [D 20:01:00.013236] done with segments or bytes [D 20:01:00.013242] of 255852544 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.013249] next loff: 255852544 next poff: 255852544 [D 20:01:00.013255] return value 2097152 [D 20:01:00.013261] finished [D 20:01:00.013267] retval = 2097152 [D 20:01:00.013273] exiting distribute returned less than expected [D 20:01:00.013279] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 255852544 fo 268435456 eof 0 [D 20:01:00.013286] ========================================================= [D 20:01:00.013293] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf0d0010, size: 2097152, tag: 15 [D 20:01:00.013300] openib_check_cq: completed send wr_id 6833b0. [D 20:01:00.013312] post_sr: 10.1.4.56:39990 bh 60 len 48 wr_id 6833e0. [D 20:01:00.013332] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:01:00.013342] get_data_sync_mode returning 0 [D 20:01:00.013349] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.013356] dbpf_bstream_rw_list: mem_offset: 0x2aaaad6c3010, mem_size: 2097152 [D 20:01:00.013363] dbpf_bstream_rw_list: stream_offset: 285212672, stream_size: 2097152 [D 20:01:00.013370] dbpf_open_cache_get: called [D 20:01:00.013376] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.013382] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.013402] DBPF I/O ops in progress: 14 [D 20:01:00.013409] lio_listio called with the following aiocbs: [D 20:01:00.013415] aiocb_ptr_array[0]: fd: 13, off: 285212672, bytes: 2097152, buf: 0x2aaaad6c3010, type: 1 [D 20:01:00.013424] issue_or_delay_io_operation: lio_listio posted 0x6eaec0 (handle 2863311531, ret 0) [D 20:01:00.013433] ========================================================= [D 20:01:00.013440] PINT_process_request [D 20:01:00.013446] tiling 10 copies [D 20:01:00.013453] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.013460] lvl 0 el 0 blk 0 by 287309824 [D 20:01:00.013467] to 287309824 ta 268435456 fi 301989888 [D 20:01:00.013474] basic type or contiguous data [D 20:01:00.013480] Request disp [D 20:01:00.013486] contig_offset = 287309824 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.013493] calling distribute [D 20:01:00.013499] PINT_distribute [D 20:01:00.013505] of 287309824 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 287309824 exfl 1 [D 20:01:00.013512] begin iteration loff: 287309824 [D 20:01:00.013519] segment exceeds byte limit [D 20:01:00.013525] file being extended [D 20:01:00.013531] process a segment [D 20:01:00.013537] of 287309824 sz 2097152 [D 20:01:00.013543] add a segment 0 [D 20:01:00.013549] end iteration [D 20:01:00.013556] done with segments or bytes [D 20:01:00.013562] of 289406976 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.013569] next loff: 289406976 next poff: 289406976 [D 20:01:00.013575] return value 2097152 [D 20:01:00.013582] finished [D 20:01:00.013587] retval = 2097152 [D 20:01:00.013593] exiting distribute returned less than expected [D 20:01:00.013599] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 289406976 fo 301989888 eof 0 [D 20:01:00.013607] ========================================================= [D 20:01:00.013613] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf2d1010, size: 2097152, tag: 16 [D 20:01:00.013621] openib_check_cq: completed send wr_id 6833e0. [D 20:01:00.013632] post_sr: 10.1.4.56:39990 bh 61 len 48 wr_id 683410. [D 20:01:00.013640] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:01:00.013648] get_data_sync_mode returning 0 [D 20:01:00.013656] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.013662] dbpf_bstream_rw_list: mem_offset: 0x2aaaad8c4010, mem_size: 2097152 [D 20:01:00.013669] dbpf_bstream_rw_list: stream_offset: 318767104, stream_size: 2097152 [D 20:01:00.013676] dbpf_open_cache_get: called [D 20:01:00.013682] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.013688] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.013706] DBPF I/O ops in progress: 15 [D 20:01:00.013713] lio_listio called with the following aiocbs: [D 20:01:00.013719] aiocb_ptr_array[0]: fd: 13, off: 318767104, bytes: 2097152, buf: 0x2aaaad8c4010, type: 1 [D 20:01:00.013728] issue_or_delay_io_operation: lio_listio posted 0x70b7e0 (handle 2863311531, ret 0) [D 20:01:00.013736] ========================================================= [D 20:01:00.013743] PINT_process_request [D 20:01:00.013749] tiling 11 copies [D 20:01:00.013755] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.013763] lvl 0 el 0 blk 0 by 320864256 [D 20:01:00.013769] to 320864256 ta 301989888 fi 335544320 [D 20:01:00.013775] basic type or contiguous data [D 20:01:00.013781] Request disp [D 20:01:00.013787] contig_offset = 320864256 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.013794] calling distribute [D 20:01:00.013800] PINT_distribute [D 20:01:00.013806] of 320864256 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 320864256 exfl 1 [D 20:01:00.013813] begin iteration loff: 320864256 [D 20:01:00.013819] segment exceeds byte limit [D 20:01:00.013825] file being extended [D 20:01:00.013831] process a segment [D 20:01:00.013837] of 320864256 sz 2097152 [D 20:01:00.013843] add a segment 0 [D 20:01:00.013849] end iteration [D 20:01:00.013855] done with segments or bytes [D 20:01:00.013861] of 322961408 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.013868] next loff: 322961408 next poff: 322961408 [D 20:01:00.013874] return value 2097152 [D 20:01:00.013881] finished [D 20:01:00.013886] retval = 2097152 [D 20:01:00.013892] exiting distribute returned less than expected [D 20:01:00.013898] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 322961408 fo 335544320 eof 0 [D 20:01:00.013906] ========================================================= [D 20:01:00.013912] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf4d2010, size: 2097152, tag: 17 [D 20:01:00.013919] openib_check_cq: completed send wr_id 683410. [D 20:01:00.013930] post_sr: 10.1.4.56:39990 bh 62 len 48 wr_id 683440. [D 20:01:00.013939] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:01:00.013946] get_data_sync_mode returning 0 [D 20:01:00.013953] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.013960] dbpf_bstream_rw_list: mem_offset: 0x2aaaadac5010, mem_size: 2097152 [D 20:01:00.013967] dbpf_bstream_rw_list: stream_offset: 352321536, stream_size: 2097152 [D 20:01:00.013973] dbpf_open_cache_get: called [D 20:01:00.013979] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.013986] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.014005] DBPF I/O ops in progress: 16 [D 20:01:00.014013] lio_listio called with the following aiocbs: [D 20:01:00.014019] aiocb_ptr_array[0]: fd: 13, off: 352321536, bytes: 2097152, buf: 0x2aaaadac5010, type: 1 [D 20:01:00.014027] issue_or_delay_io_operation: lio_listio posted 0x70e340 (handle 2863311531, ret 0) [D 20:01:00.014036] ========================================================= [D 20:01:00.014042] PINT_process_request [D 20:01:00.014049] tiling 12 copies [D 20:01:00.014055] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.014063] lvl 0 el 0 blk 0 by 354418688 [D 20:01:00.014069] to 354418688 ta 335544320 fi 369098752 [D 20:01:00.014076] basic type or contiguous data [D 20:01:00.014082] Request disp [D 20:01:00.014088] contig_offset = 354418688 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.014094] calling distribute [D 20:01:00.014100] PINT_distribute [D 20:01:00.014106] of 354418688 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 354418688 exfl 1 [D 20:01:00.014115] begin iteration loff: 354418688 [D 20:01:00.014121] segment exceeds byte limit [D 20:01:00.014127] file being extended [D 20:01:00.014133] process a segment [D 20:01:00.014139] of 354418688 sz 2097152 [D 20:01:00.014145] add a segment 0 [D 20:01:00.014150] end iteration [D 20:01:00.014156] done with segments or bytes [D 20:01:00.014162] of 356515840 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.014169] next loff: 356515840 next poff: 356515840 [D 20:01:00.014175] return value 2097152 [D 20:01:00.014181] finished [D 20:01:00.014187] retval = 2097152 [D 20:01:00.014192] exiting distribute returned less than expected [D 20:01:00.014198] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 356515840 fo 369098752 eof 0 [D 20:01:00.014206] ========================================================= [D 20:01:00.014212] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf6d3010, size: 2097152, tag: 18 [D 20:01:00.014220] openib_check_cq: completed send wr_id 683440. [D 20:01:00.014230] post_sr: 10.1.4.56:39990 bh 63 len 48 wr_id 683470. [D 20:01:00.014239] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:01:00.014247] get_data_sync_mode returning 0 [D 20:01:00.014254] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.014261] dbpf_bstream_rw_list: mem_offset: 0x2aaaadcc6010, mem_size: 2097152 [D 20:01:00.014268] dbpf_bstream_rw_list: stream_offset: 385875968, stream_size: 2097152 [D 20:01:00.014275] dbpf_open_cache_get: called [D 20:01:00.014281] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.014287] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.014306] op_queue add: 0x710ea0 [D 20:01:00.014315] delayed I/O operation 0x710ea0 (16 already in progress) [D 20:01:00.014321] DBPF I/O ops in progress: 16 [D 20:01:00.014328] ========================================================= [D 20:01:00.014335] PINT_process_request [D 20:01:00.014341] tiling 13 copies [D 20:01:00.014348] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.014355] lvl 0 el 0 blk 0 by 387973120 [D 20:01:00.014361] to 387973120 ta 369098752 fi 402653184 [D 20:01:00.014368] basic type or contiguous data [D 20:01:00.014374] Request disp [D 20:01:00.014380] contig_offset = 387973120 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.014387] calling distribute [D 20:01:00.014393] PINT_distribute [D 20:01:00.014399] of 387973120 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 387973120 exfl 1 [D 20:01:00.014406] begin iteration loff: 387973120 [D 20:01:00.014412] segment exceeds byte limit [D 20:01:00.014418] file being extended [D 20:01:00.014424] process a segment [D 20:01:00.014430] of 387973120 sz 2097152 [D 20:01:00.014436] add a segment 0 [D 20:01:00.014442] end iteration [D 20:01:00.014448] done with segments or bytes [D 20:01:00.014454] of 390070272 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.014462] next loff: 390070272 next poff: 390070272 [D 20:01:00.014468] return value 2097152 [D 20:01:00.014474] finished [D 20:01:00.014480] retval = 2097152 [D 20:01:00.014486] exiting distribute returned less than expected [D 20:01:00.014492] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 390070272 fo 402653184 eof 0 [D 20:01:00.014499] ========================================================= [D 20:01:00.014506] BMI_post_recv: addr: 6843072, offset: 0x2aaaaf8d4010, size: 2097152, tag: 19 [D 20:01:00.014513] openib_check_cq: completed send wr_id 683470. [D 20:01:00.014525] post_sr: 10.1.4.56:39990 bh 64 len 48 wr_id 6834a0. [D 20:01:00.014543] openib_check_cq: completed send wr_id 6834a0. [A 20:01:00.014606] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66c390: io: start [D 20:01:00.014623] (0x66c390) io (prelude sm) state: getattr_if_needed [D 20:01:00.014630] About to retrieve attributes for handle 2863311531 [D 20:01:00.014642] op_queue add: 0x2aaab15137f0 [D 20:01:00.014656] [SM Exiting]: (0x66c290) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:01:00.014667] [SM Entering]: (0x66d550) unexpected_sm:map_request (status: 0) [D 20:01:00.014674] [SM frame get]: (0x66d550) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.014681] server_post_unexpected_recv [D 20:01:00.014690] [SM Frame PUSH]: (0x2aaab1513940) frame: 0x2aaab1513a40 [D 20:01:00.014697] [SM Locating]: (0x2aaab1513940) op-id: 999 [D 20:01:00.014704] server_op_state_get_machine 999 [D 20:01:00.014711] [SM Locating]: (0x2aaab1513940) located: unexpected_sm:post_unexpected [D 20:01:00.014717] [SM frame get]: (0x2aaab1513940) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.014724] [SM Entering]: (0x2aaab1513940) unexpected_sm:post_unexpected (status: 0) [D 20:01:00.014732] [SM frame get]: (0x2aaab1513940) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.014741] [SM Exiting]: (0x2aaab1513940) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:01:00.014748] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:01:00.014755] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:00.014762] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 28 [D 20:01:00.014768] [SM frame get]: (0x66d550) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.014775] server_state_machine_start 0x66d550 [D 20:01:00.014782] PINT_decode [D 20:01:00.014789] lebf_decode_req [D 20:01:00.014800] [SM Locating]: (0x66d550) op-id: 3 [D 20:01:00.014807] server_op_state_get_machine 3 [D 20:01:00.014814] [SM push_state]: (0x66d550) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:01:00.014820] [SM Locating]: (0x66d550) located: prelude_sm:setup [D 20:01:00.014827] [SM Entering]: (0x66d550) prelude_sm:setup (status: 0) [D 20:01:00.014834] [SM frame get]: (0x66d550) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.014842] [SM Exiting]: (0x66d550) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:01:00.014849] [SM Exiting]: (0x66d550) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:01:00.014856] [SM push_state]: (0x66d550) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.014863] [SM Entering]: (0x66d550) prelude_work_sm:req_sched (status: 0) [D 20:01:00.014869] [SM frame get]: (0x66d550) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.014876] (0x66d650) io (prelude sm) state: req_sched [D 20:01:00.015014] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:01:00.015025] dbpf_open_cache_get: called [D 20:01:00.015032] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.015039] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.015046] dbpf_open_cache_put: cached entry. [D 20:01:00.015080] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 478150656 [D 20:01:00.015091] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 478150656) [D 20:01:00.015099] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:01:00.015107] op_queue add: 0x2aaab15137f0 [D 20:01:00.020742] DBPF I/O ops in progress: 15 [D 20:01:00.020791] starting delayed I/O operation 0x710ea0 (15 in progress) [D 20:01:00.020802] lio_listio called with 1 following aiocbs: [D 20:01:00.020809] aiocb_ptr_array[0]: fd: 13, off: 385875968, bytes: 2097152, buf: 0x2aaaadcc6010, type: 1 [D 20:01:00.020823] start_delayed_ops_if_any: lio_listio posted 0x710ea0 (handle 2863311531, ret 0)) [A 20:01:00.020862] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66d650: io: request [D 20:01:00.020896] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:01:00.020903] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab1513450 [D 20:01:00.020910] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab1513450 [D 20:01:00.020919] [SM Exiting]: (0x66d550) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:01:00.020929] [SM Entering]: (0x66d550) prelude_work_sm:getattr_if_needed (status: 0) [D 20:01:00.020936] [SM frame get]: (0x66d550) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.010250] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.021159] dbpf_open_cache_put: cached entry. [D 20:01:00.021167] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.021176] op_queue add: 0x6ec5d0 [D 20:01:00.021206] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:00.021232] DBPF I/O ops in progress: 15 [D 20:01:00.021253] DBPF I/O ops in progress: 14 [D 20:01:00.021633] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:01:00.021650] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.021661] dbpf_open_cache_put: cached entry. [D 20:01:00.021668] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.021674] op_queue add: 0x6eda80 [D 20:01:00.021684] DBPF I/O ops in progress: 13 [D 20:01:00.021688] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:00.021709] --- aio_progress_notification called with handle 2863311531 (0x6f0a90) [D 20:01:00.021720] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.021730] dbpf_open_cache_put: cached entry. [D 20:01:00.021736] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.021743] op_queue add: 0x6f0a90 [D 20:01:00.021752] DBPF I/O ops in progress: 12 [D 20:01:00.021755] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:00.021778] --- aio_progress_notification called with handle 2863311531 (0x6f0f00) [D 20:01:00.021789] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.021798] dbpf_open_cache_put: cached entry. [D 20:01:00.021805] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.021811] op_queue add: 0x6f0f00 [D 20:01:00.021820] DBPF I/O ops in progress: 11 [D 20:01:00.021823] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. -[D 20:01:00.024164] openib_check_cq: completed recv wr_id 683cf0. [D 20:01:00.024286] openib_post_rr: 10.1.4.56:39990 bh 8 wr_id 683cf0. [D 20:01:00.024311] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:00.024324] get_data_sync_mode returning 0 [D 20:01:00.024338] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.024347] dbpf_bstream_rw_list: mem_offset: 0x2aaabb051010, mem_size: 2097152 [D 20:01:00.024354] dbpf_bstream_rw_list: stream_offset: 478150656, stream_size: 2097152 [D 20:01:00.024362] dbpf_open_cache_get: called [D 20:01:00.024370] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.024377] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.024386] dbpf_attr_cache_remove: removing 2863311531 [D 20:01:00.024410] DBPF I/O ops in progress: 12 [D 20:01:00.024417] lio_listio called with the following aiocbs: [D 20:01:00.024426] aiocb_ptr_array[0]: fd: 13, off: 478150656, bytes: 2097152, buf: 0x2aaabb051010, type: 1 [D 20:01:00.024443] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [A 20:01:00.027293] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66d650: io: start [D 20:01:00.027370] (0x66d650) io (prelude sm) state: getattr_if_needed [D 20:01:00.027378] About to retrieve attributes for handle 2863311531 [D 20:01:00.027402] op_queue add: 0x2aaab1514d20 [D 20:01:00.027746] [SM Exiting]: (0x66d550) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:01:00.027759] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:01:00.027761] [SM Entering]: (0x66e810) unexpected_sm:map_request (status: 0) [D 20:01:00.027787] [SM frame get]: (0x66e810) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.027778] dbpf_open_cache_get: called [D 20:01:00.027795] server_post_unexpected_recv [D 20:01:00.027824] [SM Frame PUSH]: (0x2aaab1514e70) frame: 0x2aaab1514f70 [D 20:01:00.027809] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.027832] [SM Locating]: (0x2aaab1514e70) op-id: 999 [D 20:01:00.027853] server_op_state_get_machine 999 [D 20:01:00.027844] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.027860] [SM Locating]: (0x2aaab1514e70) located: unexpected_sm:post_unexpected [D 20:01:00.027881] [SM frame get]: (0x2aaab1514e70) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.027872] dbpf_open_cache_put: cached entry. [D 20:01:00.027900] ========================================================= [D 20:01:00.027914] PINT_process_request [D 20:01:00.027921] tiling 16 copies [D 20:01:00.027929] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.027932] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 478150656 [D 20:01:00.027889] [SM Entering]: (0x2aaab1514e70) unexpected_sm:post_unexpected (status: 0) [D 20:01:00.027948] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 478150656) [D 20:01:00.027949] [SM frame get]: (0x2aaab1514e70) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.027972] [SM Exiting]: (0x2aaab1514e70) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:01:00.027980] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:01:00.027987] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:00.027994] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 29 [D 20:01:00.028002] [SM frame get]: (0x66e810) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.028010] server_state_machine_start 0x66e810 [D 20:01:00.028017] PINT_decode [D 20:01:00.028029] lebf_decode_req [D 20:01:00.028044] [SM Locating]: (0x66e810) op-id: 3 [D 20:01:00.028051] server_op_state_get_machine 3 [D 20:01:00.028059] [SM push_state]: (0x66e810) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:01:00.028066] [SM Locating]: (0x66e810) located: prelude_sm:setup [D 20:01:00.028074] [SM Entering]: (0x66e810) prelude_sm:setup (status: 0) [D 20:01:00.028082] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.028091] [SM Exiting]: (0x66e810) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:01:00.028098] [SM Exiting]: (0x66e810) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:01:00.028106] [SM push_state]: (0x66e810) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.028113] [SM Entering]: (0x66e810) prelude_work_sm:req_sched (status: 0) [D 20:01:00.028119] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.028127] (0x66e910) io (prelude sm) state: req_sched [D 20:01:00.027936] lvl 0 el 0 blk 0 by 480247808 [D 20:01:00.028395] to 480247808 ta 469762048 fi 503316480 [D 20:01:00.028406] basic type or contiguous data [D 20:01:00.028415] Request disp [D 20:01:00.027961] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:01:00.028425] contig_offset = 480247808 contig_size = 56623104 lvl_flag = 1 [D 20:01:00.028450] calling distribute [D 20:01:00.028458] PINT_distribute [D 20:01:00.028441] op_queue add: 0x2aaab1514d20 [D 20:01:00.028483] of 480247808 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 480247808 exfl 1 [D 20:01:00.028523] begin iteration loff: 480247808 [D 20:01:00.028531] segment exceeds byte limit [D 20:01:00.028537] file being extended [D 20:01:00.028544] process a segment [D 20:01:00.028550] of 480247808 sz 2097152 [D 20:01:00.028556] add a segment 0 [D 20:01:00.028562] end iteration [D 20:01:00.028568] done with segments or bytes [D 20:01:00.028575] of 482344960 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.028582] next loff: 482344960 next poff: 482344960 [D 20:01:00.028589] return value 2097152 [D 20:01:00.028596] finished [D 20:01:00.028602] retval = 2097152 [D 20:01:00.028608] exiting distribute returned less than expected [D 20:01:00.028615] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 482344960 fo 503316480 eof 0 [D 20:01:00.028623] ========================================================= [D 20:01:00.028632] BMI_post_recv: addr: 6843072, offset: 0x2aaabb252010, size: 2097152, tag: 22 [A 20:01:00.034384] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66e910: io: request [D 20:01:00.034463] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:01:00.034472] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab150f6a0 [D 20:01:00.034480] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab150f6a0 [D 20:01:00.034488] [SM Exiting]: (0x66e810) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:01:00.034499] [SM Entering]: (0x66e810) prelude_work_sm:getattr_if_needed (status: 0) [D 20:01:00.034506] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.036503] --- aio_progress_notification called with handle 2863311531 (0x6ea5d0) [D 20:01:00.036572] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.036587] dbpf_open_cache_put: cached entry. [D 20:01:00.036595] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.036603] op_queue add: 0x6ea5d0 [D 20:01:00.036647] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:00.036674] DBPF I/O ops in progress: 11 [D 20:01:00.036720] --- aio_progress_notification called with handle 2863311531 (0x6ea480) [D 20:01:00.036734] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.036747] dbpf_open_cache_put: cached entry. [D 20:01:00.036754] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.036760] op_queue add: 0x6ea480 [D 20:01:00.036777] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:01:00.036789] DBPF I/O ops in progress: 10 [D 20:01:00.036810] --- aio_progress_notification called with handle 2863311531 (0x6ea140) [D 20:01:00.036824] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.036834] dbpf_open_cache_put: cached entry. [D 20:01:00.036840] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.036847] op_queue add: 0x6ea140 [D 20:01:00.036863] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:00.036875] DBPF I/O ops in progress: 9 [D 20:01:00.037080] --- aio_progress_notification called with handle 2863311531 (0x6ea290) [D 20:01:00.037095] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.037105] dbpf_open_cache_put: cached entry. [D 20:01:00.037111] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.037118] op_queue add: 0x6ea290 [D 20:01:00.037127] DBPF I/O ops in progress: 8 [D 20:01:00.037130] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [A 20:01:00.040626] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66e910: io: start [D 20:01:00.040691] (0x66e910) io (prelude sm) state: getattr_if_needed [D 20:01:00.040702] About to retrieve attributes for handle 2863311531 [D 20:01:00.040720] dspace_getattr fast path attr cache hit on 2863311531 (dfile_count=0 | dist_size=0 | data_size=478150656) [D 20:01:00.040736] [SM Exiting]: (0x66e810) prelude_work_sm:getattr_if_needed (error code: 0), (action: COMPLETE) [D 20:01:00.040747] [SM Entering]: (0x66e810) prelude_work_sm:perm_check (status: 0) [D 20:01:00.040755] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.040766] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.040777] Final permission check for "io" set error code to 0 [D 20:01:00.040783] (0x66e910) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.040790] [SM Exiting]: (0x66e810) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.040798] [SM pop_state]: (0x66e810) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.040805] [SM pop_state]: (0x66e810) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.040813] [SM Entering]: (0x66e810) io_sm:send_positive_ack (status: 0) [D 20:01:00.040820] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.040828] PINT_encode [D 20:01:00.040836] encode_common [D 20:01:00.040845] lebf_encode_resp [D 20:01:00.040855] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 36 [D 20:01:00.040863] element 0: offset: 0x2aaab15160d0, size: 24 [D 20:01:00.040876] post_sr: 10.1.4.56:39990 bh 65 len 40 wr_id 6834d0. [D 20:01:00.040897] [SM Exiting]: (0x66e810) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.040907] [SM Entering]: (0x66fad0) unexpected_sm:map_request (status: 0) [D 20:01:00.040915] [SM frame get]: (0x66fad0) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.040923] server_post_unexpected_recv [D 20:01:00.040933] [SM Frame PUSH]: (0x2aaab15161a0) frame: 0x2aaab15162a0 [D 20:01:00.040941] [SM Locating]: (0x2aaab15161a0) op-id: 999 [D 20:01:00.040948] server_op_state_get_machine 999 [D 20:01:00.040955] [SM Locating]: (0x2aaab15161a0) located: unexpected_sm:post_unexpected [D 20:01:00.040962] [SM frame get]: (0x2aaab15161a0) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.040970] [SM Entering]: (0x2aaab15161a0) unexpected_sm:post_unexpected (status: 0) [D 20:01:00.040978] [SM frame get]: (0x2aaab15161a0) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.040988] [SM Exiting]: (0x2aaab15161a0) unexpected_sm:post_unexpected (error code: 0), (action: DEFERRED) [D 20:01:00.040996] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 5 [D 20:01:00.041003] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:00.041010] [BMI CONTROL]: BMI_set_info: incremented ref 6843072 to: 30 [D 20:01:00.041016] [SM frame get]: (0x66fad0) op-id: 999 index: 0 base-frm: 0 [D 20:01:00.041023] server_state_machine_start 0x66fad0 [D 20:01:00.041030] PINT_decode [D 20:01:00.041039] lebf_decode_req [D 20:01:00.041053] [SM Locating]: (0x66fad0) op-id: 3 [D 20:01:00.041060] server_op_state_get_machine 3 [D 20:01:00.041067] [SM push_state]: (0x66fad0) op-id: 3 stk-ptr: 0 base-frm: 0 [D 20:01:00.041073] [SM Locating]: (0x66fad0) located: prelude_sm:setup [D 20:01:00.041081] [SM Entering]: (0x66fad0) prelude_sm:setup (status: 0) [D 20:01:00.041087] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.041095] [SM Exiting]: (0x66fad0) prelude_sm:setup (error code: 0), (action: COMPLETE) [D 20:01:00.041102] [SM Exiting]: (0x66fad0) unexpected_sm:map_request (error code: 0), (action: COMPLETE) [D 20:01:00.041110] [SM push_state]: (0x66fad0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.041117] [SM Entering]: (0x66fad0) prelude_work_sm:req_sched (status: 0) [D 20:01:00.041124] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.041131] (0x66fbd0) io (prelude sm) state: req_sched -[D 20:01:00.041393] openib_check_cq: completed send wr_id 6834d0. [D 20:01:00.043929] openib_check_cq: completed recv wr_id 683d20. [D 20:01:00.043976] openib_post_rr: 10.1.4.56:39990 bh 9 wr_id 683d20. [D 20:01:00.044005] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:00.044019] get_data_sync_mode returning 0 [D 20:01:00.044037] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.044048] dbpf_bstream_rw_list: mem_offset: 0x2aaaadec7010, mem_size: 2097152 [D 20:01:00.044056] dbpf_bstream_rw_list: stream_offset: 419430400, stream_size: 2097152 [D 20:01:00.044063] dbpf_open_cache_get: called [D 20:01:00.044071] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.044078] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.044088] dbpf_attr_cache_remove: removing 2863311531 [D 20:01:00.044119] DBPF I/O ops in progress: 9 [D 20:01:00.044132] lio_listio called with the following aiocbs: [D 20:01:00.044139] aiocb_ptr_array[0]: fd: 13, off: 419430400, bytes: 2097152, buf: 0x2aaaadec7010, type: 1 [D 20:01:00.044154] issue_or_delay_io_operation: lio_listio posted 0x6f0f00 (handle 2863311531, ret 0) [D 20:01:00.044164] ========================================================= [D 20:01:00.044171] PINT_process_request [D 20:01:00.044178] tiling 14 copies [D 20:01:00.044185] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.044193] lvl 0 el 0 blk 0 by 421527552 [D 20:01:00.044200] to 421527552 ta 402653184 fi 436207616 [D 20:01:00.044208] basic type or contiguous data [D 20:01:00.044214] Request disp [D 20:01:00.044220] contig_offset = 421527552 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.044228] calling distribute [D 20:01:00.044234] PINT_distribute [D 20:01:00.044240] of 421527552 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 421527552 exfl 1 [D 20:01:00.044249] begin iteration loff: 421527552 [D 20:01:00.044256] segment exceeds byte limit [D 20:01:00.044262] file being extended [D 20:01:00.044269] process a segment [D 20:01:00.044274] of 421527552 sz 2097152 [D 20:01:00.044281] add a segment 0 [D 20:01:00.044287] end iteration [D 20:01:00.044294] done with segments or bytes [D 20:01:00.044301] of 423624704 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.044308] next loff: 423624704 next poff: 423624704 [D 20:01:00.044316] return value 2097152 [D 20:01:00.044322] finished [D 20:01:00.044329] retval = 2097152 [D 20:01:00.044335] exiting distribute returned less than expected [D 20:01:00.044342] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 423624704 fo 436207616 eof 0 [D 20:01:00.044349] ========================================================= [D 20:01:00.044356] BMI_post_recv: addr: 6843072, offset: 0x2aaaafad5010, size: 2097152, tag: 20 [D 20:01:00.046629] openib_check_cq: completed recv wr_id 683d50. [D 20:01:00.046682] openib_post_rr: 10.1.4.56:39990 bh 10 wr_id 683d50. [D 20:01:00.046695] openib_check_cq: completed recv wr_id 683d80. [D 20:01:00.046705] openib_post_rr: 10.1.4.56:39990 bh 11 wr_id 683d80. [D 20:01:00.046713] openib_check_cq: completed recv wr_id 683db0. [D 20:01:00.046720] openib_post_rr: 10.1.4.56:39990 bh 12 wr_id 683db0. [D 20:01:00.046728] openib_check_cq: completed recv wr_id 683de0. [D 20:01:00.046735] openib_post_rr: 10.1.4.56:39990 bh 13 wr_id 683de0. [D 20:01:00.046743] openib_check_cq: completed recv wr_id 683e10. [D 20:01:00.046752] openib_post_rr: 10.1.4.56:39990 bh 14 wr_id 683e10. [D 20:01:00.046762] post_sr: 10.1.4.56:39990 bh 66 len 48 wr_id 683500. [D 20:01:00.046773] openib_check_cq: completed recv wr_id 683e40. [D 20:01:00.046780] openib_post_rr: 10.1.4.56:39990 bh 15 wr_id 683e40. [D 20:01:00.046788] openib_check_cq: completed recv wr_id 683e70. [D 20:01:00.046796] openib_post_rr: 10.1.4.56:39990 bh 16 wr_id 683e70. [D 20:01:00.046803] openib_check_cq: completed recv wr_id 683ea0. [D 20:01:00.046811] openib_post_rr: 10.1.4.56:39990 bh 17 wr_id 683ea0. [D 20:01:00.046818] openib_check_cq: completed recv wr_id 683ed0. [D 20:01:00.046826] openib_post_rr: 10.1.4.56:39990 bh 18 wr_id 683ed0. [D 20:01:00.046836] openib_check_cq: completed recv wr_id 683f00. [D 20:01:00.046844] openib_post_rr: 10.1.4.56:39990 bh 19 wr_id 683f00. [D 20:01:00.046852] openib_check_cq: completed recv wr_id 683f30. [D 20:01:00.046860] openib_post_rr: 10.1.4.56:39990 bh 20 wr_id 683f30. [D 20:01:00.046867] openib_check_cq: completed send wr_id 683500. [D 20:01:00.046888] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:00.046898] get_data_sync_mode returning 0 [D 20:01:00.046912] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.046920] dbpf_bstream_rw_list: mem_offset: 0x2aaaae0c8010, mem_size: 2097152 [D 20:01:00.046928] dbpf_bstream_rw_list: stream_offset: 452984832, stream_size: 2097152 [D 20:01:00.046935] dbpf_open_cache_get: called [D 20:01:00.046942] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.046949] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.046974] DBPF I/O ops in progress: 10 [D 20:01:00.046982] lio_listio called with the following aiocbs: [D 20:01:00.046989] aiocb_ptr_array[0]: fd: 13, off: 452984832, bytes: 2097152, buf: 0x2aaaae0c8010, type: 1 [D 20:01:00.047002] issue_or_delay_io_operation: lio_listio posted 0x710ff0 (handle 2863311531, ret 0) [D 20:01:00.047011] ========================================================= [D 20:01:00.047062] PINT_process_request [D 20:01:00.047071] tiling 15 copies [D 20:01:00.047079] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.047086] lvl 0 el 0 blk 0 by 455081984 [D 20:01:00.047093] to 455081984 ta 436207616 fi 469762048 [D 20:01:00.047100] basic type or contiguous data [D 20:01:00.047106] Request disp [D 20:01:00.047113] contig_offset = 455081984 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.047120] calling distribute [D 20:01:00.047126] PINT_distribute [D 20:01:00.047132] of 455081984 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 455081984 exfl 1 [D 20:01:00.047140] begin iteration loff: 455081984 [D 20:01:00.047147] segment exceeds byte limit [D 20:01:00.047153] file being extended [D 20:01:00.047160] process a segment [D 20:01:00.047166] of 455081984 sz 2097152 [D 20:01:00.047172] add a segment 0 [D 20:01:00.047179] end iteration [D 20:01:00.047185] done with segments or bytes [D 20:01:00.047191] of 457179136 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.047198] next loff: 457179136 next poff: 457179136 [D 20:01:00.047205] return value 2097152 [D 20:01:00.047211] finished [D 20:01:00.047217] retval = 2097152 [D 20:01:00.047224] exiting distribute returned less than expected [D 20:01:00.047230] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 457179136 fo 469762048 eof 0 [D 20:01:00.047238] ========================================================= [D 20:01:00.047244] BMI_post_recv: addr: 6843072, offset: 0x2aaaafcd6010, size: 2097152, tag: 21 [D 20:01:00.047272] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:01:00.047281] get_data_sync_mode returning 0 [D 20:01:00.047288] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.047295] dbpf_bstream_rw_list: mem_offset: 0x2aaaae2c9010, mem_size: 2097152 [D 20:01:00.047302] dbpf_bstream_rw_list: stream_offset: 18874368, stream_size: 2097152 [D 20:01:00.047308] dbpf_open_cache_get: called [D 20:01:00.047315] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.047321] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.047336] DBPF I/O ops in progress: 11 [D 20:01:00.047344] lio_listio called with the following aiocbs: [D 20:01:00.047350] aiocb_ptr_array[0]: fd: 13, off: 18874368, bytes: 2097152, buf: 0x2aaaae2c9010, type: 1 [D 20:01:00.047359] issue_or_delay_io_operation: lio_listio posted 0x6eda80 (handle 2863311531, ret 0) [D 20:01:00.047367] ========================================================= [D 20:01:00.047374] PINT_process_request [D 20:01:00.047488] tiling 2 copies [D 20:01:00.047495] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [A 20:01:00.047470] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66fbd0: io: request [D 20:01:00.125356] REQ SCHED allowing concurrent I/O, handle: 2863311531 [D 20:01:00.125366] REQ SCHED POSTING, handle: 2863311531, queue_element: 0x2aaab15160f0 [D 20:01:00.125373] REQ SCHED SCHEDULING, handle: 2863311531, queue_element: 0x2aaab15160f0 [D 20:01:00.125381] [SM Exiting]: (0x66fad0) prelude_work_sm:req_sched (error code: 0), (action: COMPLETE) [D 20:01:00.125391] [SM Entering]: (0x66fad0) prelude_work_sm:getattr_if_needed (status: 0) [D 20:01:00.125398] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.125548] lvl 0 el 0 blk 0 by 20971520 [D 20:01:00.125559] to 20971520 ta 0 fi 33554432 [D 20:01:00.125566] basic type or contiguous data [D 20:01:00.125573] Request disp [D 20:01:00.125579] contig_offset = 20971520 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.125586] calling distribute [D 20:01:00.125592] PINT_distribute [D 20:01:00.125599] of 20971520 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 20971520 exfl 1 [D 20:01:00.125607] begin iteration loff: 20971520 [D 20:01:00.125613] segment exceeds byte limit [D 20:01:00.125619] file being extended [D 20:01:00.125626] process a segment [D 20:01:00.125631] of 20971520 sz 2097152 [D 20:01:00.125638] add a segment 0 [D 20:01:00.125644] end iteration [D 20:01:00.125650] done with segments or bytes [D 20:01:00.125655] of 23068672 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.125663] next loff: 23068672 next poff: 23068672 [D 20:01:00.125669] return value 2097152 [D 20:01:00.125675] finished [D 20:01:00.125681] retval = 2097152 [D 20:01:00.125687] exiting distribute returned less than expected [D 20:01:00.125693] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 23068672 fo 33554432 eof 0 [D 20:01:00.125701] ========================================================= [D 20:01:00.125708] BMI_post_recv: addr: 6843072, offset: 0x2aaaafed7010, size: 2097152, tag: 8 [D 20:01:00.125722] openib_check_cq: completed recv wr_id 683f60. [D 20:01:00.125733] openib_post_rr: 10.1.4.56:39990 bh 21 wr_id 683f60. [D 20:01:00.125741] openib_check_cq: completed recv wr_id 683f90. [D 20:01:00.125748] openib_post_rr: 10.1.4.56:39990 bh 22 wr_id 683f90. [D 20:01:00.125755] openib_check_cq: completed recv wr_id 683fc0. [D 20:01:00.125763] openib_post_rr: 10.1.4.56:39990 bh 23 wr_id 683fc0. [D 20:01:00.125770] openib_check_cq: completed recv wr_id 683ff0. [D 20:01:00.125778] openib_post_rr: 10.1.4.56:39990 bh 24 wr_id 683ff0. [D 20:01:00.125786] post_sr: 10.1.4.56:39990 bh 67 len 48 wr_id 683530. [D 20:01:00.125796] openib_check_cq: completed recv wr_id 684020. [D 20:01:00.125803] openib_post_rr: 10.1.4.56:39990 bh 25 wr_id 684020. [D 20:01:00.125810] post_sr: 10.1.4.56:39990 bh 68 len 48 wr_id 683560. [D 20:01:00.125818] openib_check_cq: completed recv wr_id 684050. [D 20:01:00.125826] openib_post_rr: 10.1.4.56:39990 bh 26 wr_id 684050. [D 20:01:00.125833] openib_check_cq: completed recv wr_id 684080. [D 20:01:00.125841] openib_post_rr: 10.1.4.56:39990 bh 27 wr_id 684080. [D 20:01:00.125848] openib_check_cq: completed recv wr_id 6840b0. [D 20:01:00.125855] openib_post_rr: 10.1.4.56:39990 bh 28 wr_id 6840b0. [D 20:01:00.125862] openib_check_cq: completed recv wr_id 6840e0. [D 20:01:00.125869] openib_post_rr: 10.1.4.56:39990 bh 29 wr_id 6840e0. [D 20:01:00.125876] openib_check_cq: completed recv wr_id 684110. [D 20:01:00.125884] openib_post_rr: 10.1.4.56:39990 bh 30 wr_id 684110. [D 20:01:00.125891] openib_check_cq: completed recv wr_id 684140. [D 20:01:00.125898] openib_post_rr: 10.1.4.56:39990 bh 31 wr_id 684140. [D 20:01:00.125905] openib_check_cq: completed recv wr_id 684170. [D 20:01:00.125913] openib_post_rr: 10.1.4.56:39990 bh 32 wr_id 684170. [D 20:01:00.125920] openib_check_cq: completed recv wr_id 6841a0. [D 20:01:00.125928] openib_post_rr: 10.1.4.56:39990 bh 33 wr_id 6841a0. [D 20:01:00.125935] openib_check_cq: completed recv wr_id 6841d0. [D 20:01:00.125942] openib_post_rr: 10.1.4.56:39990 bh 34 wr_id 6841d0. [D 20:01:00.125950] openib_check_cq: completed recv wr_id 684200. [D 20:01:00.125958] openib_post_rr: 10.1.4.56:39990 bh 35 wr_id 684200. [D 20:01:00.125965] openib_check_cq: completed recv wr_id 684230. [D 20:01:00.125972] openib_post_rr: 10.1.4.56:39990 bh 36 wr_id 684230. [D 20:01:00.125980] openib_check_cq: completed recv wr_id 684260. [D 20:01:00.125987] openib_post_rr: 10.1.4.56:39990 bh 37 wr_id 684260. [D 20:01:00.125994] openib_check_cq: completed recv wr_id 684290. [D 20:01:00.126001] openib_post_rr: 10.1.4.56:39990 bh 38 wr_id 684290. [D 20:01:00.126008] openib_check_cq: completed recv wr_id 6842c0. [D 20:01:00.126015] openib_post_rr: 10.1.4.56:39990 bh 39 wr_id 6842c0. [D 20:01:00.126023] openib_check_cq: completed send wr_id 683530. [D 20:01:00.126030] openib_check_cq: completed send wr_id 683560. [D 20:01:00.126052] post_sr: 10.1.4.56:39990 bh 69 len 48 wr_id 683590. [D 20:01:00.126062] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:01:00.126070] get_data_sync_mode returning 0 [D 20:01:00.126080] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.126087] dbpf_bstream_rw_list: mem_offset: 0x2aaaae4ca010, mem_size: 2097152 [D 20:01:00.126094] dbpf_bstream_rw_list: stream_offset: 52428800, stream_size: 2097152 [D 20:01:00.126101] dbpf_open_cache_get: called [D 20:01:00.126107] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.126114] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.126131] DBPF I/O ops in progress: 12 [D 20:01:00.126139] lio_listio called with the following aiocbs: [D 20:01:00.126145] aiocb_ptr_array[0]: fd: 13, off: 52428800, bytes: 2097152, buf: 0x2aaaae4ca010, type: 1 [D 20:01:00.126157] issue_or_delay_io_operation: lio_listio posted 0x6f6810 (handle 2863311531, ret 0) [D 20:01:00.126167] ========================================================= [D 20:01:00.126173] PINT_process_request [D 20:01:00.126180] tiling 3 copies [D 20:01:00.126187] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.126194] lvl 0 el 0 blk 0 by 54525952 [D 20:01:00.126201] to 54525952 ta 33554432 fi 67108864 [D 20:01:00.126207] basic type or contiguous data [D 20:01:00.126213] Request disp [D 20:01:00.126219] contig_offset = 54525952 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.126226] calling distribute [D 20:01:00.126232] PINT_distribute [D 20:01:00.126238] of 54525952 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 54525952 exfl 1 [D 20:01:00.126245] begin iteration loff: 54525952 [D 20:01:00.126251] segment exceeds byte limit [D 20:01:00.126257] file being extended [D 20:01:00.126263] process a segment [D 20:01:00.126269] of 54525952 sz 2097152 [D 20:01:00.126275] add a segment 0 [D 20:01:00.126281] end iteration [D 20:01:00.126287] done with segments or bytes [D 20:01:00.126293] of 56623104 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.126300] next loff: 56623104 next poff: 56623104 [D 20:01:00.126306] return value 2097152 [D 20:01:00.126312] finished [D 20:01:00.126318] retval = 2097152 [D 20:01:00.126324] exiting distribute returned less than expected [D 20:01:00.126330] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 56623104 fo 67108864 eof 0 [D 20:01:00.126337] ========================================================= [D 20:01:00.126344] BMI_post_recv: addr: 6843072, offset: 0x2aaab00d8010, size: 2097152, tag: 9 [D 20:01:00.126352] openib_check_cq: completed send wr_id 683590. [D 20:01:00.126363] post_sr: 10.1.4.56:39990 bh 70 len 48 wr_id 6835c0. [D 20:01:00.126372] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:01:00.126379] get_data_sync_mode returning 0 [D 20:01:00.126387] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.126394] dbpf_bstream_rw_list: mem_offset: 0x2aaaae6cb010, mem_size: 2097152 [D 20:01:00.126401] dbpf_bstream_rw_list: stream_offset: 85983232, stream_size: 2097152 [D 20:01:00.126408] dbpf_open_cache_get: called [D 20:01:00.126567] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.126576] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.126589] DBPF I/O ops in progress: 13 [D 20:01:00.126596] lio_listio called with the following aiocbs: [D 20:01:00.126602] aiocb_ptr_array[0]: fd: 13, off: 85983232, bytes: 2097152, buf: 0x2aaaae6cb010, type: 1 [D 20:01:00.126610] issue_or_delay_io_operation: lio_listio posted 0x6f9370 (handle 2863311531, ret 0) [D 20:01:00.126618] ========================================================= [D 20:01:00.126624] PINT_process_request [D 20:01:00.126630] tiling 4 copies [D 20:01:00.126636] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.126643] lvl 0 el 0 blk 0 by 88080384 [D 20:01:00.126650] to 88080384 ta 67108864 fi 100663296 [D 20:01:00.126656] basic type or contiguous data [D 20:01:00.126662] Request disp [D 20:01:00.126668] contig_offset = 88080384 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.126675] calling distribute [D 20:01:00.126681] PINT_distribute [D 20:01:00.126687] of 88080384 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 88080384 exfl 1 [D 20:01:00.126694] begin iteration loff: 88080384 [D 20:01:00.126700] segment exceeds byte limit [D 20:01:00.126706] file being extended [D 20:01:00.126712] process a segment [D 20:01:00.126718] of 88080384 sz 2097152 [D 20:01:00.126724] add a segment 0 [D 20:01:00.126730] end iteration [D 20:01:00.126736] done with segments or bytes [D 20:01:00.126742] of 90177536 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.126749] next loff: 90177536 next poff: 90177536 [D 20:01:00.126755] return value 2097152 [D 20:01:00.126761] finished [D 20:01:00.126767] retval = 2097152 [D 20:01:00.126773] exiting distribute returned less than expected [D 20:01:00.126779] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 90177536 fo 100663296 eof 0 [D 20:01:00.126786] ========================================================= [D 20:01:00.126792] BMI_post_recv: addr: 6843072, offset: 0x2aaab02d9010, size: 2097152, tag: 10 [D 20:01:00.126800] openib_check_cq: completed send wr_id 6835c0. [D 20:01:00.126812] post_sr: 10.1.4.56:39990 bh 71 len 48 wr_id 6835f0. [D 20:01:00.126821] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:01:00.126830] get_data_sync_mode returning 0 [D 20:01:00.126837] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.126844] dbpf_bstream_rw_list: mem_offset: 0x2aaaae8cc010, mem_size: 2097152 [D 20:01:00.126850] dbpf_bstream_rw_list: stream_offset: 119537664, stream_size: 2097152 [D 20:01:00.126857] dbpf_open_cache_get: called [D 20:01:00.126863] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.126869] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.126885] DBPF I/O ops in progress: 14 [D 20:01:00.126892] lio_listio called with the following aiocbs: [D 20:01:00.126898] aiocb_ptr_array[0]: fd: 13, off: 119537664, bytes: 2097152, buf: 0x2aaaae8cc010, type: 1 [D 20:01:00.126907] issue_or_delay_io_operation: lio_listio posted 0x6fbf80 (handle 2863311531, ret 0) [D 20:01:00.126915] ========================================================= [D 20:01:00.126922] PINT_process_request [D 20:01:00.126928] tiling 5 copies [D 20:01:00.126934] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.126941] lvl 0 el 0 blk 0 by 121634816 [D 20:01:00.126948] to 121634816 ta 100663296 fi 134217728 [D 20:01:00.126954] basic type or contiguous data [D 20:01:00.126961] Request disp [D 20:01:00.126966] contig_offset = 121634816 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.126973] calling distribute [D 20:01:00.126979] PINT_distribute [D 20:01:00.126985] of 121634816 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 121634816 exfl 1 [D 20:01:00.126992] begin iteration loff: 121634816 [D 20:01:00.126999] segment exceeds byte limit [D 20:01:00.127004] file being extended [D 20:01:00.127010] process a segment [D 20:01:00.127017] of 121634816 sz 2097152 [D 20:01:00.127023] add a segment 0 [D 20:01:00.127029] end iteration [D 20:01:00.127035] done with segments or bytes [D 20:01:00.127041] of 123731968 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.127048] next loff: 123731968 next poff: 123731968 [D 20:01:00.127054] return value 2097152 [D 20:01:00.127060] finished [D 20:01:00.127066] retval = 2097152 [D 20:01:00.127071] exiting distribute returned less than expected [D 20:01:00.127077] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 123731968 fo 134217728 eof 0 [D 20:01:00.127085] ========================================================= [D 20:01:00.127091] BMI_post_recv: addr: 6843072, offset: 0x2aaab04da010, size: 2097152, tag: 11 [D 20:01:00.127099] openib_check_cq: completed send wr_id 6835f0. [D 20:01:00.127110] post_sr: 10.1.4.56:39990 bh 72 len 48 wr_id 683620. [D 20:01:00.127135] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:01:00.127143] get_data_sync_mode returning 0 [D 20:01:00.127151] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.127157] dbpf_bstream_rw_list: mem_offset: 0x2aaaaeacd010, mem_size: 2097152 [D 20:01:00.127164] dbpf_bstream_rw_list: stream_offset: 153092096, stream_size: 2097152 [D 20:01:00.127171] dbpf_open_cache_get: called [D 20:01:00.127177] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.127183] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.127213] DBPF I/O ops in progress: 15 [D 20:01:00.127221] lio_listio called with the following aiocbs: [D 20:01:00.127227] aiocb_ptr_array[0]: fd: 13, off: 153092096, bytes: 2097152, buf: 0x2aaaaeacd010, type: 1 [D 20:01:00.127235] issue_or_delay_io_operation: lio_listio posted 0x6ed430 (handle 2863311531, ret 0) [D 20:01:00.127243] ========================================================= [D 20:01:00.127249] PINT_process_request [D 20:01:00.127255] tiling 6 copies [D 20:01:00.127261] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.127269] lvl 0 el 0 blk 0 by 155189248 [D 20:01:00.127275] to 155189248 ta 134217728 fi 167772160 [D 20:01:00.127282] basic type or contiguous data [D 20:01:00.127288] Request disp [D 20:01:00.127294] contig_offset = 155189248 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.127301] calling distribute [D 20:01:00.127307] PINT_distribute [D 20:01:00.127313] of 155189248 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 155189248 exfl 1 [D 20:01:00.127320] begin iteration loff: 155189248 [D 20:01:00.127326] segment exceeds byte limit [D 20:01:00.127332] file being extended [D 20:01:00.127338] process a segment [D 20:01:00.127344] of 155189248 sz 2097152 [D 20:01:00.127350] add a segment 0 [D 20:01:00.127356] end iteration [D 20:01:00.127362] done with segments or bytes [D 20:01:00.127367] of 157286400 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.127375] next loff: 157286400 next poff: 157286400 [D 20:01:00.127381] return value 2097152 [D 20:01:00.127387] finished [D 20:01:00.127393] retval = 2097152 [D 20:01:00.127398] exiting distribute returned less than expected [D 20:01:00.127404] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 157286400 fo 167772160 eof 0 [D 20:01:00.127412] ========================================================= [D 20:01:00.127418] BMI_post_recv: addr: 6843072, offset: 0x2aaab06db010, size: 2097152, tag: 12 [D 20:01:00.127426] openib_check_cq: completed send wr_id 683620. [D 20:01:00.127437] post_sr: 10.1.4.56:39990 bh 73 len 48 wr_id 683650. [D 20:01:00.127447] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:01:00.127455] get_data_sync_mode returning 0 [D 20:01:00.127462] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.127469] dbpf_bstream_rw_list: mem_offset: 0x2aaaaecce010, mem_size: 2097152 [D 20:01:00.127475] dbpf_bstream_rw_list: stream_offset: 186646528, stream_size: 2097152 [D 20:01:00.127482] dbpf_open_cache_get: called [D 20:01:00.127646] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.127656] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.127671] DBPF I/O ops in progress: 16 [D 20:01:00.127679] lio_listio called with the following aiocbs: [D 20:01:00.127685] aiocb_ptr_array[0]: fd: 13, off: 186646528, bytes: 2097152, buf: 0x2aaaaecce010, type: 1 [D 20:01:00.127693] issue_or_delay_io_operation: lio_listio posted 0x6ea6c0 (handle 2863311531, ret 0) [D 20:01:00.127700] ========================================================= [D 20:01:00.127707] PINT_process_request [D 20:01:00.127713] tiling 7 copies [D 20:01:00.127719] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.127726] lvl 0 el 0 blk 0 by 188743680 [D 20:01:00.127733] to 188743680 ta 167772160 fi 201326592 [D 20:01:00.127740] basic type or contiguous data [D 20:01:00.127746] Request disp [D 20:01:00.127751] contig_offset = 188743680 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.127758] calling distribute [D 20:01:00.127764] PINT_distribute [D 20:01:00.127770] of 188743680 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 188743680 exfl 1 [D 20:01:00.127777] begin iteration loff: 188743680 [D 20:01:00.127783] segment exceeds byte limit [D 20:01:00.127789] file being extended [D 20:01:00.127795] process a segment [D 20:01:00.127801] of 188743680 sz 2097152 [D 20:01:00.127807] add a segment 0 [D 20:01:00.127813] end iteration [D 20:01:00.127819] done with segments or bytes [D 20:01:00.127825] of 190840832 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.127832] next loff: 190840832 next poff: 190840832 [D 20:01:00.127838] return value 2097152 [D 20:01:00.127844] finished [D 20:01:00.127850] retval = 2097152 [D 20:01:00.127856] exiting distribute returned less than expected [D 20:01:00.127862] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 190840832 fo 201326592 eof 0 [D 20:01:00.127869] ========================================================= [D 20:01:00.127875] BMI_post_recv: addr: 6843072, offset: 0x2aaab08dc010, size: 2097152, tag: 13 [D 20:01:00.127884] openib_check_cq: completed send wr_id 683650. [D 20:01:00.127896] post_sr: 10.1.4.56:39990 bh 74 len 48 wr_id 683680. [D 20:01:00.127905] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:01:00.127913] get_data_sync_mode returning 0 [D 20:01:00.127921] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.127927] dbpf_bstream_rw_list: mem_offset: 0x2aaaaeecf010, mem_size: 2097152 [D 20:01:00.127934] dbpf_bstream_rw_list: stream_offset: 220200960, stream_size: 2097152 [D 20:01:00.127941] dbpf_open_cache_get: called [D 20:01:00.127947] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.127953] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.127981] op_queue add: 0x6feb90 [D 20:01:00.127989] delayed I/O operation 0x6feb90 (16 already in progress) [D 20:01:00.127995] DBPF I/O ops in progress: 16 [D 20:01:00.128003] ========================================================= [D 20:01:00.128009] PINT_process_request [D 20:01:00.128015] tiling 8 copies [D 20:01:00.128021] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.128029] lvl 0 el 0 blk 0 by 222298112 [D 20:01:00.128035] to 222298112 ta 201326592 fi 234881024 [D 20:01:00.128042] basic type or contiguous data [D 20:01:00.128048] Request disp [D 20:01:00.128054] contig_offset = 222298112 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.128061] calling distribute [D 20:01:00.128067] PINT_distribute [D 20:01:00.128072] of 222298112 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 222298112 exfl 1 [D 20:01:00.128080] begin iteration loff: 222298112 [D 20:01:00.128086] segment exceeds byte limit [D 20:01:00.128092] file being extended [D 20:01:00.128099] process a segment [D 20:01:00.128105] of 222298112 sz 2097152 [D 20:01:00.128111] add a segment 0 [D 20:01:00.128117] end iteration [D 20:01:00.128123] done with segments or bytes [D 20:01:00.128130] of 224395264 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.128137] next loff: 224395264 next poff: 224395264 [D 20:01:00.128143] return value 2097152 [D 20:01:00.128149] finished [D 20:01:00.128155] retval = 2097152 [D 20:01:00.128161] exiting distribute returned less than expected [D 20:01:00.128167] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 224395264 fo 234881024 eof 0 [D 20:01:00.128174] ========================================================= [D 20:01:00.128180] BMI_post_recv: addr: 6843072, offset: 0x2aaab0add010, size: 2097152, tag: 14 [D 20:01:00.128188] openib_check_cq: completed send wr_id 683680. [D 20:01:00.128198] post_sr: 10.1.4.56:39990 bh 75 len 48 wr_id 6836b0. [D 20:01:00.128207] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:01:00.128214] get_data_sync_mode returning 0 [D 20:01:00.128222] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.128229] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf0d0010, mem_size: 2097152 [D 20:01:00.128236] dbpf_bstream_rw_list: stream_offset: 253755392, stream_size: 2097152 [D 20:01:00.128243] dbpf_open_cache_get: called [D 20:01:00.128249] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.128255] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.128279] op_queue add: 0x6fece0 [D 20:01:00.128288] delayed I/O operation 0x6fece0 (16 already in progress) [D 20:01:00.128294] DBPF I/O ops in progress: 16 [D 20:01:00.128302] ========================================================= [D 20:01:00.128308] PINT_process_request [D 20:01:00.128314] tiling 9 copies [D 20:01:00.128320] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.128328] lvl 0 el 0 blk 0 by 255852544 [D 20:01:00.128334] to 255852544 ta 234881024 fi 268435456 [D 20:01:00.128341] basic type or contiguous data [D 20:01:00.128347] Request disp [D 20:01:00.128353] contig_offset = 255852544 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.128359] calling distribute [D 20:01:00.128365] PINT_distribute [D 20:01:00.128371] of 255852544 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 255852544 exfl 1 [D 20:01:00.128379] begin iteration loff: 255852544 [D 20:01:00.128386] segment exceeds byte limit [D 20:01:00.128392] file being extended [D 20:01:00.128397] process a segment [D 20:01:00.128403] of 255852544 sz 2097152 [D 20:01:00.128410] add a segment 0 [D 20:01:00.128416] end iteration [D 20:01:00.128422] done with segments or bytes [D 20:01:00.128427] of 257949696 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.128434] next loff: 257949696 next poff: 257949696 [D 20:01:00.128440] return value 2097152 [D 20:01:00.128446] finished [D 20:01:00.128452] retval = 2097152 [D 20:01:00.128458] exiting distribute returned less than expected [D 20:01:00.128464] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 257949696 fo 268435456 eof 0 [D 20:01:00.128632] ========================================================= [D 20:01:00.128642] BMI_post_recv: addr: 6843072, offset: 0x2aaab0cde010, size: 2097152, tag: 15 [D 20:01:00.128650] openib_check_cq: completed send wr_id 6836b0. [D 20:01:00.128661] post_sr: 10.1.4.56:39990 bh 76 len 48 wr_id 6836e0. [D 20:01:00.128670] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:01:00.128677] get_data_sync_mode returning 0 [D 20:01:00.128684] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.128691] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf2d1010, mem_size: 2097152 [D 20:01:00.128697] dbpf_bstream_rw_list: stream_offset: 287309824, stream_size: 2097152 [D 20:01:00.128704] dbpf_open_cache_get: called [D 20:01:00.128710] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.128717] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.128735] op_queue add: 0x6fee30 [D 20:01:00.128743] delayed I/O operation 0x6fee30 (16 already in progress) [D 20:01:00.128750] DBPF I/O ops in progress: 16 [D 20:01:00.128757] ========================================================= [D 20:01:00.128764] PINT_process_request [D 20:01:00.128770] tiling 10 copies [D 20:01:00.128776] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.128784] lvl 0 el 0 blk 0 by 289406976 [D 20:01:00.128791] to 289406976 ta 268435456 fi 301989888 [D 20:01:00.128798] basic type or contiguous data [D 20:01:00.128804] Request disp [D 20:01:00.128810] contig_offset = 289406976 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.128816] calling distribute [D 20:01:00.128822] PINT_distribute [D 20:01:00.128828] of 289406976 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 289406976 exfl 1 [D 20:01:00.128836] begin iteration loff: 289406976 [D 20:01:00.128842] segment exceeds byte limit [D 20:01:00.128848] file being extended [D 20:01:00.128854] process a segment [D 20:01:00.128859] of 289406976 sz 2097152 [D 20:01:00.128865] add a segment 0 [D 20:01:00.128871] end iteration [D 20:01:00.128877] done with segments or bytes [D 20:01:00.128883] of 291504128 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.128890] next loff: 291504128 next poff: 291504128 [D 20:01:00.128896] return value 2097152 [D 20:01:00.128902] finished [D 20:01:00.128908] retval = 2097152 [D 20:01:00.128914] exiting distribute returned less than expected [D 20:01:00.128920] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 291504128 fo 301989888 eof 0 [D 20:01:00.129026] ========================================================= [D 20:01:00.129034] BMI_post_recv: addr: 6843072, offset: 0x2aaab0edf010, size: 2097152, tag: 16 [D 20:01:00.129043] openib_check_cq: completed send wr_id 6836e0. [D 20:01:00.129054] post_sr: 10.1.4.56:39990 bh 77 len 48 wr_id 683710. [D 20:01:00.129071] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:01:00.129081] get_data_sync_mode returning 0 [D 20:01:00.129091] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.129098] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf4d2010, mem_size: 2097152 [D 20:01:00.129105] dbpf_bstream_rw_list: stream_offset: 320864256, stream_size: 2097152 [D 20:01:00.129112] dbpf_open_cache_get: called [D 20:01:00.129118] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.129124] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.129228] op_queue add: 0x6ec210 [D 20:01:00.129237] delayed I/O operation 0x6ec210 (16 already in progress) [D 20:01:00.129243] DBPF I/O ops in progress: 16 [D 20:01:00.129250] ========================================================= [D 20:01:00.129256] PINT_process_request [D 20:01:00.129262] tiling 11 copies [D 20:01:00.129268] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.129276] lvl 0 el 0 blk 0 by 322961408 [D 20:01:00.129282] to 322961408 ta 301989888 fi 335544320 [D 20:01:00.129289] basic type or contiguous data [D 20:01:00.129295] Request disp [D 20:01:00.129301] contig_offset = 322961408 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.129308] calling distribute [D 20:01:00.129314] PINT_distribute [D 20:01:00.129320] of 322961408 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 322961408 exfl 1 [D 20:01:00.129327] begin iteration loff: 322961408 [D 20:01:00.129334] segment exceeds byte limit [D 20:01:00.129340] file being extended [D 20:01:00.129345] process a segment [D 20:01:00.129351] of 322961408 sz 2097152 [D 20:01:00.129357] add a segment 0 [D 20:01:00.129363] end iteration [D 20:01:00.129369] done with segments or bytes [D 20:01:00.129375] of 325058560 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.129382] next loff: 325058560 next poff: 325058560 [D 20:01:00.129388] return value 2097152 [D 20:01:00.129394] finished [D 20:01:00.129400] retval = 2097152 [D 20:01:00.129406] exiting distribute returned less than expected [D 20:01:00.129412] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 325058560 fo 335544320 eof 0 [D 20:01:00.129420] ========================================================= [D 20:01:00.129426] BMI_post_recv: addr: 6843072, offset: 0x2aaab10e0010, size: 2097152, tag: 17 [D 20:01:00.129434] openib_check_cq: completed send wr_id 683710. [D 20:01:00.129446] post_sr: 10.1.4.56:39990 bh 78 len 48 wr_id 683740. [D 20:01:00.129456] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:01:00.129465] get_data_sync_mode returning 0 [D 20:01:00.129472] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.129478] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf6d3010, mem_size: 2097152 [D 20:01:00.129485] dbpf_bstream_rw_list: stream_offset: 354418688, stream_size: 2097152 [D 20:01:00.129492] dbpf_open_cache_get: called [D 20:01:00.129498] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.129505] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.129530] op_queue add: 0x6ea920 [D 20:01:00.129537] delayed I/O operation 0x6ea920 (16 already in progress) [D 20:01:00.129544] DBPF I/O ops in progress: 16 [D 20:01:00.129551] ========================================================= [D 20:01:00.129557] PINT_process_request [D 20:01:00.129563] tiling 12 copies [D 20:01:00.129570] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.129577] lvl 0 el 0 blk 0 by 356515840 [D 20:01:00.129584] to 356515840 ta 335544320 fi 369098752 [D 20:01:00.129590] basic type or contiguous data [D 20:01:00.129597] Request disp [D 20:01:00.129603] contig_offset = 356515840 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.129610] calling distribute [D 20:01:00.129615] PINT_distribute [D 20:01:00.129621] of 356515840 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 356515840 exfl 1 [D 20:01:00.129628] begin iteration loff: 356515840 [D 20:01:00.129635] segment exceeds byte limit [D 20:01:00.129641] file being extended [D 20:01:00.129646] process a segment [D 20:01:00.129652] of 356515840 sz 2097152 [D 20:01:00.129658] add a segment 0 [D 20:01:00.129664] end iteration [D 20:01:00.129669] done with segments or bytes [D 20:01:00.129675] of 358612992 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.129682] next loff: 358612992 next poff: 358612992 [D 20:01:00.129688] return value 2097152 [D 20:01:00.129694] finished [D 20:01:00.129700] retval = 2097152 [D 20:01:00.129706] exiting distribute returned less than expected [D 20:01:00.129712] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 358612992 fo 369098752 eof 0 [D 20:01:00.129719] ========================================================= [D 20:01:00.129725] BMI_post_recv: addr: 6843072, offset: 0x2aaab12e1010, size: 2097152, tag: 18 [D 20:01:00.129733] openib_check_cq: completed send wr_id 683740. [D 20:01:00.129744] post_sr: 10.1.4.56:39990 bh 79 len 48 wr_id 683770. [D 20:01:00.129752] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:01:00.129760] get_data_sync_mode returning 0 [D 20:01:00.129767] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.129774] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf8d4010, mem_size: 2097152 [D 20:01:00.129781] dbpf_bstream_rw_list: stream_offset: 387973120, stream_size: 2097152 [D 20:01:00.129787] dbpf_open_cache_get: called [D 20:01:00.129793] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.129800] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.129821] op_queue add: 0x6fef80 [D 20:01:00.129829] delayed I/O operation 0x6fef80 (16 already in progress) [D 20:01:00.129835] DBPF I/O ops in progress: 16 [D 20:01:00.129843] ========================================================= [D 20:01:00.129849] PINT_process_request [D 20:01:00.129855] tiling 13 copies [D 20:01:00.129862] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.129869] lvl 0 el 0 blk 0 by 390070272 [D 20:01:00.129875] to 390070272 ta 369098752 fi 402653184 [D 20:01:00.129882] basic type or contiguous data [D 20:01:00.129888] Request disp [D 20:01:00.129893] contig_offset = 390070272 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.129901] calling distribute [D 20:01:00.129907] PINT_distribute [D 20:01:00.129913] of 390070272 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 390070272 exfl 1 [D 20:01:00.129920] begin iteration loff: 390070272 [D 20:01:00.129927] segment exceeds byte limit [D 20:01:00.129933] file being extended [D 20:01:00.129939] process a segment [D 20:01:00.129944] of 390070272 sz 2097152 [D 20:01:00.129951] add a segment 0 [D 20:01:00.129956] end iteration [D 20:01:00.129962] done with segments or bytes [D 20:01:00.129968] of 392167424 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.129975] next loff: 392167424 next poff: 392167424 [D 20:01:00.129981] return value 2097152 [D 20:01:00.129987] finished [D 20:01:00.129992] retval = 2097152 [D 20:01:00.129999] exiting distribute returned less than expected [D 20:01:00.130005] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 392167424 fo 402653184 eof 0 [D 20:01:00.130012] ========================================================= [D 20:01:00.130018] BMI_post_recv: addr: 6843072, offset: 0x2aaab1600010, size: 2097152, tag: 19 [D 20:01:00.130027] openib_check_cq: completed send wr_id 683770. [D 20:01:00.130038] post_sr: 10.1.4.56:39990 bh 80 len 48 wr_id 6837a0. [D 20:01:00.130047] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:00.130055] get_data_sync_mode returning 0 [D 20:01:00.130063] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.130070] dbpf_bstream_rw_list: mem_offset: 0x2aaabb252010, mem_size: 2097152 [D 20:01:00.130077] dbpf_bstream_rw_list: stream_offset: 480247808, stream_size: 2097152 [D 20:01:00.130083] dbpf_open_cache_get: called [D 20:01:00.130090] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.130096] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.130122] op_queue add: 0x6ff0d0 [D 20:01:00.130129] delayed I/O operation 0x6ff0d0 (16 already in progress) [D 20:01:00.130136] DBPF I/O ops in progress: 16 [A 20:01:00.131617] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66fbd0: io: start [D 20:01:00.131684] (0x66fbd0) io (prelude sm) state: getattr_if_needed [D 20:01:00.131693] About to retrieve attributes for handle 2863311531 [D 20:01:00.131746] op_queue add: 0x2aaab15177c0 [D 20:01:00.047523] --- aio_progress_notification called with handle 2863311531 (0x6ead70) [D 20:01:00.132357] ========================================================= [D 20:01:00.132435] PINT_process_request [D 20:01:00.132502] tiling 16 copies [D 20:01:00.132510] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.132518] lvl 0 el 0 blk 0 by 482344960 [D 20:01:00.132524] to 482344960 ta 469762048 fi 503316480 [D 20:01:00.132531] basic type or contiguous data [D 20:01:00.132537] Request disp [D 20:01:00.132618] contig_offset = 482344960 contig_size = 54525952 lvl_flag = 1 [D 20:01:00.132681] calling distribute [D 20:01:00.132688] PINT_distribute [D 20:01:00.132694] of 482344960 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 482344960 exfl 1 [D 20:01:00.132786] begin iteration loff: 482344960 [D 20:01:00.132794] segment exceeds byte limit [D 20:01:00.132800] file being extended [D 20:01:00.132806] process a segment [D 20:01:00.132812] of 482344960 sz 2097152 [D 20:01:00.132818] add a segment 0 [D 20:01:00.132824] end iteration [D 20:01:00.132829] done with segments or bytes [D 20:01:00.132835] of 484442112 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.132842] next loff: 484442112 next poff: 484442112 [D 20:01:00.132848] return value 2097152 [D 20:01:00.132854] finished [D 20:01:00.132860] retval = 2097152 [D 20:01:00.132866] exiting distribute returned less than expected [D 20:01:00.132872] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 484442112 fo 503316480 eof 0 [D 20:01:00.132880] ========================================================= [D 20:01:00.132887] BMI_post_recv: addr: 6843072, offset: 0x2aaabb453010, size: 2097152, tag: 22 [D 20:01:00.132900] [SM Exiting]: (0x66fad0) prelude_work_sm:getattr_if_needed (error code: 0), (action: DEFERRED) [D 20:01:00.132906] [DBPF THREAD]: STARTING TROVE SERVICE ROUTINE (DSPACE_GETATTR) [D 20:01:00.132937] dbpf_open_cache_get: called [D 20:01:00.132947] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.132955] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.047539] --- aio_progress_notification called with handle 2863311531 (0x6eac20) [D 20:01:00.047552] --- aio_progress_notification called with handle 2863311531 (0x6eaad0) [D 20:01:00.132920] [SM Entering]: (0x65f450) prelude_work_sm:perm_check (status: 0) [D 20:01:00.133373] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.132910] openib_check_cq: completed send wr_id 6837a0. [D 20:01:00.133388] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.133403] Final permission check for "io" set error code to 0 [D 20:01:00.133410] (0x65f550) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.050295] --- aio_progress_notification called with handle 2863311531 (0x6eaec0) [D 20:01:00.132963] dbpf_open_cache_put: cached entry. [D 20:01:00.133418] [SM Exiting]: (0x65f450) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.133441] [SM pop_state]: (0x65f450) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.133463] ATTRIB: retrieved attributes from DISK for key 2863311531 uid = 0, mode = 0, type = 2, dfile_count = 0, dist_size = 0 b_size = 480247808 [D 20:01:00.133482] dbpf_attr_cache_insert: inserting 2863311531 (b_size is 480247808) [D 20:01:00.133494] [DBPF THREAD]: FINISHED TROVE SERVICE ROUTINE (DSPACE_GETATTR) (ret: 1) [D 20:01:00.133502] op_queue add: 0x2aaab15177c0 [D 20:01:00.133464] [SM pop_state]: (0x65f450) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.133413] post_sr: 10.1.4.56:39990 bh 81 len 48 wr_id 6837d0. [D 20:01:00.133528] [SM Entering]: (0x65f450) io_sm:send_positive_ack (status: 0) [D 20:01:00.133539] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.133543] openib_check_cq: completed send wr_id 6837d0. [D 20:01:00.133546] PINT_encode [D 20:01:00.133581] encode_common [D 20:01:00.133590] lebf_encode_resp [D 20:01:00.133598] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 23 [D 20:01:00.133606] element 0: offset: 0x2aaab15177a0, size: 24 [D 20:01:00.133467] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.133145] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.133617] post_sr: 10.1.4.56:39990 bh 82 len 40 wr_id 683800. [D 20:01:00.133634] dbpf_open_cache_put: cached entry. [D 20:01:00.133549] --- aio_progress_notification called with handle 2863311531 (0x6f6810) [D 20:01:00.133668] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.060511] --- aio_progress_notification called with handle 2863311531 (0x710ea0) [D 20:01:00.133722] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.060591] --- aio_progress_notification called with handle 2863311531 (0x70e340) [D 20:01:00.133748] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.060627] --- aio_progress_notification called with handle 2863311531 (0x70b7e0) [D 20:01:00.133776] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.068534] --- aio_progress_notification called with handle 2863311531 (0x710ff0) [D 20:01:00.133802] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.068613] --- aio_progress_notification called with handle 2863311531 (0x6f0f00) [D 20:01:00.133829] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.071400] --- aio_progress_notification called with handle 2863311531 (0x6eda80) [D 20:01:00.133644] [SM Exiting]: (0x65f450) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.133648] openib_check_cq: completed send wr_id 683800. [D 20:01:00.133670] dbpf_open_cache_put: cached entry. [D 20:01:00.133678] op_queue add: 0x6ead70 [D 20:01:00.133153] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.133236] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.136506] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.062210] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:01:00.136528] [SM Entering]: (0x660710) prelude_work_sm:perm_check (status: 0) [D 20:01:00.142783] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:00.142894] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.142908] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.142789] DBPF I/O ops in progress: 15 [D 20:01:00.142935] starting delayed I/O operation 0x6feb90 (15 in progress) [D 20:01:00.142946] lio_listio called with 1 following aiocbs: [D 20:01:00.142957] aiocb_ptr_array[0]: fd: 13, off: 220200960, bytes: 2097152, buf: 0x2aaaaeecf010, type: 1 [D 20:01:00.142972] start_delayed_ops_if_any: lio_listio posted 0x6feb90 (handle 2863311531, ret 0)) [D 20:01:00.142920] Final permission check for "io" set error code to 0 [D 20:01:00.142997] (0x660810) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.142808] dbpf_open_cache_put: cached entry. [D 20:01:00.143005] [SM Exiting]: (0x660710) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.143009] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.143022] op_queue add: 0x6eac20 [D 20:01:00.143013] [SM pop_state]: (0x660710) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.143037] [SM pop_state]: (0x660710) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.143042] DBPF I/O ops in progress: 15 [D 20:01:00.143039] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:00.143056] starting delayed I/O operation 0x6fece0 (15 in progress) [D 20:01:00.143048] [SM Entering]: (0x660710) io_sm:send_positive_ack (status: 0) [D 20:01:00.136560] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.143081] lio_listio called with 1 following aiocbs: [D 20:01:00.142861] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.143085] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.143113] aiocb_ptr_array[0]: fd: 13, off: 253755392, bytes: 2097152, buf: 0x2aaaaf0d0010, type: 1 [D 20:01:00.143126] PINT_encode [D 20:01:00.143155] encode_common [D 20:01:00.143164] lebf_encode_resp [D 20:01:00.143174] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 24 [D 20:01:00.143182] element 0: offset: 0x2aaab15016c0, size: 24 [D 20:01:00.143193] post_sr: 10.1.4.56:39990 bh 83 len 40 wr_id 683830. [D 20:01:00.143207] [SM Exiting]: (0x660710) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) -[D 20:01:00.143227] [SM Entering]: (0x6619d0) prelude_work_sm:perm_check (status: 0) [D 20:01:00.143235] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.143237] openib_check_cq: completed send wr_id 683830. [D 20:01:00.143245] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.143272] Final permission check for "io" set error code to 0 [D 20:01:00.143279] (0x661ad0) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.143285] [SM Exiting]: (0x6619d0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.143293] [SM pop_state]: (0x6619d0) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.143300] [SM pop_state]: (0x6619d0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.143306] [SM Entering]: (0x6619d0) io_sm:send_positive_ack (status: 0) [D 20:01:00.143313] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.143320] PINT_encode [D 20:01:00.143326] encode_common [D 20:01:00.143332] lebf_encode_resp [D 20:01:00.143340] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 25 [D 20:01:00.143347] element 0: offset: 0x2aaab150f3c0, size: 24 [D 20:01:00.143145] start_delayed_ops_if_any: lio_listio posted 0x6fece0 (handle 2863311531, ret 0)) [D 20:01:00.143355] post_sr: 10.1.4.56:39990 bh 84 len 40 wr_id 683860. [D 20:01:00.143367] [SM Exiting]: (0x6619d0) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.143374] openib_check_cq: completed send wr_id 683860. [D 20:01:00.143376] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:01:00.143463] Start times (hr:min:sec): 20:00:55.876 20:00:54.832 20:00:53.796 20:00:52.756 20:00:51.720 20:00:50.688 [P 20:01:00.143470] Intervals (hr:min:sec) : 00:00:04.267 00:00:01.044 00:00:01.036 00:00:01.040 00:00:01.036 00:00:01.032 [P 20:01:00.143477] ------------------------------------------------------------------------------------------------------------- [P 20:01:00.143483] bytes read : 0 0 0 0 0 0 [P 20:01:00.143489] bytes written : 257949696 0 0 0 0 0 [P 20:01:00.143496] metadata reads : 42 26 25 24 24 23 [P 20:01:00.143502] metadata writes : 7 7 7 7 7 7 [P 20:01:00.143508] metadata dspace ops : 0 0 0 0 0 0 [P 20:01:00.143514] metadata keyval ops : 0 0 0 0 0 0 [P 20:01:00.143520] request scheduler : 30 14 13 12 12 11 [D 20:01:00.143531] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:01:00.143539] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:01:00.143548] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) [D 20:01:00.143573] [SM Entering]: (0x662c90) prelude_work_sm:perm_check (status: 0) [D 20:01:00.143581] [SM frame get]: (0x662c90) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.143589] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.143597] Final permission check for "io" set error code to 0 [D 20:01:00.143603] (0x662d90) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.143609] [SM Exiting]: (0x662c90) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.143616] [SM pop_state]: (0x662c90) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.143623] [SM pop_state]: (0x662c90) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.143635] [SM Entering]: (0x662c90) io_sm:send_positive_ack (status: 0) [D 20:01:00.143642] [SM frame get]: (0x662c90) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.143649] PINT_encode [D 20:01:00.143655] encode_common [D 20:01:00.143661] lebf_encode_resp [D 20:01:00.143668] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 26 [D 20:01:00.143675] element 0: offset: 0x2aaab1509e60, size: 24 [D 20:01:00.143097] dbpf_open_cache_put: cached entry. [D 20:01:00.143683] post_sr: 10.1.4.56:39990 bh 85 len 40 wr_id 683890. [D 20:01:00.143699] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.143716] [SM Exiting]: (0x662c90) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.143722] openib_check_cq: completed send wr_id 683890. [D 20:01:00.143743] [SM Entering]: (0x663f50) prelude_work_sm:perm_check (status: 0) [D 20:01:00.143762] [SM frame get]: (0x663f50) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.143770] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.143778] Final permission check for "io" set error code to 0 [D 20:01:00.143784] (0x664050) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.143791] [SM Exiting]: (0x663f50) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.143798] [SM pop_state]: (0x663f50) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.143804] [SM pop_state]: (0x663f50) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.143811] [SM Entering]: (0x663f50) io_sm:send_positive_ack (status: 0) [D 20:01:00.143817] [SM frame get]: (0x663f50) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.143824] PINT_encode [D 20:01:00.143830] encode_common [D 20:01:00.143836] lebf_encode_resp [D 20:01:00.143843] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 27 [D 20:01:00.143850] element 0: offset: 0x2aaab150faa0, size: 24 [D 20:01:00.143107] op_queue add: 0x6eaec0 [D 20:01:00.143705] dbpf_open_cache_put: cached entry. [D 20:01:00.143859] post_sr: 10.1.4.56:39990 bh 86 len 40 wr_id 6838c0. [D 20:01:00.143878] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.143884] [SM Exiting]: (0x663f50) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.143892] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:00.143895] openib_check_cq: completed send wr_id 6838c0. [D 20:01:00.143914] [SM Entering]: (0x665210) prelude_work_sm:perm_check (status: 0) [D 20:01:00.148551] [SM frame get]: (0x665210) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.148562] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.148572] Final permission check for "io" set error code to 0 [D 20:01:00.148578] (0x665310) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.148585] [SM Exiting]: (0x665210) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.148593] [SM pop_state]: (0x665210) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.148600] [SM pop_state]: (0x665210) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.148608] [SM Entering]: (0x665210) io_sm:send_positive_ack (status: 0) [D 20:01:00.148614] [SM frame get]: (0x665210) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.148621] PINT_encode [D 20:01:00.148628] encode_common [D 20:01:00.148637] lebf_encode_resp [D 20:01:00.148647] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 28 [D 20:01:00.148654] element 0: offset: 0x2aaab150ff90, size: 24 [D 20:01:00.148668] post_sr: 10.1.4.56:39990 bh 87 len 40 wr_id 6838f0. [D 20:01:00.148690] --- aio_progress_notification called with handle 2863311531 (0x6f9370) [D 20:01:00.148702] [SM Exiting]: (0x665210) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.148712] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.148721] openib_check_cq: completed send wr_id 6838f0. [D 20:01:00.148738] --- aio_progress_notification called with handle 2863311531 (0x6fbf80) [D 20:01:00.148759] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.148772] --- aio_progress_notification called with handle 2863311531 (0x6ed430) [D 20:01:00.148781] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.148794] --- aio_progress_notification called with handle 2863311531 (0x6ea6c0) [D 20:01:00.148802] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.133688] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.148715] [SM Entering]: (0x6664d0) prelude_work_sm:perm_check (status: 0) [D 20:01:00.148834] [SM frame get]: (0x6664d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.148843] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.148851] Final permission check for "io" set error code to 0 [D 20:01:00.148858] (0x6665d0) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.148864] [SM Exiting]: (0x6664d0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.148871] [SM pop_state]: (0x6664d0) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.148878] [SM pop_state]: (0x6664d0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.148885] [SM Entering]: (0x6664d0) io_sm:send_positive_ack (status: 0) [D 20:01:00.148892] [SM frame get]: (0x6664d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.148898] PINT_encode [D 20:01:00.148904] encode_common [D 20:01:00.148911] lebf_encode_resp [D 20:01:00.148918] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 29 [D 20:01:00.148925] element 0: offset: 0x2aaab15138f0, size: 24 [D 20:01:00.149350] openib_check_cq: completed recv wr_id 6842f0. [D 20:01:00.149364] openib_post_rr: 10.1.4.56:39990 bh 40 wr_id 6842f0. [D 20:01:00.149378] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:00.149390] get_data_sync_mode returning 0 [D 20:01:00.149403] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.149411] dbpf_bstream_rw_list: mem_offset: 0x2aaaafad5010, mem_size: 2097152 [D 20:01:00.149418] dbpf_bstream_rw_list: stream_offset: 421527552, stream_size: 2097152 [D 20:01:00.149427] dbpf_open_cache_get: called [D 20:01:00.149440] post_sr: 10.1.4.56:39990 bh 88 len 40 wr_id 683920. [D 20:01:00.149450] [SM Exiting]: (0x6664d0) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.149460] [SM Entering]: (0x667790) prelude_work_sm:perm_check (status: 0) [D 20:01:00.149468] [SM frame get]: (0x667790) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149476] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.149484] Final permission check for "io" set error code to 0 [D 20:01:00.149490] (0x667890) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.149497] [SM Exiting]: (0x667790) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.149504] [SM pop_state]: (0x667790) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.149511] [SM pop_state]: (0x667790) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.149518] [SM Entering]: (0x667790) io_sm:send_positive_ack (status: 0) [D 20:01:00.149525] [SM frame get]: (0x667790) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149532] PINT_encode [D 20:01:00.149538] encode_common [D 20:01:00.149545] lebf_encode_resp [D 20:01:00.149552] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 30 [D 20:01:00.149560] element 0: offset: 0x2aaab1514c00, size: 24 [D 20:01:00.149568] post_sr: 10.1.4.56:39990 bh 89 len 40 wr_id 683950. [D 20:01:00.149577] [SM Exiting]: (0x667790) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.149584] [SM Entering]: (0x668a50) prelude_work_sm:perm_check (status: 0) [D 20:01:00.149592] [SM frame get]: (0x668a50) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149599] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.149608] Final permission check for "io" set error code to 0 [D 20:01:00.149614] (0x668b50) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.149621] [SM Exiting]: (0x668a50) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.149630] [SM pop_state]: (0x668a50) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.149637] [SM pop_state]: (0x668a50) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.149643] [SM Entering]: (0x668a50) io_sm:send_positive_ack (status: 0) [D 20:01:00.149650] [SM frame get]: (0x668a50) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149657] PINT_encode [D 20:01:00.149663] encode_common [D 20:01:00.149669] lebf_encode_resp [D 20:01:00.149677] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 31 [D 20:01:00.149684] element 0: offset: 0x2aaab1501630, size: 24 [D 20:01:00.149692] post_sr: 10.1.4.56:39990 bh 90 len 40 wr_id 683980. [D 20:01:00.149701] [SM Exiting]: (0x668a50) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.149709] [SM Entering]: (0x669d10) prelude_work_sm:perm_check (status: 0) [D 20:01:00.149716] [SM frame get]: (0x669d10) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149723] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.149731] Final permission check for "io" set error code to 0 [D 20:01:00.149738] (0x669e10) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.149744] [SM Exiting]: (0x669d10) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.149751] [SM pop_state]: (0x669d10) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.149758] [SM pop_state]: (0x669d10) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.149764] [SM Entering]: (0x669d10) io_sm:send_positive_ack (status: 0) [D 20:01:00.149771] [SM frame get]: (0x669d10) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149777] PINT_encode [D 20:01:00.149783] encode_common [D 20:01:00.149790] lebf_encode_resp [D 20:01:00.149797] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 32 [D 20:01:00.149804] element 0: offset: 0x2aaab150ea10, size: 24 [D 20:01:00.149813] post_sr: 10.1.4.56:39990 bh 91 len 40 wr_id 6839b0. [D 20:01:00.149821] [SM Exiting]: (0x669d10) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.149829] [SM Entering]: (0x66afd0) prelude_work_sm:perm_check (status: 0) [D 20:01:00.149836] [SM frame get]: (0x66afd0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149844] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.149852] Final permission check for "io" set error code to 0 [D 20:01:00.149858] (0x66b0d0) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.149865] [SM Exiting]: (0x66afd0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.149872] [SM pop_state]: (0x66afd0) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.149879] [SM pop_state]: (0x66afd0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.149886] [SM Entering]: (0x66afd0) io_sm:send_positive_ack (status: 0) [D 20:01:00.149892] [SM frame get]: (0x66afd0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149899] PINT_encode [D 20:01:00.149905] encode_common [D 20:01:00.149911] lebf_encode_resp [D 20:01:00.149919] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 33 [D 20:01:00.149926] element 0: offset: 0x2aaab1500e10, size: 24 [D 20:01:00.149935] post_sr: 10.1.4.56:39990 bh 92 len 40 wr_id 6839e0. [D 20:01:00.149943] [SM Exiting]: (0x66afd0) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.149952] [SM Entering]: (0x66c290) prelude_work_sm:perm_check (status: 0) [D 20:01:00.149959] [SM frame get]: (0x66c290) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.149967] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.149975] Final permission check for "io" set error code to 0 [D 20:01:00.149981] (0x66c390) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.149988] [SM Exiting]: (0x66c290) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.149996] [SM pop_state]: (0x66c290) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.150003] [SM pop_state]: (0x66c290) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.150010] [SM Entering]: (0x66c290) io_sm:send_positive_ack (status: 0) [D 20:01:00.150016] [SM frame get]: (0x66c290) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.150024] PINT_encode [D 20:01:00.150030] encode_common [D 20:01:00.150036] lebf_encode_resp [D 20:01:00.150043] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 34 [D 20:01:00.150051] element 0: offset: 0x2aaab1501ce0, size: 24 [D 20:01:00.150060] post_sr: 10.1.4.56:39990 bh 93 len 40 wr_id 683a10. [D 20:01:00.150069] [SM Exiting]: (0x66c290) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.150077] [SM Entering]: (0x66d550) prelude_work_sm:perm_check (status: 0) [D 20:01:00.150084] [SM frame get]: (0x66d550) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.150092] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.150100] Final permission check for "io" set error code to 0 [D 20:01:00.150106] (0x66d650) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.150113] [SM Exiting]: (0x66d550) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.150120] [SM pop_state]: (0x66d550) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.150127] [SM pop_state]: (0x66d550) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.150133] [SM Entering]: (0x66d550) io_sm:send_positive_ack (status: 0) [D 20:01:00.150140] [SM frame get]: (0x66d550) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.150147] PINT_encode [D 20:01:00.150153] encode_common [D 20:01:00.150158] lebf_encode_resp [D 20:01:00.150165] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 35 [D 20:01:00.150173] element 0: offset: 0x2aaab15135d0, size: 24 [D 20:01:00.150181] post_sr: 10.1.4.56:39990 bh 94 len 40 wr_id 683a40. [D 20:01:00.150190] [SM Exiting]: (0x66d550) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.150198] [SM Entering]: (0x66e810) io_sm:start_flow (status: 0) [D 20:01:00.150205] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.150213] io_start_flow() issuing flow to read data. [D 20:01:00.150221] flow: fsize: 478150656, server_nr: 0, server_ct: 1 [D 20:01:00.150228] file_req_offset: 436207616, aggregate_size: 33554432, handle: 2863311531 [D 20:01:00.150239] PINT_new_request_states n=1 [D 20:01:00.150248] flowproto posting 0x2aaab1518270 [D 20:01:00.150279] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.150290] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518270. [D 20:01:00.150299] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:00.143902] op_queue add: 0x70b7e0 [D 20:01:00.143889] DBPF I/O ops in progress: 15 [D 20:01:00.143907] dbpf_open_cache_put: cached entry. [D 20:01:00.152927] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.152901] starting delayed I/O operation 0x6fee30 (15 in progress) [D 20:01:00.152954] lio_listio called with 1 following aiocbs: [D 20:01:00.152937] op_queue add: 0x710ff0 [D 20:01:00.153008] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:00.153027] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:00.152953] dbpf_open_cache_put: cached entry. [D 20:01:00.152982] ========================================================= [D 20:01:00.153062] dbpf_open_cache_put: cached entry. [D 20:01:00.152967] aiocb_ptr_array[0]: fd: 13, off: 287309824, bytes: 2097152, buf: 0x2aaaaf2d1010, type: 1 [D 20:01:00.153085] PINT_process_request [D 20:01:00.153100] start_delayed_ops_if_any: lio_listio posted 0x6fee30 (handle 2863311531, ret 0)) [D 20:01:00.153106] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.153124] dbpf_open_cache_put: cached entry. [D 20:01:00.153146] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.153117] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.153104] tiling 15 copies [D 20:01:00.153169] DBPF I/O ops in progress: 15 [D 20:01:00.153180] starting delayed I/O operation 0x6ec210 (15 in progress) [D 20:01:00.153193] lio_listio called with 1 following aiocbs: [D 20:01:00.153203] aiocb_ptr_array[0]: fd: 13, off: 320864256, bytes: 2097152, buf: 0x2aaaaf4d2010, type: 1 [D 20:01:00.153170] skipping ahead to target_offset [D 20:01:00.153121] op_queue add: 0x70e340 [D 20:01:00.153227] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.153239] lvl 0 el 0 blk 0 by 0 [D 20:01:00.153241] op_queue add: 0x6eaad0 [D 20:01:00.153249] to 0 ta 436207616 fi 469762048 [D 20:01:00.153220] dbpf_open_cache_put: cached entry. [D 20:01:00.153287] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.153267] op_queue add: 0x6eda80 [D 20:01:00.153315] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:00.153327] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:00.153336] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:01:00.153353] dbpf_open_cache_put: cached entry. [D 20:01:00.153367] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.153375] op_queue add: 0x6ec5d0 [D 20:01:00.153215] start_delayed_ops_if_any: lio_listio posted 0x6ec210 (handle 2863311531, ret 0)) [D 20:01:00.153277] basic type or contiguous data [D 20:01:00.153406] op_queue add: 0x6f0f00 [D 20:01:00.153407] Request disp [D 20:01:00.153428] DBPF I/O ops in progress: 15 [D 20:01:00.153432] contig_offset = 0 contig_size = 503316480 lvl_flag = 1 [D 20:01:00.153425] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:00.153449] starting delayed I/O operation 0x6ea920 (15 in progress) [D 20:01:00.153484] lio_listio called with 1 following aiocbs: [D 20:01:00.153490] aiocb_ptr_array[0]: fd: 13, off: 354418688, bytes: 2097152, buf: 0x2aaaaf6d3010, type: 1 [D 20:01:00.153499] start_delayed_ops_if_any: lio_listio posted 0x6ea920 (handle 2863311531, ret 0)) [D 20:01:00.153451] op_queue add: 0x710ea0 [D 20:01:00.153458] process logical skip [D 20:01:00.153525] DBPF I/O ops in progress: 15 [D 20:01:00.153546] starting delayed I/O operation 0x6fef80 (15 in progress) [D 20:01:00.153536] retval = 436207616 [D 20:01:00.153560] lio_listio called with 1 following aiocbs: [D 20:01:00.153582] aiocb_ptr_array[0]: fd: 13, off: 387973120, bytes: 2097152, buf: 0x2aaaaf8d4010, type: 1 [D 20:01:00.153594] start_delayed_ops_if_any: lio_listio posted 0x6fef80 (handle 2863311531, ret 0)) [D 20:01:00.153573] exiting logical skip because distribute indicates done [D 20:01:00.153606] DBPF I/O ops in progress: 15 [D 20:01:00.153627] starting delayed I/O operation 0x6ff0d0 (15 in progress) [D 20:01:00.153640] lio_listio called with 1 following aiocbs: [D 20:01:00.153617] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.153650] aiocb_ptr_array[0]: fd: 13, off: 480247808, bytes: 2097152, buf: 0x2aaabb252010, type: 1 [D 20:01:00.153663] lvl 0 el 0 blk 0 by 436207616 [D 20:01:00.153671] start_delayed_ops_if_any: lio_listio posted 0x6ff0d0 (handle 2863311531, ret 0)) [D 20:01:00.153672] to 436207616 ta 436207616 fi 469762048 [D 20:01:00.153721] DBPF I/O ops in progress: 15 [D 20:01:00.153726] basic type or contiguous data [D 20:01:00.153760] DBPF I/O ops in progress: 14 [D 20:01:00.153770] Request disp [D 20:01:00.153795] contig_offset = 436207616 contig_size = 67108864 lvl_flag = 1 [D 20:01:00.153799] DBPF I/O ops in progress: 13 [D 20:01:00.153835] DBPF I/O ops in progress: 12 [D 20:01:00.153803] calling distribute [D 20:01:00.153869] PINT_distribute [D 20:01:00.153876] of 436207616 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.153885] begin iteration loff: 436207616 [D 20:01:00.153892] segment exceeds byte limit [D 20:01:00.153899] process a segment [D 20:01:00.153905] of 436207616 sz 2097152 [D 20:01:00.153911] add a segment 0 [D 20:01:00.153917] end iteration [D 20:01:00.153923] done with segments or bytes [D 20:01:00.153929] of 438304768 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.153936] next loff: 438304768 next poff: 438304768 [D 20:01:00.153943] return value 2097152 [D 20:01:00.153949] finished [D 20:01:00.153955] retval = 2097152 [D 20:01:00.153961] exiting distribute returned less than expected [D 20:01:00.153967] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 438304768 fo 469762048 eof 0 [D 20:01:00.153975] ========================================================= [D 20:01:00.153986] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.153993] dbpf_bstream_rw_list: mem_offset: 0x2aaabb654010, mem_size: 2097152 [D 20:01:00.154000] dbpf_bstream_rw_list: stream_offset: 436207616, stream_size: 2097152 [D 20:01:00.154007] dbpf_open_cache_get: called [D 20:01:00.154014] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.154021] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.154046] DBPF I/O ops in progress: 13 [D 20:01:00.154053] lio_listio called with the following aiocbs: [D 20:01:00.154060] aiocb_ptr_array[0]: fd: 13, off: 436207616, bytes: 2097152, buf: 0x2aaabb654010, type: 0 [D 20:01:00.154070] issue_or_delay_io_operation: lio_listio posted 0x2aaab151ad60 (handle 2863311531, ret 0) [D 20:01:00.154078] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.154085] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518270. [D 20:01:00.154093] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.155769] ========================================================= [D 20:01:00.155799] PINT_process_request [D 20:01:00.155806] tiling 15 copies [D 20:01:00.155813] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.155821] lvl 0 el 0 blk 0 by 438304768 [D 20:01:00.155827] to 438304768 ta 436207616 fi 469762048 [D 20:01:00.155834] basic type or contiguous data [D 20:01:00.155840] Request disp [D 20:01:00.155847] contig_offset = 438304768 contig_size = 65011712 lvl_flag = 1 [D 20:01:00.155853] calling distribute [D 20:01:00.155859] PINT_distribute [D 20:01:00.155865] of 438304768 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.155873] begin iteration loff: 438304768 [D 20:01:00.155880] segment exceeds byte limit [D 20:01:00.155886] process a segment [D 20:01:00.155892] of 438304768 sz 2097152 [D 20:01:00.155899] add a segment 0 [D 20:01:00.155905] end iteration [D 20:01:00.155911] done with segments or bytes [D 20:01:00.155917] of 440401920 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.155924] next loff: 440401920 next poff: 440401920 [D 20:01:00.155930] return value 2097152 [D 20:01:00.155937] finished [D 20:01:00.155943] retval = 2097152 [D 20:01:00.155949] exiting distribute returned less than expected [D 20:01:00.155955] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 440401920 fo 469762048 eof 0 [D 20:01:00.155962] ========================================================= [D 20:01:00.155971] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.155977] dbpf_bstream_rw_list: mem_offset: 0x2aaabb855010, mem_size: 2097152 [D 20:01:00.155984] dbpf_bstream_rw_list: stream_offset: 438304768, stream_size: 2097152 [D 20:01:00.155991] dbpf_open_cache_get: called [D 20:01:00.155998] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.156004] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.156028] DBPF I/O ops in progress: 14 [D 20:01:00.156036] lio_listio called with the following aiocbs: [D 20:01:00.156042] aiocb_ptr_array[0]: fd: 13, off: 438304768, bytes: 2097152, buf: 0x2aaabb855010, type: 0 [D 20:01:00.156054] issue_or_delay_io_operation: lio_listio posted 0x2aaab151d8c0 (handle 2863311531, ret 0) [D 20:01:00.156061] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.156068] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518270. [D 20:01:00.156076] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.157756] dbpf_open_cache_put: cached entry. [D 20:01:00.157831] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.157844] op_queue add: 0x6f9370 [D 20:01:00.157857] DBPF I/O ops in progress: 13 [D 20:01:00.157882] --- aio_progress_notification called with handle 2863311531 (0x6feb90) [D 20:01:00.157944] ========================================================= [D 20:01:00.157959] PINT_process_request [D 20:01:00.157982] tiling 15 copies [D 20:01:00.157989] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.157997] lvl 0 el 0 blk 0 by 440401920 [D 20:01:00.158004] to 440401920 ta 436207616 fi 469762048 [D 20:01:00.158011] basic type or contiguous data [D 20:01:00.158018] Request disp [D 20:01:00.158024] contig_offset = 440401920 contig_size = 62914560 lvl_flag = 1 [D 20:01:00.158033] calling distribute [D 20:01:00.158040] PINT_distribute [D 20:01:00.158047] of 440401920 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.158057] begin iteration loff: 440401920 [D 20:01:00.158066] segment exceeds byte limit [D 20:01:00.158073] process a segment [D 20:01:00.158079] of 440401920 sz 2097152 [D 20:01:00.158086] add a segment 0 [D 20:01:00.158092] end iteration [D 20:01:00.158099] done with segments or bytes [D 20:01:00.158105] of 442499072 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.158112] next loff: 442499072 next poff: 442499072 [D 20:01:00.158120] return value 2097152 [D 20:01:00.158126] finished [D 20:01:00.158132] retval = 2097152 [D 20:01:00.158139] exiting distribute returned less than expected [D 20:01:00.158145] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 442499072 fo 469762048 eof 0 [D 20:01:00.158152] ========================================================= [D 20:01:00.158170] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.158179] dbpf_bstream_rw_list: mem_offset: 0x2aaabba56010, mem_size: 2097152 [D 20:01:00.158186] dbpf_bstream_rw_list: stream_offset: 440401920, stream_size: 2097152 [D 20:01:00.158194] dbpf_open_cache_get: called [D 20:01:00.158203] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.158210] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.158254] DBPF I/O ops in progress: 14 [D 20:01:00.158264] lio_listio called with the following aiocbs: [D 20:01:00.158270] aiocb_ptr_array[0]: fd: 13, off: 440401920, bytes: 2097152, buf: 0x2aaabba56010, type: 0 [D 20:01:00.158287] issue_or_delay_io_operation: lio_listio posted 0x2aaab1520420 (handle 2863311531, ret 0) [D 20:01:00.158295] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.158302] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518270. [D 20:01:00.158312] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.157957] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.161529] dbpf_open_cache_put: cached entry. [D 20:01:00.161604] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.161631] ========================================================= [D 20:01:00.161654] PINT_process_request [D 20:01:00.161663] tiling 15 copies [D 20:01:00.161672] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.161681] lvl 0 el 0 blk 0 by 442499072 [D 20:01:00.161688] to 442499072 ta 436207616 fi 469762048 [D 20:01:00.161697] basic type or contiguous data [D 20:01:00.161704] Request disp [D 20:01:00.161714] contig_offset = 442499072 contig_size = 60817408 lvl_flag = 1 [D 20:01:00.161722] calling distribute [D 20:01:00.161728] PINT_distribute [D 20:01:00.161735] of 442499072 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.161743] begin iteration loff: 442499072 [D 20:01:00.161751] segment exceeds byte limit [D 20:01:00.161757] process a segment [D 20:01:00.161763] of 442499072 sz 2097152 [D 20:01:00.161769] add a segment 0 [D 20:01:00.161776] end iteration [D 20:01:00.161782] done with segments or bytes [D 20:01:00.161788] of 444596224 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.161795] next loff: 444596224 next poff: 444596224 [D 20:01:00.161802] return value 2097152 [D 20:01:00.161808] finished [D 20:01:00.161814] retval = 2097152 [D 20:01:00.161820] exiting distribute returned less than expected [D 20:01:00.161826] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 444596224 fo 469762048 eof 0 [D 20:01:00.161834] ========================================================= [D 20:01:00.161882] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.161893] dbpf_bstream_rw_list: mem_offset: 0x2aaabbc57010, mem_size: 2097152 [D 20:01:00.161900] dbpf_bstream_rw_list: stream_offset: 442499072, stream_size: 2097152 [D 20:01:00.161908] dbpf_open_cache_get: called [D 20:01:00.161915] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.161922] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.161952] DBPF I/O ops in progress: 15 [D 20:01:00.161960] lio_listio called with the following aiocbs: [D 20:01:00.161967] aiocb_ptr_array[0]: fd: 13, off: 442499072, bytes: 2097152, buf: 0x2aaabbc57010, type: 0 [D 20:01:00.161988] issue_or_delay_io_operation: lio_listio posted 0x2aaab1522f80 (handle 2863311531, ret 0) [D 20:01:00.161996] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.162003] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518270. [D 20:01:00.162013] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.164946] op_queue add: 0x6feb90 [D 20:01:00.165050] DBPF I/O ops in progress: 14 [D 20:01:00.165076] ========================================================= [D 20:01:00.165093] PINT_process_request [D 20:01:00.165102] tiling 15 copies [D 20:01:00.165110] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.165118] lvl 0 el 0 blk 0 by 444596224 [D 20:01:00.165125] to 444596224 ta 436207616 fi 469762048 [D 20:01:00.165133] basic type or contiguous data [D 20:01:00.165139] Request disp [D 20:01:00.165146] contig_offset = 444596224 contig_size = 58720256 lvl_flag = 1 [D 20:01:00.165153] calling distribute [D 20:01:00.165159] PINT_distribute [D 20:01:00.165165] of 444596224 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.165175] begin iteration loff: 444596224 [D 20:01:00.165183] segment exceeds byte limit [D 20:01:00.165190] process a segment [D 20:01:00.165196] of 444596224 sz 2097152 [D 20:01:00.165202] add a segment 0 [D 20:01:00.165208] end iteration [D 20:01:00.165214] done with segments or bytes [D 20:01:00.165220] of 446693376 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.165227] next loff: 446693376 next poff: 446693376 [D 20:01:00.165233] return value 2097152 [D 20:01:00.165240] finished [D 20:01:00.165246] retval = 2097152 [D 20:01:00.165252] exiting distribute returned less than expected [D 20:01:00.165258] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 446693376 fo 469762048 eof 0 [D 20:01:00.165265] ========================================================= [D 20:01:00.165282] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.165289] dbpf_bstream_rw_list: mem_offset: 0x2aaabbe58010, mem_size: 2097152 [D 20:01:00.165296] dbpf_bstream_rw_list: stream_offset: 444596224, stream_size: 2097152 [D 20:01:00.165304] dbpf_open_cache_get: called [D 20:01:00.165311] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.165320] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.165376] DBPF I/O ops in progress: 15 [D 20:01:00.165387] lio_listio called with the following aiocbs: [D 20:01:00.165393] aiocb_ptr_array[0]: fd: 13, off: 444596224, bytes: 2097152, buf: 0x2aaabbe58010, type: 0 [D 20:01:00.165408] issue_or_delay_io_operation: lio_listio posted 0x2aaab1525bd0 (handle 2863311531, ret 0) [D 20:01:00.165416] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.165423] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518270. [D 20:01:00.165432] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.167788] ========================================================= [D 20:01:00.167851] PINT_process_request [D 20:01:00.167859] tiling 15 copies [D 20:01:00.167867] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.167876] lvl 0 el 0 blk 0 by 446693376 [D 20:01:00.167882] to 446693376 ta 436207616 fi 469762048 [D 20:01:00.167890] basic type or contiguous data [D 20:01:00.167897] Request disp [D 20:01:00.167904] contig_offset = 446693376 contig_size = 56623104 lvl_flag = 1 [D 20:01:00.167912] calling distribute [D 20:01:00.167919] PINT_distribute [D 20:01:00.167927] of 446693376 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.167935] begin iteration loff: 446693376 [D 20:01:00.167942] segment exceeds byte limit [D 20:01:00.167948] process a segment [D 20:01:00.167954] of 446693376 sz 2097152 [D 20:01:00.167961] add a segment 0 [D 20:01:00.167968] end iteration [D 20:01:00.167975] done with segments or bytes [D 20:01:00.167980] of 448790528 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.167987] next loff: 448790528 next poff: 448790528 [D 20:01:00.167994] return value 2097152 [D 20:01:00.168001] finished [D 20:01:00.168006] retval = 2097152 [D 20:01:00.168012] exiting distribute returned less than expected [D 20:01:00.168019] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 448790528 fo 469762048 eof 0 [D 20:01:00.168026] ========================================================= [D 20:01:00.168042] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.168050] dbpf_bstream_rw_list: mem_offset: 0x2aaabc059010, mem_size: 2097152 [D 20:01:00.168058] dbpf_bstream_rw_list: stream_offset: 446693376, stream_size: 2097152 [D 20:01:00.168065] dbpf_open_cache_get: called [D 20:01:00.168072] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.168079] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.168124] DBPF I/O ops in progress: 16 [D 20:01:00.168134] lio_listio called with the following aiocbs: [D 20:01:00.168141] aiocb_ptr_array[0]: fd: 13, off: 446693376, bytes: 2097152, buf: 0x2aaabc059010, type: 0 [D 20:01:00.168154] issue_or_delay_io_operation: lio_listio posted 0x2aaab1528820 (handle 2863311531, ret 0) [D 20:01:00.168163] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.168170] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518270. [D 20:01:00.168180] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.170553] ========================================================= [D 20:01:00.170619] PINT_process_request [D 20:01:00.170628] tiling 15 copies [D 20:01:00.170636] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.170645] lvl 0 el 0 blk 0 by 448790528 [D 20:01:00.170651] to 448790528 ta 436207616 fi 469762048 [D 20:01:00.170659] basic type or contiguous data [D 20:01:00.170666] Request disp [D 20:01:00.170673] contig_offset = 448790528 contig_size = 54525952 lvl_flag = 1 [D 20:01:00.170681] calling distribute [D 20:01:00.170687] PINT_distribute [D 20:01:00.170694] of 448790528 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.170703] begin iteration loff: 448790528 [D 20:01:00.170713] segment exceeds byte limit [D 20:01:00.170721] process a segment [D 20:01:00.170726] of 448790528 sz 2097152 [D 20:01:00.170733] add a segment 0 [D 20:01:00.170740] end iteration [D 20:01:00.170746] done with segments or bytes [D 20:01:00.170752] of 450887680 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.170759] next loff: 450887680 next poff: 450887680 [D 20:01:00.170765] return value 2097152 [D 20:01:00.170772] finished [D 20:01:00.170779] retval = 2097152 [D 20:01:00.170785] exiting distribute returned less than expected [D 20:01:00.170791] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 450887680 fo 469762048 eof 0 [D 20:01:00.170798] ========================================================= [D 20:01:00.170816] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.170824] dbpf_bstream_rw_list: mem_offset: 0x2aaabc25a010, mem_size: 2097152 [D 20:01:00.170833] dbpf_bstream_rw_list: stream_offset: 448790528, stream_size: 2097152 [D 20:01:00.170840] dbpf_open_cache_get: called [D 20:01:00.170848] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.170855] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.170902] op_queue add: 0x2aaab152b470 [D 20:01:00.170914] delayed I/O operation 0x2aaab152b470 (16 already in progress) [D 20:01:00.170921] DBPF I/O ops in progress: 16 [D 20:01:00.170929] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.170936] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518270. [D 20:01:00.170946] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.171734] --- aio_progress_notification called with handle 2863311531 (0x6fece0) [D 20:01:00.171773] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.171792] --- aio_progress_notification called with handle 2863311531 (0x6fee30) [D 20:01:00.171804] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.171818] --- aio_progress_notification called with handle 2863311531 (0x6ec210) [D 20:01:00.171829] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.173660] ========================================================= [D 20:01:00.173720] PINT_process_request [D 20:01:00.173729] tiling 15 copies [D 20:01:00.173738] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.173747] lvl 0 el 0 blk 0 by 450887680 [D 20:01:00.173754] to 450887680 ta 436207616 fi 469762048 [D 20:01:00.173762] basic type or contiguous data [D 20:01:00.173769] Request disp [D 20:01:00.173776] contig_offset = 450887680 contig_size = 52428800 lvl_flag = 1 [D 20:01:00.173783] calling distribute [D 20:01:00.173789] PINT_distribute [D 20:01:00.173795] of 450887680 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.173804] begin iteration loff: 450887680 [D 20:01:00.173812] segment exceeds byte limit [D 20:01:00.173819] process a segment [D 20:01:00.173825] of 450887680 sz 2097152 [D 20:01:00.173832] add a segment 0 [D 20:01:00.173838] end iteration [D 20:01:00.173844] done with segments or bytes [D 20:01:00.173850] of 452984832 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.173858] next loff: 452984832 next poff: 452984832 [D 20:01:00.173864] return value 2097152 [D 20:01:00.173871] finished [D 20:01:00.173877] retval = 2097152 [D 20:01:00.173883] exiting distribute returned less than expected [D 20:01:00.173889] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 452984832 fo 469762048 eof 0 [D 20:01:00.173896] ========================================================= [D 20:01:00.173913] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.173920] dbpf_bstream_rw_list: mem_offset: 0x2aaabc45b010, mem_size: 2097152 [D 20:01:00.173928] dbpf_bstream_rw_list: stream_offset: 450887680, stream_size: 2097152 [D 20:01:00.173938] dbpf_open_cache_get: called [D 20:01:00.176521] --- aio_progress_notification called with handle 2863311531 (0x6ea920) [D 20:01:00.176573] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.176614] dbpf_open_cache_put: cached entry. [D 20:01:00.176636] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.176644] op_queue add: 0x6fbf80 [D 20:01:00.176654] DBPF I/O ops in progress: 15 [D 20:01:00.176662] starting delayed I/O operation 0x2aaab152b470 (15 in progress) [D 20:01:00.176673] lio_listio called with 1 following aiocbs: [D 20:01:00.176680] aiocb_ptr_array[0]: fd: 13, off: 448790528, bytes: 2097152, buf: 0x2aaabc25a010, type: 0 [D 20:01:00.176696] start_delayed_ops_if_any: lio_listio posted 0x2aaab152b470 (handle 2863311531, ret 0)) [D 20:01:00.176735] dbpf_open_cache_put: cached entry. [D 20:01:00.176748] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.176755] op_queue add: 0x6ed430 [D 20:01:00.176763] DBPF I/O ops in progress: 15 [D 20:01:00.176781] dbpf_open_cache_put: cached entry. [D 20:01:00.176796] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.176803] op_queue add: 0x6ea6c0 [D 20:01:00.176812] DBPF I/O ops in progress: 14 [D 20:01:00.176836] dbpf_open_cache_put: cached entry. [D 20:01:00.176855] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.176867] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.176879] dbpf_attr_cache_remove: removing 2863311531 [D 20:01:00.176903] DBPF I/O ops in progress: 15 [D 20:01:00.176910] lio_listio called with the following aiocbs: [D 20:01:00.176916] aiocb_ptr_array[0]: fd: 13, off: 421527552, bytes: 2097152, buf: 0x2aaaafad5010, type: 1 [D 20:01:00.176926] issue_or_delay_io_operation: lio_listio posted 0x6eac20 (handle 2863311531, ret 0) [D 20:01:00.176938] ========================================================= [D 20:01:00.176946] PINT_process_request [D 20:01:00.176953] tiling 14 copies [D 20:01:00.176959] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.176967] lvl 0 el 0 blk 0 by 423624704 [D 20:01:00.176974] to 423624704 ta 402653184 fi 436207616 [D 20:01:00.176981] basic type or contiguous data [D 20:01:00.176987] Request disp [D 20:01:00.176993] contig_offset = 423624704 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.177001] calling distribute [D 20:01:00.177007] PINT_distribute [D 20:01:00.177013] of 423624704 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 423624704 exfl 1 [D 20:01:00.177021] begin iteration loff: 423624704 [D 20:01:00.177029] segment exceeds byte limit [D 20:01:00.177035] file being extended [D 20:01:00.177041] process a segment [D 20:01:00.177047] of 423624704 sz 2097152 [D 20:01:00.177053] add a segment 0 [D 20:01:00.177059] end iteration [D 20:01:00.177065] done with segments or bytes [D 20:01:00.177071] of 425721856 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.177078] next loff: 425721856 next poff: 425721856 [D 20:01:00.177084] return value 2097152 [D 20:01:00.177090] finished [D 20:01:00.177096] retval = 2097152 [D 20:01:00.177103] exiting distribute returned less than expected [D 20:01:00.177109] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 425721856 fo 436207616 eof 0 [D 20:01:00.177116] ========================================================= [D 20:01:00.177124] BMI_post_recv: addr: 6843072, offset: 0x2aaab1801010, size: 2097152, tag: 20 [D 20:01:00.177141] openib_check_cq: completed send wr_id 683920. [D 20:01:00.177152] openib_check_cq: completed send wr_id 683950. [D 20:01:00.177159] openib_check_cq: completed send wr_id 683980. [D 20:01:00.177166] openib_check_cq: completed send wr_id 6839b0. [D 20:01:00.177173] openib_check_cq: completed send wr_id 6839e0. [D 20:01:00.177180] openib_check_cq: completed send wr_id 683a10. [D 20:01:00.177186] openib_check_cq: completed send wr_id 683a40. [D 20:01:00.177196] openib_check_cq: completed recv wr_id 684320. [D 20:01:00.177207] openib_post_rr: 10.1.4.56:39990 bh 41 wr_id 684320. [D 20:01:00.177217] openib_check_cq: completed recv wr_id 684350. [D 20:01:00.177225] openib_post_rr: 10.1.4.56:39990 bh 42 wr_id 684350. [D 20:01:00.177232] openib_check_cq: completed recv wr_id 684380. [D 20:01:00.177241] openib_post_rr: 10.1.4.56:39990 bh 43 wr_id 684380. [D 20:01:00.177248] openib_check_cq: completed recv wr_id 6843b0. [D 20:01:00.177256] openib_post_rr: 10.1.4.56:39990 bh 44 wr_id 6843b0. [D 20:01:00.177263] openib_check_cq: completed recv wr_id 6843e0. [D 20:01:00.177271] openib_post_rr: 10.1.4.56:39990 bh 45 wr_id 6843e0. [D 20:01:00.177278] openib_check_cq: completed recv wr_id 684410. [D 20:01:00.177286] openib_post_rr: 10.1.4.56:39990 bh 46 wr_id 684410. [D 20:01:00.177294] openib_check_cq: completed recv wr_id 684440. [D 20:01:00.177301] openib_post_rr: 10.1.4.56:39990 bh 47 wr_id 684440. [D 20:01:00.177308] openib_check_cq: completed recv wr_id 684470. [D 20:01:00.177316] openib_post_rr: 10.1.4.56:39990 bh 48 wr_id 684470. [D 20:01:00.177323] openib_check_cq: completed recv wr_id 6844a0. [D 20:01:00.177331] openib_post_rr: 10.1.4.56:39990 bh 49 wr_id 6844a0. [D 20:01:00.177338] openib_check_cq: completed recv wr_id 6844d0. [D 20:01:00.177347] openib_post_rr: 10.1.4.56:39990 bh 50 wr_id 6844d0. [D 20:01:00.177354] openib_check_cq: completed recv wr_id 684500. [D 20:01:00.177362] openib_post_rr: 10.1.4.56:39990 bh 51 wr_id 684500. [D 20:01:00.177369] openib_check_cq: completed recv wr_id 684530. [D 20:01:00.177377] openib_post_rr: 10.1.4.56:39990 bh 52 wr_id 684530. [D 20:01:00.177408] post_sr: 10.1.4.56:39990 bh 95 len 48 wr_id 683a70. [D 20:01:00.177427] openib_check_cq: completed send wr_id 683a70. [D 20:01:00.177458] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:00.177469] get_data_sync_mode returning 0 [D 20:01:00.177481] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.177488] dbpf_bstream_rw_list: mem_offset: 0x2aaaafcd6010, mem_size: 2097152 [D 20:01:00.177495] dbpf_bstream_rw_list: stream_offset: 455081984, stream_size: 2097152 [D 20:01:00.177502] dbpf_open_cache_get: called [D 20:01:00.177508] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.177514] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.177532] DBPF I/O ops in progress: 16 [D 20:01:00.177540] lio_listio called with the following aiocbs: [D 20:01:00.177546] aiocb_ptr_array[0]: fd: 13, off: 455081984, bytes: 2097152, buf: 0x2aaaafcd6010, type: 1 [D 20:01:00.177555] issue_or_delay_io_operation: lio_listio posted 0x710ff0 (handle 2863311531, ret 0) [D 20:01:00.177563] ========================================================= [D 20:01:00.177570] PINT_process_request [D 20:01:00.177576] tiling 15 copies [D 20:01:00.177583] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.177590] lvl 0 el 0 blk 0 by 457179136 [D 20:01:00.177596] to 457179136 ta 436207616 fi 469762048 [D 20:01:00.177603] basic type or contiguous data [D 20:01:00.177609] Request disp [D 20:01:00.177615] contig_offset = 457179136 contig_size = 46137344 lvl_flag = 1 [D 20:01:00.177622] calling distribute [D 20:01:00.177628] PINT_distribute [D 20:01:00.177634] of 457179136 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 457179136 exfl 1 [D 20:01:00.177643] begin iteration loff: 457179136 [D 20:01:00.177649] segment exceeds byte limit [D 20:01:00.177655] file being extended [D 20:01:00.177660] process a segment [D 20:01:00.177666] of 457179136 sz 2097152 [D 20:01:00.177673] add a segment 0 [D 20:01:00.177678] end iteration [D 20:01:00.177684] done with segments or bytes [D 20:01:00.177690] of 459276288 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.177697] next loff: 459276288 next poff: 459276288 [D 20:01:00.177703] return value 2097152 [D 20:01:00.177709] finished [D 20:01:00.177714] retval = 2097152 [D 20:01:00.177720] exiting distribute returned less than expected [D 20:01:00.177733] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.236608] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.236640] op_queue add: 0x2aaab152e050 [D 20:01:00.236649] delayed I/O operation 0x2aaab152e050 (16 already in progress) [D 20:01:00.236656] DBPF I/O ops in progress: 16 [D 20:01:00.236665] flowproto posted 0x2aaab1518270 [D 20:01:00.236672] Job flows in progress (post time): 16 [D 20:01:00.236683] [SM Exiting]: (0x66e810) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:00.236694] [SM Entering]: (0x66fad0) prelude_work_sm:perm_check (status: 0) [D 20:01:00.236702] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.236713] PVFS operation "io" got attr mask 127 (attr_uid_valid? yes, attr_owner = 0, credentials_uid = 2840) (attr_gid_valid? yes, attr_group = 0, credentials.gid = 4001) [D 20:01:00.236722] Final permission check for "io" set error code to 0 [D 20:01:00.236728] (0x66fbd0) io (prelude sm) state: perm_check (status = 0) [D 20:01:00.236735] [SM Exiting]: (0x66fad0) prelude_work_sm:perm_check (error code: 0), (action: COMPLETE) [D 20:01:00.236742] [SM pop_state]: (0x66fad0) op-id: 3 stk-ptr: 2 base-frm: 0 [D 20:01:00.236749] [SM pop_state]: (0x66fad0) op-id: 3 stk-ptr: 1 base-frm: 0 [D 20:01:00.236756] [SM Entering]: (0x66fad0) io_sm:send_positive_ack (status: 0) [D 20:01:00.236763] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.236770] PINT_encode [D 20:01:00.236777] encode_common [D 20:01:00.236785] lebf_encode_resp [D 20:01:00.236795] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 37 [D 20:01:00.236802] element 0: offset: 0x2aaab1517780, size: 24 [D 20:01:00.236813] post_sr: 10.1.4.56:39990 bh 96 len 40 wr_id 683aa0. [D 20:01:00.236825] [SM Exiting]: (0x66fad0) io_sm:send_positive_ack (error code: 0), (action: DEFERRED) [D 20:01:00.236833] [SM Entering]: (0x65f450) io_sm:start_flow (status: 0) [D 20:01:00.236840] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.236848] io_start_flow() issuing flow to read data. [D 20:01:00.236855] flow: fsize: 440401920, server_nr: 0, server_ct: 1 [D 20:01:00.236862] file_req_offset: 0, aggregate_size: 33554432, handle: 2863311531 [D 20:01:00.236871] PINT_new_request_states n=1 [D 20:01:00.236879] flowproto posting 0x2aaab1517ef0 [D 20:01:00.236899] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.236908] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517ef0. [D 20:01:00.236917] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:00.177749] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:00.238140] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:00.238167] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:00.238175] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:00.238183] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:01:00.238191] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:01:00.238199] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:00.238213] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 459276288 fo 469762048 eof 0 [D 20:01:00.238223] ========================================================= [D 20:01:00.238231] BMI_post_recv: addr: 6843072, offset: 0x2aaab1a02010, size: 2097152, tag: 21 [D 20:01:00.238243] openib_check_cq: completed recv wr_id 684560. [D 20:01:00.238354] dbpf_open_cache_put: cached entry. [D 20:01:00.238503] openib_post_rr: 10.1.4.56:39990 bh 53 wr_id 684560. [D 20:01:00.238520] openib_check_cq: completed recv wr_id 684590. [D 20:01:00.238527] openib_post_rr: 10.1.4.56:39990 bh 54 wr_id 684590. [D 20:01:00.238535] openib_check_cq: completed recv wr_id 6845c0. [D 20:01:00.238542] openib_post_rr: 10.1.4.56:39990 bh 55 wr_id 6845c0. [D 20:01:00.238552] openib_check_cq: completed recv wr_id 6845f0. [D 20:01:00.238559] openib_post_rr: 10.1.4.56:39990 bh 56 wr_id 6845f0. [D 20:01:00.238566] openib_check_cq: completed send wr_id 683aa0. [D 20:01:00.238588] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:01:00.238596] get_data_sync_mode returning 0 [D 20:01:00.238605] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.238612] dbpf_bstream_rw_list: mem_offset: 0x2aaaafed7010, mem_size: 2097152 [D 20:01:00.238618] dbpf_bstream_rw_list: stream_offset: 20971520, stream_size: 2097152 [D 20:01:00.238625] dbpf_open_cache_get: called [D 20:01:00.238632] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.238638] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.238655] op_queue add: 0x6f0f00 [D 20:01:00.238662] delayed I/O operation 0x6f0f00 (16 already in progress) [D 20:01:00.238669] DBPF I/O ops in progress: 16 [D 20:01:00.238676] ========================================================= [D 20:01:00.238682] PINT_process_request [D 20:01:00.238689] tiling 2 copies [D 20:01:00.238695] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.238702] lvl 0 el 0 blk 0 by 23068672 [D 20:01:00.238709] to 23068672 ta 0 fi 33554432 [D 20:01:00.238715] basic type or contiguous data [D 20:01:00.238721] Request disp [D 20:01:00.238727] contig_offset = 23068672 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.238734] calling distribute [D 20:01:00.238740] PINT_distribute [D 20:01:00.238746] of 23068672 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 23068672 exfl 1 [D 20:01:00.238754] begin iteration loff: 23068672 [D 20:01:00.238761] segment exceeds byte limit [D 20:01:00.238767] file being extended [D 20:01:00.238773] process a segment [D 20:01:00.238779] of 23068672 sz 2097152 [D 20:01:00.238785] add a segment 0 [D 20:01:00.238791] end iteration [D 20:01:00.238797] done with segments or bytes [D 20:01:00.238802] of 25165824 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.238809] next loff: 25165824 next poff: 25165824 [D 20:01:00.238815] return value 2097152 [D 20:01:00.238821] finished [D 20:01:00.238827] retval = 2097152 [D 20:01:00.238833] exiting distribute returned less than expected [D 20:01:00.238839] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 25165824 fo 33554432 eof 0 [D 20:01:00.238846] ========================================================= [D 20:01:00.238852] BMI_post_recv: addr: 6843072, offset: 0x2aaab1c03010, size: 2097152, tag: 8 [D 20:01:00.238863] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:01:00.238871] get_data_sync_mode returning 0 [D 20:01:00.238877] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.238884] dbpf_bstream_rw_list: mem_offset: 0x2aaab00d8010, mem_size: 2097152 [D 20:01:00.238890] dbpf_bstream_rw_list: stream_offset: 54525952, stream_size: 2097152 [D 20:01:00.238897] dbpf_open_cache_get: called [D 20:01:00.238903] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.238909] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.238922] op_queue add: 0x6feb90 [D 20:01:00.238929] delayed I/O operation 0x6feb90 (16 already in progress) [D 20:01:00.238935] DBPF I/O ops in progress: 16 [D 20:01:00.238942] ========================================================= [D 20:01:00.238948] PINT_process_request [D 20:01:00.238954] tiling 3 copies [D 20:01:00.238960] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.238967] lvl 0 el 0 blk 0 by 56623104 [D 20:01:00.238973] to 56623104 ta 33554432 fi 67108864 [D 20:01:00.238980] basic type or contiguous data [D 20:01:00.238986] Request disp [D 20:01:00.238992] contig_offset = 56623104 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.238998] calling distribute [D 20:01:00.239004] PINT_distribute [D 20:01:00.239009] of 56623104 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 56623104 exfl 1 [D 20:01:00.239017] begin iteration loff: 56623104 [D 20:01:00.239024] segment exceeds byte limit [D 20:01:00.239029] file being extended [D 20:01:00.239035] process a segment [D 20:01:00.239041] of 56623104 sz 2097152 [D 20:01:00.239047] add a segment 0 [D 20:01:00.239053] end iteration [D 20:01:00.239058] done with segments or bytes [D 20:01:00.239064] of 58720256 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.239071] next loff: 58720256 next poff: 58720256 [D 20:01:00.239077] return value 2097152 [D 20:01:00.239083] finished [D 20:01:00.239088] retval = 2097152 [D 20:01:00.239094] exiting distribute returned less than expected [D 20:01:00.239100] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 58720256 fo 67108864 eof 0 [D 20:01:00.239107] ========================================================= [D 20:01:00.239113] BMI_post_recv: addr: 6843072, offset: 0x2aaab1e04010, size: 2097152, tag: 9 [D 20:01:00.239132] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:01:00.239140] get_data_sync_mode returning 0 [D 20:01:00.239147] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.239154] dbpf_bstream_rw_list: mem_offset: 0x2aaab02d9010, mem_size: 2097152 [D 20:01:00.239161] dbpf_bstream_rw_list: stream_offset: 88080384, stream_size: 2097152 [D 20:01:00.239167] dbpf_open_cache_get: called [D 20:01:00.239173] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.239179] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.239193] op_queue add: 0x6ed430 [D 20:01:00.239200] delayed I/O operation 0x6ed430 (16 already in progress) [D 20:01:00.239206] DBPF I/O ops in progress: 16 [D 20:01:00.239213] ========================================================= [D 20:01:00.239219] PINT_process_request [D 20:01:00.239225] tiling 4 copies [D 20:01:00.239231] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.239239] lvl 0 el 0 blk 0 by 90177536 [D 20:01:00.239245] to 90177536 ta 67108864 fi 100663296 [D 20:01:00.239252] basic type or contiguous data [D 20:01:00.239257] Request disp [D 20:01:00.239263] contig_offset = 90177536 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.239270] calling distribute [D 20:01:00.239275] PINT_distribute [D 20:01:00.239281] of 90177536 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 90177536 exfl 1 [D 20:01:00.239288] begin iteration loff: 90177536 [D 20:01:00.239294] segment exceeds byte limit [D 20:01:00.239300] file being extended [D 20:01:00.239306] process a segment [D 20:01:00.239312] of 90177536 sz 2097152 [D 20:01:00.239318] add a segment 0 [D 20:01:00.239324] end iteration [D 20:01:00.239329] done with segments or bytes [D 20:01:00.239335] of 92274688 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.239342] next loff: 92274688 next poff: 92274688 [D 20:01:00.239348] return value 2097152 [D 20:01:00.239354] finished [D 20:01:00.239360] retval = 2097152 [D 20:01:00.239365] exiting distribute returned less than expected [D 20:01:00.239371] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 92274688 fo 100663296 eof 0 [D 20:01:00.239379] ========================================================= [D 20:01:00.239385] BMI_post_recv: addr: 6843072, offset: 0x2aaab2005010, size: 2097152, tag: 10 [D 20:01:00.239395] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:01:00.239403] get_data_sync_mode returning 0 [D 20:01:00.239410] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.239417] dbpf_bstream_rw_list: mem_offset: 0x2aaab04da010, mem_size: 2097152 [D 20:01:00.239423] dbpf_bstream_rw_list: stream_offset: 121634816, stream_size: 2097152 [D 20:01:00.239430] dbpf_open_cache_get: called [D 20:01:00.239436] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.239442] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.239456] op_queue add: 0x6eaa70 [D 20:01:00.239463] delayed I/O operation 0x6eaa70 (16 already in progress) [D 20:01:00.239470] DBPF I/O ops in progress: 16 [D 20:01:00.239476] ========================================================= [D 20:01:00.239482] PINT_process_request [D 20:01:00.239488] tiling 5 copies [D 20:01:00.239494] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.239502] lvl 0 el 0 blk 0 by 123731968 [D 20:01:00.239508] to 123731968 ta 100663296 fi 134217728 [D 20:01:00.239515] basic type or contiguous data [D 20:01:00.239520] Request disp [D 20:01:00.239526] contig_offset = 123731968 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.239533] calling distribute [D 20:01:00.239538] PINT_distribute [D 20:01:00.239544] of 123731968 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 123731968 exfl 1 [D 20:01:00.239551] begin iteration loff: 123731968 [D 20:01:00.239557] segment exceeds byte limit [D 20:01:00.239563] file being extended [D 20:01:00.239569] process a segment [D 20:01:00.239574] of 123731968 sz 2097152 [D 20:01:00.239580] add a segment 0 [D 20:01:00.239586] end iteration [D 20:01:00.239592] done with segments or bytes [D 20:01:00.239597] of 125829120 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.239604] next loff: 125829120 next poff: 125829120 [D 20:01:00.239610] return value 2097152 [D 20:01:00.239616] finished [D 20:01:00.239622] retval = 2097152 [D 20:01:00.239627] exiting distribute returned less than expected [D 20:01:00.239634] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 125829120 fo 134217728 eof 0 [D 20:01:00.239641] ========================================================= [D 20:01:00.239647] BMI_post_recv: addr: 6843072, offset: 0x2aaab2206010, size: 2097152, tag: 11 [D 20:01:00.239657] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:01:00.239665] get_data_sync_mode returning 0 [D 20:01:00.239672] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.239679] dbpf_bstream_rw_list: mem_offset: 0x2aaab06db010, mem_size: 2097152 [D 20:01:00.239685] dbpf_bstream_rw_list: stream_offset: 155189248, stream_size: 2097152 [D 20:01:00.239691] dbpf_open_cache_get: called [D 20:01:00.239697] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.239704] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.239717] op_queue add: 0x6f0a90 [D 20:01:00.239724] delayed I/O operation 0x6f0a90 (16 already in progress) [D 20:01:00.239730] DBPF I/O ops in progress: 16 [D 20:01:00.239737] ========================================================= [D 20:01:00.239743] PINT_process_request [D 20:01:00.239749] tiling 6 copies [D 20:01:00.239755] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.239762] lvl 0 el 0 blk 0 by 157286400 [D 20:01:00.239769] to 157286400 ta 134217728 fi 167772160 [D 20:01:00.239775] basic type or contiguous data [D 20:01:00.239781] Request disp [D 20:01:00.239787] contig_offset = 157286400 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.239793] calling distribute [D 20:01:00.239799] PINT_distribute [D 20:01:00.239805] of 157286400 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 157286400 exfl 1 [D 20:01:00.239812] begin iteration loff: 157286400 [D 20:01:00.239818] segment exceeds byte limit [D 20:01:00.239824] file being extended [D 20:01:00.239830] process a segment [D 20:01:00.239835] of 157286400 sz 2097152 [D 20:01:00.239841] add a segment 0 [D 20:01:00.239847] end iteration [D 20:01:00.239853] done with segments or bytes [D 20:01:00.239859] of 159383552 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.239865] next loff: 159383552 next poff: 159383552 [D 20:01:00.239872] return value 2097152 [D 20:01:00.239878] finished [D 20:01:00.239883] retval = 2097152 [D 20:01:00.239889] exiting distribute returned less than expected [D 20:01:00.239895] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 159383552 fo 167772160 eof 0 [D 20:01:00.239902] ========================================================= [D 20:01:00.240012] BMI_post_recv: addr: 6843072, offset: 0x2aaab2407010, size: 2097152, tag: 12 [D 20:01:00.240024] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:01:00.240032] get_data_sync_mode returning 0 [D 20:01:00.240039] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.240046] dbpf_bstream_rw_list: mem_offset: 0x2aaab08dc010, mem_size: 2097152 [D 20:01:00.240052] dbpf_bstream_rw_list: stream_offset: 188743680, stream_size: 2097152 [D 20:01:00.240058] dbpf_open_cache_get: called [D 20:01:00.240064] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.240071] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.240086] op_queue add: 0x6fea40 [D 20:01:00.240093] delayed I/O operation 0x6fea40 (16 already in progress) [D 20:01:00.240099] DBPF I/O ops in progress: 16 [D 20:01:00.240106] ========================================================= [D 20:01:00.240112] PINT_process_request [D 20:01:00.240118] tiling 7 copies [D 20:01:00.240124] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.240131] lvl 0 el 0 blk 0 by 190840832 [D 20:01:00.240137] to 190840832 ta 167772160 fi 201326592 [D 20:01:00.240144] basic type or contiguous data [D 20:01:00.240150] Request disp [D 20:01:00.240155] contig_offset = 190840832 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.240162] calling distribute [D 20:01:00.240168] PINT_distribute [D 20:01:00.240173] of 190840832 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 190840832 exfl 1 [D 20:01:00.240181] begin iteration loff: 190840832 [D 20:01:00.240187] segment exceeds byte limit [D 20:01:00.240192] file being extended [D 20:01:00.240198] process a segment [D 20:01:00.240204] of 190840832 sz 2097152 [D 20:01:00.240210] add a segment 0 [D 20:01:00.240215] end iteration [D 20:01:00.240221] done with segments or bytes [D 20:01:00.240227] of 192937984 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.240233] next loff: 192937984 next poff: 192937984 [D 20:01:00.240240] return value 2097152 [D 20:01:00.240245] finished [D 20:01:00.240251] retval = 2097152 [D 20:01:00.240257] exiting distribute returned less than expected [D 20:01:00.240263] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 192937984 fo 201326592 eof 0 [D 20:01:00.240270] ========================================================= [D 20:01:00.240276] BMI_post_recv: addr: 6843072, offset: 0x2aaab2608010, size: 2097152, tag: 13 [D 20:01:00.240290] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:01:00.240298] get_data_sync_mode returning 0 [D 20:01:00.240305] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.240312] dbpf_bstream_rw_list: mem_offset: 0x2aaab0add010, mem_size: 2097152 [D 20:01:00.240318] dbpf_bstream_rw_list: stream_offset: 222298112, stream_size: 2097152 [D 20:01:00.240325] dbpf_open_cache_get: called [D 20:01:00.240331] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.240337] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.240351] op_queue add: 0x6ea1d0 [D 20:01:00.240357] delayed I/O operation 0x6ea1d0 (16 already in progress) [D 20:01:00.240364] DBPF I/O ops in progress: 16 [D 20:01:00.240370] ========================================================= [D 20:01:00.240376] PINT_process_request [D 20:01:00.240382] tiling 8 copies [D 20:01:00.240388] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.240395] lvl 0 el 0 blk 0 by 224395264 [D 20:01:00.240402] to 224395264 ta 201326592 fi 234881024 [D 20:01:00.240408] basic type or contiguous data [D 20:01:00.240414] Request disp [D 20:01:00.240420] contig_offset = 224395264 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.240426] calling distribute [D 20:01:00.240432] PINT_distribute [D 20:01:00.240438] of 224395264 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 224395264 exfl 1 [D 20:01:00.240445] begin iteration loff: 224395264 [D 20:01:00.240452] segment exceeds byte limit [D 20:01:00.240458] file being extended [D 20:01:00.240463] process a segment [D 20:01:00.240469] of 224395264 sz 2097152 [D 20:01:00.240481] add a segment 0 [D 20:01:00.240520] end iteration [D 20:01:00.240526] done with segments or bytes [D 20:01:00.240539] of 226492416 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.240572] next loff: 226492416 next poff: 226492416 [D 20:01:00.240578] return value 2097152 [D 20:01:00.240584] finished [D 20:01:00.240590] retval = 2097152 [D 20:01:00.240595] exiting distribute returned less than expected [D 20:01:00.240601] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 226492416 fo 234881024 eof 0 [D 20:01:00.240608] ========================================================= [D 20:01:00.240614] BMI_post_recv: addr: 6843072, offset: 0x2aaab2809010, size: 2097152, tag: 14 [D 20:01:00.240625] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:01:00.240633] get_data_sync_mode returning 0 [D 20:01:00.240640] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.240647] dbpf_bstream_rw_list: mem_offset: 0x2aaab0cde010, mem_size: 2097152 [D 20:01:00.240689] dbpf_bstream_rw_list: stream_offset: 255852544, stream_size: 2097152 [D 20:01:00.240719] dbpf_open_cache_get: called [D 20:01:00.240725] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.240731] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.240744] op_queue add: 0x6ead70 [D 20:01:00.240751] delayed I/O operation 0x6ead70 (16 already in progress) [D 20:01:00.240757] DBPF I/O ops in progress: 16 [D 20:01:00.240764] ========================================================= [D 20:01:00.240770] PINT_process_request [D 20:01:00.240775] tiling 9 copies [D 20:01:00.240781] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.240789] lvl 0 el 0 blk 0 by 257949696 [D 20:01:00.240795] to 257949696 ta 234881024 fi 268435456 [D 20:01:00.240802] basic type or contiguous data [D 20:01:00.240842] Request disp [D 20:01:00.240847] contig_offset = 257949696 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.240860] calling distribute [D 20:01:00.240908] PINT_distribute [D 20:01:00.240914] of 257949696 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 257949696 exfl 1 [D 20:01:00.240921] begin iteration loff: 257949696 [D 20:01:00.240927] segment exceeds byte limit [D 20:01:00.240933] file being extended [D 20:01:00.240939] process a segment [D 20:01:00.240944] of 257949696 sz 2097152 [D 20:01:00.240950] add a segment 0 [D 20:01:00.240956] end iteration [D 20:01:00.240962] done with segments or bytes [D 20:01:00.240967] of 260046848 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.240974] next loff: 260046848 next poff: 260046848 [D 20:01:00.240980] return value 2097152 [D 20:01:00.240986] finished [D 20:01:00.240991] retval = 2097152 [D 20:01:00.240997] exiting distribute returned less than expected [D 20:01:00.241111] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 260046848 fo 268435456 eof 0 [D 20:01:00.241119] ========================================================= [D 20:01:00.241125] BMI_post_recv: addr: 6843072, offset: 0x2aaab2a0a010, size: 2097152, tag: 15 [D 20:01:00.241136] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:01:00.241144] get_data_sync_mode returning 0 [D 20:01:00.241150] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.241157] dbpf_bstream_rw_list: mem_offset: 0x2aaab0edf010, mem_size: 2097152 [D 20:01:00.241163] dbpf_bstream_rw_list: stream_offset: 289406976, stream_size: 2097152 [D 20:01:00.241170] dbpf_open_cache_get: called [D 20:01:00.241176] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.241182] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.241196] op_queue add: 0x6eaec0 [D 20:01:00.241249] delayed I/O operation 0x6eaec0 (16 already in progress) [D 20:01:00.241274] DBPF I/O ops in progress: 16 [D 20:01:00.241321] ========================================================= [D 20:01:00.241327] PINT_process_request [D 20:01:00.241333] tiling 10 copies [D 20:01:00.241339] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.241346] lvl 0 el 0 blk 0 by 291504128 [D 20:01:00.241352] to 291504128 ta 268435456 fi 301989888 [D 20:01:00.241359] basic type or contiguous data [D 20:01:00.241365] Request disp [D 20:01:00.241370] contig_offset = 291504128 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.241377] calling distribute [D 20:01:00.241382] PINT_distribute [D 20:01:00.241388] of 291504128 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 291504128 exfl 1 [D 20:01:00.241396] begin iteration loff: 291504128 [D 20:01:00.241402] segment exceeds byte limit [D 20:01:00.241407] file being extended [D 20:01:00.241500] process a segment [D 20:01:00.241506] of 291504128 sz 2097152 [D 20:01:00.241512] add a segment 0 [D 20:01:00.241518] end iteration [D 20:01:00.241524] done with segments or bytes [D 20:01:00.241529] of 293601280 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.241536] next loff: 293601280 next poff: 293601280 [D 20:01:00.241542] return value 2097152 [D 20:01:00.241548] finished [D 20:01:00.241553] retval = 2097152 [D 20:01:00.241559] exiting distribute returned less than expected [D 20:01:00.241565] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 293601280 fo 301989888 eof 0 [D 20:01:00.241572] ========================================================= [D 20:01:00.241578] BMI_post_recv: addr: 6843072, offset: 0x2aaab2c0b010, size: 2097152, tag: 16 [D 20:01:00.241589] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:01:00.241657] get_data_sync_mode returning 0 [D 20:01:00.241673] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.241715] dbpf_bstream_rw_list: mem_offset: 0x2aaab10e0010, mem_size: 2097152 [D 20:01:00.241722] dbpf_bstream_rw_list: stream_offset: 322961408, stream_size: 2097152 [D 20:01:00.241728] dbpf_open_cache_get: called [D 20:01:00.241734] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.241740] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.241754] op_queue add: 0x701c30 [D 20:01:00.241760] delayed I/O operation 0x701c30 (16 already in progress) [D 20:01:00.241767] DBPF I/O ops in progress: 16 [D 20:01:00.241773] ========================================================= [D 20:01:00.241779] PINT_process_request [D 20:01:00.241785] tiling 11 copies [D 20:01:00.241791] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.241798] lvl 0 el 0 blk 0 by 325058560 [D 20:01:00.241895] to 325058560 ta 301989888 fi 335544320 [D 20:01:00.241902] basic type or contiguous data [D 20:01:00.241908] Request disp [D 20:01:00.241913] contig_offset = 325058560 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.241920] calling distribute [D 20:01:00.241926] PINT_distribute [D 20:01:00.241932] of 325058560 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 325058560 exfl 1 [D 20:01:00.241939] begin iteration loff: 325058560 [D 20:01:00.241945] segment exceeds byte limit [D 20:01:00.241950] file being extended [D 20:01:00.241956] process a segment [D 20:01:00.241962] of 325058560 sz 2097152 [D 20:01:00.241968] add a segment 0 [D 20:01:00.241974] end iteration [D 20:01:00.242044] done with segments or bytes [D 20:01:00.242050] of 327155712 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.242057] next loff: 327155712 next poff: 327155712 [D 20:01:00.242095] return value 2097152 [D 20:01:00.242101] finished [D 20:01:00.242107] retval = 2097152 [D 20:01:00.242113] exiting distribute returned less than expected [D 20:01:00.242118] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 327155712 fo 335544320 eof 0 [D 20:01:00.242126] ========================================================= [D 20:01:00.242132] BMI_post_recv: addr: 6843072, offset: 0x2aaab2e0c010, size: 2097152, tag: 17 [D 20:01:00.242143] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:01:00.242150] get_data_sync_mode returning 0 [D 20:01:00.242157] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.242164] dbpf_bstream_rw_list: mem_offset: 0x2aaab12e1010, mem_size: 2097152 [D 20:01:00.242170] dbpf_bstream_rw_list: stream_offset: 356515840, stream_size: 2097152 [D 20:01:00.242177] dbpf_open_cache_get: called [D 20:01:00.242237] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.242259] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.242272] op_queue add: 0x704790 [D 20:01:00.242278] delayed I/O operation 0x704790 (16 already in progress) [D 20:01:00.242285] DBPF I/O ops in progress: 16 [D 20:01:00.242291] ========================================================= [D 20:01:00.242297] PINT_process_request [D 20:01:00.242303] tiling 12 copies [D 20:01:00.242310] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.242317] lvl 0 el 0 blk 0 by 358612992 [D 20:01:00.242323] to 358612992 ta 335544320 fi 369098752 [D 20:01:00.242330] basic type or contiguous data [D 20:01:00.242336] Request disp [D 20:01:00.242420] contig_offset = 358612992 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.242427] calling distribute [D 20:01:00.242432] PINT_distribute [D 20:01:00.242455] of 358612992 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 358612992 exfl 1 [D 20:01:00.242462] begin iteration loff: 358612992 [D 20:01:00.242468] segment exceeds byte limit [D 20:01:00.242474] file being extended [D 20:01:00.242480] process a segment [D 20:01:00.242485] of 358612992 sz 2097152 [D 20:01:00.242491] add a segment 0 [D 20:01:00.242497] end iteration [D 20:01:00.242503] done with segments or bytes [D 20:01:00.242508] of 360710144 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.242515] next loff: 360710144 next poff: 360710144 [D 20:01:00.242521] return value 2097152 [D 20:01:00.242527] finished [D 20:01:00.180512] --- aio_progress_notification called with handle 2863311531 (0x6fef80) [D 20:01:00.184497] --- aio_progress_notification called with handle 2863311531 (0x6ff0d0) [D 20:01:00.192504] --- aio_progress_notification called with handle 2863311531 (0x2aaab1525bd0) [D 20:01:00.196517] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b470) [D 20:01:00.242596] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.242611] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.242618] dbpf_open_cache_put: cached entry. [D 20:01:00.242596] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.192505] --- aio_progress_notification called with handle 2863311531 (0x2aaab1522f80) [D 20:01:00.242631] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.196594] --- aio_progress_notification called with handle 2863311531 (0x2aaab1528820) [D 20:01:00.238506] ========================================================= [D 20:01:00.242647] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.200507] --- aio_progress_notification called with handle 2863311531 (0x6eac20) [D 20:01:00.242663] PINT_process_request [D 20:01:00.242660] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.242673] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.184491] --- aio_progress_notification called with handle 2863311531 (0x2aaab151ad60) [D 20:01:00.242605] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.188508] --- aio_progress_notification called with handle 2863311531 (0x2aaab1520420) [D 20:01:00.201627] --- aio_progress_notification called with handle 2863311531 (0x710ff0) [D 20:01:00.242562] retval = 2097152 [D 20:01:00.242639] dbpf_open_cache_put: cached entry. [D 20:01:00.242722] exiting distribute returned less than expected [D 20:01:00.242731] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.188587] --- aio_progress_notification called with handle 2863311531 (0x2aaab151d8c0) [D 20:01:00.242680] tiling 2 copies [D 20:01:00.242721] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.242735] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 360710144 fo 369098752 eof 0 [D 20:01:00.242762] ========================================================= [D 20:01:00.242769] BMI_post_recv: addr: 6843072, offset: 0x2aaab300e010, size: 2097152, tag: 18 [D 20:01:00.242780] dbpf_open_cache_put: cached entry. [D 20:01:00.177760] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.242784] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:01:00.238448] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.242755] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.242804] get_data_sync_mode returning 0 [D 20:01:00.242814] lvl 0 el 0 blk 0 by 0 [D 20:01:00.242828] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.242833] to 0 ta 0 fi 33554432 [D 20:01:00.242739] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.242855] basic type or contiguous data [D 20:01:00.242796] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.242788] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.242875] Request disp [D 20:01:00.242824] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.242649] op_queue add: 0x6fef80 [D 20:01:00.242912] DBPF I/O ops in progress: 15 [D 20:01:00.242902] dbpf_bstream_rw_list: mem_offset: 0x2aaab1600010, mem_size: 2097152 [D 20:01:00.242893] contig_offset = 0 contig_size = 67108864 lvl_flag = 1 [D 20:01:00.242931] dbpf_bstream_rw_list: stream_offset: 390070272, stream_size: 2097152 [D 20:01:00.242939] calling distribute [D 20:01:00.242924] starting delayed I/O operation 0x2aaab152e050 (15 in progress) [D 20:01:00.242915] op_queue add: 0x6ea920 [D 20:01:00.242837] dbpf_open_cache_put: cached entry. [D 20:01:00.242948] dbpf_open_cache_get: called [D 20:01:00.242988] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.242993] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.242962] PINT_distribute [D 20:01:00.243010] of 0 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.243018] begin iteration loff: 0 [D 20:01:00.243024] segment exceeds byte limit [D 20:01:00.243030] process a segment [D 20:01:00.243036] of 0 sz 2097152 [D 20:01:00.243042] add a segment 0 [D 20:01:00.243048] end iteration [D 20:01:00.243054] done with segments or bytes [D 20:01:00.243059] of 2097152 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.243066] next loff: 2097152 next poff: 2097152 [D 20:01:00.243072] return value 2097152 [D 20:01:00.243078] finished [D 20:01:00.243084] retval = 2097152 [D 20:01:00.243090] exiting distribute returned less than expected [D 20:01:00.243096] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 2097152 fo 33554432 eof 0 [D 20:01:00.243103] ========================================================= [D 20:01:00.243112] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.243119] dbpf_bstream_rw_list: mem_offset: 0x2aaabc65c010, mem_size: 2097152 [D 20:01:00.243126] dbpf_bstream_rw_list: stream_offset: 0, stream_size: 2097152 [D 20:01:00.243132] dbpf_open_cache_get: called [D 20:01:00.242971] lio_listio called with 1 following aiocbs: [D 20:01:00.243003] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.243151] aiocb_ptr_array[0]: fd: 13, off: 450887680, bytes: 2097152, buf: 0x2aaabc45b010, type: 0 [D 20:01:00.243173] start_delayed_ops_if_any: lio_listio posted 0x2aaab152e050 (handle 2863311531, ret 0)) [D 20:01:00.243185] DBPF I/O ops in progress: 15 [D 20:01:00.243197] starting delayed I/O operation 0x6f0f00 (15 in progress) [D 20:01:00.242997] op_queue add: 0x6fece0 [D 20:01:00.243204] lio_listio called with 1 following aiocbs: [D 20:01:00.243217] dbpf_open_cache_put: cached entry. [D 20:01:00.243219] aiocb_ptr_array[0]: fd: 13, off: 20971520, bytes: 2097152, buf: 0x2aaaafed7010, type: 1 [D 20:01:00.243241] start_delayed_ops_if_any: lio_listio posted 0x6f0f00 (handle 2863311531, ret 0)) [D 20:01:00.243251] op_queue add: 0x707430 [D 20:01:00.243258] delayed I/O operation 0x707430 (16 already in progress) [D 20:01:00.243265] DBPF I/O ops in progress: 16 [D 20:01:00.243272] ========================================================= [D 20:01:00.243279] PINT_process_request [D 20:01:00.243285] tiling 13 copies [D 20:01:00.243291] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.243299] lvl 0 el 0 blk 0 by 392167424 [D 20:01:00.243305] to 392167424 ta 369098752 fi 402653184 [D 20:01:00.243312] basic type or contiguous data [D 20:01:00.243318] Request disp [D 20:01:00.243324] contig_offset = 392167424 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.243330] calling distribute [D 20:01:00.243336] PINT_distribute [D 20:01:00.243342] of 392167424 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 392167424 exfl 1 [D 20:01:00.243349] begin iteration loff: 392167424 [D 20:01:00.243355] segment exceeds byte limit [D 20:01:00.243361] file being extended [D 20:01:00.243367] process a segment [D 20:01:00.243373] of 392167424 sz 2097152 [D 20:01:00.243379] add a segment 0 [D 20:01:00.243385] end iteration [D 20:01:00.243390] done with segments or bytes [D 20:01:00.243396] of 394264576 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.243403] next loff: 394264576 next poff: 394264576 [D 20:01:00.243409] return value 2097152 [D 20:01:00.243415] finished [D 20:01:00.243421] retval = 2097152 [D 20:01:00.243427] exiting distribute returned less than expected [D 20:01:00.243433] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 394264576 fo 402653184 eof 0 [D 20:01:00.243440] ========================================================= [D 20:01:00.243447] BMI_post_recv: addr: 6843072, offset: 0x2aaab320f010, size: 2097152, tag: 19 [D 20:01:00.243461] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:00.243471] get_data_sync_mode returning 0 [D 20:01:00.243478] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.243484] dbpf_bstream_rw_list: mem_offset: 0x2aaabb453010, mem_size: 2097152 [D 20:01:00.243491] dbpf_bstream_rw_list: stream_offset: 482344960, stream_size: 2097152 [D 20:01:00.243498] dbpf_open_cache_get: called [D 20:01:00.243504] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.243510] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.243531] op_queue add: 0x6fef80 [D 20:01:00.243538] delayed I/O operation 0x6fef80 (16 already in progress) [D 20:01:00.243544] DBPF I/O ops in progress: 16 [D 20:01:00.246311] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:00.246386] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:00.246396] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:00.246486] ========================================================= [D 20:01:00.246498] PINT_process_request [D 20:01:00.246507] tiling 16 copies [D 20:01:00.246514] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.246522] lvl 0 el 0 blk 0 by 484442112 [D 20:01:00.246529] to 484442112 ta 469762048 fi 503316480 [D 20:01:00.246541] basic type or contiguous data [D 20:01:00.246549] Request disp [D 20:01:00.246555] contig_offset = 484442112 contig_size = 52428800 lvl_flag = 1 [D 20:01:00.246563] calling distribute [D 20:01:00.246569] PINT_distribute [D 20:01:00.246576] of 484442112 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 484442112 exfl 1 [D 20:01:00.246584] begin iteration loff: 484442112 [D 20:01:00.246592] segment exceeds byte limit [D 20:01:00.246598] file being extended [D 20:01:00.246604] process a segment [D 20:01:00.246610] of 484442112 sz 2097152 [D 20:01:00.246616] add a segment 0 [D 20:01:00.246622] end iteration [D 20:01:00.246628] done with segments or bytes [D 20:01:00.246634] of 486539264 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.246642] next loff: 486539264 next poff: 486539264 [D 20:01:00.246649] return value 2097152 [D 20:01:00.246657] finished [D 20:01:00.246662] retval = 2097152 [D 20:01:00.246669] exiting distribute returned less than expected [D 20:01:00.246675] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 486539264 fo 503316480 eof 0 [D 20:01:00.246682] ========================================================= [D 20:01:00.246690] BMI_post_recv: addr: 6843072, offset: 0x2aaabc85d010, size: 2097152, tag: 22 [D 20:01:00.246734] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:00.246745] get_data_sync_mode returning 0 [D 20:01:00.246756] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.246764] dbpf_bstream_rw_list: mem_offset: 0x2aaab1801010, mem_size: 2097152 [D 20:01:00.246771] dbpf_bstream_rw_list: stream_offset: 423624704, stream_size: 2097152 [D 20:01:00.246779] dbpf_open_cache_get: called [D 20:01:00.246787] dbpf_open_cache_put: cached entry. [D 20:01:00.246800] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.246809] op_queue add: 0x2aaab1522f80 [D 20:01:00.246826] DBPF I/O ops in progress: 15 [D 20:01:00.246835] starting delayed I/O operation 0x6feb90 (15 in progress) [D 20:01:00.246845] lio_listio called with 1 following aiocbs: [D 20:01:00.246852] aiocb_ptr_array[0]: fd: 13, off: 54525952, bytes: 2097152, buf: 0x2aaab00d8010, type: 1 [D 20:01:00.246865] start_delayed_ops_if_any: lio_listio posted 0x6feb90 (handle 2863311531, ret 0)) [D 20:01:00.246885] dbpf_open_cache_put: cached entry. [D 20:01:00.246898] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.246905] op_queue add: 0x6fee30 [D 20:01:00.246913] DBPF I/O ops in progress: 15 [D 20:01:00.246920] starting delayed I/O operation 0x6ed430 (15 in progress) [D 20:01:00.246930] lio_listio called with 1 following aiocbs: [D 20:01:00.246936] aiocb_ptr_array[0]: fd: 13, off: 88080384, bytes: 2097152, buf: 0x2aaab02d9010, type: 1 [D 20:01:00.246945] start_delayed_ops_if_any: lio_listio posted 0x6ed430 (handle 2863311531, ret 0)) [D 20:01:00.248503] dbpf_open_cache_put: cached entry. [D 20:01:00.248544] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.248553] op_queue add: 0x2aaab1528820 [D 20:01:00.248561] DBPF I/O ops in progress: 15 [D 20:01:00.248568] starting delayed I/O operation 0x6eaa70 (15 in progress) [D 20:01:00.248579] lio_listio called with 1 following aiocbs: [D 20:01:00.248586] aiocb_ptr_array[0]: fd: 13, off: 121634816, bytes: 2097152, buf: 0x2aaab04da010, type: 1 [D 20:01:00.248598] start_delayed_ops_if_any: lio_listio posted 0x6eaa70 (handle 2863311531, ret 0)) [D 20:01:00.248631] op_queue add: 0x6ff0d0 [D 20:01:00.248649] op_queue add: 0x6f6810 [D 20:01:00.248674] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.248687] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:00.248697] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.248705] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:00.248715] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:00.248734] DBPF I/O ops in progress: 15 [D 20:01:00.248742] starting delayed I/O operation 0x6f0a90 (15 in progress) [D 20:01:00.248751] lio_listio called with 1 following aiocbs: [D 20:01:00.248758] aiocb_ptr_array[0]: fd: 13, off: 155189248, bytes: 2097152, buf: 0x2aaab06db010, type: 1 [D 20:01:00.248769] start_delayed_ops_if_any: lio_listio posted 0x6f0a90 (handle 2863311531, ret 0)) [D 20:01:00.248820] DBPF I/O ops in progress: 15 [D 20:01:00.248833] starting delayed I/O operation 0x6fea40 (15 in progress) [D 20:01:00.248843] lio_listio called with 1 following aiocbs: [D 20:01:00.248850] aiocb_ptr_array[0]: fd: 13, off: 188743680, bytes: 2097152, buf: 0x2aaab08dc010, type: 1 [D 20:01:00.248861] start_delayed_ops_if_any: lio_listio posted 0x6fea40 (handle 2863311531, ret 0)) [D 20:01:00.248899] --- aio_progress_notification called with handle 2863311531 (0x2aaab152e050) [D 20:01:00.248913] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.248925] dbpf_open_cache_put: cached entry. [D 20:01:00.248932] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.248939] op_queue add: 0x2aaab152e050 [D 20:01:00.248956] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.248969] DBPF I/O ops in progress: 15 [D 20:01:00.248977] starting delayed I/O operation 0x6ea1d0 (15 in progress) [D 20:01:00.248987] lio_listio called with 1 following aiocbs: [D 20:01:00.248993] aiocb_ptr_array[0]: fd: 13, off: 222298112, bytes: 2097152, buf: 0x2aaab0add010, type: 1 [D 20:01:00.249002] start_delayed_ops_if_any: lio_listio posted 0x6ea1d0 (handle 2863311531, ret 0)) [D 20:01:00.249038] dbpf_open_cache_put: cached entry. [D 20:01:00.249052] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.249059] op_queue add: 0x6eac20 [D 20:01:00.249077] DBPF I/O ops in progress: 15 [D 20:01:00.249085] starting delayed I/O operation 0x6ead70 (15 in progress) [D 20:01:00.249094] lio_listio called with 1 following aiocbs: [D 20:01:00.249100] aiocb_ptr_array[0]: fd: 13, off: 255852544, bytes: 2097152, buf: 0x2aaab0cde010, type: 1 [D 20:01:00.249110] start_delayed_ops_if_any: lio_listio posted 0x6ead70 (handle 2863311531, ret 0)) [D 20:01:00.249144] dbpf_open_cache_put: cached entry. [D 20:01:00.249156] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.249164] op_queue add: 0x2aaab1525bd0 [D 20:01:00.249171] DBPF I/O ops in progress: 15 [D 20:01:00.249178] starting delayed I/O operation 0x6eaec0 (15 in progress) [D 20:01:00.249188] lio_listio called with 1 following aiocbs: [D 20:01:00.249195] aiocb_ptr_array[0]: fd: 13, off: 289406976, bytes: 2097152, buf: 0x2aaab0edf010, type: 1 [D 20:01:00.249203] start_delayed_ops_if_any: lio_listio posted 0x6eaec0 (handle 2863311531, ret 0)) [D 20:01:00.249231] dbpf_open_cache_put: cached entry. [D 20:01:00.249244] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.249251] op_queue add: 0x2aaab151ad60 [D 20:01:00.249259] DBPF I/O ops in progress: 15 [D 20:01:00.249266] starting delayed I/O operation 0x701c30 (15 in progress) [D 20:01:00.249275] lio_listio called with 1 following aiocbs: [D 20:01:00.249282] aiocb_ptr_array[0]: fd: 13, off: 322961408, bytes: 2097152, buf: 0x2aaab10e0010, type: 1 [D 20:01:00.249290] start_delayed_ops_if_any: lio_listio posted 0x701c30 (handle 2863311531, ret 0)) [D 20:01:00.249323] dbpf_open_cache_put: cached entry. [D 20:01:00.249337] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.249345] op_queue add: 0x2aaab1520420 [D 20:01:00.249353] DBPF I/O ops in progress: 15 [D 20:01:00.249360] starting delayed I/O operation 0x704790 (15 in progress) [D 20:01:00.249369] lio_listio called with 1 following aiocbs: [D 20:01:00.249375] aiocb_ptr_array[0]: fd: 13, off: 356515840, bytes: 2097152, buf: 0x2aaab12e1010, type: 1 [D 20:01:00.249384] start_delayed_ops_if_any: lio_listio posted 0x704790 (handle 2863311531, ret 0)) [D 20:01:00.249414] dbpf_open_cache_put: cached entry. [D 20:01:00.249428] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.249436] op_queue add: 0x2aaab151d8c0 [D 20:01:00.249443] DBPF I/O ops in progress: 15 [D 20:01:00.249450] starting delayed I/O operation 0x707430 (15 in progress) [D 20:01:00.249460] lio_listio called with 1 following aiocbs: [D 20:01:00.249466] aiocb_ptr_array[0]: fd: 13, off: 390070272, bytes: 2097152, buf: 0x2aaab1600010, type: 1 [D 20:01:00.249475] start_delayed_ops_if_any: lio_listio posted 0x707430 (handle 2863311531, ret 0)) [D 20:01:00.249505] dbpf_open_cache_put: cached entry. [D 20:01:00.249523] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.249533] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.249552] op_queue add: 0x2aaab15178c0 [D 20:01:00.249560] delayed I/O operation 0x2aaab15178c0 (16 already in progress) [D 20:01:00.249567] DBPF I/O ops in progress: 16 [D 20:01:00.249575] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.249582] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517ef0. [D 20:01:00.249590] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.251660] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.251726] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.251758] op_queue add: 0x6ea920 [D 20:01:00.251767] delayed I/O operation 0x6ea920 (16 already in progress) [D 20:01:00.251774] DBPF I/O ops in progress: 16 [D 20:01:00.251787] ========================================================= [D 20:01:00.251794] PINT_process_request [D 20:01:00.251802] tiling 14 copies [D 20:01:00.251810] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.251818] lvl 0 el 0 blk 0 by 425721856 [D 20:01:00.251825] to 425721856 ta 402653184 fi 436207616 [D 20:01:00.251833] basic type or contiguous data [D 20:01:00.251839] Request disp [D 20:01:00.251846] contig_offset = 425721856 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.251853] calling distribute [D 20:01:00.251859] PINT_distribute [D 20:01:00.251865] of 425721856 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 425721856 exfl 1 [D 20:01:00.251875] begin iteration loff: 425721856 [D 20:01:00.251883] segment exceeds byte limit [D 20:01:00.251888] file being extended [D 20:01:00.251895] process a segment [D 20:01:00.251901] of 425721856 sz 2097152 [D 20:01:00.251908] add a segment 0 [D 20:01:00.251915] end iteration [D 20:01:00.251921] done with segments or bytes [D 20:01:00.251926] of 427819008 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.251934] next loff: 427819008 next poff: 427819008 [D 20:01:00.251940] return value 2097152 [D 20:01:00.251947] finished [D 20:01:00.251953] retval = 2097152 [D 20:01:00.251959] exiting distribute returned less than expected [D 20:01:00.251966] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 427819008 fo 436207616 eof 0 [D 20:01:00.251973] ========================================================= [D 20:01:00.251981] BMI_post_recv: addr: 6843072, offset: 0x2aaab3410010, size: 2097152, tag: 20 [D 20:01:00.252017] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.252037] op_queue add: 0x2aaab152b470 [D 20:01:00.252047] DBPF I/O ops in progress: 15 [D 20:01:00.258272] starting delayed I/O operation 0x6fef80 (15 in progress) [D 20:01:00.258315] lio_listio called with 1 following aiocbs: [D 20:01:00.258322] aiocb_ptr_array[0]: fd: 13, off: 482344960, bytes: 2097152, buf: 0x2aaabb453010, type: 1 [D 20:01:00.258338] start_delayed_ops_if_any: lio_listio posted 0x6fef80 (handle 2863311531, ret 0)) [D 20:01:00.260509] --- aio_progress_notification called with handle 2863311531 (0x6eaa70) [D 20:01:00.260559] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.260572] dbpf_open_cache_put: cached entry. [D 20:01:00.260583] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.260599] DBPF I/O ops in progress: 15 [D 20:01:00.260609] starting delayed I/O operation 0x2aaab15178c0 (15 in progress) [D 20:01:00.260620] lio_listio called with 1 following aiocbs: [D 20:01:00.260627] aiocb_ptr_array[0]: fd: 13, off: 0, bytes: 2097152, buf: 0x2aaabc65c010, type: 0 [D 20:01:00.260640] start_delayed_ops_if_any: lio_listio posted 0x2aaab15178c0 (handle 2863311531, ret 0)) -[D 20:01:00.261230] openib_check_cq: completed recv wr_id 684620. [D 20:01:00.261250] openib_post_rr: 10.1.4.56:39990 bh 57 wr_id 684620. [D 20:01:00.261263] post_sr: 10.1.4.56:39990 bh 97 len 48 wr_id 683ad0. [D 20:01:00.261286] openib_check_cq: completed send wr_id 683ad0. [D 20:01:00.262361] openib_check_cq: completed recv wr_id 684650. [D 20:01:00.262375] openib_post_rr: 10.1.4.56:39990 bh 58 wr_id 684650. [D 20:01:00.262386] post_sr: 10.1.4.56:39990 bh 98 len 48 wr_id 683b00. [D 20:01:00.262402] openib_check_cq: completed send wr_id 683b00. [D 20:01:00.263531] openib_check_cq: completed recv wr_id 684680. [D 20:01:00.263548] openib_post_rr: 10.1.4.56:39990 bh 59 wr_id 684680. [D 20:01:00.263558] post_sr: 10.1.4.56:39990 bh 99 len 48 wr_id 683b30. [D 20:01:00.263573] openib_check_cq: completed send wr_id 683b30. [D 20:01:00.264719] openib_check_cq: completed recv wr_id 6846b0. [D 20:01:00.264747] openib_post_rr: 10.1.4.56:39990 bh 60 wr_id 6846b0. [D 20:01:00.264757] post_sr: 10.1.4.56:39990 bh 0 len 48 wr_id 6828a0. [D 20:01:00.264774] openib_check_cq: completed send wr_id 6828a0. [D 20:01:00.265884] openib_check_cq: completed recv wr_id 6846e0. [D 20:01:00.265902] openib_post_rr: 10.1.4.56:39990 bh 61 wr_id 6846e0. [D 20:01:00.265912] post_sr: 10.1.4.56:39990 bh 1 len 48 wr_id 6828d0. [D 20:01:00.265928] openib_check_cq: completed send wr_id 6828d0. [D 20:01:00.256497] op_queue add: 0x710ff0 [D 20:01:00.268551] DBPF I/O ops in progress: 15 [D 20:01:00.268561] starting delayed I/O operation 0x6ea920 (15 in progress) [D 20:01:00.268569] lio_listio called with 1 following aiocbs: [D 20:01:00.268576] aiocb_ptr_array[0]: fd: 13, off: 423624704, bytes: 2097152, buf: 0x2aaab1801010, type: 1 [D 20:01:00.268587] start_delayed_ops_if_any: lio_listio posted 0x6ea920 (handle 2863311531, ret 0)) [D 20:01:00.268619] --- aio_progress_notification called with handle 2863311531 (0x6f0a90) [D 20:01:00.268631] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.268641] dbpf_open_cache_put: cached entry. [D 20:01:00.268648] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.268655] op_queue add: 0x6f0a90 [D 20:01:00.268662] DBPF I/O ops in progress: 15 [D 20:01:00.268677] --- aio_progress_notification called with handle 2863311531 (0x6fea40) [D 20:01:00.268687] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.268696] dbpf_open_cache_put: cached entry. [D 20:01:00.268703] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.268709] op_queue add: 0x6fea40 [D 20:01:00.268716] DBPF I/O ops in progress: 14 [D 20:01:00.243230] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.268740] op_queue add: 0x6ec210 [D 20:01:00.268748] DBPF I/O ops in progress: 13 [D 20:01:00.256514] --- aio_progress_notification called with handle 2863311531 (0x6f0f00) [D 20:01:00.268773] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.268783] dbpf_open_cache_put: cached entry. [D 20:01:00.268789] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.268795] op_queue add: 0x6f0f00 [D 20:01:00.268802] DBPF I/O ops in progress: 12 [D 20:01:00.256519] --- aio_progress_notification called with handle 2863311531 (0x6feb90) [D 20:01:00.268849] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.268859] dbpf_open_cache_put: cached entry. [D 20:01:00.268867] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.268874] op_queue add: 0x6feb90 [D 20:01:00.268881] DBPF I/O ops in progress: 11 [D 20:01:00.256522] --- aio_progress_notification called with handle 2863311531 (0x6ed430) [D 20:01:00.268916] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.268925] dbpf_open_cache_put: cached entry. [D 20:01:00.268932] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.268938] op_queue add: 0x6ed430 [D 20:01:00.268945] DBPF I/O ops in progress: 10 [D 20:01:00.258229] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:00.272207] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.272219] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.272226] BMI_post_send: addr: 6843072, offset: 0x2aaabb654010, size: 2097152, tag: 36 [D 20:01:00.272250] post_sr: 10.1.4.56:39990 bh 2 len 32 wr_id 682900. [D 20:01:00.272289] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:00.272300] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.272307] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.272315] BMI_post_send: addr: 6843072, offset: 0x2aaabb855010, size: 2097152, tag: 36 [D 20:01:00.272324] post_sr: 10.1.4.56:39990 bh 3 len 32 wr_id 682930. [D 20:01:00.272334] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:00.272342] BMI_post_send: addr: 6843072, offset: 0x2aaabba56010, size: 2097152, tag: 36 [D 20:01:00.272350] post_sr: 10.1.4.56:39990 bh 4 len 32 wr_id 682960. [D 20:01:00.272361] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:00.272368] BMI_post_send: addr: 6843072, offset: 0x2aaabbc57010, size: 2097152, tag: 36 [D 20:01:00.272376] post_sr: 10.1.4.56:39990 bh 5 len 32 wr_id 682990. [D 20:01:00.272386] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:00.272393] BMI_post_send: addr: 6843072, offset: 0x2aaabbe58010, size: 2097152, tag: 36 [D 20:01:00.272401] post_sr: 10.1.4.56:39990 bh 6 len 32 wr_id 6829c0. [D 20:01:00.272411] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:00.272418] BMI_post_send: addr: 6843072, offset: 0x2aaabc059010, size: 2097152, tag: 36 [D 20:01:00.272427] post_sr: 10.1.4.56:39990 bh 7 len 32 wr_id 6829f0. [D 20:01:00.272436] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:00.272444] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.272451] BMI_post_send: addr: 6843072, offset: 0x2aaabc25a010, size: 2097152, tag: 36 [D 20:01:00.272462] post_sr: 10.1.4.56:39990 bh 8 len 32 wr_id 682a20. [D 20:01:00.272471] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:00.272491] BMI_post_send: addr: 6843072, offset: 0x2aaabc45b010, size: 2097152, tag: 36 [D 20:01:00.272501] post_sr: 10.1.4.56:39990 bh 9 len 32 wr_id 682a50. [D 20:01:00.272511] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:00.272528] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:00.272538] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:01:00.272546] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:00.272554] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:00.272563] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:01:00.272576] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:00.272584] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:00.258240] ========================================================= [D 20:01:00.272616] PINT_process_request [D 20:01:00.272624] tiling 2 copies [D 20:01:00.272631] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.272639] lvl 0 el 0 blk 0 by 2097152 [D 20:01:00.272646] to 2097152 ta 0 fi 33554432 [D 20:01:00.272653] basic type or contiguous data [D 20:01:00.272661] Request disp [D 20:01:00.272667] contig_offset = 2097152 contig_size = 65011712 lvl_flag = 1 [D 20:01:00.272675] calling distribute [D 20:01:00.272681] PINT_distribute [D 20:01:00.272688] of 2097152 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.272697] begin iteration loff: 2097152 [D 20:01:00.272705] segment exceeds byte limit [D 20:01:00.272712] process a segment [D 20:01:00.272718] of 2097152 sz 2097152 [D 20:01:00.272726] add a segment 0 [D 20:01:00.272732] end iteration [D 20:01:00.272738] done with segments or bytes [D 20:01:00.272745] of 4194304 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.272752] next loff: 4194304 next poff: 4194304 [D 20:01:00.272759] return value 2097152 [D 20:01:00.272766] finished [D 20:01:00.272772] retval = 2097152 [D 20:01:00.272779] exiting distribute returned less than expected [D 20:01:00.272785] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 4194304 fo 33554432 eof 0 [D 20:01:00.272792] ========================================================= [D 20:01:00.272807] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.272816] dbpf_bstream_rw_list: mem_offset: 0x2aaabca5e010, mem_size: 2097152 [D 20:01:00.272822] dbpf_bstream_rw_list: stream_offset: 2097152, stream_size: 2097152 [D 20:01:00.272830] dbpf_open_cache_get: called [D 20:01:00.272838] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.272845] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.272870] DBPF I/O ops in progress: 11 [D 20:01:00.272878] lio_listio called with the following aiocbs: [D 20:01:00.272885] aiocb_ptr_array[0]: fd: 13, off: 2097152, bytes: 2097152, buf: 0x2aaabca5e010, type: 0 [D 20:01:00.272898] issue_or_delay_io_operation: lio_listio posted 0x2aaab1530a60 (handle 2863311531, ret 0) [D 20:01:00.272906] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.272914] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517ef0. [D 20:01:00.272922] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no -[D 20:01:00.275704] openib_check_cq: completed send wr_id 682900. [D 20:01:00.275775] openib_check_cq: completed send wr_id 682930. [D 20:01:00.275784] openib_check_cq: completed send wr_id 682960. [D 20:01:00.275791] openib_check_cq: completed send wr_id 682990. [D 20:01:00.275798] openib_check_cq: completed send wr_id 6829c0. [D 20:01:00.275805] openib_check_cq: completed send wr_id 6829f0. [D 20:01:00.275812] openib_check_cq: completed send wr_id 682a20. [D 20:01:00.275819] openib_check_cq: completed send wr_id 682a50. [D 20:01:00.275821] ========================================================= [D 20:01:00.275844] PINT_process_request [D 20:01:00.275851] tiling 2 copies [D 20:01:00.275858] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.275866] lvl 0 el 0 blk 0 by 4194304 [D 20:01:00.275872] to 4194304 ta 0 fi 33554432 [D 20:01:00.275879] basic type or contiguous data [D 20:01:00.275886] Request disp [D 20:01:00.275892] contig_offset = 4194304 contig_size = 62914560 lvl_flag = 1 [D 20:01:00.275899] calling distribute [D 20:01:00.275905] PINT_distribute [D 20:01:00.275911] of 4194304 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.275919] begin iteration loff: 4194304 [D 20:01:00.275926] segment exceeds byte limit [D 20:01:00.275932] process a segment [D 20:01:00.275938] of 4194304 sz 2097152 [D 20:01:00.275944] add a segment 0 [D 20:01:00.275950] end iteration [D 20:01:00.275956] done with segments or bytes [D 20:01:00.275962] of 6291456 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.275970] next loff: 6291456 next poff: 6291456 [D 20:01:00.275977] return value 2097152 [D 20:01:00.275983] finished [D 20:01:00.275989] retval = 2097152 [D 20:01:00.275994] exiting distribute returned less than expected [D 20:01:00.276000] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 6291456 fo 33554432 eof 0 [D 20:01:00.276008] ========================================================= [D 20:01:00.276020] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.276028] dbpf_bstream_rw_list: mem_offset: 0x2aaabcc5f010, mem_size: 2097152 [D 20:01:00.276035] dbpf_bstream_rw_list: stream_offset: 4194304, stream_size: 2097152 [D 20:01:00.276041] dbpf_open_cache_get: called [D 20:01:00.276048] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.276055] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.276074] DBPF I/O ops in progress: 12 [D 20:01:00.276081] lio_listio called with the following aiocbs: [D 20:01:00.276088] aiocb_ptr_array[0]: fd: 13, off: 4194304, bytes: 2097152, buf: 0x2aaabcc5f010, type: 0 [D 20:01:00.276100] issue_or_delay_io_operation: lio_listio posted 0x2aaab152de80 (handle 2863311531, ret 0) [D 20:01:00.276108] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.276115] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517ef0. [D 20:01:00.276123] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.277858] ========================================================= [D 20:01:00.277893] PINT_process_request [D 20:01:00.277900] tiling 2 copies [D 20:01:00.277906] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.277914] lvl 0 el 0 blk 0 by 6291456 [D 20:01:00.277920] to 6291456 ta 0 fi 33554432 [D 20:01:00.277927] basic type or contiguous data [D 20:01:00.277933] Request disp [D 20:01:00.277939] contig_offset = 6291456 contig_size = 60817408 lvl_flag = 1 [D 20:01:00.277946] calling distribute [D 20:01:00.277951] PINT_distribute [D 20:01:00.277957] of 6291456 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.277965] begin iteration loff: 6291456 [D 20:01:00.277972] segment exceeds byte limit [D 20:01:00.277978] process a segment [D 20:01:00.277984] of 6291456 sz 2097152 [D 20:01:00.277990] add a segment 0 [D 20:01:00.277996] end iteration [D 20:01:00.278002] done with segments or bytes [D 20:01:00.278008] of 8388608 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.278015] next loff: 8388608 next poff: 8388608 [D 20:01:00.278021] return value 2097152 [D 20:01:00.278028] finished [D 20:01:00.278033] retval = 2097152 [D 20:01:00.278039] exiting distribute returned less than expected [D 20:01:00.278045] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 8388608 fo 33554432 eof 0 [D 20:01:00.278052] ========================================================= [D 20:01:00.278063] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.278069] dbpf_bstream_rw_list: mem_offset: 0x2aaabce60010, mem_size: 2097152 [D 20:01:00.278076] dbpf_bstream_rw_list: stream_offset: 6291456, stream_size: 2097152 [D 20:01:00.278083] dbpf_open_cache_get: called [D 20:01:00.278090] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.278096] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.278115] DBPF I/O ops in progress: 13 [D 20:01:00.278122] lio_listio called with the following aiocbs: [D 20:01:00.278128] aiocb_ptr_array[0]: fd: 13, off: 6291456, bytes: 2097152, buf: 0x2aaabce60010, type: 0 [D 20:01:00.278139] issue_or_delay_io_operation: lio_listio posted 0x2aaab15285e0 (handle 2863311531, ret 0) [D 20:01:00.278147] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.278153] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517ef0. [D 20:01:00.278161] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.278232] openib_check_cq: completed recv wr_id 684710. [D 20:01:00.278251] openib_post_rr: 10.1.4.56:39990 bh 62 wr_id 684710. [D 20:01:00.278265] post_sr: 10.1.4.56:39990 bh 10 len 48 wr_id 682a80. [D 20:01:00.278276] openib_check_cq: completed recv wr_id 684740. [D 20:01:00.278284] openib_post_rr: 10.1.4.56:39990 bh 63 wr_id 684740. [D 20:01:00.278291] post_sr: 10.1.4.56:39990 bh 11 len 48 wr_id 682ab0. [D 20:01:00.278299] openib_check_cq: completed recv wr_id 684770. [D 20:01:00.278306] openib_post_rr: 10.1.4.56:39990 bh 64 wr_id 684770. [D 20:01:00.278315] post_sr: 10.1.4.56:39990 bh 12 len 48 wr_id 682ae0. [D 20:01:00.278322] openib_check_cq: completed recv wr_id 6847a0. [D 20:01:00.278330] openib_post_rr: 10.1.4.56:39990 bh 65 wr_id 6847a0. [D 20:01:00.278338] post_sr: 10.1.4.56:39990 bh 13 len 48 wr_id 682b10. [D 20:01:00.278346] openib_check_cq: completed recv wr_id 6847d0. [D 20:01:00.278353] openib_post_rr: 10.1.4.56:39990 bh 66 wr_id 6847d0. [D 20:01:00.278360] post_sr: 10.1.4.56:39990 bh 14 len 48 wr_id 682b40. [D 20:01:00.278368] openib_check_cq: completed recv wr_id 684800. [D 20:01:00.278406] op_queue add: 0x6eaa70 [D 20:01:00.279704] --- aio_progress_notification called with handle 2863311531 (0x6ea1d0) [D 20:01:00.279775] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.279799] dbpf_open_cache_put: cached entry. [D 20:01:00.279809] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.279837] DBPF I/O ops in progress: 12 [D 20:01:00.279887] ========================================================= [D 20:01:00.279891] openib_post_rr: 10.1.4.56:39990 bh 67 wr_id 684800. [D 20:01:00.279904] PINT_process_request [D 20:01:00.279924] openib_check_cq: completed recv wr_id 684830. [D 20:01:00.279946] openib_post_rr: 10.1.4.56:39990 bh 68 wr_id 684830. [D 20:01:00.279958] openib_check_cq: completed recv wr_id 684860. [D 20:01:00.279934] tiling 2 copies [D 20:01:00.279967] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea630 to 10.1.4.56:39990 remote addr 88437000 rkey 108a27 nic_wr_credit 1. [D 20:01:00.279977] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.279990] openib_post_rr: 10.1.4.56:39990 bh 69 wr_id 684860. [D 20:01:00.280006] lvl 0 el 0 blk 0 by 8388608 [D 20:01:00.280013] openib_check_cq: completed recv wr_id 684890. [D 20:01:00.280030] openib_post_rr: 10.1.4.56:39990 bh 70 wr_id 684890. [D 20:01:00.279895] op_queue add: 0x6ea1d0 [D 20:01:00.280017] to 8388608 ta 0 fi 33554432 [D 20:01:00.280071] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:01:00.280092] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:00.280075] basic type or contiguous data [D 20:01:00.280044] post_sr: 10.1.4.56:39990 bh 15 len 48 wr_id 682b70. [D 20:01:00.280127] openib_check_cq: completed recv wr_id 6848c0. [D 20:01:00.280139] openib_post_rr: 10.1.4.56:39990 bh 71 wr_id 6848c0. [D 20:01:00.280151] post_sr: 10.1.4.56:39990 bh 16 len 48 wr_id 682ba0. [D 20:01:00.280162] openib_check_cq: completed recv wr_id 6848f0. [D 20:01:00.280173] openib_post_rr: 10.1.4.56:39990 bh 72 wr_id 6848f0. [D 20:01:00.280184] post_sr: 10.1.4.56:39990 bh 17 len 48 wr_id 682bd0. [D 20:01:00.280116] Request disp [D 20:01:00.280193] openib_check_cq: completed recv wr_id 684920. [D 20:01:00.280213] openib_post_rr: 10.1.4.56:39990 bh 73 wr_id 684920. [D 20:01:00.280223] post_sr: 10.1.4.56:39990 bh 18 len 48 wr_id 682c00. [D 20:01:00.280201] contig_offset = 8388608 contig_size = 58720256 lvl_flag = 1 [D 20:01:00.280231] openib_check_cq: completed send wr_id 682a80. [D 20:01:00.280252] openib_check_cq: completed send wr_id 682ab0. [D 20:01:00.280262] openib_check_cq: completed send wr_id 682ae0. [D 20:01:00.280273] openib_check_cq: completed send wr_id 682b10. [D 20:01:00.280241] calling distribute [D 20:01:00.280280] openib_check_cq: completed send wr_id 682b40. [D 20:01:00.280301] openib_check_cq: completed recv wr_id 684950. [D 20:01:00.336809] openib_post_rr: 10.1.4.56:39990 bh 74 wr_id 684950. [D 20:01:00.336841] post_sr: 10.1.4.56:39990 bh 19 len 48 wr_id 682c30. [D 20:01:00.336853] openib_check_cq: completed recv wr_id 684980. [D 20:01:00.336861] openib_post_rr: 10.1.4.56:39990 bh 75 wr_id 684980. [D 20:01:00.336869] openib_check_cq: completed recv wr_id 6849b0. [D 20:01:00.336876] openib_post_rr: 10.1.4.56:39990 bh 76 wr_id 6849b0. [D 20:01:00.336883] openib_check_cq: completed recv wr_id 6849e0. [D 20:01:00.336890] openib_post_rr: 10.1.4.56:39990 bh 77 wr_id 6849e0. [D 20:01:00.336898] openib_check_cq: completed rdma wr_id 6ea630, nic_wr_credit 0. [D 20:01:00.336906] post_sr: 10.1.4.56:39990 bh 20 len 16 wr_id 682c60. [D 20:01:00.336913] openib_check_cq: completed send wr_id 682b70. [D 20:01:00.336920] openib_check_cq: completed send wr_id 682ba0. [D 20:01:00.336928] openib_check_cq: completed send wr_id 682bd0. [D 20:01:00.337028] openib_check_cq: completed send wr_id 682c00. [D 20:01:00.337035] openib_check_cq: completed recv wr_id 684a10. [D 20:01:00.337043] openib_post_rr: 10.1.4.56:39990 bh 78 wr_id 684a10. [D 20:01:00.337051] openib_check_cq: completed recv wr_id 684a40. [D 20:01:00.337058] openib_post_rr: 10.1.4.56:39990 bh 79 wr_id 684a40. [D 20:01:00.337065] openib_check_cq: completed recv wr_id 684a70. [D 20:01:00.337073] openib_post_rr: 10.1.4.56:39990 bh 80 wr_id 684a70. [D 20:01:00.337080] openib_check_cq: completed recv wr_id 684aa0. [D 20:01:00.337088] openib_post_rr: 10.1.4.56:39990 bh 81 wr_id 684aa0. [D 20:01:00.337095] openib_check_cq: completed recv wr_id 684ad0. [D 20:01:00.337102] openib_post_rr: 10.1.4.56:39990 bh 82 wr_id 684ad0. [D 20:01:00.337109] openib_check_cq: completed recv wr_id 684b00. [D 20:01:00.337117] openib_post_rr: 10.1.4.56:39990 bh 83 wr_id 684b00. [D 20:01:00.337198] openib_check_cq: completed recv wr_id 684b30. [D 20:01:00.280290] PINT_distribute [D 20:01:00.280505] --- aio_progress_notification called with handle 2863311531 (0x6ead70) [D 20:01:00.284496] --- aio_progress_notification called with handle 2863311531 (0x6eaec0) [D 20:01:00.337259] of 8388608 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.337274] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.337289] dbpf_open_cache_put: cached entry. [D 20:01:00.337297] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.337303] op_queue add: 0x6eaec0 [D 20:01:00.337321] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:00.337267] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.337348] dbpf_open_cache_put: cached entry. [D 20:01:00.337355] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.337361] op_queue add: 0x6ead70 [D 20:01:00.337378] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:00.288494] --- aio_progress_notification called with handle 2863311531 (0x704790) [D 20:01:00.337278] begin iteration loff: 8388608 [D 20:01:00.292493] --- aio_progress_notification called with handle 2863311531 (0x707430) [D 20:01:00.337497] segment exceeds byte limit [D 20:01:00.288504] --- aio_progress_notification called with handle 2863311531 (0x701c30) [D 20:01:00.337512] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.337524] dbpf_open_cache_put: cached entry. [D 20:01:00.337530] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.337536] op_queue add: 0x701c30 [D 20:01:00.337551] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:00.296628] --- aio_progress_notification called with handle 2863311531 (0x6fef80) [D 20:01:00.337568] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.337584] dbpf_open_cache_put: cached entry. [D 20:01:00.337591] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.337680] op_queue add: 0x6fef80 [D 20:01:00.337695] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:00.337498] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.337728] dbpf_open_cache_put: cached entry. [D 20:01:00.337735] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.337741] op_queue add: 0x704790 [D 20:01:00.337755] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:00.300536] --- aio_progress_notification called with handle 2863311531 (0x2aaab15178c0) [D 20:01:00.337773] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.337788] dbpf_open_cache_put: cached entry. [D 20:01:00.337795] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.337907] op_queue add: 0x2aaab15178c0 [D 20:01:00.304514] --- aio_progress_notification called with handle 2863311531 (0x2aaab1530a60) [D 20:01:00.337930] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.337944] dbpf_open_cache_put: cached entry. [D 20:01:00.337951] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.337957] op_queue add: 0x2aaab1530a60 [D 20:01:00.280110] DBPF I/O ops in progress: 11 [D 20:01:00.337219] openib_post_rr: 10.1.4.56:39990 bh 84 wr_id 684b30. [D 20:01:00.337971] openib_check_cq: completed recv wr_id 684b60. [D 20:01:00.337979] openib_post_rr: 10.1.4.56:39990 bh 85 wr_id 684b60. [D 20:01:00.337987] openib_check_cq: completed recv wr_id 684b90. [D 20:01:00.338043] openib_post_rr: 10.1.4.56:39990 bh 86 wr_id 684b90. [D 20:01:00.338058] openib_check_cq: completed recv wr_id 684bc0. [D 20:01:00.338076] process a segment [D 20:01:00.338098] of 8388608 sz 2097152 [D 20:01:00.338077] openib_post_rr: 10.1.4.56:39990 bh 87 wr_id 684bc0. [D 20:01:00.338120] openib_check_cq: completed recv wr_id 684bf0. [D 20:01:00.338128] openib_post_rr: 10.1.4.56:39990 bh 88 wr_id 684bf0. [D 20:01:00.338135] openib_check_cq: completed recv wr_id 684c20. [D 20:01:00.338142] openib_post_rr: 10.1.4.56:39990 bh 89 wr_id 684c20. [D 20:01:00.338149] openib_check_cq: completed recv wr_id 684c50. [D 20:01:00.338156] openib_post_rr: 10.1.4.56:39990 bh 90 wr_id 684c50. [D 20:01:00.338163] openib_check_cq: completed recv wr_id 684c80. [D 20:01:00.338170] openib_post_rr: 10.1.4.56:39990 bh 91 wr_id 684c80. [D 20:01:00.338177] openib_check_cq: completed send wr_id 682c30. [D 20:01:00.338184] openib_check_cq: completed send wr_id 682c60. [D 20:01:00.338108] add a segment 0 [D 20:01:00.338201] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:00.338292] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:00.338315] ========================================================= [D 20:01:00.338322] PINT_process_request [D 20:01:00.338328] tiling 15 copies [D 20:01:00.338335] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.338342] lvl 0 el 0 blk 0 by 452984832 [D 20:01:00.338348] to 452984832 ta 436207616 fi 469762048 [D 20:01:00.338355] basic type or contiguous data [D 20:01:00.338361] Request disp [D 20:01:00.338367] contig_offset = 452984832 contig_size = 50331648 lvl_flag = 1 [D 20:01:00.338374] calling distribute [D 20:01:00.338380] PINT_distribute [D 20:01:00.338386] of 452984832 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.338103] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.338484] dbpf_open_cache_put: cached entry. [D 20:01:00.338393] begin iteration loff: 452984832 [D 20:01:00.338514] segment exceeds byte limit [D 20:01:00.338520] process a segment [D 20:01:00.338525] of 452984832 sz 2097152 [D 20:01:00.338531] add a segment 0 [D 20:01:00.338537] end iteration [D 20:01:00.338543] done with segments or bytes [D 20:01:00.338549] of 455081984 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.338556] next loff: 455081984 next poff: 455081984 [D 20:01:00.338562] return value 2097152 [D 20:01:00.338568] finished [D 20:01:00.338573] retval = 2097152 [D 20:01:00.338579] exiting distribute returned less than expected [D 20:01:00.338587] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 455081984 fo 469762048 eof 0 [D 20:01:00.338491] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.338203] end iteration [D 20:01:00.338652] ========================================================= [D 20:01:00.338672] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.338680] dbpf_bstream_rw_list: mem_offset: 0x2aaabb654010, mem_size: 2097152 [D 20:01:00.338686] dbpf_bstream_rw_list: stream_offset: 452984832, stream_size: 2097152 [D 20:01:00.338693] dbpf_open_cache_get: called [D 20:01:00.338700] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.338706] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.338722] DBPF I/O ops in progress: 12 [D 20:01:00.338729] lio_listio called with the following aiocbs: [D 20:01:00.338735] aiocb_ptr_array[0]: fd: 13, off: 452984832, bytes: 2097152, buf: 0x2aaabb654010, type: 0 [D 20:01:00.338661] done with segments or bytes [D 20:01:00.338747] issue_or_delay_io_operation: lio_listio posted 0x6ec5d0 (handle 2863311531, ret 0) [D 20:01:00.338788] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:00.338822] get_data_sync_mode returning 0 [D 20:01:00.338836] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.338858] dbpf_bstream_rw_list: mem_offset: 0x2aaab1a02010, mem_size: 2097152 [D 20:01:00.338865] dbpf_bstream_rw_list: stream_offset: 457179136, stream_size: 2097152 [D 20:01:00.338871] dbpf_open_cache_get: called [D 20:01:00.338877] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.338883] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.338752] DBPF I/O ops in progress: 11 [D 20:01:00.338749] op_queue add: 0x707430 [D 20:01:00.338917] DBPF I/O ops in progress: 10 [D 20:01:00.338930] DBPF I/O ops in progress: 9 [D 20:01:00.339018] DBPF I/O ops in progress: 8 [D 20:01:00.339047] DBPF I/O ops in progress: 7 [D 20:01:00.339070] DBPF I/O ops in progress: 6 [D 20:01:00.339095] DBPF I/O ops in progress: 5 [D 20:01:00.339150] DBPF I/O ops in progress: 4 [D 20:01:00.338757] of 10485760 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.339247] next loff: 10485760 next poff: 10485760 [D 20:01:00.339257] return value 2097152 [D 20:01:00.339256] DBPF I/O ops in progress: 5 [D 20:01:00.339360] lio_listio called with the following aiocbs: [D 20:01:00.339367] aiocb_ptr_array[0]: fd: 13, off: 457179136, bytes: 2097152, buf: 0x2aaab1a02010, type: 1 [D 20:01:00.339375] issue_or_delay_io_operation: lio_listio posted 0x6ed8e0 (handle 2863311531, ret 0) [D 20:01:00.339384] ========================================================= [D 20:01:00.339390] PINT_process_request [D 20:01:00.339396] tiling 15 copies [D 20:01:00.339402] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.339409] lvl 0 el 0 blk 0 by 459276288 [D 20:01:00.339415] to 459276288 ta 436207616 fi 469762048 [D 20:01:00.339422] basic type or contiguous data [D 20:01:00.339269] finished [D 20:01:00.339428] Request disp [D 20:01:00.339448] contig_offset = 459276288 contig_size = 44040192 lvl_flag = 1 [D 20:01:00.339535] calling distribute [D 20:01:00.339541] PINT_distribute [D 20:01:00.339547] of 459276288 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 459276288 exfl 1 [D 20:01:00.339565] begin iteration loff: 459276288 [D 20:01:00.339572] segment exceeds byte limit [D 20:01:00.339577] file being extended [D 20:01:00.339583] process a segment [D 20:01:00.339589] of 459276288 sz 2097152 [D 20:01:00.339595] add a segment 0 [D 20:01:00.339601] end iteration [D 20:01:00.339606] done with segments or bytes [D 20:01:00.339612] of 461373440 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.339619] next loff: 461373440 next poff: 461373440 [D 20:01:00.339441] retval = 2097152 [D 20:01:00.339625] return value 2097152 [D 20:01:00.339644] exiting distribute returned less than expected [D 20:01:00.339648] finished [D 20:01:00.339683] retval = 2097152 [D 20:01:00.339656] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 10485760 fo 33554432 eof 0 [D 20:01:00.300756] --- aio_progress_notification called with handle 2863311531 (0x6ea920) [D 20:01:00.339706] exiting distribute returned less than expected [D 20:01:00.339717] ========================================================= [D 20:01:00.339735] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.339737] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 461373440 fo 469762048 eof 0 [D 20:01:00.339746] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.339763] ========================================================= [D 20:01:00.339773] dbpf_bstream_rw_list: mem_offset: 0x2aaabd061010, mem_size: 2097152 [D 20:01:00.307072] --- aio_progress_notification called with handle 2863311531 (0x2aaab152de80) [D 20:01:00.339782] BMI_post_recv: addr: 6843072, offset: 0x2aaab3611010, size: 2097152, tag: 21 [D 20:01:00.339790] dbpf_bstream_rw_list: stream_offset: 8388608, stream_size: 2097152 [D 20:01:00.339807] dbpf_open_cache_get: called [D 20:01:00.339815] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.307114] --- aio_progress_notification called with handle 2863311531 (0x2aaab15285e0) [D 20:01:00.339815] post_sr: 10.1.4.56:39990 bh 21 len 48 wr_id 682c90. [D 20:01:00.339839] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:01:00.339848] get_data_sync_mode returning 0 [D 20:01:00.339856] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.339863] dbpf_bstream_rw_list: mem_offset: 0x2aaab1c03010, mem_size: 2097152 [D 20:01:00.339869] dbpf_bstream_rw_list: stream_offset: 23068672, stream_size: 2097152 [D 20:01:00.339876] dbpf_open_cache_get: called [D 20:01:00.340488] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.339775] dbpf_open_cache_put: cached entry. [D 20:01:00.340530] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.340535] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.340538] op_queue add: 0x6ea920 [D 20:01:00.340550] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.340561] DBPF I/O ops in progress: 4 [D 20:01:00.340577] DBPF I/O ops in progress: 5 [D 20:01:00.340591] lio_listio called with the following aiocbs: [D 20:01:00.340598] aiocb_ptr_array[0]: fd: 13, off: 8388608, bytes: 2097152, buf: 0x2aaabd061010, type: 0 [D 20:01:00.340607] dbpf_open_cache_put: cached entry. [D 20:01:00.340608] issue_or_delay_io_operation: lio_listio posted 0x2aaab15178c0 (handle 2863311531, ret 0) [D 20:01:00.340625] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.340631] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.340635] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.340643] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517ef0. [D 20:01:00.340672] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.340637] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.340713] DBPF I/O ops in progress: 6 [D 20:01:00.341848] lio_listio called with the following aiocbs: [D 20:01:00.341875] aiocb_ptr_array[0]: fd: 13, off: 23068672, bytes: 2097152, buf: 0x2aaab1c03010, type: 1 [D 20:01:00.341892] issue_or_delay_io_operation: lio_listio posted 0x6ea5a0 (handle 2863311531, ret 0) [D 20:01:00.341905] ========================================================= [D 20:01:00.341917] PINT_process_request [D 20:01:00.341925] tiling 2 copies [D 20:01:00.341932] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.341940] lvl 0 el 0 blk 0 by 25165824 [D 20:01:00.341947] to 25165824 ta 0 fi 33554432 [D 20:01:00.341955] basic type or contiguous data [D 20:01:00.341961] Request disp [D 20:01:00.341967] contig_offset = 25165824 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.341975] calling distribute [D 20:01:00.341981] PINT_distribute [D 20:01:00.341987] of 25165824 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 25165824 exfl 1 [D 20:01:00.341990] ========================================================= [D 20:01:00.341995] begin iteration loff: 25165824 [D 20:01:00.342026] segment exceeds byte limit [D 20:01:00.342008] dbpf_open_cache_put: cached entry. [D 20:01:00.342014] PINT_process_request [D 20:01:00.342048] file being extended [D 20:01:00.342072] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.342058] tiling 2 copies [D 20:01:00.342069] process a segment [D 20:01:00.342103] of 25165824 sz 2097152 [D 20:01:00.342110] add a segment 0 [D 20:01:00.342117] end iteration [D 20:01:00.342123] done with segments or bytes [D 20:01:00.342129] of 27262976 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.342137] next loff: 27262976 next poff: 27262976 [D 20:01:00.342144] return value 2097152 [D 20:01:00.342151] finished [D 20:01:00.342157] retval = 2097152 [D 20:01:00.342164] exiting distribute returned less than expected [D 20:01:00.342171] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 27262976 fo 33554432 eof 0 [D 20:01:00.342178] ========================================================= [D 20:01:00.342186] BMI_post_recv: addr: 6843072, offset: 0x2aaab3812010, size: 2097152, tag: 8 [D 20:01:00.342202] openib_check_cq: completed send wr_id 682c90. [D 20:01:00.342244] post_sr: 10.1.4.56:39990 bh 22 len 48 wr_id 682cc0. [D 20:01:00.342258] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:01:00.342268] get_data_sync_mode returning 0 [D 20:01:00.342281] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.342289] dbpf_bstream_rw_list: mem_offset: 0x2aaab1e04010, mem_size: 2097152 [D 20:01:00.342296] dbpf_bstream_rw_list: stream_offset: 56623104, stream_size: 2097152 [D 20:01:00.342304] dbpf_open_cache_get: called [D 20:01:00.342311] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.342317] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.342350] DBPF I/O ops in progress: 7 [D 20:01:00.342358] lio_listio called with the following aiocbs: [D 20:01:00.342364] aiocb_ptr_array[0]: fd: 13, off: 56623104, bytes: 2097152, buf: 0x2aaab1e04010, type: 1 [D 20:01:00.342379] issue_or_delay_io_operation: lio_listio posted 0x2aaabd303440 (handle 2863311531, ret 0) [D 20:01:00.342388] ========================================================= [D 20:01:00.342394] PINT_process_request [D 20:01:00.342400] tiling 3 copies [D 20:01:00.342407] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.342414] lvl 0 el 0 blk 0 by 58720256 [D 20:01:00.342420] to 58720256 ta 33554432 fi 67108864 [D 20:01:00.342427] basic type or contiguous data [D 20:01:00.342434] Request disp [D 20:01:00.342439] contig_offset = 58720256 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.342446] calling distribute [D 20:01:00.342452] PINT_distribute [D 20:01:00.342457] of 58720256 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 58720256 exfl 1 [D 20:01:00.342465] begin iteration loff: 58720256 [D 20:01:00.342472] segment exceeds byte limit [D 20:01:00.342478] file being extended [D 20:01:00.342484] process a segment [D 20:01:00.342489] of 58720256 sz 2097152 [D 20:01:00.342496] add a segment 0 [D 20:01:00.342502] end iteration [D 20:01:00.342508] done with segments or bytes [D 20:01:00.342513] of 60817408 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.342521] next loff: 60817408 next poff: 60817408 [D 20:01:00.342527] return value 2097152 [D 20:01:00.342534] finished [D 20:01:00.342539] retval = 2097152 [D 20:01:00.342545] exiting distribute returned less than expected [D 20:01:00.342551] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 60817408 fo 67108864 eof 0 [D 20:01:00.342559] ========================================================= [D 20:01:00.342565] BMI_post_recv: addr: 6843072, offset: 0x2aaab3a13010, size: 2097152, tag: 9 [D 20:01:00.342573] openib_check_cq: completed send wr_id 682cc0. [D 20:01:00.342585] post_sr: 10.1.4.56:39990 bh 23 len 48 wr_id 682cf0. [D 20:01:00.342595] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:01:00.342604] get_data_sync_mode returning 0 [D 20:01:00.342611] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.342619] dbpf_bstream_rw_list: mem_offset: 0x2aaab2005010, mem_size: 2097152 [D 20:01:00.342625] dbpf_bstream_rw_list: stream_offset: 90177536, stream_size: 2097152 [D 20:01:00.342632] dbpf_open_cache_get: called [D 20:01:00.342638] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.342645] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.342668] DBPF I/O ops in progress: 8 [D 20:01:00.342677] lio_listio called with the following aiocbs: [D 20:01:00.342683] aiocb_ptr_array[0]: fd: 13, off: 90177536, bytes: 2097152, buf: 0x2aaab2005010, type: 1 [D 20:01:00.342691] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3060e0 (handle 2863311531, ret 0) [D 20:01:00.342699] ========================================================= [D 20:01:00.342706] PINT_process_request [D 20:01:00.342712] tiling 4 copies [D 20:01:00.342718] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.342726] lvl 0 el 0 blk 0 by 92274688 [D 20:01:00.342732] to 92274688 ta 67108864 fi 100663296 [D 20:01:00.342739] basic type or contiguous data [D 20:01:00.342746] Request disp [D 20:01:00.342751] contig_offset = 92274688 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.342758] calling distribute [D 20:01:00.342764] PINT_distribute [D 20:01:00.342770] of 92274688 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 92274688 exfl 1 [D 20:01:00.342778] begin iteration loff: 92274688 [D 20:01:00.342784] segment exceeds byte limit [D 20:01:00.342791] file being extended [D 20:01:00.342797] process a segment [D 20:01:00.342803] of 92274688 sz 2097152 [D 20:01:00.342809] add a segment 0 [D 20:01:00.342815] end iteration [D 20:01:00.342821] done with segments or bytes [D 20:01:00.342827] of 94371840 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.342834] next loff: 94371840 next poff: 94371840 [D 20:01:00.342841] return value 2097152 [D 20:01:00.342848] finished [D 20:01:00.342853] retval = 2097152 [D 20:01:00.342860] exiting distribute returned less than expected [D 20:01:00.342866] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 94371840 fo 100663296 eof 0 [D 20:01:00.342874] ========================================================= [D 20:01:00.342880] BMI_post_recv: addr: 6843072, offset: 0x2aaab3c14010, size: 2097152, tag: 10 [D 20:01:00.342888] openib_check_cq: completed send wr_id 682cf0. [D 20:01:00.342900] post_sr: 10.1.4.56:39990 bh 24 len 48 wr_id 682d20. [D 20:01:00.342918] openib_check_cq: completed send wr_id 682d20. [D 20:01:00.342943] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:01:00.342953] get_data_sync_mode returning 0 [D 20:01:00.342960] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.342967] dbpf_bstream_rw_list: mem_offset: 0x2aaab2206010, mem_size: 2097152 [D 20:01:00.342973] dbpf_bstream_rw_list: stream_offset: 123731968, stream_size: 2097152 [D 20:01:00.342980] dbpf_open_cache_get: called [D 20:01:00.342986] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.342993] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.343018] DBPF I/O ops in progress: 9 [D 20:01:00.343026] lio_listio called with the following aiocbs: [D 20:01:00.343032] aiocb_ptr_array[0]: fd: 13, off: 123731968, bytes: 2097152, buf: 0x2aaab2206010, type: 1 [D 20:01:00.343040] issue_or_delay_io_operation: lio_listio posted 0x2aaabd308d80 (handle 2863311531, ret 0) [D 20:01:00.343049] ========================================================= [D 20:01:00.343056] PINT_process_request [D 20:01:00.343062] tiling 5 copies [D 20:01:00.343068] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.343076] lvl 0 el 0 blk 0 by 125829120 [D 20:01:00.343082] to 125829120 ta 100663296 fi 134217728 [D 20:01:00.343089] basic type or contiguous data [D 20:01:00.343095] Request disp [D 20:01:00.343101] contig_offset = 125829120 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.343108] calling distribute [D 20:01:00.343114] PINT_distribute [D 20:01:00.343120] of 125829120 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 125829120 exfl 1 [D 20:01:00.343128] begin iteration loff: 125829120 [D 20:01:00.343134] segment exceeds byte limit [D 20:01:00.343140] file being extended [D 20:01:00.343146] process a segment [D 20:01:00.343152] of 125829120 sz 2097152 [D 20:01:00.343158] add a segment 0 [D 20:01:00.343165] end iteration [D 20:01:00.343170] done with segments or bytes [D 20:01:00.343176] of 127926272 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.343184] next loff: 127926272 next poff: 127926272 [D 20:01:00.343190] return value 2097152 [D 20:01:00.343196] finished [D 20:01:00.343202] retval = 2097152 [D 20:01:00.343208] exiting distribute returned less than expected [D 20:01:00.343214] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 127926272 fo 134217728 eof 0 [D 20:01:00.343222] ========================================================= [D 20:01:00.343228] BMI_post_recv: addr: 6843072, offset: 0x2aaab3e15010, size: 2097152, tag: 11 [D 20:01:00.343240] post_sr: 10.1.4.56:39990 bh 25 len 48 wr_id 682d50. [D 20:01:00.343252] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:01:00.343261] get_data_sync_mode returning 0 [D 20:01:00.343268] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.343274] dbpf_bstream_rw_list: mem_offset: 0x2aaab2407010, mem_size: 2097152 [D 20:01:00.343282] dbpf_bstream_rw_list: stream_offset: 157286400, stream_size: 2097152 [D 20:01:00.343288] dbpf_open_cache_get: called [D 20:01:00.343294] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.343301] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.343328] DBPF I/O ops in progress: 10 [D 20:01:00.343336] lio_listio called with the following aiocbs: [D 20:01:00.343342] aiocb_ptr_array[0]: fd: 13, off: 157286400, bytes: 2097152, buf: 0x2aaab2407010, type: 1 [D 20:01:00.343356] --- aio_progress_notification called with handle 2863311531 (0x6ec5d0) [D 20:01:00.342000] op_queue add: 0x2aaab152de80 [D 20:01:00.342096] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.343388] DBPF I/O ops in progress: 9 [D 20:01:00.343400] lvl 0 el 0 blk 0 by 10485760 [D 20:01:00.343420] to 10485760 ta 0 fi 33554432 [D 20:01:00.343427] basic type or contiguous data [D 20:01:00.343410] issue_or_delay_io_operation: lio_listio posted 0x2aaabd30ba20 (handle 2863311531, ret 0) [D 20:01:00.343446] ========================================================= [D 20:01:00.343453] PINT_process_request [D 20:01:00.343459] tiling 6 copies [D 20:01:00.343465] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.343434] Request disp [D 20:01:00.343472] lvl 0 el 0 blk 0 by 159383552 [D 20:01:00.343491] to 159383552 ta 134217728 fi 167772160 [D 20:01:00.343501] basic type or contiguous data [D 20:01:00.343509] Request disp [D 20:01:00.343517] contig_offset = 159383552 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.343527] calling distribute [D 20:01:00.343536] PINT_distribute [D 20:01:00.343481] contig_offset = 10485760 contig_size = 56623104 lvl_flag = 1 [D 20:01:00.343545] of 159383552 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 159383552 exfl 1 [D 20:01:00.343565] begin iteration loff: 159383552 [D 20:01:00.343572] segment exceeds byte limit [D 20:01:00.343578] file being extended [D 20:01:00.343585] process a segment [D 20:01:00.343590] of 159383552 sz 2097152 [D 20:01:00.343597] add a segment 0 [D 20:01:00.343603] end iteration [D 20:01:00.343609] done with segments or bytes [D 20:01:00.343615] of 161480704 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.343622] next loff: 161480704 next poff: 161480704 [D 20:01:00.343628] return value 2097152 [D 20:01:00.343638] finished [D 20:01:00.343644] retval = 2097152 [D 20:01:00.343650] exiting distribute returned less than expected [D 20:01:00.343656] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 161480704 fo 167772160 eof 0 [D 20:01:00.343664] ========================================================= [D 20:01:00.343670] BMI_post_recv: addr: 6843072, offset: 0x2aaab4016010, size: 2097152, tag: 12 [D 20:01:00.343678] openib_check_cq: completed send wr_id 682d50. [D 20:01:00.343691] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:01:00.343699] get_data_sync_mode returning 0 [D 20:01:00.343707] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.343713] dbpf_bstream_rw_list: mem_offset: 0x2aaab2608010, mem_size: 2097152 [D 20:01:00.343720] dbpf_bstream_rw_list: stream_offset: 190840832, stream_size: 2097152 [D 20:01:00.343726] dbpf_open_cache_get: called [D 20:01:00.343733] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.343739] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.343763] DBPF I/O ops in progress: 10 [D 20:01:00.343771] lio_listio called with the following aiocbs: [D 20:01:00.343777] aiocb_ptr_array[0]: fd: 13, off: 190840832, bytes: 2097152, buf: 0x2aaab2608010, type: 1 [D 20:01:00.343786] issue_or_delay_io_operation: lio_listio posted 0x2aaabd30e750 (handle 2863311531, ret 0) [D 20:01:00.343794] ========================================================= [D 20:01:00.343801] PINT_process_request [D 20:01:00.343807] tiling 7 copies [D 20:01:00.343813] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.343820] lvl 0 el 0 blk 0 by 192937984 [D 20:01:00.343827] to 192937984 ta 167772160 fi 201326592 [D 20:01:00.343834] basic type or contiguous data [D 20:01:00.343840] Request disp [D 20:01:00.343846] contig_offset = 192937984 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.343853] calling distribute [D 20:01:00.343859] PINT_distribute [D 20:01:00.343864] of 192937984 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 192937984 exfl 1 [D 20:01:00.343872] begin iteration loff: 192937984 [D 20:01:00.343878] segment exceeds byte limit [D 20:01:00.343884] file being extended [D 20:01:00.343890] process a segment [D 20:01:00.343896] of 192937984 sz 2097152 [D 20:01:00.343902] add a segment 0 [D 20:01:00.343908] end iteration [D 20:01:00.343914] done with segments or bytes [D 20:01:00.343920] of 195035136 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.343927] next loff: 195035136 next poff: 195035136 [D 20:01:00.343933] return value 2097152 [D 20:01:00.343939] finished [D 20:01:00.343945] retval = 2097152 [D 20:01:00.343951] exiting distribute returned less than expected [D 20:01:00.343957] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 195035136 fo 201326592 eof 0 [D 20:01:00.343965] ========================================================= [D 20:01:00.343971] BMI_post_recv: addr: 6843072, offset: 0x2aaab4217010, size: 2097152, tag: 13 [D 20:01:00.343982] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:01:00.343991] get_data_sync_mode returning 0 [D 20:01:00.343999] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.344005] dbpf_bstream_rw_list: mem_offset: 0x2aaab2809010, mem_size: 2097152 [D 20:01:00.344014] dbpf_bstream_rw_list: stream_offset: 224395264, stream_size: 2097152 [D 20:01:00.344021] dbpf_open_cache_get: called [D 20:01:00.344027] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.344034] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.344052] DBPF I/O ops in progress: 11 [D 20:01:00.344060] lio_listio called with the following aiocbs: [D 20:01:00.344066] aiocb_ptr_array[0]: fd: 13, off: 224395264, bytes: 2097152, buf: 0x2aaab2809010, type: 1 [D 20:01:00.344078] issue_or_delay_io_operation: lio_listio posted 0x2aaabd311480 (handle 2863311531, ret 0) [D 20:01:00.344087] ========================================================= [D 20:01:00.344093] PINT_process_request [D 20:01:00.344099] tiling 8 copies [D 20:01:00.344106] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.344113] lvl 0 el 0 blk 0 by 226492416 [D 20:01:00.344119] to 226492416 ta 201326592 fi 234881024 [D 20:01:00.344127] basic type or contiguous data [D 20:01:00.344133] Request disp [D 20:01:00.344139] contig_offset = 226492416 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.344146] calling distribute [D 20:01:00.344151] PINT_distribute [D 20:01:00.344157] of 226492416 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 226492416 exfl 1 [D 20:01:00.344165] begin iteration loff: 226492416 [D 20:01:00.344171] segment exceeds byte limit [D 20:01:00.344177] file being extended [D 20:01:00.344183] process a segment [D 20:01:00.344189] of 226492416 sz 2097152 [D 20:01:00.344195] add a segment 0 [D 20:01:00.344202] end iteration [D 20:01:00.344208] done with segments or bytes [D 20:01:00.344214] of 228589568 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.344220] next loff: 228589568 next poff: 228589568 [D 20:01:00.344227] return value 2097152 [D 20:01:00.344233] finished [D 20:01:00.344239] retval = 2097152 [D 20:01:00.344245] exiting distribute returned less than expected [D 20:01:00.344252] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 228589568 fo 234881024 eof 0 [D 20:01:00.344259] ========================================================= [D 20:01:00.344265] BMI_post_recv: addr: 6843072, offset: 0x2aaab4418010, size: 2097152, tag: 14 [D 20:01:00.344278] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:01:00.344286] get_data_sync_mode returning 0 [D 20:01:00.344293] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.344300] dbpf_bstream_rw_list: mem_offset: 0x2aaab2a0a010, mem_size: 2097152 [D 20:01:00.344307] dbpf_bstream_rw_list: stream_offset: 257949696, stream_size: 2097152 [D 20:01:00.344313] dbpf_open_cache_get: called [D 20:01:00.344319] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.344326] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.344344] DBPF I/O ops in progress: 12 [D 20:01:00.344352] lio_listio called with the following aiocbs: [D 20:01:00.344358] aiocb_ptr_array[0]: fd: 13, off: 257949696, bytes: 2097152, buf: 0x2aaab2a0a010, type: 1 [D 20:01:00.344367] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3141b0 (handle 2863311531, ret 0) [D 20:01:00.344375] ========================================================= [D 20:01:00.344381] PINT_process_request [D 20:01:00.344387] tiling 9 copies [D 20:01:00.344393] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.344401] lvl 0 el 0 blk 0 by 260046848 [D 20:01:00.344407] to 260046848 ta 234881024 fi 268435456 [D 20:01:00.344414] basic type or contiguous data [D 20:01:00.344420] Request disp [D 20:01:00.344426] contig_offset = 260046848 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.344433] calling distribute [D 20:01:00.344439] PINT_distribute [D 20:01:00.344445] of 260046848 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 260046848 exfl 1 [D 20:01:00.344453] begin iteration loff: 260046848 [D 20:01:00.344459] segment exceeds byte limit [D 20:01:00.344466] file being extended [D 20:01:00.344473] process a segment [D 20:01:00.344502] of 260046848 sz 2097152 [D 20:01:00.344509] add a segment 0 [D 20:01:00.344515] end iteration [D 20:01:00.344521] done with segments or bytes [D 20:01:00.344527] of 262144000 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.344534] next loff: 262144000 next poff: 262144000 [D 20:01:00.344540] return value 2097152 [D 20:01:00.344546] finished [D 20:01:00.344552] retval = 2097152 [D 20:01:00.344558] exiting distribute returned less than expected [D 20:01:00.344564] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 262144000 fo 268435456 eof 0 [D 20:01:00.344571] ========================================================= [D 20:01:00.344578] BMI_post_recv: addr: 6843072, offset: 0x2aaab4619010, size: 2097152, tag: 15 [D 20:01:00.344603] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:01:00.344613] get_data_sync_mode returning 0 [D 20:01:00.344627] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.344634] dbpf_bstream_rw_list: mem_offset: 0x2aaab2c0b010, mem_size: 2097152 [D 20:01:00.344641] dbpf_bstream_rw_list: stream_offset: 291504128, stream_size: 2097152 [D 20:01:00.344648] dbpf_open_cache_get: called [D 20:01:00.344654] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.344661] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.344681] DBPF I/O ops in progress: 13 [D 20:01:00.344688] lio_listio called with the following aiocbs: [D 20:01:00.344694] aiocb_ptr_array[0]: fd: 13, off: 291504128, bytes: 2097152, buf: 0x2aaab2c0b010, type: 1 [D 20:01:00.344703] issue_or_delay_io_operation: lio_listio posted 0x2aaabd316ee0 (handle 2863311531, ret 0) [D 20:01:00.344710] ========================================================= [D 20:01:00.344717] PINT_process_request [D 20:01:00.344723] tiling 10 copies [D 20:01:00.344729] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.344737] lvl 0 el 0 blk 0 by 293601280 [D 20:01:00.344743] to 293601280 ta 268435456 fi 301989888 [D 20:01:00.344750] basic type or contiguous data [D 20:01:00.344756] Request disp [D 20:01:00.344761] contig_offset = 293601280 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.344768] calling distribute [D 20:01:00.344775] PINT_distribute [D 20:01:00.344780] of 293601280 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 293601280 exfl 1 [D 20:01:00.344788] begin iteration loff: 293601280 [D 20:01:00.344795] segment exceeds byte limit [D 20:01:00.344801] file being extended [D 20:01:00.344807] process a segment [D 20:01:00.344813] of 293601280 sz 2097152 [D 20:01:00.344819] add a segment 0 [D 20:01:00.344825] end iteration [D 20:01:00.344831] done with segments or bytes [D 20:01:00.344837] of 295698432 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.344844] next loff: 295698432 next poff: 295698432 [D 20:01:00.344850] return value 2097152 [D 20:01:00.344856] finished [D 20:01:00.344862] retval = 2097152 [D 20:01:00.344868] exiting distribute returned less than expected [D 20:01:00.344874] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 295698432 fo 301989888 eof 0 [D 20:01:00.344881] ========================================================= [D 20:01:00.344887] BMI_post_recv: addr: 6843072, offset: 0x2aaab481a010, size: 2097152, tag: 16 [D 20:01:00.344900] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:01:00.344908] get_data_sync_mode returning 0 [D 20:01:00.344914] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.344921] dbpf_bstream_rw_list: mem_offset: 0x2aaab2e0c010, mem_size: 2097152 [D 20:01:00.344928] dbpf_bstream_rw_list: stream_offset: 325058560, stream_size: 2097152 [D 20:01:00.344934] dbpf_open_cache_get: called [D 20:01:00.344940] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.344947] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.344974] DBPF I/O ops in progress: 14 [D 20:01:00.344987] lio_listio called with the following aiocbs: [D 20:01:00.344993] aiocb_ptr_array[0]: fd: 13, off: 325058560, bytes: 2097152, buf: 0x2aaab2e0c010, type: 1 [D 20:01:00.345002] issue_or_delay_io_operation: lio_listio posted 0x2aaabd319c10 (handle 2863311531, ret 0) [D 20:01:00.345010] ========================================================= [D 20:01:00.345017] PINT_process_request [D 20:01:00.345023] tiling 11 copies [D 20:01:00.345029] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.345036] lvl 0 el 0 blk 0 by 327155712 [D 20:01:00.345042] to 327155712 ta 301989888 fi 335544320 [D 20:01:00.345049] basic type or contiguous data [D 20:01:00.345055] Request disp [D 20:01:00.345061] contig_offset = 327155712 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.345068] calling distribute [D 20:01:00.345074] PINT_distribute [D 20:01:00.345079] of 327155712 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 327155712 exfl 1 [D 20:01:00.345087] begin iteration loff: 327155712 [D 20:01:00.345093] segment exceeds byte limit [D 20:01:00.345099] file being extended [D 20:01:00.345105] process a segment [D 20:01:00.345111] of 327155712 sz 2097152 [D 20:01:00.345117] add a segment 0 [D 20:01:00.345123] end iteration [D 20:01:00.345129] done with segments or bytes [D 20:01:00.345135] of 329252864 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.345142] next loff: 329252864 next poff: 329252864 [D 20:01:00.345148] return value 2097152 [D 20:01:00.345154] finished [D 20:01:00.345160] retval = 2097152 [D 20:01:00.345165] exiting distribute returned less than expected [D 20:01:00.345172] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 329252864 fo 335544320 eof 0 [D 20:01:00.345179] ========================================================= [D 20:01:00.345185] BMI_post_recv: addr: 6843072, offset: 0x2aaab4a1b010, size: 2097152, tag: 17 [D 20:01:00.345197] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:01:00.345206] get_data_sync_mode returning 0 [D 20:01:00.345213] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.345220] dbpf_bstream_rw_list: mem_offset: 0x2aaab300e010, mem_size: 2097152 [D 20:01:00.345227] dbpf_bstream_rw_list: stream_offset: 358612992, stream_size: 2097152 [D 20:01:00.345233] dbpf_open_cache_get: called [D 20:01:00.345239] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.345246] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.345268] DBPF I/O ops in progress: 15 [D 20:01:00.345276] lio_listio called with the following aiocbs: [D 20:01:00.345282] aiocb_ptr_array[0]: fd: 13, off: 358612992, bytes: 2097152, buf: 0x2aaab300e010, type: 1 [D 20:01:00.345290] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31c940 (handle 2863311531, ret 0) [D 20:01:00.345298] ========================================================= [D 20:01:00.345305] PINT_process_request [D 20:01:00.345311] tiling 12 copies [D 20:01:00.345317] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.345325] lvl 0 el 0 blk 0 by 360710144 [D 20:01:00.345331] to 360710144 ta 335544320 fi 369098752 [D 20:01:00.345337] basic type or contiguous data [D 20:01:00.345343] Request disp [D 20:01:00.345349] contig_offset = 360710144 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.345356] calling distribute [D 20:01:00.345362] PINT_distribute [D 20:01:00.345368] of 360710144 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 360710144 exfl 1 [D 20:01:00.345375] begin iteration loff: 360710144 [D 20:01:00.345381] segment exceeds byte limit [D 20:01:00.345387] file being extended [D 20:01:00.345393] process a segment [D 20:01:00.345398] of 360710144 sz 2097152 [D 20:01:00.345404] add a segment 0 [D 20:01:00.345410] end iteration [D 20:01:00.345416] done with segments or bytes [D 20:01:00.345422] of 362807296 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.345429] next loff: 362807296 next poff: 362807296 [D 20:01:00.345436] return value 2097152 [D 20:01:00.345442] finished [D 20:01:00.345448] retval = 2097152 [D 20:01:00.345454] exiting distribute returned less than expected [D 20:01:00.345460] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 362807296 fo 369098752 eof 0 [D 20:01:00.345467] ========================================================= [D 20:01:00.345473] BMI_post_recv: addr: 6843072, offset: 0x2aaab4c1c010, size: 2097152, tag: 18 [D 20:01:00.345485] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:01:00.345493] get_data_sync_mode returning 0 [D 20:01:00.345500] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.345507] dbpf_bstream_rw_list: mem_offset: 0x2aaab320f010, mem_size: 2097152 [D 20:01:00.345514] dbpf_bstream_rw_list: stream_offset: 392167424, stream_size: 2097152 [D 20:01:00.345520] dbpf_open_cache_get: called [D 20:01:00.345527] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.345533] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.345557] DBPF I/O ops in progress: 16 [D 20:01:00.345564] lio_listio called with the following aiocbs: [D 20:01:00.345571] aiocb_ptr_array[0]: fd: 13, off: 392167424, bytes: 2097152, buf: 0x2aaab320f010, type: 1 [D 20:01:00.345579] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31f670 (handle 2863311531, ret 0) [D 20:01:00.345588] ========================================================= [D 20:01:00.345594] PINT_process_request [D 20:01:00.345601] tiling 13 copies [D 20:01:00.345607] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.345614] lvl 0 el 0 blk 0 by 394264576 [D 20:01:00.345620] to 394264576 ta 369098752 fi 402653184 [D 20:01:00.345628] basic type or contiguous data [D 20:01:00.345634] Request disp [D 20:01:00.345640] contig_offset = 394264576 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.345647] calling distribute [D 20:01:00.345653] PINT_distribute [D 20:01:00.345659] of 394264576 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 394264576 exfl 1 [D 20:01:00.345667] begin iteration loff: 394264576 [D 20:01:00.345673] segment exceeds byte limit [D 20:01:00.345679] file being extended [D 20:01:00.345685] process a segment [D 20:01:00.345691] of 394264576 sz 2097152 [D 20:01:00.345697] add a segment 0 [D 20:01:00.345703] end iteration [D 20:01:00.345709] done with segments or bytes [D 20:01:00.345715] of 396361728 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.345722] next loff: 396361728 next poff: 396361728 [D 20:01:00.345728] return value 2097152 [D 20:01:00.345734] finished [D 20:01:00.345740] retval = 2097152 [D 20:01:00.345746] exiting distribute returned less than expected [D 20:01:00.345752] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 396361728 fo 402653184 eof 0 [D 20:01:00.345760] ========================================================= [D 20:01:00.345767] BMI_post_recv: addr: 6843072, offset: 0x2aaab4e1d010, size: 2097152, tag: 19 [D 20:01:00.345779] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:00.345787] get_data_sync_mode returning 0 [D 20:01:00.345794] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.345801] dbpf_bstream_rw_list: mem_offset: 0x2aaabc85d010, mem_size: 2097152 [D 20:01:00.345808] dbpf_bstream_rw_list: stream_offset: 484442112, stream_size: 2097152 [D 20:01:00.345815] dbpf_open_cache_get: called [D 20:01:00.345821] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.345827] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.345863] op_queue add: 0x2aaabd3223a0 [D 20:01:00.345875] delayed I/O operation 0x2aaabd3223a0 (16 already in progress) [D 20:01:00.345882] DBPF I/O ops in progress: 16 [D 20:01:00.345890] ========================================================= [D 20:01:00.345896] PINT_process_request [D 20:01:00.345902] tiling 16 copies [D 20:01:00.345909] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.345917] lvl 0 el 0 blk 0 by 486539264 [D 20:01:00.345923] to 486539264 ta 469762048 fi 503316480 [D 20:01:00.345930] basic type or contiguous data [D 20:01:00.345936] Request disp [D 20:01:00.345942] contig_offset = 486539264 contig_size = 50331648 lvl_flag = 1 [D 20:01:00.345949] calling distribute [D 20:01:00.345954] PINT_distribute [D 20:01:00.345960] of 486539264 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 486539264 exfl 1 [D 20:01:00.345968] begin iteration loff: 486539264 [D 20:01:00.345974] segment exceeds byte limit [D 20:01:00.345980] file being extended [D 20:01:00.345986] process a segment [D 20:01:00.345991] of 486539264 sz 2097152 [D 20:01:00.345998] add a segment 0 [D 20:01:00.346004] end iteration [D 20:01:00.346010] done with segments or bytes [D 20:01:00.346016] of 488636416 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.346022] next loff: 488636416 next poff: 488636416 [D 20:01:00.346029] return value 2097152 [D 20:01:00.346034] finished [D 20:01:00.346040] retval = 2097152 [D 20:01:00.346046] exiting distribute returned less than expected [D 20:01:00.346052] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 488636416 fo 503316480 eof 0 [D 20:01:00.346060] ========================================================= [D 20:01:00.346066] BMI_post_recv: addr: 6843072, offset: 0x2aaab521f010, size: 2097152, tag: 22 [D 20:01:00.343559] calling distribute [D 20:01:00.353123] PINT_distribute [D 20:01:00.353132] of 10485760 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.353142] begin iteration loff: 10485760 [D 20:01:00.353149] segment exceeds byte limit [D 20:01:00.353156] process a segment [D 20:01:00.353162] of 10485760 sz 2097152 [D 20:01:00.353168] add a segment 0 [D 20:01:00.353175] end iteration [D 20:01:00.353181] done with segments or bytes [D 20:01:00.353187] of 12582912 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.353195] next loff: 12582912 next poff: 12582912 [D 20:01:00.353202] return value 2097152 [D 20:01:00.353209] finished [D 20:01:00.353215] retval = 2097152 [D 20:01:00.353222] exiting distribute returned less than expected [D 20:01:00.353229] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 12582912 fo 33554432 eof 0 [D 20:01:00.353236] ========================================================= [D 20:01:00.353256] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.353264] dbpf_bstream_rw_list: mem_offset: 0x2aaabd400010, mem_size: 2097152 [D 20:01:00.353271] dbpf_bstream_rw_list: stream_offset: 10485760, stream_size: 2097152 [D 20:01:00.353280] dbpf_open_cache_get: called [D 20:01:00.353287] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.353295] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.353317] op_queue add: 0x2aaab152b230 [D 20:01:00.353325] delayed I/O operation 0x2aaab152b230 (16 already in progress) [D 20:01:00.353332] DBPF I/O ops in progress: 16 [D 20:01:00.353340] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.353348] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517ef0. [D 20:01:00.353357] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.354833] --- aio_progress_notification called with handle 2863311531 (0x6ed8e0) [D 20:01:00.354873] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.354888] dbpf_open_cache_put: cached entry. [D 20:01:00.354896] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.354911] --- aio_progress_notification called with handle 2863311531 (0x2aaab15178c0) [D 20:01:00.354923] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.354934] dbpf_open_cache_put: cached entry. [D 20:01:00.354941] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.354953] --- aio_progress_notification called with handle 2863311531 (0x6ea5a0) [D 20:01:00.354966] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.354977] dbpf_open_cache_put: cached entry. [D 20:01:00.354983] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.354994] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303440) [D 20:01:00.355004] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.355015] dbpf_open_cache_put: cached entry. [D 20:01:00.355021] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.343391] op_queue add: 0x2aaab15285e0 [D 20:01:00.355300] DBPF I/O ops in progress: 15 [D 20:01:00.355309] starting delayed I/O operation 0x2aaabd3223a0 (15 in progress) [D 20:01:00.355321] lio_listio called with 1 following aiocbs: [D 20:01:00.355327] aiocb_ptr_array[0]: fd: 13, off: 484442112, bytes: 2097152, buf: 0x2aaabc85d010, type: 1 [D 20:01:00.355342] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3223a0 (handle 2863311531, ret 0)) [D 20:01:00.355396] op_queue add: 0x2aaab15178c0 [D 20:01:00.355421] ========================================================= [D 20:01:00.355431] PINT_process_request [D 20:01:00.355439] tiling 2 copies [D 20:01:00.355446] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.355454] lvl 0 el 0 blk 0 by 12582912 [D 20:01:00.355460] to 12582912 ta 0 fi 33554432 [D 20:01:00.355467] basic type or contiguous data [D 20:01:00.355473] Request disp [D 20:01:00.355479] contig_offset = 12582912 contig_size = 54525952 lvl_flag = 1 [D 20:01:00.355487] calling distribute [D 20:01:00.355493] PINT_distribute [D 20:01:00.355499] of 12582912 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.355508] begin iteration loff: 12582912 [D 20:01:00.355516] segment exceeds byte limit [D 20:01:00.355523] process a segment [D 20:01:00.355529] of 12582912 sz 2097152 [D 20:01:00.355535] add a segment 0 [D 20:01:00.355541] end iteration [D 20:01:00.355547] done with segments or bytes [D 20:01:00.355554] of 14680064 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.355562] next loff: 14680064 next poff: 14680064 [D 20:01:00.355568] return value 2097152 [D 20:01:00.355575] finished [D 20:01:00.355581] retval = 2097152 [D 20:01:00.355587] exiting distribute returned less than expected [D 20:01:00.355593] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 14680064 fo 33554432 eof 0 [D 20:01:00.355600] ========================================================= [D 20:01:00.355615] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.355623] dbpf_bstream_rw_list: mem_offset: 0x2aaabd601010, mem_size: 2097152 [D 20:01:00.355630] dbpf_bstream_rw_list: stream_offset: 12582912, stream_size: 2097152 [D 20:01:00.355639] dbpf_open_cache_get: called [D 20:01:00.355645] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.355653] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.355676] DBPF I/O ops in progress: 15 [D 20:01:00.355684] starting delayed I/O operation 0x2aaab152b230 (15 in progress) [D 20:01:00.355696] lio_listio called with 1 following aiocbs: [D 20:01:00.355702] aiocb_ptr_array[0]: fd: 13, off: 10485760, bytes: 2097152, buf: 0x2aaabd400010, type: 0 [D 20:01:00.355710] start_delayed_ops_if_any: lio_listio posted 0x2aaab152b230 (handle 2863311531, ret 0)) [D 20:01:00.355720] op_queue add: 0x2aaab151d770 [D 20:01:00.355728] delayed I/O operation 0x2aaab151d770 (16 already in progress) [D 20:01:00.355734] DBPF I/O ops in progress: 16 [D 20:01:00.355741] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.355749] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517ef0. [D 20:01:00.355758] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.343379] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.355773] op_queue add: 0x2aaabd303440 [D 20:01:00.358730] DBPF I/O ops in progress: 15 [D 20:01:00.358758] starting delayed I/O operation 0x2aaab151d770 (15 in progress) [D 20:01:00.358780] ========================================================= [D 20:01:00.358795] PINT_process_request [D 20:01:00.358804] tiling 2 copies [D 20:01:00.358812] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.358820] lvl 0 el 0 blk 0 by 14680064 [D 20:01:00.358826] to 14680064 ta 0 fi 33554432 [D 20:01:00.358834] basic type or contiguous data [D 20:01:00.358841] Request disp [D 20:01:00.358847] contig_offset = 14680064 contig_size = 52428800 lvl_flag = 1 [D 20:01:00.358855] calling distribute [D 20:01:00.358862] PINT_distribute [D 20:01:00.358869] of 14680064 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.358878] begin iteration loff: 14680064 [D 20:01:00.358884] segment exceeds byte limit [D 20:01:00.358891] process a segment [D 20:01:00.358897] of 14680064 sz 2097152 [D 20:01:00.358903] add a segment 0 [D 20:01:00.358910] end iteration [D 20:01:00.358916] done with segments or bytes [D 20:01:00.358923] of 16777216 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.358930] next loff: 16777216 next poff: 16777216 [D 20:01:00.358936] return value 2097152 [D 20:01:00.358943] finished [D 20:01:00.358950] retval = 2097152 [D 20:01:00.358956] exiting distribute returned less than expected [D 20:01:00.358962] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 16777216 fo 33554432 eof 0 [D 20:01:00.358969] ========================================================= [D 20:01:00.358985] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.358992] dbpf_bstream_rw_list: mem_offset: 0x2aaabd802010, mem_size: 2097152 [D 20:01:00.359000] dbpf_bstream_rw_list: stream_offset: 14680064, stream_size: 2097152 [D 20:01:00.359009] dbpf_open_cache_get: called [D 20:01:00.359022] lio_listio called with 1 following aiocbs: [D 20:01:00.359030] aiocb_ptr_array[0]: fd: 13, off: 12582912, bytes: 2097152, buf: 0x2aaabd601010, type: 0 [D 20:01:00.359043] start_delayed_ops_if_any: lio_listio posted 0x2aaab151d770 (handle 2863311531, ret 0)) [D 20:01:00.356564] dbpf_open_cache_put: cached entry. [D 20:01:00.359092] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.359100] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.359122] op_queue add: 0x2aaab15202d0 [D 20:01:00.359130] delayed I/O operation 0x2aaab15202d0 (16 already in progress) [D 20:01:00.359137] DBPF I/O ops in progress: 16 [D 20:01:00.359151] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:00.359164] BMI_post_send: addr: 6843072, offset: 0x2aaabc65c010, size: 2097152, tag: 23 [D 20:01:00.359180] post_sr: 10.1.4.56:39990 bh 26 len 32 wr_id 682d80. [D 20:01:00.359217] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:00.359240] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:00.359249] BMI_post_send: addr: 6843072, offset: 0x2aaabca5e010, size: 2097152, tag: 23 [D 20:01:00.359258] post_sr: 10.1.4.56:39990 bh 27 len 32 wr_id 682db0. [D 20:01:00.359268] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:00.359278] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:00.359289] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:00.359297] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:00.359305] BMI_post_send: addr: 6843072, offset: 0x2aaabcc5f010, size: 2097152, tag: 23 [D 20:01:00.359313] post_sr: 10.1.4.56:39990 bh 28 len 32 wr_id 682de0. [D 20:01:00.359324] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:00.359331] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:00.359341] BMI_post_send: addr: 6843072, offset: 0x2aaabce60010, size: 2097152, tag: 23 [D 20:01:00.359350] post_sr: 10.1.4.56:39990 bh 29 len 32 wr_id 682e10. [D 20:01:00.359361] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:00.359371] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:00.359378] BMI_post_send: addr: 6843072, offset: 0x2aaabd061010, size: 2097152, tag: 23 [D 20:01:00.359388] post_sr: 10.1.4.56:39990 bh 30 len 32 wr_id 682e40. [D 20:01:00.359398] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:00.359408] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:00.359421] flowproto posted 0x2aaab1517ef0 [D 20:01:00.359430] Job flows in progress (post time): 17 [D 20:01:00.359443] [SM Exiting]: (0x65f450) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:00.359455] [SM Entering]: (0x660710) io_sm:start_flow (status: 0) [D 20:01:00.359462] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.359474] io_start_flow() issuing flow to read data. [D 20:01:00.359481] flow: fsize: 440401920, server_nr: 0, server_ct: 1 [D 20:01:00.359487] file_req_offset: 33554432, aggregate_size: 33554432, handle: 2863311531 [D 20:01:00.359500] PINT_new_request_states n=1 [D 20:01:00.359510] flowproto posting 0x2aaab1518130 [D 20:01:00.359523] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.359530] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518130. [D 20:01:00.359538] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes -[D 20:01:00.359574] openib_check_cq: completed send wr_id 682d80. [D 20:01:00.362111] openib_check_cq: completed send wr_id 682db0. [D 20:01:00.362132] openib_check_cq: completed send wr_id 682de0. [D 20:01:00.362139] openib_check_cq: completed send wr_id 682e10. [D 20:01:00.362147] openib_check_cq: completed send wr_id 682e40. [D 20:01:00.369736] ========================================================= [D 20:01:00.369761] PINT_process_request [D 20:01:00.369769] tiling 3 copies [D 20:01:00.369777] skipping ahead to target_offset [D 20:01:00.369783] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.369791] lvl 0 el 0 blk 0 by 0 [D 20:01:00.369798] to 0 ta 33554432 fi 67108864 [D 20:01:00.369804] basic type or contiguous data [D 20:01:00.369811] Request disp [D 20:01:00.369817] contig_offset = 0 contig_size = 100663296 lvl_flag = 1 [D 20:01:00.369824] process logical skip [D 20:01:00.369830] retval = 33554432 [D 20:01:00.369837] exiting logical skip because distribute indicates done [D 20:01:00.369843] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.369850] lvl 0 el 0 blk 0 by 33554432 [D 20:01:00.369856] to 33554432 ta 33554432 fi 67108864 [D 20:01:00.369862] basic type or contiguous data [D 20:01:00.369868] Request disp [D 20:01:00.369874] contig_offset = 33554432 contig_size = 67108864 lvl_flag = 1 [D 20:01:00.369881] calling distribute [D 20:01:00.369887] PINT_distribute [D 20:01:00.369893] of 33554432 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.369903] begin iteration loff: 33554432 [D 20:01:00.369910] segment exceeds byte limit [D 20:01:00.369917] process a segment [D 20:01:00.369923] of 33554432 sz 2097152 [D 20:01:00.369929] add a segment 0 [D 20:01:00.369936] end iteration [D 20:01:00.369942] done with segments or bytes [D 20:01:00.369948] of 35651584 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.369955] next loff: 35651584 next poff: 35651584 [D 20:01:00.369961] return value 2097152 [D 20:01:00.369973] finished [D 20:01:00.369980] retval = 2097152 [D 20:01:00.369986] exiting distribute returned less than expected [D 20:01:00.359625] op_queue add: 0x6ea5a0 [D 20:01:00.369989] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3060e0) [D 20:01:00.369992] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 35651584 fo 67108864 eof 0 [D 20:01:00.425668] ========================================================= [D 20:01:00.425677] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.425686] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.425698] dbpf_open_cache_put: cached entry. [D 20:01:00.370014] --- aio_progress_notification called with handle 2863311531 (0x2aaabd308d80) [D 20:01:00.425699] dbpf_bstream_rw_list: mem_offset: 0x2aaabda03010, mem_size: 2097152 [D 20:01:00.370024] DBPF I/O ops in progress: 15 [D 20:01:00.425712] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.370035] --- aio_progress_notification called with handle 2863311531 (0x2aaabd30e750) [D 20:01:00.425722] dbpf_bstream_rw_list: stream_offset: 33554432, stream_size: 2097152 [D 20:01:00.359578] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) -[D 20:01:00.425741] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.425766] dbpf_open_cache_put: cached entry. [D 20:01:00.370028] --- aio_progress_notification called with handle 2863311531 (0x2aaabd30ba20) [D 20:01:00.425744] dbpf_open_cache_get: called [D 20:01:00.425776] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.382700] --- aio_progress_notification called with handle 2863311531 (0x2aaabd319c10) [D 20:01:00.425786] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.425806] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.425769] openib_check_cq: completed recv wr_id 684cb0. [D 20:01:00.425837] openib_post_sr_rdmaw: ibv_post_send wr_id 66f9e0 to 10.1.4.56:39990 remote addr a2451000 rkey 108c27 nic_wr_credit 1. [D 20:01:00.425848] openib_post_rr: 10.1.4.56:39990 bh 92 wr_id 684cb0. [D 20:01:00.425857] openib_check_cq: completed recv wr_id 684ce0. [D 20:01:00.425866] openib_post_rr: 10.1.4.56:39990 bh 93 wr_id 684ce0. [D 20:01:00.425874] openib_check_cq: completed recv wr_id 684d10. [D 20:01:00.425881] openib_post_rr: 10.1.4.56:39990 bh 94 wr_id 684d10. [D 20:01:00.425889] openib_check_cq: completed recv wr_id 684d40. [D 20:01:00.425896] openib_post_rr: 10.1.4.56:39990 bh 95 wr_id 684d40. [D 20:01:00.425903] openib_check_cq: completed recv wr_id 684d70. [D 20:01:00.425911] openib_post_rr: 10.1.4.56:39990 bh 96 wr_id 684d70. [D 20:01:00.425918] openib_check_cq: completed recv wr_id 684da0. [D 20:01:00.425926] openib_post_rr: 10.1.4.56:39990 bh 97 wr_id 684da0. [D 20:01:00.425933] openib_check_cq: completed recv wr_id 684dd0. [D 20:01:00.425940] openib_post_rr: 10.1.4.56:39990 bh 98 wr_id 684dd0. [D 20:01:00.425947] openib_check_cq: completed recv wr_id 684e00. [D 20:01:00.425955] openib_post_rr: 10.1.4.56:39990 bh 99 wr_id 684e00. [D 20:01:00.425964] post_sr: 10.1.4.56:39990 bh 31 len 48 wr_id 682e70. [D 20:01:00.425972] openib_check_cq: completed recv wr_id 683b70. [D 20:01:00.425979] openib_post_rr: 10.1.4.56:39990 bh 0 wr_id 683b70. [D 20:01:00.425987] post_sr: 10.1.4.56:39990 bh 32 len 48 wr_id 682ea0. [D 20:01:00.425994] openib_check_cq: completed recv wr_id 683ba0. [D 20:01:00.426001] openib_post_rr: 10.1.4.56:39990 bh 1 wr_id 683ba0. [D 20:01:00.426009] post_sr: 10.1.4.56:39990 bh 33 len 48 wr_id 682ed0. [D 20:01:00.426016] openib_check_cq: completed recv wr_id 683bd0. [D 20:01:00.426023] openib_post_rr: 10.1.4.56:39990 bh 2 wr_id 683bd0. [D 20:01:00.426030] post_sr: 10.1.4.56:39990 bh 34 len 48 wr_id 682f00. [D 20:01:00.426038] openib_check_cq: completed recv wr_id 683c00. [D 20:01:00.426045] openib_post_rr: 10.1.4.56:39990 bh 3 wr_id 683c00. [D 20:01:00.426053] post_sr: 10.1.4.56:39990 bh 35 len 48 wr_id 682f30. [D 20:01:00.426061] openib_check_cq: completed recv wr_id 683c30. [D 20:01:00.426068] openib_post_rr: 10.1.4.56:39990 bh 4 wr_id 683c30. [D 20:01:00.426076] post_sr: 10.1.4.56:39990 bh 36 len 48 wr_id 682f60. [D 20:01:00.426084] openib_check_cq: completed recv wr_id 683c60. [D 20:01:00.426092] openib_post_rr: 10.1.4.56:39990 bh 5 wr_id 683c60. [D 20:01:00.426099] post_sr: 10.1.4.56:39990 bh 37 len 48 wr_id 682f90. [D 20:01:00.426107] openib_check_cq: completed recv wr_id 683c90. [D 20:01:00.426114] openib_post_rr: 10.1.4.56:39990 bh 6 wr_id 683c90. [D 20:01:00.426121] post_sr: 10.1.4.56:39990 bh 38 len 48 wr_id 682fc0. [D 20:01:00.426129] openib_check_cq: completed recv wr_id 683cc0. [D 20:01:00.426136] openib_post_rr: 10.1.4.56:39990 bh 7 wr_id 683cc0. [D 20:01:00.426144] post_sr: 10.1.4.56:39990 bh 39 len 48 wr_id 682ff0. [D 20:01:00.426152] openib_check_cq: completed recv wr_id 683cf0. [D 20:01:00.426161] openib_post_rr: 10.1.4.56:39990 bh 8 wr_id 683cf0. [D 20:01:00.426168] openib_check_cq: completed recv wr_id 683d20. [D 20:01:00.426176] openib_post_rr: 10.1.4.56:39990 bh 9 wr_id 683d20. [D 20:01:00.426193] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:00.426202] get_data_sync_mode returning 0 [D 20:01:00.426210] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.426217] dbpf_bstream_rw_list: mem_offset: 0x2aaab3410010, mem_size: 2097152 [D 20:01:00.426224] dbpf_bstream_rw_list: stream_offset: 425721856, stream_size: 2097152 [D 20:01:00.426231] dbpf_open_cache_get: called [D 20:01:00.426237] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.426243] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.426260] starting delayed I/O operation 0x2aaab15202d0 (15 in progress) [D 20:01:00.426271] lio_listio called with 1 following aiocbs: [D 20:01:00.426278] aiocb_ptr_array[0]: fd: 13, off: 14680064, bytes: 2097152, buf: 0x2aaabd802010, type: 0 [D 20:01:00.426290] start_delayed_ops_if_any: lio_listio posted 0x2aaab15202d0 (handle 2863311531, ret 0)) [D 20:01:00.426300] op_queue add: 0x2aaab152b040 [D 20:01:00.426308] delayed I/O operation 0x2aaab152b040 (16 already in progress) [D 20:01:00.426315] DBPF I/O ops in progress: 16 [D 20:01:00.426324] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.426332] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518130. [D 20:01:00.426342] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.429213] op_queue add: 0x2aaabd303560 [D 20:01:00.429291] delayed I/O operation 0x2aaabd303560 (16 already in progress) [D 20:01:00.429298] DBPF I/O ops in progress: 16 [D 20:01:00.429315] ========================================================= [D 20:01:00.429322] PINT_process_request [D 20:01:00.429324] ========================================================= [D 20:01:00.429343] PINT_process_request [D 20:01:00.429351] tiling 3 copies [D 20:01:00.370042] op_queue add: 0x6ed8e0 [D 20:01:00.429357] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.429366] lvl 0 el 0 blk 0 by 35651584 [D 20:01:00.429373] to 35651584 ta 33554432 fi 67108864 [D 20:01:00.429376] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:01:00.377257] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3141b0) [D 20:01:00.429330] tiling 14 copies [D 20:01:00.377329] --- aio_progress_notification called with handle 2863311531 (0x2aaabd311480) [D 20:01:00.429408] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.380504] --- aio_progress_notification called with handle 2863311531 (0x2aaabd316ee0) [D 20:01:00.392510] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3223a0) [D 20:01:00.429395] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:00.429436] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.429380] basic type or contiguous data [D 20:01:00.429456] Request disp [D 20:01:00.429460] dbpf_open_cache_put: cached entry. [D 20:01:00.392597] --- aio_progress_notification called with handle 2863311531 (0x2aaabd31f670) [D 20:01:00.433690] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.433708] dbpf_open_cache_put: cached entry. [D 20:01:00.433715] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.433726] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.433743] dbpf_open_cache_put: cached entry. [D 20:01:00.433751] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.392631] --- aio_progress_notification called with handle 2863311531 (0x2aaabd31c940) [D 20:01:00.433773] lvl 0 el 0 blk 0 by 427819008 [D 20:01:00.429462] contig_offset = 35651584 contig_size = 65011712 lvl_flag = 1 [D 20:01:00.433779] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.433810] calling distribute [D 20:01:00.433818] dbpf_open_cache_put: cached entry. [D 20:01:00.433830] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.429400] DBPF I/O ops in progress: 15 [D 20:01:00.433925] starting delayed I/O operation 0x2aaab152b040 (15 in progress) [D 20:01:00.433935] lio_listio called with 1 following aiocbs: [D 20:01:00.433941] aiocb_ptr_array[0]: fd: 13, off: 33554432, bytes: 2097152, buf: 0x2aaabda03010, type: 0 [D 20:01:00.433953] start_delayed_ops_if_any: lio_listio posted 0x2aaab152b040 (handle 2863311531, ret 0)) [D 20:01:00.429411] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.433978] dbpf_open_cache_put: cached entry. [D 20:01:00.433986] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.429427] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.429474] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.433821] PINT_distribute [D 20:01:00.434005] dbpf_open_cache_put: cached entry. [D 20:01:00.434022] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.434015] of 35651584 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.434204] begin iteration loff: 35651584 [D 20:01:00.434212] segment exceeds byte limit [D 20:01:00.434218] process a segment [D 20:01:00.434224] of 35651584 sz 2097152 [D 20:01:00.434230] add a segment 0 [D 20:01:00.434236] end iteration [D 20:01:00.434242] done with segments or bytes [D 20:01:00.434248] of 37748736 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.434255] next loff: 37748736 next poff: 37748736 [D 20:01:00.434261] return value 2097152 [D 20:01:00.434267] finished [D 20:01:00.434273] retval = 2097152 [D 20:01:00.434279] exiting distribute returned less than expected [D 20:01:00.434291] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 37748736 fo 67108864 eof 0 [D 20:01:00.434298] ========================================================= [D 20:01:00.434309] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.434317] dbpf_bstream_rw_list: mem_offset: 0x2aaabdc04010, mem_size: 2097152 [D 20:01:00.434323] dbpf_bstream_rw_list: stream_offset: 35651584, stream_size: 2097152 [D 20:01:00.434330] dbpf_open_cache_get: called [D 20:01:00.434337] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.434344] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.434362] op_queue add: 0x2aaab152de80 [D 20:01:00.434369] delayed I/O operation 0x2aaab152de80 (16 already in progress) [D 20:01:00.434375] DBPF I/O ops in progress: 16 [D 20:01:00.434382] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.434389] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518130. [D 20:01:00.434397] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.433804] to 427819008 ta 402653184 fi 436207616 [D 20:01:00.435906] basic type or contiguous data [D 20:01:00.435914] Request disp [D 20:01:00.435921] contig_offset = 427819008 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.435928] calling distribute [D 20:01:00.435934] PINT_distribute [D 20:01:00.435940] of 427819008 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 427819008 exfl 1 [D 20:01:00.435955] begin iteration loff: 427819008 [D 20:01:00.435962] segment exceeds byte limit [D 20:01:00.435968] file being extended [D 20:01:00.435975] process a segment [D 20:01:00.435981] of 427819008 sz 2097152 [D 20:01:00.435987] add a segment 0 [D 20:01:00.435993] end iteration [D 20:01:00.435999] done with segments or bytes [D 20:01:00.436005] of 429916160 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.436012] next loff: 429916160 next poff: 429916160 [D 20:01:00.436019] return value 2097152 [D 20:01:00.436025] finished [D 20:01:00.436031] retval = 2097152 [D 20:01:00.436037] exiting distribute returned less than expected [D 20:01:00.436043] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 429916160 fo 436207616 eof 0 [D 20:01:00.436050] ========================================================= [D 20:01:00.436058] BMI_post_recv: addr: 6843072, offset: 0x2aaab501e010, size: 2097152, tag: 20 [D 20:01:00.436071] openib_check_cq: completed recv wr_id 683d50. [D 20:01:00.436085] openib_post_rr: 10.1.4.56:39990 bh 10 wr_id 683d50. [D 20:01:00.436094] openib_check_cq: completed recv wr_id 683d80. [D 20:01:00.436102] openib_post_rr: 10.1.4.56:39990 bh 11 wr_id 683d80. [D 20:01:00.436110] openib_check_cq: completed rdma wr_id 66f9e0, nic_wr_credit 0. [D 20:01:00.429715] op_queue add: 0x2aaabd3060e0 [D 20:01:00.436504] op_queue add: 0x6ec5d0 [D 20:01:00.436517] DBPF I/O ops in progress: 15 [D 20:01:00.436524] starting delayed I/O operation 0x2aaabd303560 (15 in progress) [D 20:01:00.436534] lio_listio called with 1 following aiocbs: [D 20:01:00.436541] aiocb_ptr_array[0]: fd: 13, off: 425721856, bytes: 2097152, buf: 0x2aaab3410010, type: 1 [D 20:01:00.436551] start_delayed_ops_if_any: lio_listio posted 0x2aaabd303560 (handle 2863311531, ret 0)) [D 20:01:00.436576] DBPF I/O ops in progress: 15 [D 20:01:00.436584] starting delayed I/O operation 0x2aaab152de80 (15 in progress) [D 20:01:00.436593] lio_listio called with 1 following aiocbs: [D 20:01:00.436600] aiocb_ptr_array[0]: fd: 13, off: 35651584, bytes: 2097152, buf: 0x2aaabdc04010, type: 0 [D 20:01:00.436608] start_delayed_ops_if_any: lio_listio posted 0x2aaab152de80 (handle 2863311531, ret 0)) [D 20:01:00.393609] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b230) [D 20:01:00.436714] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.436733] dbpf_open_cache_put: cached entry. [D 20:01:00.436740] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.436746] op_queue add: 0x2aaab152b230 [D 20:01:00.436753] DBPF I/O ops in progress: 15 [D 20:01:00.395614] --- aio_progress_notification called with handle 2863311531 (0x2aaab151d770) [D 20:01:00.436771] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.436791] dbpf_open_cache_put: cached entry. [D 20:01:00.436799] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.436863] op_queue add: 0x2aaab151d770 [D 20:01:00.436918] DBPF I/O ops in progress: 14 [D 20:01:00.436925] post_sr: 10.1.4.56:39990 bh 40 len 16 wr_id 683020. [D 20:01:00.436945] openib_check_cq: completed send wr_id 682e70. [D 20:01:00.436955] openib_check_cq: completed send wr_id 682ea0. [D 20:01:00.436963] openib_check_cq: completed send wr_id 682ed0. [D 20:01:00.436970] openib_check_cq: completed send wr_id 682f00. [D 20:01:00.436976] openib_check_cq: completed send wr_id 682f30. [D 20:01:00.436983] openib_check_cq: completed send wr_id 682f60. [D 20:01:00.436990] openib_check_cq: completed send wr_id 682f90. [D 20:01:00.436996] openib_check_cq: completed send wr_id 682fc0. [D 20:01:00.437003] openib_check_cq: completed send wr_id 682ff0. [D 20:01:00.437010] openib_check_cq: completed recv wr_id 683db0. [D 20:01:00.437017] openib_post_rr: 10.1.4.56:39990 bh 12 wr_id 683db0. [D 20:01:00.437025] openib_check_cq: completed recv wr_id 683de0. [D 20:01:00.437042] openib_post_rr: 10.1.4.56:39990 bh 13 wr_id 683de0. [D 20:01:00.437050] openib_check_cq: completed send wr_id 683020. [D 20:01:00.437070] post_sr: 10.1.4.56:39990 bh 41 len 48 wr_id 683050. [D 20:01:00.437080] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:00.437088] get_data_sync_mode returning 0 [D 20:01:00.437098] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.437104] dbpf_bstream_rw_list: mem_offset: 0x2aaab3611010, mem_size: 2097152 [D 20:01:00.437111] dbpf_bstream_rw_list: stream_offset: 459276288, stream_size: 2097152 [D 20:01:00.437118] dbpf_open_cache_get: called [D 20:01:00.437124] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.437131] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.437159] DBPF I/O ops in progress: 15 [D 20:01:00.437168] lio_listio called with the following aiocbs: [D 20:01:00.437174] aiocb_ptr_array[0]: fd: 13, off: 459276288, bytes: 2097152, buf: 0x2aaab3611010, type: 1 [D 20:01:00.437183] issue_or_delay_io_operation: lio_listio posted 0x2aaabd300980 (handle 2863311531, ret 0) [D 20:01:00.437192] ========================================================= [D 20:01:00.437198] PINT_process_request [D 20:01:00.437205] tiling 15 copies [D 20:01:00.437212] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.437219] lvl 0 el 0 blk 0 by 461373440 [D 20:01:00.437226] to 461373440 ta 436207616 fi 469762048 [D 20:01:00.437232] basic type or contiguous data [D 20:01:00.437238] Request disp [D 20:01:00.437244] contig_offset = 461373440 contig_size = 41943040 lvl_flag = 1 [D 20:01:00.437251] calling distribute [D 20:01:00.437257] PINT_distribute [D 20:01:00.437262] of 461373440 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 461373440 exfl 1 [D 20:01:00.437270] begin iteration loff: 461373440 [D 20:01:00.437276] segment exceeds byte limit [D 20:01:00.437282] file being extended [D 20:01:00.437288] process a segment [D 20:01:00.437294] of 461373440 sz 2097152 [D 20:01:00.437300] add a segment 0 [D 20:01:00.437306] end iteration [D 20:01:00.437312] done with segments or bytes [D 20:01:00.437318] of 463470592 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.437325] next loff: 463470592 next poff: 463470592 [D 20:01:00.437331] return value 2097152 [D 20:01:00.437337] finished [D 20:01:00.437343] retval = 2097152 [D 20:01:00.437349] exiting distribute returned less than expected [D 20:01:00.437355] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 463470592 fo 469762048 eof 0 [D 20:01:00.437362] ========================================================= [D 20:01:00.437368] BMI_post_recv: addr: 6843072, offset: 0x2aaab5420010, size: 2097152, tag: 21 [D 20:01:00.437376] openib_check_cq: completed send wr_id 683050. [D 20:01:00.437387] post_sr: 10.1.4.56:39990 bh 42 len 48 wr_id 683080. [D 20:01:00.437395] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:01:00.437403] get_data_sync_mode returning 0 [D 20:01:00.437410] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.437417] dbpf_bstream_rw_list: mem_offset: 0x2aaab3812010, mem_size: 2097152 [D 20:01:00.437423] dbpf_bstream_rw_list: stream_offset: 25165824, stream_size: 2097152 [D 20:01:00.437430] dbpf_open_cache_get: called [D 20:01:00.437436] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.437442] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.437460] DBPF I/O ops in progress: 16 [D 20:01:00.437469] lio_listio called with the following aiocbs: [D 20:01:00.437475] aiocb_ptr_array[0]: fd: 13, off: 25165824, bytes: 2097152, buf: 0x2aaab3812010, type: 1 [D 20:01:00.437483] issue_or_delay_io_operation: lio_listio posted 0x2aaabd300b80 (handle 2863311531, ret 0) [D 20:01:00.437491] ========================================================= [D 20:01:00.437498] PINT_process_request [D 20:01:00.437505] tiling 2 copies [D 20:01:00.437511] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.437518] lvl 0 el 0 blk 0 by 27262976 [D 20:01:00.437524] to 27262976 ta 0 fi 33554432 [D 20:01:00.437531] basic type or contiguous data [D 20:01:00.437537] Request disp [D 20:01:00.437543] contig_offset = 27262976 contig_size = 39845888 lvl_flag = 1 [D 20:01:00.437550] calling distribute [D 20:01:00.437556] PINT_distribute [D 20:01:00.437562] of 27262976 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 27262976 exfl 1 [D 20:01:00.437569] begin iteration loff: 27262976 [D 20:01:00.437575] segment exceeds byte limit [D 20:01:00.437581] file being extended [D 20:01:00.437587] process a segment [D 20:01:00.437593] of 27262976 sz 2097152 [D 20:01:00.437599] add a segment 0 [D 20:01:00.437605] end iteration [D 20:01:00.437611] done with segments or bytes [D 20:01:00.437617] of 29360128 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.437624] next loff: 29360128 next poff: 29360128 [D 20:01:00.437630] return value 2097152 [D 20:01:00.437636] finished [D 20:01:00.437642] retval = 2097152 [D 20:01:00.437648] exiting distribute returned less than expected [D 20:01:00.437654] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 29360128 fo 33554432 eof 0 [D 20:01:00.437662] ========================================================= [D 20:01:00.437668] BMI_post_recv: addr: 6843072, offset: 0x2aaab5621010, size: 2097152, tag: 8 [D 20:01:00.437676] openib_check_cq: completed send wr_id 683080. [D 20:01:00.437686] post_sr: 10.1.4.56:39990 bh 43 len 48 wr_id 6830b0. [D 20:01:00.437695] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:01:00.437703] get_data_sync_mode returning 0 [D 20:01:00.437710] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.437716] dbpf_bstream_rw_list: mem_offset: 0x2aaab3a13010, mem_size: 2097152 [D 20:01:00.437723] dbpf_bstream_rw_list: stream_offset: 58720256, stream_size: 2097152 [D 20:01:00.437730] dbpf_open_cache_get: called [D 20:01:00.437736] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.437742] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.437760] op_queue add: 0x2aaabd300d80 [D 20:01:00.437768] delayed I/O operation 0x2aaabd300d80 (16 already in progress) [D 20:01:00.437775] DBPF I/O ops in progress: 16 [D 20:01:00.437782] ========================================================= [D 20:01:00.437788] PINT_process_request [D 20:01:00.437794] tiling 3 copies [D 20:01:00.437801] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.437808] lvl 0 el 0 blk 0 by 60817408 [D 20:01:00.437814] to 60817408 ta 33554432 fi 67108864 [D 20:01:00.437821] basic type or contiguous data [D 20:01:00.437827] Request disp [D 20:01:00.437833] contig_offset = 60817408 contig_size = 39845888 lvl_flag = 1 [D 20:01:00.437840] calling distribute [D 20:01:00.437846] PINT_distribute [D 20:01:00.437851] of 60817408 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 60817408 exfl 1 [D 20:01:00.437859] begin iteration loff: 60817408 [D 20:01:00.437865] segment exceeds byte limit [D 20:01:00.437871] file being extended [D 20:01:00.437877] process a segment [D 20:01:00.437883] of 60817408 sz 2097152 [D 20:01:00.437889] add a segment 0 [D 20:01:00.437895] end iteration [D 20:01:00.437901] done with segments or bytes [D 20:01:00.437907] of 62914560 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.437914] next loff: 62914560 next poff: 62914560 [D 20:01:00.437921] return value 2097152 [D 20:01:00.437927] finished [D 20:01:00.437933] retval = 2097152 [D 20:01:00.437938] exiting distribute returned less than expected [D 20:01:00.437945] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 62914560 fo 67108864 eof 0 [D 20:01:00.437952] ========================================================= [D 20:01:00.437958] BMI_post_recv: addr: 6843072, offset: 0x2aaab5822010, size: 2097152, tag: 9 [D 20:01:00.437967] openib_check_cq: completed send wr_id 6830b0. [D 20:01:00.437977] post_sr: 10.1.4.56:39990 bh 44 len 48 wr_id 6830e0. [D 20:01:00.437986] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:01:00.437994] get_data_sync_mode returning 0 [D 20:01:00.438000] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.438007] dbpf_bstream_rw_list: mem_offset: 0x2aaab3c14010, mem_size: 2097152 [D 20:01:00.438013] dbpf_bstream_rw_list: stream_offset: 92274688, stream_size: 2097152 [D 20:01:00.438020] dbpf_open_cache_get: called [D 20:01:00.438026] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.438032] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.438058] op_queue add: 0x2aaabd300f80 [D 20:01:00.438066] delayed I/O operation 0x2aaabd300f80 (16 already in progress) [D 20:01:00.438073] DBPF I/O ops in progress: 16 [D 20:01:00.438079] ========================================================= [D 20:01:00.438086] PINT_process_request [D 20:01:00.438092] tiling 4 copies [D 20:01:00.438098] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.438105] lvl 0 el 0 blk 0 by 94371840 [D 20:01:00.438112] to 94371840 ta 67108864 fi 100663296 [D 20:01:00.438118] basic type or contiguous data [D 20:01:00.438124] Request disp [D 20:01:00.438130] contig_offset = 94371840 contig_size = 39845888 lvl_flag = 1 [D 20:01:00.438137] calling distribute [D 20:01:00.438143] PINT_distribute [D 20:01:00.438149] of 94371840 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 94371840 exfl 1 [D 20:01:00.438156] begin iteration loff: 94371840 [D 20:01:00.438162] segment exceeds byte limit [D 20:01:00.438168] file being extended [D 20:01:00.438174] process a segment [D 20:01:00.438180] of 94371840 sz 2097152 [D 20:01:00.438186] add a segment 0 [D 20:01:00.438192] end iteration [D 20:01:00.438198] done with segments or bytes [D 20:01:00.438203] of 96468992 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.438210] next loff: 96468992 next poff: 96468992 [D 20:01:00.438217] return value 2097152 [D 20:01:00.438223] finished [D 20:01:00.438228] retval = 2097152 [D 20:01:00.438234] exiting distribute returned less than expected [D 20:01:00.438241] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 96468992 fo 100663296 eof 0 [D 20:01:00.438248] ========================================================= [D 20:01:00.438254] BMI_post_recv: addr: 6843072, offset: 0x2aaab5a23010, size: 2097152, tag: 10 [D 20:01:00.438262] openib_check_cq: completed send wr_id 6830e0. [D 20:01:00.438273] post_sr: 10.1.4.56:39990 bh 45 len 48 wr_id 683110. [D 20:01:00.438291] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:00.438301] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:00.438308] ========================================================= [D 20:01:00.438315] PINT_process_request [D 20:01:00.438320] tiling 2 copies [D 20:01:00.438326] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.438334] lvl 0 el 0 blk 0 by 16777216 [D 20:01:00.438340] to 16777216 ta 0 fi 33554432 [D 20:01:00.438346] basic type or contiguous data [D 20:01:00.438352] Request disp [D 20:01:00.438358] contig_offset = 16777216 contig_size = 50331648 lvl_flag = 1 [D 20:01:00.438365] calling distribute [D 20:01:00.438370] PINT_distribute [D 20:01:00.438376] of 16777216 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.438384] begin iteration loff: 16777216 [D 20:01:00.438390] segment exceeds byte limit [D 20:01:00.438395] process a segment [D 20:01:00.438401] of 16777216 sz 2097152 [D 20:01:00.438407] add a segment 0 [D 20:01:00.438413] end iteration [D 20:01:00.438419] done with segments or bytes [D 20:01:00.438425] of 18874368 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.438431] next loff: 18874368 next poff: 18874368 [D 20:01:00.438438] return value 2097152 [D 20:01:00.438444] finished [D 20:01:00.438451] retval = 2097152 [D 20:01:00.438456] exiting distribute returned less than expected [D 20:01:00.438462] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 18874368 fo 33554432 eof 0 [D 20:01:00.438470] ========================================================= [D 20:01:00.438478] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.438484] dbpf_bstream_rw_list: mem_offset: 0x2aaabc65c010, mem_size: 2097152 [D 20:01:00.438491] dbpf_bstream_rw_list: stream_offset: 16777216, stream_size: 2097152 [D 20:01:00.438497] dbpf_open_cache_get: called [D 20:01:00.438503] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.438510] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.438532] op_queue add: 0x2aaabd3011a0 [D 20:01:00.438540] delayed I/O operation 0x2aaabd3011a0 (16 already in progress) [D 20:01:00.438547] DBPF I/O ops in progress: 16 [D 20:01:00.438554] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:01:00.438561] get_data_sync_mode returning 0 [D 20:01:00.438568] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.438574] dbpf_bstream_rw_list: mem_offset: 0x2aaab3e15010, mem_size: 2097152 [D 20:01:00.438581] dbpf_bstream_rw_list: stream_offset: 125829120, stream_size: 2097152 [D 20:01:00.438587] dbpf_open_cache_get: called [D 20:01:00.438593] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.438600] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.438618] op_queue add: 0x2aaabd3012f0 [D 20:01:00.438627] delayed I/O operation 0x2aaabd3012f0 (16 already in progress) [D 20:01:00.438633] DBPF I/O ops in progress: 16 [D 20:01:00.438640] ========================================================= [D 20:01:00.438646] PINT_process_request [D 20:01:00.438652] tiling 5 copies [D 20:01:00.438658] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.438665] lvl 0 el 0 blk 0 by 127926272 [D 20:01:00.438672] to 127926272 ta 100663296 fi 134217728 [D 20:01:00.438678] basic type or contiguous data [D 20:01:00.438684] Request disp [D 20:01:00.438690] contig_offset = 127926272 contig_size = 39845888 lvl_flag = 1 [D 20:01:00.438697] calling distribute [D 20:01:00.438703] PINT_distribute [D 20:01:00.438708] of 127926272 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 127926272 exfl 1 [D 20:01:00.438716] begin iteration loff: 127926272 [D 20:01:00.438722] segment exceeds byte limit [D 20:01:00.438728] file being extended [D 20:01:00.438734] process a segment [D 20:01:00.438740] of 127926272 sz 2097152 [D 20:01:00.438746] add a segment 0 [D 20:01:00.438752] end iteration [D 20:01:00.438758] done with segments or bytes [D 20:01:00.438764] of 130023424 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.438771] next loff: 130023424 next poff: 130023424 [D 20:01:00.438777] return value 2097152 [D 20:01:00.438783] finished [D 20:01:00.438789] retval = 2097152 [D 20:01:00.438795] exiting distribute returned less than expected [D 20:01:00.438801] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 130023424 fo 134217728 eof 0 [D 20:01:00.438809] ========================================================= [D 20:01:00.438815] BMI_post_recv: addr: 6843072, offset: 0x2aaab5e26010, size: 2097152, tag: 11 [D 20:01:00.438823] openib_check_cq: completed send wr_id 683110. [D 20:01:00.438833] post_sr: 10.1.4.56:39990 bh 46 len 48 wr_id 683140. [D 20:01:00.438851] openib_check_cq: completed send wr_id 683140. [D 20:01:00.440572] ========================================================= [D 20:01:00.440583] PINT_process_request [D 20:01:00.440590] tiling 3 copies [D 20:01:00.440596] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.440603] lvl 0 el 0 blk 0 by 37748736 [D 20:01:00.440610] to 37748736 ta 33554432 fi 67108864 [D 20:01:00.440616] basic type or contiguous data [D 20:01:00.440622] Request disp [D 20:01:00.440847] contig_offset = 37748736 contig_size = 62914560 lvl_flag = 1 [D 20:01:00.440858] calling distribute [D 20:01:00.440864] PINT_distribute [D 20:01:00.440870] of 37748736 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.440877] begin iteration loff: 37748736 [D 20:01:00.440884] segment exceeds byte limit [D 20:01:00.440890] process a segment [D 20:01:00.440895] of 37748736 sz 2097152 [D 20:01:00.440902] add a segment 0 [D 20:01:00.440908] end iteration [D 20:01:00.440913] done with segments or bytes [D 20:01:00.440919] of 39845888 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.440926] next loff: 39845888 next poff: 39845888 [D 20:01:00.440932] return value 2097152 [D 20:01:00.440938] finished [D 20:01:00.440944] retval = 2097152 [D 20:01:00.440950] exiting distribute returned less than expected [D 20:01:00.440956] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 39845888 fo 67108864 eof 0 [D 20:01:00.440963] ========================================================= [D 20:01:00.440972] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.440979] dbpf_bstream_rw_list: mem_offset: 0x2aaabde05010, mem_size: 2097152 [D 20:01:00.440986] dbpf_bstream_rw_list: stream_offset: 37748736, stream_size: 2097152 [D 20:01:00.440992] dbpf_open_cache_get: called [D 20:01:00.440998] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.441005] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.441021] op_queue add: 0x2aaab152e050 [D 20:01:00.441027] delayed I/O operation 0x2aaab152e050 (16 already in progress) [D 20:01:00.441034] DBPF I/O ops in progress: 16 [D 20:01:00.441040] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.441047] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518130. [D 20:01:00.441054] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.443380] ========================================================= [D 20:01:00.443407] PINT_process_request [D 20:01:00.443414] tiling 3 copies [D 20:01:00.443421] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.443429] lvl 0 el 0 blk 0 by 39845888 [D 20:01:00.443435] to 39845888 ta 33554432 fi 67108864 [D 20:01:00.443442] basic type or contiguous data [D 20:01:00.443448] Request disp [D 20:01:00.443455] contig_offset = 39845888 contig_size = 60817408 lvl_flag = 1 [D 20:01:00.443462] calling distribute [D 20:01:00.443468] PINT_distribute [D 20:01:00.443474] of 39845888 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.443481] begin iteration loff: 39845888 [D 20:01:00.443488] segment exceeds byte limit [D 20:01:00.443494] process a segment [D 20:01:00.443500] of 39845888 sz 2097152 [D 20:01:00.443506] add a segment 0 [D 20:01:00.443512] end iteration [D 20:01:00.443518] done with segments or bytes [D 20:01:00.443524] of 41943040 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.443531] next loff: 41943040 next poff: 41943040 [D 20:01:00.443537] return value 2097152 [D 20:01:00.443544] finished [D 20:01:00.443550] retval = 2097152 [D 20:01:00.443556] exiting distribute returned less than expected [D 20:01:00.443562] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 41943040 fo 67108864 eof 0 [D 20:01:00.443569] ========================================================= [D 20:01:00.443580] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.443587] dbpf_bstream_rw_list: mem_offset: 0x2aaabe006010, mem_size: 2097152 [D 20:01:00.443594] dbpf_bstream_rw_list: stream_offset: 39845888, stream_size: 2097152 [D 20:01:00.443601] dbpf_open_cache_get: called [D 20:01:00.443607] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.443614] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.443632] op_queue add: 0x2aaab1522e30 [D 20:01:00.443640] delayed I/O operation 0x2aaab1522e30 (16 already in progress) [D 20:01:00.443647] DBPF I/O ops in progress: 16 [D 20:01:00.443653] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.443660] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518130. [D 20:01:00.443668] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.446091] ========================================================= [D 20:01:00.446125] PINT_process_request [D 20:01:00.446133] tiling 3 copies [D 20:01:00.446140] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.446147] lvl 0 el 0 blk 0 by 41943040 [D 20:01:00.446154] to 41943040 ta 33554432 fi 67108864 [D 20:01:00.446160] basic type or contiguous data [D 20:01:00.446166] Request disp [D 20:01:00.446172] contig_offset = 41943040 contig_size = 58720256 lvl_flag = 1 [D 20:01:00.446179] calling distribute [D 20:01:00.446185] PINT_distribute [D 20:01:00.446191] of 41943040 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.446198] begin iteration loff: 41943040 [D 20:01:00.446205] segment exceeds byte limit [D 20:01:00.446211] process a segment [D 20:01:00.446217] of 41943040 sz 2097152 [D 20:01:00.446223] add a segment 0 [D 20:01:00.446229] end iteration [D 20:01:00.446234] done with segments or bytes [D 20:01:00.446240] of 44040192 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.446247] next loff: 44040192 next poff: 44040192 [D 20:01:00.446254] return value 2097152 [D 20:01:00.446260] finished [D 20:01:00.446266] retval = 2097152 [D 20:01:00.446272] exiting distribute returned less than expected [D 20:01:00.446278] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 44040192 fo 67108864 eof 0 [D 20:01:00.446285] ========================================================= [D 20:01:00.446297] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.446304] dbpf_bstream_rw_list: mem_offset: 0x2aaabe207010, mem_size: 2097152 [D 20:01:00.446311] dbpf_bstream_rw_list: stream_offset: 41943040, stream_size: 2097152 [D 20:01:00.446318] dbpf_open_cache_get: called [D 20:01:00.446326] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.446332] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.446384] op_queue add: 0x2aaab15333d0 [D 20:01:00.446394] delayed I/O operation 0x2aaab15333d0 (16 already in progress) [D 20:01:00.446401] DBPF I/O ops in progress: 16 [D 20:01:00.446408] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.446415] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518130. [D 20:01:00.446426] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.450400] ========================================================= [D 20:01:00.450483] PINT_process_request [D 20:01:00.450491] tiling 3 copies [D 20:01:00.450498] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.450506] lvl 0 el 0 blk 0 by 44040192 [D 20:01:00.450513] to 44040192 ta 33554432 fi 67108864 [D 20:01:00.450520] basic type or contiguous data [D 20:01:00.450537] Request disp [D 20:01:00.450544] contig_offset = 44040192 contig_size = 56623104 lvl_flag = 1 [D 20:01:00.450551] calling distribute [D 20:01:00.450557] PINT_distribute [D 20:01:00.450563] of 44040192 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.450571] begin iteration loff: 44040192 [D 20:01:00.450577] segment exceeds byte limit [D 20:01:00.450584] process a segment [D 20:01:00.450589] of 44040192 sz 2097152 [D 20:01:00.450595] add a segment 0 [D 20:01:00.450602] end iteration [D 20:01:00.450607] done with segments or bytes [D 20:01:00.450613] of 46137344 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.450620] next loff: 46137344 next poff: 46137344 [D 20:01:00.450627] return value 2097152 [D 20:01:00.450633] finished [D 20:01:00.450639] retval = 2097152 [D 20:01:00.450645] exiting distribute returned less than expected [D 20:01:00.450651] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 46137344 fo 67108864 eof 0 [D 20:01:00.450659] ========================================================= [D 20:01:00.450689] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.450698] dbpf_bstream_rw_list: mem_offset: 0x2aaabe408010, mem_size: 2097152 [D 20:01:00.450705] dbpf_bstream_rw_list: stream_offset: 44040192, stream_size: 2097152 [D 20:01:00.450713] dbpf_open_cache_get: called [D 20:01:00.450719] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.450726] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.450747] op_queue add: 0x2aaab1535f30 [D 20:01:00.450755] delayed I/O operation 0x2aaab1535f30 (16 already in progress) [D 20:01:00.450762] DBPF I/O ops in progress: 16 [D 20:01:00.450769] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.450775] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518130. [D 20:01:00.450784] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.452448] ========================================================= [D 20:01:00.452499] op_queue add: 0x2aaabd30e750 [D 20:01:00.452503] PINT_process_request [D 20:01:00.452531] DBPF I/O ops in progress: 15 [D 20:01:00.452533] tiling 3 copies [D 20:01:00.452556] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.452558] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.452585] op_queue add: 0x2aaabd31f670 [D 20:01:00.452590] lvl 0 el 0 blk 0 by 46137344 [D 20:01:00.452609] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.452613] to 46137344 ta 33554432 fi 67108864 [D 20:01:00.452580] dbpf_open_cache_put: cached entry. [D 20:01:00.452636] basic type or contiguous data [D 20:01:00.452647] op_queue add: 0x2aaabd311480 [D 20:01:00.452649] Request disp [D 20:01:00.452668] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.452669] contig_offset = 46137344 contig_size = 54525952 lvl_flag = 1 [D 20:01:00.452699] calling distribute [D 20:01:00.452642] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.452722] op_queue add: 0x2aaabd308d80 [D 20:01:00.429736] --- aio_progress_notification called with handle 2863311531 (0x2aaab15202d0) [D 20:01:00.452708] PINT_distribute [D 20:01:00.452743] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.452746] of 46137344 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.452701] dbpf_open_cache_put: cached entry. [D 20:01:00.452767] begin iteration loff: 46137344 [D 20:01:00.452773] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.452778] segment exceeds byte limit [D 20:01:00.452546] starting delayed I/O operation 0x2aaabd300d80 (15 in progress) [D 20:01:00.452818] lio_listio called with 1 following aiocbs: [D 20:01:00.452825] aiocb_ptr_array[0]: fd: 13, off: 58720256, bytes: 2097152, buf: 0x2aaab3a13010, type: 1 [D 20:01:00.452847] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300d80 (handle 2863311531, ret 0)) [D 20:01:00.452737] op_queue add: 0x2aaabd31c940 [D 20:01:00.452800] process a segment [D 20:01:00.452980] DBPF I/O ops in progress: 15 [D 20:01:00.452984] of 46137344 sz 2097152 [D 20:01:00.453002] op_queue add: 0x2aaabd3141b0 [D 20:01:00.452995] starting delayed I/O operation 0x2aaabd300f80 (15 in progress) [D 20:01:00.453007] add a segment 0 [D 20:01:00.453039] lio_listio called with 1 following aiocbs: [D 20:01:00.453061] aiocb_ptr_array[0]: fd: 13, off: 92274688, bytes: 2097152, buf: 0x2aaab3c14010, type: 1 [D 20:01:00.453052] end iteration [D 20:01:00.453070] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300f80 (handle 2863311531, ret 0)) [D 20:01:00.453075] done with segments or bytes [D 20:01:00.453092] of 48234496 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.452792] dbpf_open_cache_put: cached entry. [D 20:01:00.453116] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.453046] op_queue add: 0x2aaabd3223a0 [D 20:01:00.453140] DBPF I/O ops in progress: 15 [D 20:01:00.453149] starting delayed I/O operation 0x2aaabd3011a0 (15 in progress) [D 20:01:00.453159] lio_listio called with 1 following aiocbs: [D 20:01:00.453165] aiocb_ptr_array[0]: fd: 13, off: 16777216, bytes: 2097152, buf: 0x2aaabc65c010, type: 0 [D 20:01:00.453101] next loff: 48234496 next poff: 48234496 [D 20:01:00.453178] dbpf_open_cache_put: cached entry. [D 20:01:00.453184] return value 2097152 [D 20:01:00.453192] op_queue add: 0x2aaabd316ee0 [D 20:01:00.453195] finished [D 20:01:00.453220] retval = 2097152 [D 20:01:00.453229] exiting distribute returned less than expected [D 20:01:00.453236] op_queue add: 0x2aaabd319c10 [D 20:01:00.453222] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:00.453174] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3011a0 (handle 2863311531, ret 0)) [D 20:01:00.453240] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 48234496 fo 67108864 eof 0 [D 20:01:00.453273] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:00.453297] BMI_post_send: addr: 6843072, offset: 0x2aaabb654010, size: 2097152, tag: 36 [D 20:01:00.453284] ========================================================= [D 20:01:00.453313] post_sr: 10.1.4.56:39990 bh 47 len 32 wr_id 683170. [D 20:01:00.453323] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 -[D 20:01:00.453334] dbpf_bstream_rw_list: mem_offset: 0x2aaabe609010, mem_size: 2097152 [D 20:01:00.453342] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:00.453383] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:00.453392] BMI_post_send: addr: 6843072, offset: 0x2aaabd400010, size: 2097152, tag: 23 [D 20:01:00.453188] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.453352] op_queue add: 0x2aaabd30ba20 [D 20:01:00.453364] openib_check_cq: completed send wr_id 683170. [D 20:01:00.453355] DBPF I/O ops in progress: 15 [D 20:01:00.453442] starting delayed I/O operation 0x2aaabd3012f0 (15 in progress) [D 20:01:00.453451] post_sr: 10.1.4.56:39990 bh 48 len 32 wr_id 6831a0. [D 20:01:00.453485] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:00.453490] openib_check_cq: completed send wr_id 6831a0. [D 20:01:00.453495] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:00.453509] BMI_post_send: addr: 6843072, offset: 0x2aaabd601010, size: 2097152, tag: 23 [D 20:01:00.453371] dbpf_bstream_rw_list: stream_offset: 46137344, stream_size: 2097152 [D 20:01:00.453518] post_sr: 10.1.4.56:39990 bh 49 len 32 wr_id 6831d0. [D 20:01:00.453522] dbpf_open_cache_get: called [D 20:01:00.453531] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:00.453424] op_queue add: 0x2aaab15202d0 [D 20:01:00.453464] lio_listio called with 1 following aiocbs: [D 20:01:00.453536] openib_check_cq: completed send wr_id 6831d0. [D 20:01:00.453550] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.453576] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.454537] openib_check_cq: completed recv wr_id 683e10. [D 20:01:00.454556] openib_post_sr_rdmaw: ibv_post_send wr_id 6eda80 to 10.1.4.56:39990 remote addr 88637000 rkey 108a27 nic_wr_credit 1. [D 20:01:00.454565] openib_post_rr: 10.1.4.56:39990 bh 14 wr_id 683e10. [D 20:01:00.456897] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b040) [D 20:01:00.456941] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.456952] dbpf_open_cache_put: cached entry. [D 20:01:00.456959] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.456967] op_queue add: 0x2aaab152b040 [D 20:01:00.456980] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303560) [D 20:01:00.456989] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.456999] dbpf_open_cache_put: cached entry. [D 20:01:00.457005] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.457011] op_queue add: 0x2aaabd303560 [D 20:01:00.457372] openib_check_cq: completed recv wr_id 683e40. [D 20:01:00.457391] openib_post_rr: 10.1.4.56:39990 bh 15 wr_id 683e40. [D 20:01:00.457405] openib_check_cq: completed rdma wr_id 6eda80, nic_wr_credit 0. [D 20:01:00.457415] post_sr: 10.1.4.56:39990 bh 50 len 16 wr_id 683200. [D 20:01:00.457427] openib_check_cq: completed recv wr_id 683e70. [D 20:01:00.457435] openib_post_rr: 10.1.4.56:39990 bh 16 wr_id 683e70. [D 20:01:00.457443] openib_check_cq: completed send wr_id 683200. [D 20:01:00.457466] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:00.457478] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:00.457487] ========================================================= [D 20:01:00.457494] PINT_process_request [D 20:01:00.457502] tiling 15 copies [D 20:01:00.457509] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.457517] lvl 0 el 0 blk 0 by 455081984 [D 20:01:00.457524] to 455081984 ta 436207616 fi 469762048 [D 20:01:00.457532] basic type or contiguous data [D 20:01:00.457538] Request disp [D 20:01:00.457546] contig_offset = 455081984 contig_size = 48234496 lvl_flag = 1 [D 20:01:00.457554] calling distribute [D 20:01:00.457561] PINT_distribute [D 20:01:00.457568] of 455081984 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:00.457579] begin iteration loff: 455081984 [D 20:01:00.457587] segment exceeds byte limit [D 20:01:00.457594] process a segment [D 20:01:00.457600] of 455081984 sz 2097152 [D 20:01:00.457607] add a segment 0 [D 20:01:00.457614] end iteration [D 20:01:00.457620] done with segments or bytes [D 20:01:00.457627] of 457179136 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.457634] next loff: 457179136 next poff: 457179136 [D 20:01:00.457641] return value 2097152 [D 20:01:00.457648] finished [D 20:01:00.457654] retval = 2097152 [D 20:01:00.457660] exiting distribute returned less than expected [D 20:01:00.457667] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 457179136 fo 469762048 eof 0 [D 20:01:00.457675] ========================================================= [D 20:01:00.457697] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.457705] dbpf_bstream_rw_list: mem_offset: 0x2aaabb855010, mem_size: 2097152 [D 20:01:00.457712] dbpf_bstream_rw_list: stream_offset: 455081984, stream_size: 2097152 [D 20:01:00.457721] dbpf_open_cache_get: called [D 20:01:00.457730] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.457738] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.453558] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:00.457817] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:00.457828] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:00.457836] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:01:00.457846] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:00.457856] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:00.457874] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:00.457884] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:00.457893] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:00.457902] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:01:00.457918] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:00.457930] BMI_post_send: addr: 6843072, offset: 0x2aaabd802010, size: 2097152, tag: 23 [D 20:01:00.457949] post_sr: 10.1.4.56:39990 bh 51 len 32 wr_id 683230. [D 20:01:00.457999] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:00.458022] aiocb_ptr_array[0]: fd: 13, off: 125829120, bytes: 2097152, buf: 0x2aaab3e15010, type: 1 [D 20:01:00.458044] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3012f0 (handle 2863311531, ret 0)) [D 20:01:00.458105] DBPF I/O ops in progress: 15 [D 20:01:00.458120] starting delayed I/O operation 0x2aaab152e050 (15 in progress) [D 20:01:00.458131] lio_listio called with 1 following aiocbs: [D 20:01:00.458138] aiocb_ptr_array[0]: fd: 13, off: 37748736, bytes: 2097152, buf: 0x2aaabde05010, type: 0 [D 20:01:00.458149] start_delayed_ops_if_any: lio_listio posted 0x2aaab152e050 (handle 2863311531, ret 0)) [D 20:01:00.458221] DBPF I/O ops in progress: 15 [D 20:01:00.458237] starting delayed I/O operation 0x2aaab1522e30 (15 in progress) [D 20:01:00.458250] lio_listio called with 1 following aiocbs: [D 20:01:00.458257] aiocb_ptr_array[0]: fd: 13, off: 39845888, bytes: 2097152, buf: 0x2aaabe006010, type: 0 [D 20:01:00.458266] start_delayed_ops_if_any: lio_listio posted 0x2aaab1522e30 (handle 2863311531, ret 0)) [D 20:01:00.458310] DBPF I/O ops in progress: 15 [D 20:01:00.458323] starting delayed I/O operation 0x2aaab15333d0 (15 in progress) [D 20:01:00.458336] lio_listio called with 1 following aiocbs: [D 20:01:00.458343] aiocb_ptr_array[0]: fd: 13, off: 41943040, bytes: 2097152, buf: 0x2aaabe207010, type: 0 [D 20:01:00.458354] start_delayed_ops_if_any: lio_listio posted 0x2aaab15333d0 (handle 2863311531, ret 0)) [D 20:01:00.458393] DBPF I/O ops in progress: 15 [D 20:01:00.458405] starting delayed I/O operation 0x2aaab1535f30 (15 in progress) [D 20:01:00.458417] lio_listio called with 1 following aiocbs: [D 20:01:00.458423] aiocb_ptr_array[0]: fd: 13, off: 44040192, bytes: 2097152, buf: 0x2aaabe408010, type: 0 [D 20:01:00.458433] start_delayed_ops_if_any: lio_listio posted 0x2aaab1535f30 (handle 2863311531, ret 0)) [D 20:01:00.458476] DBPF I/O ops in progress: 15 [D 20:01:00.458521] DBPF I/O ops in progress: 14 [D 20:01:00.458564] DBPF I/O ops in progress: 13 [D 20:01:00.458589] DBPF I/O ops in progress: 14 [D 20:01:00.458603] lio_listio called with the following aiocbs: [D 20:01:00.458610] aiocb_ptr_array[0]: fd: 13, off: 46137344, bytes: 2097152, buf: 0x2aaabe609010, type: 0 [D 20:01:00.458620] issue_or_delay_io_operation: lio_listio posted 0x2aaab152b210 (handle 2863311531, ret 0) [D 20:01:00.458629] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.458636] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1518130. [D 20:01:00.458644] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.460216] ========================================================= [D 20:01:00.460247] PINT_process_request [D 20:01:00.460254] tiling 3 copies [D 20:01:00.460261] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.460269] lvl 0 el 0 blk 0 by 48234496 [D 20:01:00.460275] to 48234496 ta 33554432 fi 67108864 [D 20:01:00.460282] basic type or contiguous data [D 20:01:00.460289] Request disp [D 20:01:00.460295] contig_offset = 48234496 contig_size = 52428800 lvl_flag = 1 [D 20:01:00.460302] calling distribute [D 20:01:00.460308] PINT_distribute [D 20:01:00.460314] of 48234496 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.460322] begin iteration loff: 48234496 [D 20:01:00.460329] segment exceeds byte limit [D 20:01:00.460335] process a segment [D 20:01:00.460341] of 48234496 sz 2097152 [D 20:01:00.460347] add a segment 0 [D 20:01:00.460353] end iteration [D 20:01:00.460359] done with segments or bytes [D 20:01:00.460365] of 50331648 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.460373] next loff: 50331648 next poff: 50331648 [D 20:01:00.460380] return value 2097152 [D 20:01:00.460386] finished [D 20:01:00.460392] retval = 2097152 [D 20:01:00.460398] exiting distribute returned less than expected [D 20:01:00.460404] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 50331648 fo 67108864 eof 0 [D 20:01:00.460411] ========================================================= [D 20:01:00.460423] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.460430] dbpf_bstream_rw_list: mem_offset: 0x2aaabe80a010, mem_size: 2097152 [D 20:01:00.460437] dbpf_bstream_rw_list: stream_offset: 48234496, stream_size: 2097152 [D 20:01:00.460444] dbpf_open_cache_get: called [D 20:01:00.460451] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.460457] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.460473] DBPF I/O ops in progress: 15 [D 20:01:00.460492] lio_listio called with the following aiocbs: [D 20:01:00.460499] aiocb_ptr_array[0]: fd: 13, off: 48234496, bytes: 2097152, buf: 0x2aaabe80a010, type: 0 [D 20:01:00.460510] issue_or_delay_io_operation: lio_listio posted 0x2aaab152b470 (handle 2863311531, ret 0) [D 20:01:00.460521] flowproto posted 0x2aaab1518130 [D 20:01:00.460524] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:00.460551] BMI_post_send: addr: 6843072, offset: 0x2aaabda03010, size: 2097152, tag: 24 [D 20:01:00.460572] post_sr: 10.1.4.56:39990 bh 52 len 32 wr_id 683260. [D 20:01:00.460614] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:00.460626] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:00.460531] Job flows in progress (post time): 18 [D 20:01:00.460672] [SM Exiting]: (0x660710) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:00.460686] [SM Entering]: (0x6619d0) io_sm:start_flow (status: 0) [D 20:01:00.460695] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:00.460705] io_start_flow() issuing flow to read data. [D 20:01:00.460712] flow: fsize: 440401920, server_nr: 0, server_ct: 1 [D 20:01:00.460719] file_req_offset: 67108864, aggregate_size: 33554432, handle: 2863311531 [D 20:01:00.460734] PINT_new_request_states n=1 [D 20:01:00.460745] flowproto posting 0x2aaab152b630 [D 20:01:00.460764] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.460772] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b630. [D 20:01:00.460779] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:00.464042] ========================================================= [D 20:01:00.464127] PINT_process_request [D 20:01:00.464136] tiling 4 copies [D 20:01:00.464143] skipping ahead to target_offset [D 20:01:00.464151] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.464159] lvl 0 el 0 blk 0 by 0 [D 20:01:00.464166] to 0 ta 67108864 fi 100663296 [D 20:01:00.464173] basic type or contiguous data [D 20:01:00.464179] Request disp [D 20:01:00.464186] contig_offset = 0 contig_size = 134217728 lvl_flag = 1 [D 20:01:00.464193] process logical skip [D 20:01:00.464199] retval = 67108864 [D 20:01:00.464205] exiting logical skip because distribute indicates done [D 20:01:00.464212] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.464219] lvl 0 el 0 blk 0 by 67108864 [D 20:01:00.464226] to 67108864 ta 67108864 fi 100663296 [D 20:01:00.464232] basic type or contiguous data [D 20:01:00.464238] Request disp [D 20:01:00.464244] contig_offset = 67108864 contig_size = 67108864 lvl_flag = 1 [D 20:01:00.464252] calling distribute [D 20:01:00.464259] PINT_distribute [D 20:01:00.464267] of 67108864 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.464276] begin iteration loff: 67108864 [D 20:01:00.464283] segment exceeds byte limit [D 20:01:00.464290] process a segment [D 20:01:00.464298] of 67108864 sz 2097152 [D 20:01:00.464306] add a segment 0 [D 20:01:00.464312] end iteration [D 20:01:00.464319] done with segments or bytes [D 20:01:00.464324] of 69206016 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.464332] next loff: 69206016 next poff: 69206016 [D 20:01:00.464339] return value 2097152 [D 20:01:00.464347] finished [D 20:01:00.464353] retval = 2097152 [D 20:01:00.464359] exiting distribute returned less than expected [D 20:01:00.464366] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 69206016 fo 100663296 eof 0 [D 20:01:00.464373] ========================================================= [D 20:01:00.464394] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.464402] dbpf_bstream_rw_list: mem_offset: 0x2aaabea0b010, mem_size: 2097152 [D 20:01:00.464411] dbpf_bstream_rw_list: stream_offset: 67108864, stream_size: 2097152 [D 20:01:00.464418] dbpf_open_cache_get: called [D 20:01:00.464426] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.464432] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.464528] DBPF I/O ops in progress: 16 [D 20:01:00.464539] lio_listio called with the following aiocbs: [D 20:01:00.464546] aiocb_ptr_array[0]: fd: 13, off: 67108864, bytes: 2097152, buf: 0x2aaabea0b010, type: 0 [D 20:01:00.464565] issue_or_delay_io_operation: lio_listio posted 0x2aaab1522750 (handle 2863311531, ret 0) [D 20:01:00.464574] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.464582] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b630. [D 20:01:00.464592] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.467610] ========================================================= [D 20:01:00.467686] PINT_process_request [D 20:01:00.467697] tiling 4 copies [D 20:01:00.467705] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.467713] lvl 0 el 0 blk 0 by 69206016 [D 20:01:00.467720] to 69206016 ta 67108864 fi 100663296 [D 20:01:00.467727] basic type or contiguous data [D 20:01:00.467735] Request disp [D 20:01:00.467743] contig_offset = 69206016 contig_size = 65011712 lvl_flag = 1 [D 20:01:00.467751] calling distribute [D 20:01:00.467758] PINT_distribute [D 20:01:00.467767] of 69206016 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.467776] begin iteration loff: 69206016 [D 20:01:00.467783] segment exceeds byte limit [D 20:01:00.467789] process a segment [D 20:01:00.467795] of 69206016 sz 2097152 [D 20:01:00.467802] add a segment 0 [D 20:01:00.467810] end iteration [D 20:01:00.467816] done with segments or bytes [D 20:01:00.467822] of 71303168 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.467830] next loff: 71303168 next poff: 71303168 [D 20:01:00.467838] return value 2097152 [D 20:01:00.467846] finished [D 20:01:00.467853] retval = 2097152 [D 20:01:00.467860] exiting distribute returned less than expected [D 20:01:00.467867] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 71303168 fo 100663296 eof 0 [D 20:01:00.467874] ========================================================= [D 20:01:00.467895] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.467904] dbpf_bstream_rw_list: mem_offset: 0x2aaabec0c010, mem_size: 2097152 [D 20:01:00.467911] dbpf_bstream_rw_list: stream_offset: 69206016, stream_size: 2097152 [D 20:01:00.467922] dbpf_open_cache_get: called [D 20:01:00.467930] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.467937] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.467995] op_queue add: 0x2aaab1522990 [D 20:01:00.468007] delayed I/O operation 0x2aaab1522990 (16 already in progress) [D 20:01:00.468014] DBPF I/O ops in progress: 16 [D 20:01:00.468021] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.468029] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b630. [D 20:01:00.468039] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.470985] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300980) [D 20:01:00.471123] ========================================================= [D 20:01:00.471135] PINT_process_request [D 20:01:00.471142] tiling 4 copies [D 20:01:00.471150] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.471158] lvl 0 el 0 blk 0 by 71303168 [D 20:01:00.471165] to 71303168 ta 67108864 fi 100663296 [D 20:01:00.471172] basic type or contiguous data [D 20:01:00.471178] Request disp [D 20:01:00.471185] contig_offset = 71303168 contig_size = 62914560 lvl_flag = 1 [D 20:01:00.471192] calling distribute [D 20:01:00.471199] PINT_distribute [D 20:01:00.471207] of 71303168 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.471215] begin iteration loff: 71303168 [D 20:01:00.471223] segment exceeds byte limit [D 20:01:00.471230] process a segment [D 20:01:00.471236] of 71303168 sz 2097152 [D 20:01:00.471243] add a segment 0 [D 20:01:00.471250] end iteration [D 20:01:00.471256] done with segments or bytes [D 20:01:00.471262] of 73400320 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.471269] next loff: 73400320 next poff: 73400320 [D 20:01:00.471276] return value 2097152 [D 20:01:00.471283] finished [D 20:01:00.471289] retval = 2097152 [D 20:01:00.471296] exiting distribute returned less than expected [D 20:01:00.471302] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 73400320 fo 100663296 eof 0 [D 20:01:00.471309] ========================================================= [D 20:01:00.471327] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.471335] dbpf_bstream_rw_list: mem_offset: 0x2aaabee0d010, mem_size: 2097152 [D 20:01:00.471342] dbpf_bstream_rw_list: stream_offset: 71303168, stream_size: 2097152 [D 20:01:00.471350] dbpf_open_cache_get: called [D 20:01:00.471358] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.471366] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.471422] op_queue add: 0x2aaab1522b60 [D 20:01:00.471435] delayed I/O operation 0x2aaab1522b60 (16 already in progress) [D 20:01:00.471442] DBPF I/O ops in progress: 16 [D 20:01:00.471450] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.471459] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b630. [D 20:01:00.471470] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.471100] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.474174] dbpf_open_cache_put: cached entry. [D 20:01:00.474230] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.474246] op_queue add: 0x2aaabd300980 [D 20:01:00.474281] ========================================================= [D 20:01:00.474292] PINT_process_request [D 20:01:00.474300] tiling 4 copies [D 20:01:00.474308] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.474316] lvl 0 el 0 blk 0 by 73400320 [D 20:01:00.474322] to 73400320 ta 67108864 fi 100663296 [D 20:01:00.474330] basic type or contiguous data [D 20:01:00.474337] Request disp [D 20:01:00.474344] contig_offset = 73400320 contig_size = 60817408 lvl_flag = 1 [D 20:01:00.474352] calling distribute [D 20:01:00.474359] PINT_distribute [D 20:01:00.474365] of 73400320 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:00.474374] begin iteration loff: 73400320 [D 20:01:00.474381] segment exceeds byte limit [D 20:01:00.474387] process a segment [D 20:01:00.474393] of 73400320 sz 2097152 [D 20:01:00.474400] add a segment 0 [D 20:01:00.474406] end iteration [D 20:01:00.474412] done with segments or bytes [D 20:01:00.474419] of 75497472 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:00.474429] next loff: 75497472 next poff: 75497472 [D 20:01:00.474436] return value 2097152 [D 20:01:00.474443] finished [D 20:01:00.474449] retval = 2097152 [D 20:01:00.474455] exiting distribute returned less than expected [D 20:01:00.474461] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 75497472 fo 100663296 eof 0 [D 20:01:00.474468] ========================================================= [D 20:01:00.474485] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.474493] dbpf_bstream_rw_list: mem_offset: 0x2aaabf00e010, mem_size: 2097152 [D 20:01:00.474499] dbpf_bstream_rw_list: stream_offset: 73400320, stream_size: 2097152 [D 20:01:00.474507] dbpf_open_cache_get: called [D 20:01:00.474514] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.474521] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.474570] op_queue add: 0x2aaab152bb90 [D 20:01:00.474581] delayed I/O operation 0x2aaab152bb90 (16 already in progress) [D 20:01:00.474588] DBPF I/O ops in progress: 16 [D 20:01:00.474596] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.474604] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b630. [D 20:01:00.474613] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.476518] DBPF I/O ops in progress: 15 [D 20:01:00.476600] starting delayed I/O operation 0x2aaab1522990 (15 in progress) [D 20:01:00.476613] lio_listio called with 1 following aiocbs: [D 20:01:00.476620] aiocb_ptr_array[0]: fd: 13, off: 69206016, bytes: 2097152, buf: 0x2aaabec0c010, type: 0 [D 20:01:00.476640] start_delayed_ops_if_any: lio_listio posted 0x2aaab1522990 (handle 2863311531, ret 0)) [D 20:01:00.476665] --- aio_progress_notification called with handle 2863311531 (0x2aaab152de80) [D 20:01:00.476680] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.476693] dbpf_open_cache_put: cached entry. [D 20:01:00.476700] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.476715] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300b80) [D 20:01:00.476727] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.476737] dbpf_open_cache_put: cached entry. [D 20:01:00.476744] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.476766] op_queue add: 0x2aaabd3112d0 [D 20:01:00.476782] delayed I/O operation 0x2aaabd3112d0 (16 already in progress) [D 20:01:00.476789] DBPF I/O ops in progress: 16 [D 20:01:00.476802] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:01:00.476814] get_data_sync_mode returning 0 [D 20:01:00.476828] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:00.476835] dbpf_bstream_rw_list: mem_offset: 0x2aaab4016010, mem_size: 2097152 [D 20:01:00.476842] dbpf_bstream_rw_list: stream_offset: 159383552, stream_size: 2097152 [D 20:01:00.476850] dbpf_open_cache_get: called [D 20:01:00.476856] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:00.476863] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.476888] op_queue add: 0x2aaabd308af0 [D 20:01:00.476895] delayed I/O operation 0x2aaabd308af0 (16 already in progress) [D 20:01:00.476901] DBPF I/O ops in progress: 16 [D 20:01:00.476911] ========================================================= [D 20:01:00.476919] PINT_process_request [D 20:01:00.476927] tiling 6 copies [D 20:01:00.476934] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:00.476941] lvl 0 el 0 blk 0 by 161480704 [D 20:01:00.476948] to 161480704 ta 134217728 fi 167772160 [D 20:01:00.476956] basic type or contiguous data [D 20:01:00.476962] Request disp [D 20:01:00.476969] contig_offset = 161480704 contig_size = 39845888 lvl_flag = 1 [D 20:01:00.476977] calling distribute [D 20:01:00.476983] PINT_distribute [D 20:01:00.476990] of 161480704 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 161480704 exfl 1 [D 20:01:02.233221] begin iteration loff: 161480704 [D 20:01:02.233315] segment exceeds byte limit [D 20:01:02.233323] file being extended [D 20:01:02.233329] process a segment [D 20:01:02.233395] of 161480704 sz 2097152 [D 20:01:02.233402] add a segment 0 [D 20:01:02.233489] end iteration [D 20:01:02.233496] done with segments or bytes [D 20:01:02.233502] of 163577856 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.233509] next loff: 163577856 next poff: 163577856 [D 20:01:02.233515] return value 2097152 [D 20:01:02.233522] finished [D 20:01:02.233528] retval = 2097152 [D 20:01:02.233534] exiting distribute returned less than expected [D 20:01:02.233540] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 163577856 fo 167772160 eof 0 [D 20:01:02.233547] ========================================================= [D 20:01:02.233554] BMI_post_recv: addr: 6843072, offset: 0x2aaab6027010, size: 2097152, tag: 12 [D 20:01:02.233569] openib_check_cq: completed send wr_id 683230. [D 20:01:02.233577] openib_check_cq: completed send wr_id 683260. [D 20:01:02.233585] openib_check_cq: completed recv wr_id 683ea0. [D 20:01:02.233594] openib_post_rr: 10.1.4.56:39990 bh 17 wr_id 683ea0. [D 20:01:02.233603] openib_check_cq: completed recv wr_id 683ed0. [D 20:01:02.233610] openib_post_rr: 10.1.4.56:39990 bh 18 wr_id 683ed0. [D 20:01:02.233617] openib_check_cq: completed recv wr_id 683f00. [D 20:01:02.233625] openib_post_rr: 10.1.4.56:39990 bh 19 wr_id 683f00. [D 20:01:02.233632] openib_check_cq: completed recv wr_id 683f30. [D 20:01:02.233639] openib_post_rr: 10.1.4.56:39990 bh 20 wr_id 683f30. [D 20:01:02.233646] openib_check_cq: completed recv wr_id 683f60. [D 20:01:02.233654] openib_post_rr: 10.1.4.56:39990 bh 21 wr_id 683f60. [D 20:01:02.233661] openib_check_cq: completed recv wr_id 683f90. [D 20:01:02.233668] openib_post_rr: 10.1.4.56:39990 bh 22 wr_id 683f90. [D 20:01:02.233675] openib_check_cq: completed recv wr_id 683fc0. [D 20:01:02.233683] openib_post_rr: 10.1.4.56:39990 bh 23 wr_id 683fc0. [D 20:01:02.233690] openib_check_cq: completed recv wr_id 683ff0. [D 20:01:02.233697] openib_post_rr: 10.1.4.56:39990 bh 24 wr_id 683ff0. [D 20:01:02.233704] openib_check_cq: completed recv wr_id 684020. [D 20:01:02.233715] openib_post_sr_rdmaw: ibv_post_send wr_id 6df4e0 to 10.1.4.56:39990 remote addr a044f000 rkey 108d27 nic_wr_credit 1. [D 20:01:02.233724] openib_post_rr: 10.1.4.56:39990 bh 25 wr_id 684020. [D 20:01:02.233731] openib_check_cq: completed recv wr_id 684050. [D 20:01:02.233738] openib_post_rr: 10.1.4.56:39990 bh 26 wr_id 684050. [D 20:01:02.233745] openib_check_cq: completed recv wr_id 684080. [D 20:01:02.233753] openib_post_rr: 10.1.4.56:39990 bh 27 wr_id 684080. [D 20:01:02.233760] openib_check_cq: completed recv wr_id 6840b0. [D 20:01:02.233767] openib_post_rr: 10.1.4.56:39990 bh 28 wr_id 6840b0. [D 20:01:02.233774] openib_check_cq: completed recv wr_id 6840e0. [D 20:01:02.233782] openib_post_rr: 10.1.4.56:39990 bh 29 wr_id 6840e0. [D 20:01:02.233789] openib_check_cq: completed recv wr_id 684110. [D 20:01:02.233796] openib_post_rr: 10.1.4.56:39990 bh 30 wr_id 684110. [D 20:01:02.233803] openib_check_cq: completed recv wr_id 684140. [D 20:01:02.233812] openib_post_rr: 10.1.4.56:39990 bh 31 wr_id 684140. [D 20:01:02.233819] openib_check_cq: completed recv wr_id 684170. [D 20:01:02.233826] openib_post_rr: 10.1.4.56:39990 bh 32 wr_id 684170. [D 20:01:02.233833] openib_check_cq: completed recv wr_id 6841a0. [D 20:01:02.233841] openib_post_rr: 10.1.4.56:39990 bh 33 wr_id 6841a0. [D 20:01:02.233848] openib_check_cq: completed recv wr_id 6841d0. [D 20:01:02.233855] openib_post_rr: 10.1.4.56:39990 bh 34 wr_id 6841d0. [D 20:01:02.233862] openib_check_cq: completed recv wr_id 684200. [D 20:01:02.233869] openib_post_rr: 10.1.4.56:39990 bh 35 wr_id 684200. [D 20:01:02.233876] openib_check_cq: completed recv wr_id 684230. [D 20:01:02.233884] openib_post_rr: 10.1.4.56:39990 bh 36 wr_id 684230. [D 20:01:02.233891] openib_check_cq: completed recv wr_id 684260. [D 20:01:02.233898] openib_post_rr: 10.1.4.56:39990 bh 37 wr_id 684260. [D 20:01:02.233905] openib_check_cq: completed recv wr_id 684290. [D 20:01:02.233912] openib_post_rr: 10.1.4.56:39990 bh 38 wr_id 684290. [D 20:01:02.233919] openib_check_cq: completed recv wr_id 6842c0. [D 20:01:02.233927] openib_post_rr: 10.1.4.56:39990 bh 39 wr_id 6842c0. [D 20:01:02.233934] openib_check_cq: completed recv wr_id 6842f0. [D 20:01:02.233941] openib_post_rr: 10.1.4.56:39990 bh 40 wr_id 6842f0. [D 20:01:02.233948] openib_check_cq: completed recv wr_id 684320. [D 20:01:02.233956] openib_post_rr: 10.1.4.56:39990 bh 41 wr_id 684320. [D 20:01:02.233963] openib_check_cq: completed recv wr_id 684350. [D 20:01:02.233970] openib_post_rr: 10.1.4.56:39990 bh 42 wr_id 684350. [D 20:01:02.233977] openib_check_cq: completed recv wr_id 684380. [D 20:01:02.233984] openib_post_rr: 10.1.4.56:39990 bh 43 wr_id 684380. [D 20:01:02.233991] openib_check_cq: completed recv wr_id 6843b0. [D 20:01:02.233999] openib_post_rr: 10.1.4.56:39990 bh 44 wr_id 6843b0. [D 20:01:02.234006] openib_check_cq: completed recv wr_id 6843e0. [D 20:01:02.234013] openib_post_rr: 10.1.4.56:39990 bh 45 wr_id 6843e0. [D 20:01:02.234021] openib_check_cq: completed recv wr_id 684410. [D 20:01:02.234028] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec210 to 10.1.4.56:39990 remote addr a2651000 rkey 108c27 nic_wr_credit 2. [D 20:01:02.234036] openib_post_rr: 10.1.4.56:39990 bh 46 wr_id 684410. [D 20:01:02.234043] openib_check_cq: completed recv wr_id 684440. [D 20:01:02.234050] openib_post_sr_rdmaw: ibv_post_send wr_id 711140 to 10.1.4.56:39990 remote addr 88837000 rkey 108a27 nic_wr_credit 3. [D 20:01:02.234057] openib_post_rr: 10.1.4.56:39990 bh 47 wr_id 684440. [D 20:01:02.234085] post_sr: 10.1.4.56:39990 bh 53 len 48 wr_id 683290. [D 20:01:02.234095] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:01:02.234103] get_data_sync_mode returning 0 [D 20:01:02.234114] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.234120] dbpf_bstream_rw_list: mem_offset: 0x2aaab4217010, mem_size: 2097152 [D 20:01:02.234127] dbpf_bstream_rw_list: stream_offset: 192937984, stream_size: 2097152 [D 20:01:02.234135] dbpf_open_cache_get: called [D 20:01:02.234141] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.234148] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:00.477018] DBPF I/O ops in progress: 15 [D 20:01:00.477055] ========================================================= [D 20:01:02.234280] PINT_process_request [D 20:01:02.234290] tiling 4 copies [D 20:01:02.234297] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.234305] lvl 0 el 0 blk 0 by 75497472 [D 20:01:02.234311] to 75497472 ta 67108864 fi 100663296 [D 20:01:02.234318] basic type or contiguous data [D 20:01:02.234324] Request disp [D 20:01:02.234330] contig_offset = 75497472 contig_size = 58720256 lvl_flag = 1 [D 20:01:02.234337] calling distribute [D 20:01:02.234343] PINT_distribute [D 20:01:02.234350] of 75497472 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.234357] begin iteration loff: 75497472 [D 20:01:02.234363] segment exceeds byte limit [D 20:01:02.234369] process a segment [D 20:01:02.234375] of 75497472 sz 2097152 [D 20:01:02.234381] add a segment 0 [D 20:01:02.234387] end iteration [D 20:01:02.234392] done with segments or bytes [D 20:01:02.234398] of 77594624 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.234405] next loff: 77594624 next poff: 77594624 [D 20:01:02.234411] return value 2097152 [D 20:01:02.234417] finished [D 20:01:02.234422] retval = 2097152 [D 20:01:02.234428] exiting distribute returned less than expected [D 20:01:02.234434] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 77594624 fo 100663296 eof 0 [D 20:01:02.234441] ========================================================= [D 20:01:02.234452] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.234459] dbpf_bstream_rw_list: mem_offset: 0x2aaabf20f010, mem_size: 2097152 [D 20:01:02.234466] dbpf_bstream_rw_list: stream_offset: 75497472, stream_size: 2097152 [D 20:01:02.234472] dbpf_open_cache_get: called [D 20:01:02.234479] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.234485] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.234592] starting delayed I/O operation 0x2aaab1522b60 (15 in progress) [D 20:01:02.234725] lio_listio called with 1 following aiocbs: [D 20:01:02.234733] aiocb_ptr_array[0]: fd: 13, off: 71303168, bytes: 2097152, buf: 0x2aaabee0d010, type: 0 [D 20:01:02.234808] start_delayed_ops_if_any: lio_listio posted 0x2aaab1522b60 (handle 2863311531, ret 0)) [D 20:01:00.477039] op_queue add: 0x2aaab152de80 [D 20:01:02.234845] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:02.234858] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.234866] BMI_post_send: addr: 6843072, offset: 0x2aaabdc04010, size: 2097152, tag: 24 [D 20:01:02.234876] post_sr: 10.1.4.56:39990 bh 54 len 32 wr_id 6832c0. [D 20:01:02.234888] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.234960] DBPF I/O ops in progress: 15 [D 20:01:02.235023] starting delayed I/O operation 0x2aaab152bb90 (15 in progress) [D 20:01:02.235112] lio_listio called with 1 following aiocbs: [D 20:01:02.235176] aiocb_ptr_array[0]: fd: 13, off: 73400320, bytes: 2097152, buf: 0x2aaabf00e010, type: 0 [D 20:01:02.235186] start_delayed_ops_if_any: lio_listio posted 0x2aaab152bb90 (handle 2863311531, ret 0)) [D 20:01:02.235202] DBPF I/O ops in progress: 15 [D 20:01:02.235280] starting delayed I/O operation 0x2aaabd3112d0 (15 in progress) [D 20:01:02.235345] lio_listio called with 1 following aiocbs: [D 20:01:02.235353] aiocb_ptr_array[0]: fd: 13, off: 455081984, bytes: 2097152, buf: 0x2aaabb855010, type: 0 [D 20:01:02.235361] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3112d0 (handle 2863311531, ret 0)) [D 20:01:02.235371] op_queue add: 0x2aaabd301c40 [D 20:01:02.235378] delayed I/O operation 0x2aaabd301c40 (16 already in progress) [D 20:01:02.235384] DBPF I/O ops in progress: 16 [D 20:01:02.235393] ========================================================= [D 20:01:02.235400] PINT_process_request [D 20:01:02.235406] tiling 7 copies [D 20:01:02.235412] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.235419] lvl 0 el 0 blk 0 by 195035136 [D 20:01:02.235425] to 195035136 ta 167772160 fi 201326592 [D 20:01:02.235432] basic type or contiguous data [D 20:01:02.235438] Request disp [D 20:01:02.235443] contig_offset = 195035136 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.235450] calling distribute [D 20:01:02.235456] PINT_distribute [D 20:01:02.235461] of 195035136 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 195035136 exfl 1 [D 20:01:02.235469] begin iteration loff: 195035136 [D 20:01:02.235475] segment exceeds byte limit [D 20:01:02.235481] file being extended [D 20:01:02.235486] process a segment [D 20:01:02.235492] of 195035136 sz 2097152 [D 20:01:02.235498] add a segment 0 [D 20:01:02.235504] end iteration [D 20:01:02.235509] done with segments or bytes [D 20:01:02.235515] of 197132288 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.235522] next loff: 197132288 next poff: 197132288 [D 20:01:02.235528] return value 2097152 [D 20:01:02.235534] finished [D 20:01:02.235539] retval = 2097152 [D 20:01:02.235545] exiting distribute returned less than expected [D 20:01:02.235551] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 197132288 fo 201326592 eof 0 [D 20:01:02.235558] ========================================================= [D 20:01:02.235564] BMI_post_recv: addr: 6843072, offset: 0x2aaab6228010, size: 2097152, tag: 13 [D 20:01:02.235576] post_sr: 10.1.4.56:39990 bh 55 len 48 wr_id 6832f0. [D 20:01:02.235595] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:01:02.235604] get_data_sync_mode returning 0 [D 20:01:02.235611] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.235618] dbpf_bstream_rw_list: mem_offset: 0x2aaab4418010, mem_size: 2097152 [D 20:01:02.235624] dbpf_bstream_rw_list: stream_offset: 226492416, stream_size: 2097152 [D 20:01:02.235631] dbpf_open_cache_get: called [D 20:01:02.235637] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.235643] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.235657] op_queue add: 0x2aaabd324d90 [D 20:01:02.235664] delayed I/O operation 0x2aaabd324d90 (16 already in progress) [D 20:01:02.235671] DBPF I/O ops in progress: 16 [D 20:01:02.235678] ========================================================= [D 20:01:02.235684] PINT_process_request [D 20:01:02.235690] tiling 8 copies [D 20:01:02.235698] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.235705] lvl 0 el 0 blk 0 by 228589568 [D 20:01:02.235712] to 228589568 ta 201326592 fi 234881024 [D 20:01:02.235718] basic type or contiguous data [D 20:01:02.235724] Request disp [D 20:01:02.235730] contig_offset = 228589568 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.235737] calling distribute [D 20:01:02.235743] PINT_distribute [D 20:01:02.235748] of 228589568 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 228589568 exfl 1 [D 20:01:02.235756] begin iteration loff: 228589568 [D 20:01:02.235762] segment exceeds byte limit [D 20:01:02.235768] file being extended [D 20:01:02.235774] process a segment [D 20:01:02.235780] of 228589568 sz 2097152 [D 20:01:02.235786] add a segment 0 [D 20:01:02.235792] end iteration [D 20:01:02.235797] done with segments or bytes [D 20:01:02.235803] of 230686720 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.235810] next loff: 230686720 next poff: 230686720 [D 20:01:02.235816] return value 2097152 [D 20:01:02.235822] finished [D 20:01:02.235828] retval = 2097152 [D 20:01:02.235834] exiting distribute returned less than expected [D 20:01:02.235840] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 230686720 fo 234881024 eof 0 [D 20:01:02.235847] ========================================================= [D 20:01:02.235853] BMI_post_recv: addr: 6843072, offset: 0x2aaab642a010, size: 2097152, tag: 14 [D 20:01:02.235864] post_sr: 10.1.4.56:39990 bh 56 len 48 wr_id 683320. [D 20:01:02.235873] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:01:02.235881] get_data_sync_mode returning 0 [D 20:01:02.235888] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.235894] dbpf_bstream_rw_list: mem_offset: 0x2aaab4619010, mem_size: 2097152 [D 20:01:02.235901] dbpf_bstream_rw_list: stream_offset: 260046848, stream_size: 2097152 [D 20:01:02.235908] dbpf_open_cache_get: called [D 20:01:02.235914] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.235920] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.235933] op_queue add: 0x2aaabd301e40 [D 20:01:02.235940] delayed I/O operation 0x2aaabd301e40 (16 already in progress) [D 20:01:02.235946] DBPF I/O ops in progress: 16 [D 20:01:02.235953] ========================================================= [D 20:01:02.235959] PINT_process_request [D 20:01:02.235965] tiling 9 copies [D 20:01:02.235971] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.235979] lvl 0 el 0 blk 0 by 262144000 [D 20:01:02.235985] to 262144000 ta 234881024 fi 268435456 [D 20:01:02.235991] basic type or contiguous data [D 20:01:02.235997] Request disp [D 20:01:02.236003] contig_offset = 262144000 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.236010] calling distribute [D 20:01:02.236016] PINT_distribute [D 20:01:02.236022] of 262144000 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 262144000 exfl 1 [D 20:01:02.236029] begin iteration loff: 262144000 [D 20:01:02.236035] segment exceeds byte limit [D 20:01:02.236041] file being extended [D 20:01:02.236047] process a segment [D 20:01:02.236053] of 262144000 sz 2097152 [D 20:01:02.236060] add a segment 0 [D 20:01:02.236066] end iteration [D 20:01:02.236071] done with segments or bytes [D 20:01:02.236077] of 264241152 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.236084] next loff: 264241152 next poff: 264241152 [D 20:01:02.236090] return value 2097152 [D 20:01:02.236096] finished [D 20:01:02.236102] retval = 2097152 [D 20:01:02.236107] exiting distribute returned less than expected [D 20:01:02.236113] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 264241152 fo 268435456 eof 0 [D 20:01:02.236121] ========================================================= [D 20:01:02.236127] BMI_post_recv: addr: 6843072, offset: 0x2aaab662b010, size: 2097152, tag: 15 [D 20:01:02.236134] openib_check_cq: completed rdma wr_id 6df4e0, nic_wr_credit 2. [D 20:01:02.236142] post_sr: 10.1.4.56:39990 bh 57 len 16 wr_id 683350. [D 20:01:02.236153] post_sr: 10.1.4.56:39990 bh 58 len 48 wr_id 683380. [D 20:01:02.236161] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:01:02.236169] get_data_sync_mode returning 0 [D 20:01:02.236176] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.236182] dbpf_bstream_rw_list: mem_offset: 0x2aaab481a010, mem_size: 2097152 [D 20:01:02.236189] dbpf_bstream_rw_list: stream_offset: 293601280, stream_size: 2097152 [D 20:01:02.236195] dbpf_open_cache_get: called [D 20:01:02.236201] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.236207] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.236221] op_queue add: 0x2aaabd301f90 [D 20:01:02.236228] delayed I/O operation 0x2aaabd301f90 (16 already in progress) [D 20:01:02.236234] DBPF I/O ops in progress: 16 [D 20:01:02.236241] ========================================================= [D 20:01:02.236247] PINT_process_request [D 20:01:02.236253] tiling 10 copies [D 20:01:02.236259] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.236266] lvl 0 el 0 blk 0 by 295698432 [D 20:01:02.236272] to 295698432 ta 268435456 fi 301989888 [D 20:01:02.236279] basic type or contiguous data [D 20:01:02.236285] Request disp [D 20:01:02.236291] contig_offset = 295698432 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.236297] calling distribute [D 20:01:02.236303] PINT_distribute [D 20:01:02.236309] of 295698432 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 295698432 exfl 1 [D 20:01:02.236316] begin iteration loff: 295698432 [D 20:01:02.236322] segment exceeds byte limit [D 20:01:02.236328] file being extended [D 20:01:02.236334] process a segment [D 20:01:02.236340] of 295698432 sz 2097152 [D 20:01:02.236346] add a segment 0 [D 20:01:02.236352] end iteration [D 20:01:02.236357] done with segments or bytes [D 20:01:02.236363] of 297795584 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.236370] next loff: 297795584 next poff: 297795584 [D 20:01:02.236376] return value 2097152 [D 20:01:02.236382] finished [D 20:01:02.236388] retval = 2097152 [D 20:01:02.236394] exiting distribute returned less than expected [D 20:01:02.236399] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 297795584 fo 301989888 eof 0 [D 20:01:02.236407] ========================================================= [D 20:01:02.236413] BMI_post_recv: addr: 6843072, offset: 0x2aaab682c010, size: 2097152, tag: 16 [D 20:01:02.236423] post_sr: 10.1.4.56:39990 bh 59 len 48 wr_id 6833b0. [D 20:01:02.236432] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:01:02.236439] get_data_sync_mode returning 0 [D 20:01:02.236447] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.236453] dbpf_bstream_rw_list: mem_offset: 0x2aaab4a1b010, mem_size: 2097152 [D 20:01:02.236460] dbpf_bstream_rw_list: stream_offset: 327155712, stream_size: 2097152 [D 20:01:02.236466] dbpf_open_cache_get: called [D 20:01:02.236472] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.236478] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.236493] op_queue add: 0x2aaabd3020e0 [D 20:01:02.236500] delayed I/O operation 0x2aaabd3020e0 (16 already in progress) [D 20:01:02.236506] DBPF I/O ops in progress: 16 [D 20:01:02.236513] ========================================================= [D 20:01:02.236519] PINT_process_request [D 20:01:02.236528] tiling 11 copies [D 20:01:02.236534] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.236542] lvl 0 el 0 blk 0 by 329252864 [D 20:01:02.236548] to 329252864 ta 301989888 fi 335544320 [D 20:01:02.236554] basic type or contiguous data [D 20:01:02.236560] Request disp [D 20:01:02.236566] contig_offset = 329252864 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.236573] calling distribute [D 20:01:02.236579] PINT_distribute [D 20:01:02.236584] of 329252864 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 329252864 exfl 1 [D 20:01:02.236592] begin iteration loff: 329252864 [D 20:01:02.236598] segment exceeds byte limit [D 20:01:02.236604] file being extended [D 20:01:02.236610] process a segment [D 20:01:02.236615] of 329252864 sz 2097152 [D 20:01:02.236621] add a segment 0 [D 20:01:02.236627] end iteration [D 20:01:02.236633] done with segments or bytes [D 20:01:02.236638] of 331350016 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.236645] next loff: 331350016 next poff: 331350016 [D 20:01:02.236651] return value 2097152 [D 20:01:02.236657] finished [D 20:01:02.236663] retval = 2097152 [D 20:01:02.236669] exiting distribute returned less than expected [D 20:01:02.236675] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 331350016 fo 335544320 eof 0 [D 20:01:02.236682] ========================================================= [D 20:01:02.236688] BMI_post_recv: addr: 6843072, offset: 0x2aaab6a2d010, size: 2097152, tag: 17 [D 20:01:02.236699] post_sr: 10.1.4.56:39990 bh 60 len 48 wr_id 6833e0. [D 20:01:02.236708] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:01:02.236715] get_data_sync_mode returning 0 [D 20:01:02.236722] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.236729] dbpf_bstream_rw_list: mem_offset: 0x2aaab4c1c010, mem_size: 2097152 [D 20:01:02.236735] dbpf_bstream_rw_list: stream_offset: 360710144, stream_size: 2097152 [D 20:01:02.236742] dbpf_open_cache_get: called [D 20:01:02.236748] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.236754] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.236768] op_queue add: 0x2aaabd310ef0 [D 20:01:02.236774] delayed I/O operation 0x2aaabd310ef0 (16 already in progress) [D 20:01:02.236781] DBPF I/O ops in progress: 16 [D 20:01:02.236787] ========================================================= [D 20:01:02.236794] PINT_process_request [D 20:01:02.236800] tiling 12 copies [D 20:01:02.236806] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.236813] lvl 0 el 0 blk 0 by 362807296 [D 20:01:02.236820] to 362807296 ta 335544320 fi 369098752 [D 20:01:02.236826] basic type or contiguous data [D 20:01:02.236832] Request disp [D 20:01:02.236838] contig_offset = 362807296 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.236844] calling distribute [D 20:01:02.236850] PINT_distribute [D 20:01:02.236856] of 362807296 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 362807296 exfl 1 [D 20:01:02.236863] begin iteration loff: 362807296 [D 20:01:02.236869] segment exceeds byte limit [D 20:01:02.236875] file being extended [D 20:01:02.236881] process a segment [D 20:01:02.236887] of 362807296 sz 2097152 [D 20:01:02.236893] add a segment 0 [D 20:01:02.236899] end iteration [D 20:01:02.236904] done with segments or bytes [D 20:01:02.236910] of 364904448 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.236917] next loff: 364904448 next poff: 364904448 [D 20:01:02.236923] return value 2097152 [D 20:01:02.236929] finished [D 20:01:02.236935] retval = 2097152 [D 20:01:02.236941] exiting distribute returned less than expected [D 20:01:02.236947] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 364904448 fo 369098752 eof 0 [D 20:01:02.236955] ========================================================= [D 20:01:02.236961] BMI_post_recv: addr: 6843072, offset: 0x2aaab6c2e010, size: 2097152, tag: 18 [D 20:01:02.236971] post_sr: 10.1.4.56:39990 bh 61 len 48 wr_id 683410. [D 20:01:02.236984] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:01:02.236993] get_data_sync_mode returning 0 [D 20:01:02.237000] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.237006] dbpf_bstream_rw_list: mem_offset: 0x2aaab4e1d010, mem_size: 2097152 [D 20:01:02.237013] dbpf_bstream_rw_list: stream_offset: 394264576, stream_size: 2097152 [D 20:01:02.237019] dbpf_open_cache_get: called [D 20:01:02.237025] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.237032] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.237045] op_queue add: 0x2aaabd300a30 [D 20:01:02.237052] delayed I/O operation 0x2aaabd300a30 (16 already in progress) [D 20:01:02.237058] DBPF I/O ops in progress: 16 [D 20:01:02.237065] ========================================================= [D 20:01:02.237071] PINT_process_request [D 20:01:02.237077] tiling 13 copies [D 20:01:02.237083] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.237091] lvl 0 el 0 blk 0 by 396361728 [D 20:01:02.237097] to 396361728 ta 369098752 fi 402653184 [D 20:01:02.237104] basic type or contiguous data [D 20:01:02.237110] Request disp [D 20:01:02.237116] contig_offset = 396361728 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.237122] calling distribute [D 20:01:02.237128] PINT_distribute [D 20:01:02.237134] of 396361728 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 396361728 exfl 1 [D 20:01:02.237141] begin iteration loff: 396361728 [D 20:01:02.237148] segment exceeds byte limit [D 20:01:02.237153] file being extended [D 20:01:02.237159] process a segment [D 20:01:02.237165] of 396361728 sz 2097152 [D 20:01:02.237171] add a segment 0 [D 20:01:02.237177] end iteration [D 20:01:02.237183] done with segments or bytes [D 20:01:02.237188] of 398458880 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.237195] next loff: 398458880 next poff: 398458880 [D 20:01:02.237201] return value 2097152 [D 20:01:02.237207] finished [D 20:01:02.237213] retval = 2097152 [D 20:01:02.237219] exiting distribute returned less than expected [D 20:01:02.237225] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 398458880 fo 402653184 eof 0 [D 20:01:02.237232] ========================================================= [D 20:01:02.237238] BMI_post_recv: addr: 6843072, offset: 0x2aaab6e2f010, size: 2097152, tag: 19 [D 20:01:02.237249] post_sr: 10.1.4.56:39990 bh 62 len 48 wr_id 683440. [D 20:01:02.237258] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:02.237265] get_data_sync_mode returning 0 [D 20:01:02.237272] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.237279] dbpf_bstream_rw_list: mem_offset: 0x2aaab521f010, mem_size: 2097152 [D 20:01:02.237285] dbpf_bstream_rw_list: stream_offset: 486539264, stream_size: 2097152 [D 20:01:02.237292] dbpf_open_cache_get: called [D 20:01:02.237298] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.237304] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.237317] op_queue add: 0x2aaabd311110 [D 20:01:02.237324] delayed I/O operation 0x2aaabd311110 (16 already in progress) [D 20:01:02.237330] DBPF I/O ops in progress: 16 [D 20:01:02.237337] ========================================================= [D 20:01:02.237343] PINT_process_request [D 20:01:02.237349] tiling 16 copies [D 20:01:02.237355] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.237362] lvl 0 el 0 blk 0 by 488636416 [D 20:01:02.237369] to 488636416 ta 469762048 fi 503316480 [D 20:01:02.237375] basic type or contiguous data [D 20:01:02.237382] Request disp [D 20:01:02.237388] contig_offset = 488636416 contig_size = 48234496 lvl_flag = 1 [D 20:01:02.237395] calling distribute [D 20:01:02.237400] PINT_distribute [D 20:01:02.237406] of 488636416 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 488636416 exfl 1 [D 20:01:02.237414] begin iteration loff: 488636416 [D 20:01:02.237420] segment exceeds byte limit [D 20:01:02.237426] file being extended [D 20:01:02.237432] process a segment [D 20:01:02.237438] of 488636416 sz 2097152 [D 20:01:02.237444] add a segment 0 [D 20:01:02.237450] end iteration [D 20:01:02.237455] done with segments or bytes [D 20:01:02.237461] of 490733568 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.237468] next loff: 490733568 next poff: 490733568 [D 20:01:02.237474] return value 2097152 [D 20:01:02.237480] finished [D 20:01:02.237486] retval = 2097152 [D 20:01:02.237491] exiting distribute returned less than expected [D 20:01:02.237498] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 490733568 fo 503316480 eof 0 [D 20:01:02.237505] ========================================================= [D 20:01:02.237511] BMI_post_recv: addr: 6843072, offset: 0x2aaab5c25010, size: 2097152, tag: 22 [D 20:01:02.237522] post_sr: 10.1.4.56:39990 bh 63 len 48 wr_id 683470. [D 20:01:02.237530] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:02.237538] get_data_sync_mode returning 0 [D 20:01:02.237545] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.237552] dbpf_bstream_rw_list: mem_offset: 0x2aaab501e010, mem_size: 2097152 [D 20:01:02.237558] dbpf_bstream_rw_list: stream_offset: 427819008, stream_size: 2097152 [D 20:01:02.237565] dbpf_open_cache_get: called [D 20:01:02.237571] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.237577] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.237589] op_queue add: 0x2aaabd302230 [D 20:01:02.237596] delayed I/O operation 0x2aaabd302230 (16 already in progress) [D 20:01:02.237603] DBPF I/O ops in progress: 16 [D 20:01:02.237610] ========================================================= [D 20:01:02.237616] PINT_process_request [D 20:01:02.237622] tiling 14 copies [D 20:01:02.237628] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.237636] lvl 0 el 0 blk 0 by 429916160 [D 20:01:02.237642] to 429916160 ta 402653184 fi 436207616 [D 20:01:02.237649] basic type or contiguous data [D 20:01:02.237655] Request disp [D 20:01:02.237661] contig_offset = 429916160 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.237667] calling distribute [D 20:01:02.237673] PINT_distribute [D 20:01:02.237679] of 429916160 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 429916160 exfl 1 [D 20:01:02.237687] begin iteration loff: 429916160 [D 20:01:02.237693] segment exceeds byte limit [D 20:01:02.237699] file being extended [D 20:01:02.237704] process a segment [D 20:01:02.237710] of 429916160 sz 2097152 [D 20:01:02.237716] add a segment 0 [D 20:01:02.237722] end iteration [D 20:01:02.237728] done with segments or bytes [D 20:01:02.237734] of 432013312 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.237741] next loff: 432013312 next poff: 432013312 [D 20:01:02.237747] return value 2097152 [D 20:01:02.237753] finished [D 20:01:02.237759] retval = 2097152 [D 20:01:02.237764] exiting distribute returned less than expected [D 20:01:02.237770] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 432013312 fo 436207616 eof 0 [D 20:01:02.237778] ========================================================= [D 20:01:02.237784] BMI_post_recv: addr: 6843072, offset: 0x2aaab7030010, size: 2097152, tag: 20 [D 20:01:02.237795] post_sr: 10.1.4.56:39990 bh 64 len 48 wr_id 6834a0. [D 20:01:02.237803] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:02.237811] get_data_sync_mode returning 0 [D 20:01:02.237818] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.237824] dbpf_bstream_rw_list: mem_offset: 0x2aaab5420010, mem_size: 2097152 [D 20:01:02.237831] dbpf_bstream_rw_list: stream_offset: 461373440, stream_size: 2097152 [D 20:01:02.237838] dbpf_open_cache_get: called [D 20:01:02.237844] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.237850] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.237863] op_queue add: 0x2aaabd302380 [D 20:01:02.237870] delayed I/O operation 0x2aaabd302380 (16 already in progress) [D 20:01:02.237876] DBPF I/O ops in progress: 16 [D 20:01:02.237883] ========================================================= [D 20:01:02.237889] PINT_process_request [D 20:01:02.237895] tiling 15 copies [D 20:01:02.237901] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.237909] lvl 0 el 0 blk 0 by 463470592 [D 20:01:02.237915] to 463470592 ta 436207616 fi 469762048 [D 20:01:02.237922] basic type or contiguous data [D 20:01:02.237928] Request disp [D 20:01:02.237933] contig_offset = 463470592 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.237940] calling distribute [D 20:01:02.237946] PINT_distribute [D 20:01:02.237952] of 463470592 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 463470592 exfl 1 [D 20:01:02.237959] begin iteration loff: 463470592 [D 20:01:02.237965] segment exceeds byte limit [D 20:01:02.237972] file being extended [D 20:01:02.237977] process a segment [D 20:01:02.237983] of 463470592 sz 2097152 [D 20:01:02.237989] add a segment 0 [D 20:01:02.237995] end iteration [D 20:01:02.238001] done with segments or bytes [D 20:01:02.238007] of 465567744 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.238014] next loff: 465567744 next poff: 465567744 [D 20:01:02.238020] return value 2097152 [D 20:01:02.238026] finished [D 20:01:02.238032] retval = 2097152 [D 20:01:02.238038] exiting distribute returned less than expected [D 20:01:02.238044] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 465567744 fo 469762048 eof 0 [D 20:01:02.238051] ========================================================= [D 20:01:02.238057] BMI_post_recv: addr: 6843072, offset: 0x2aaab7231010, size: 2097152, tag: 21 [D 20:01:02.238068] post_sr: 10.1.4.56:39990 bh 65 len 48 wr_id 6834d0. [D 20:01:02.238076] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:01:02.238084] get_data_sync_mode returning 0 [D 20:01:02.238091] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.238098] dbpf_bstream_rw_list: mem_offset: 0x2aaab5621010, mem_size: 2097152 [D 20:01:02.238104] dbpf_bstream_rw_list: stream_offset: 27262976, stream_size: 2097152 [D 20:01:02.238111] dbpf_open_cache_get: called [D 20:01:02.238117] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.238123] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.238136] op_queue add: 0x2aaabd3024d0 [D 20:01:02.238142] delayed I/O operation 0x2aaabd3024d0 (16 already in progress) [D 20:01:02.238149] DBPF I/O ops in progress: 16 [D 20:01:02.238156] ========================================================= [D 20:01:02.238162] PINT_process_request [D 20:01:02.238168] tiling 2 copies [D 20:01:02.238174] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.238181] lvl 0 el 0 blk 0 by 29360128 [D 20:01:02.238188] to 29360128 ta 0 fi 33554432 [D 20:01:02.238194] basic type or contiguous data [D 20:01:02.238201] Request disp [D 20:01:02.238206] contig_offset = 29360128 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.238213] calling distribute [D 20:01:02.238219] PINT_distribute [D 20:01:02.238224] of 29360128 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 29360128 exfl 1 [D 20:01:02.238232] begin iteration loff: 29360128 [D 20:01:02.238238] segment exceeds byte limit [D 20:01:02.238244] file being extended [D 20:01:02.238250] process a segment [D 20:01:02.238255] of 29360128 sz 2097152 [D 20:01:02.238262] add a segment 0 [D 20:01:02.238267] end iteration [D 20:01:02.238273] done with segments or bytes [D 20:01:02.238279] of 31457280 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.238287] next loff: 31457280 next poff: 31457280 [D 20:01:02.238293] return value 2097152 [D 20:01:02.238299] finished [D 20:01:02.238304] retval = 2097152 [D 20:01:02.238310] exiting distribute returned less than expected [D 20:01:02.238316] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 31457280 fo 33554432 eof 0 [D 20:01:02.238323] ========================================================= [D 20:01:02.238329] BMI_post_recv: addr: 6843072, offset: 0x2aaab7432010, size: 2097152, tag: 8 [D 20:01:02.238337] openib_check_cq: completed rdma wr_id 6ec210, nic_wr_credit 1. [D 20:01:02.238345] post_sr: 10.1.4.56:39990 bh 66 len 16 wr_id 683500. [D 20:01:02.238355] post_sr: 10.1.4.56:39990 bh 67 len 48 wr_id 683530. [D 20:01:02.238367] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:01:02.238375] get_data_sync_mode returning 0 [D 20:01:02.238382] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.238389] dbpf_bstream_rw_list: mem_offset: 0x2aaab5822010, mem_size: 2097152 [D 20:01:02.238396] dbpf_bstream_rw_list: stream_offset: 60817408, stream_size: 2097152 [D 20:01:02.238402] dbpf_open_cache_get: called [D 20:01:02.238408] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.238414] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.238436] op_queue add: 0x2aaabd3026f0 [D 20:01:02.238445] delayed I/O operation 0x2aaabd3026f0 (16 already in progress) [D 20:01:02.238451] DBPF I/O ops in progress: 16 [D 20:01:02.238458] ========================================================= [D 20:01:02.238464] PINT_process_request [D 20:01:02.238470] tiling 3 copies [D 20:01:02.238476] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.238484] lvl 0 el 0 blk 0 by 62914560 [D 20:01:02.238490] to 62914560 ta 33554432 fi 67108864 [D 20:01:02.238497] basic type or contiguous data [D 20:01:02.238503] Request disp [D 20:01:02.238509] contig_offset = 62914560 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.238515] calling distribute [D 20:01:02.238521] PINT_distribute [D 20:01:02.238527] of 62914560 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 62914560 exfl 1 [D 20:01:02.238534] begin iteration loff: 62914560 [D 20:01:02.238540] segment exceeds byte limit [D 20:01:02.238546] file being extended [D 20:01:02.238552] process a segment [D 20:01:02.238558] of 62914560 sz 2097152 [D 20:01:02.238564] add a segment 0 [D 20:01:02.238570] end iteration [D 20:01:02.238575] done with segments or bytes [D 20:01:02.238581] of 65011712 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.238588] next loff: 65011712 next poff: 65011712 [D 20:01:02.238594] return value 2097152 [D 20:01:02.238600] finished [D 20:01:02.238606] retval = 2097152 [D 20:01:02.238612] exiting distribute returned less than expected [D 20:01:02.238618] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 65011712 fo 67108864 eof 0 [D 20:01:02.238625] ========================================================= [D 20:01:02.238631] BMI_post_recv: addr: 6843072, offset: 0x2aaab7633010, size: 2097152, tag: 9 [D 20:01:02.238642] post_sr: 10.1.4.56:39990 bh 68 len 48 wr_id 683560. [D 20:01:02.238651] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:01:02.238658] get_data_sync_mode returning 0 [D 20:01:02.238665] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.238672] dbpf_bstream_rw_list: mem_offset: 0x2aaab5a23010, mem_size: 2097152 [D 20:01:02.238679] dbpf_bstream_rw_list: stream_offset: 94371840, stream_size: 2097152 [D 20:01:02.238685] dbpf_open_cache_get: called [D 20:01:02.238691] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.238697] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.238719] op_queue add: 0x2aaabd302840 [D 20:01:02.238727] delayed I/O operation 0x2aaabd302840 (16 already in progress) [D 20:01:02.238734] DBPF I/O ops in progress: 16 [D 20:01:02.238741] ========================================================= [D 20:01:02.238748] PINT_process_request [D 20:01:02.238754] tiling 4 copies [D 20:01:02.238760] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.238767] lvl 0 el 0 blk 0 by 96468992 [D 20:01:02.238773] to 96468992 ta 67108864 fi 100663296 [D 20:01:02.238780] basic type or contiguous data [D 20:01:02.238786] Request disp [D 20:01:02.238791] contig_offset = 96468992 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.238798] calling distribute [D 20:01:02.238804] PINT_distribute [D 20:01:02.238810] of 96468992 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 96468992 exfl 1 [D 20:01:02.238817] begin iteration loff: 96468992 [D 20:01:02.238823] segment exceeds byte limit [D 20:01:02.238829] file being extended [D 20:01:02.238835] process a segment [D 20:01:02.238841] of 96468992 sz 2097152 [D 20:01:02.238847] add a segment 0 [D 20:01:02.238853] end iteration [D 20:01:02.238858] done with segments or bytes [D 20:01:02.238864] of 98566144 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.238871] next loff: 98566144 next poff: 98566144 [D 20:01:02.238877] return value 2097152 [D 20:01:02.238883] finished [D 20:01:02.238889] retval = 2097152 [D 20:01:02.238895] exiting distribute returned less than expected [D 20:01:02.238901] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 98566144 fo 100663296 eof 0 [D 20:01:02.238908] ========================================================= [D 20:01:02.238914] BMI_post_recv: addr: 6843072, offset: 0x2aaab7834010, size: 2097152, tag: 10 [D 20:01:02.238925] post_sr: 10.1.4.56:39990 bh 69 len 48 wr_id 683590. [D 20:01:02.238933] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:01:02.238941] get_data_sync_mode returning 0 [D 20:01:02.238948] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.238954] dbpf_bstream_rw_list: mem_offset: 0x2aaab5e26010, mem_size: 2097152 [D 20:01:02.238961] dbpf_bstream_rw_list: stream_offset: 127926272, stream_size: 2097152 [D 20:01:02.238967] dbpf_open_cache_get: called [D 20:01:02.238973] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.238980] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.238996] op_queue add: 0x2aaabd302990 [D 20:01:02.239005] delayed I/O operation 0x2aaabd302990 (16 already in progress) [D 20:01:02.239011] DBPF I/O ops in progress: 16 [D 20:01:02.239018] ========================================================= [D 20:01:02.239024] PINT_process_request [D 20:01:02.239030] tiling 5 copies [D 20:01:02.239036] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.239043] lvl 0 el 0 blk 0 by 130023424 [D 20:01:02.239050] to 130023424 ta 100663296 fi 134217728 [D 20:01:02.239056] basic type or contiguous data [D 20:01:02.239062] Request disp [D 20:01:02.239068] contig_offset = 130023424 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.239075] calling distribute [D 20:01:02.239080] PINT_distribute [D 20:01:02.239086] of 130023424 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 130023424 exfl 1 [D 20:01:02.239094] begin iteration loff: 130023424 [D 20:01:02.239100] segment exceeds byte limit [D 20:01:02.239106] file being extended [D 20:01:02.239111] process a segment [D 20:01:02.239117] of 130023424 sz 2097152 [D 20:01:02.239123] add a segment 0 [D 20:01:02.239129] end iteration [D 20:01:02.239135] done with segments or bytes [D 20:01:02.239141] of 132120576 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.239147] next loff: 132120576 next poff: 132120576 [D 20:01:02.239153] return value 2097152 [D 20:01:02.239159] finished [D 20:01:02.239165] retval = 2097152 [D 20:01:02.239171] exiting distribute returned less than expected [D 20:01:02.239177] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 132120576 fo 134217728 eof 0 [D 20:01:02.239184] ========================================================= [D 20:01:02.239191] BMI_post_recv: addr: 6843072, offset: 0x2aaab7a35010, size: 2097152, tag: 11 [D 20:01:02.239202] post_sr: 10.1.4.56:39990 bh 70 len 48 wr_id 6835c0. [D 20:01:02.240403] openib_check_cq: completed rdma wr_id 711140, nic_wr_credit 0. [D 20:01:02.240412] post_sr: 10.1.4.56:39990 bh 71 len 16 wr_id 6835f0. [D 20:01:02.240419] openib_check_cq: completed send wr_id 683290. [D 20:01:02.240426] openib_check_cq: completed send wr_id 6832c0. [D 20:01:02.240433] openib_check_cq: completed send wr_id 6832f0. [D 20:01:02.240439] openib_check_cq: completed send wr_id 683320. [D 20:01:02.240446] openib_check_cq: completed send wr_id 683350. [D 20:01:02.240452] openib_check_cq: completed send wr_id 683380. [D 20:01:02.240459] openib_check_cq: completed send wr_id 6833b0. [D 20:01:02.240465] openib_check_cq: completed send wr_id 6833e0. [D 20:01:02.240472] openib_check_cq: completed send wr_id 683410. [D 20:01:02.240479] openib_check_cq: completed send wr_id 683440. [D 20:01:02.240485] openib_check_cq: completed send wr_id 683470. [D 20:01:02.240492] openib_check_cq: completed send wr_id 6834a0. [D 20:01:02.240498] openib_check_cq: completed send wr_id 6834d0. [D 20:01:02.240505] openib_check_cq: completed send wr_id 683500. [D 20:01:02.240512] openib_check_cq: completed send wr_id 683530. [D 20:01:02.240518] openib_check_cq: completed send wr_id 683560. [D 20:01:02.240528] op_queue add: 0x2aaab152bce0 [D 20:01:00.480522] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300f80) [D 20:01:02.240530] openib_check_cq: completed send wr_id 683590. [D 20:01:00.480588] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300d80) [D 20:01:02.240545] delayed I/O operation 0x2aaab152bce0 (16 already in progress) [D 20:01:02.240561] openib_check_cq: completed send wr_id 6835c0. [D 20:01:00.484543] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3011a0) [D 20:01:02.240574] DBPF I/O ops in progress: 16 [D 20:01:02.240583] openib_check_cq: completed send wr_id 6835f0. [D 20:01:02.240597] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:00.485440] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3012f0) [D 20:01:02.240609] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.240612] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b630. [D 20:01:02.240627] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.240634] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:00.488515] --- aio_progress_notification called with handle 2863311531 (0x2aaab152e050) [D 20:01:02.240623] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:02.240690] ========================================================= [D 20:01:02.240638] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.240697] PINT_process_request [D 20:01:02.240655] dbpf_open_cache_put: cached entry. [D 20:01:00.492512] --- aio_progress_notification called with handle 2863311531 (0x2aaab15333d0) [D 20:01:00.492600] --- aio_progress_notification called with handle 2863311531 (0x2aaab1522e30) [D 20:01:02.240578] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.244142] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:00.496545] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b210) [D 20:01:00.500523] --- aio_progress_notification called with handle 2863311531 (0x2aaab1522750) [D 20:01:00.500621] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b470) [D 20:01:02.244277] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.240715] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.493338] --- aio_progress_notification called with handle 2863311531 (0x2aaab1535f30) [D 20:01:02.240587] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.244159] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.244317] ========================================================= [D 20:01:02.244336] PINT_process_request [D 20:01:02.240704] tiling 15 copies [D 20:01:02.240552] op_queue add: 0x2aaabd300b80 [D 20:01:02.244269] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:00.501203] --- aio_progress_notification called with handle 2863311531 (0x2aaab1522990) [D 20:01:02.244376] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.244387] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.244261] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.244349] tiling 4 copies [D 20:01:02.244429] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.244409] op_queue add: 0x2aaabd3011a0 [D 20:01:02.244389] lvl 0 el 0 blk 0 by 457179136 [D 20:01:02.244451] to 457179136 ta 436207616 fi 469762048 [D 20:01:02.244456] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:01:02.244477] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:02.244490] BMI_post_send: addr: 6843072, offset: 0x2aaabc65c010, size: 2097152, tag: 23 [D 20:01:02.244517] post_sr: 10.1.4.56:39990 bh 72 len 32 wr_id 683620. [D 20:01:02.244599] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:02.244443] lvl 0 el 0 blk 0 by 77594624 [D 20:01:02.244632] to 77594624 ta 67108864 fi 100663296 [D 20:01:02.244459] basic type or contiguous data [D 20:01:02.244646] basic type or contiguous data [D 20:01:02.244654] Request disp [D 20:01:02.244665] Request disp [D 20:01:02.244686] contig_offset = 77594624 contig_size = 56623104 lvl_flag = 1 [D 20:01:02.244695] calling distribute [D 20:01:02.244702] PINT_distribute [D 20:01:02.244710] of 77594624 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.244720] begin iteration loff: 77594624 [D 20:01:02.244728] segment exceeds byte limit [D 20:01:02.244735] process a segment [D 20:01:02.244741] of 77594624 sz 2097152 [D 20:01:02.244749] add a segment 0 [D 20:01:02.244755] end iteration [D 20:01:02.244761] done with segments or bytes [D 20:01:02.244767] of 79691776 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.244774] next loff: 79691776 next poff: 79691776 [D 20:01:02.244781] return value 2097152 [D 20:01:02.244787] finished [D 20:01:02.244793] retval = 2097152 [D 20:01:02.244800] exiting distribute returned less than expected [D 20:01:02.244807] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 79691776 fo 100663296 eof 0 [D 20:01:02.244814] ========================================================= [D 20:01:02.244833] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.244841] dbpf_bstream_rw_list: mem_offset: 0x2aaabf410010, mem_size: 2097152 [D 20:01:02.244849] dbpf_bstream_rw_list: stream_offset: 77594624, stream_size: 2097152 [D 20:01:02.244857] dbpf_open_cache_get: called [D 20:01:02.244354] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.244219] dbpf_open_cache_put: cached entry. [D 20:01:02.244900] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.244907] op_queue add: 0x2aaabd3012f0 [D 20:01:02.244924] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:01:02.244376] DBPF I/O ops in progress: 15 [D 20:01:02.244677] contig_offset = 457179136 contig_size = 46137344 lvl_flag = 1 [D 20:01:02.244949] starting delayed I/O operation 0x2aaabd308af0 (15 in progress) [D 20:01:02.244962] calling distribute [D 20:01:02.244392] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.244971] lio_listio called with 1 following aiocbs: [D 20:01:02.244979] PINT_distribute [D 20:01:02.244989] aiocb_ptr_array[0]: fd: 13, off: 159383552, bytes: 2097152, buf: 0x2aaab4016010, type: 1 [D 20:01:02.244998] of 457179136 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:02.244952] dbpf_open_cache_put: cached entry. [D 20:01:02.245022] begin iteration loff: 457179136 [D 20:01:02.245028] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.245040] op_queue add: 0x2aaabd300f80 [D 20:01:02.245031] segment exceeds byte limit [D 20:01:02.245056] process a segment [D 20:01:02.245058] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:02.245064] of 457179136 sz 2097152 [D 20:01:02.245014] start_delayed_ops_if_any: lio_listio posted 0x2aaabd308af0 (handle 2863311531, ret 0)) [D 20:01:02.245091] add a segment 0 [D 20:01:02.245112] end iteration [D 20:01:02.245118] done with segments or bytes [D 20:01:02.245122] DBPF I/O ops in progress: 15 [D 20:01:02.245124] of 459276288 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.245139] starting delayed I/O operation 0x2aaabd301c40 (15 in progress) [D 20:01:02.245043] dbpf_open_cache_put: cached entry. [D 20:01:02.245161] lio_listio called with 1 following aiocbs: [D 20:01:02.245152] next loff: 459276288 next poff: 459276288 [D 20:01:02.245180] dbpf_open_cache_put: cached entry. [D 20:01:02.245182] return value 2097152 [D 20:01:02.245195] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.245208] dbpf_open_cache_put: cached entry. [D 20:01:02.245221] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.245204] finished [D 20:01:02.245184] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.245213] op_queue add: 0x2aaab152e050 [D 20:01:02.245246] retval = 2097152 [D 20:01:02.245262] exiting distribute returned less than expected [D 20:01:02.245172] aiocb_ptr_array[0]: fd: 13, off: 192937984, bytes: 2097152, buf: 0x2aaab4217010, type: 1 [D 20:01:02.245270] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 459276288 fo 469762048 eof 0 [D 20:01:02.245284] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301c40 (handle 2863311531, ret 0)) [D 20:01:02.245291] ========================================================= [D 20:01:02.245307] DBPF I/O ops in progress: 15 [D 20:01:02.245313] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.245293] dbpf_open_cache_put: cached entry. [D 20:01:02.245318] starting delayed I/O operation 0x2aaabd324d90 (15 in progress) [D 20:01:02.245326] dbpf_bstream_rw_list: mem_offset: 0x2aaabba56010, mem_size: 2097152 [D 20:01:02.245337] lio_listio called with 1 following aiocbs: [D 20:01:02.245345] dbpf_bstream_rw_list: stream_offset: 457179136, stream_size: 2097152 [D 20:01:02.245349] op_queue add: 0x2aaabd300d80 [D 20:01:02.245353] aiocb_ptr_array[0]: fd: 13, off: 226492416, bytes: 2097152, buf: 0x2aaab4418010, type: 1 [D 20:01:02.245369] dbpf_open_cache_get: called [D 20:01:02.245379] start_delayed_ops_if_any: lio_listio posted 0x2aaabd324d90 (handle 2863311531, ret 0)) [D 20:01:02.245387] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.245403] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.245413] op_queue add: 0x2aaab152b470 [D 20:01:02.245427] DBPF I/O ops in progress: 15 [D 20:01:02.245457] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.245347] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.245484] BMI_post_send: addr: 6843072, offset: 0x2aaabde05010, size: 2097152, tag: 24 [D 20:01:02.245510] post_sr: 10.1.4.56:39990 bh 73 len 32 wr_id 683650. [D 20:01:02.245526] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:02.245537] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:02.245546] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.245468] starting delayed I/O operation 0x2aaabd301e40 (15 in progress) [D 20:01:02.245494] op_queue add: 0x2aaab15333d0 [D 20:01:02.245567] lio_listio called with 1 following aiocbs: [D 20:01:02.245594] aiocb_ptr_array[0]: fd: 13, off: 260046848, bytes: 2097152, buf: 0x2aaab4619010, type: 1 [D 20:01:02.245604] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301e40 (handle 2863311531, ret 0)) [D 20:01:02.245587] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.245438] dbpf_open_cache_put: cached entry. [D 20:01:02.245667] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.245674] op_queue add: 0x2aaab1522990 [D 20:01:02.245677] dbpf_open_cache_put: cached entry. [D 20:01:02.245691] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.245695] DBPF I/O ops in progress: 15 [D 20:01:02.245700] op_queue add: 0x2aaab152b210 [D 20:01:02.245693] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.245736] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.245712] starting delayed I/O operation 0x2aaabd301f90 (15 in progress) [D 20:01:02.245773] lio_listio called with 1 following aiocbs: [D 20:01:02.245781] aiocb_ptr_array[0]: fd: 13, off: 293601280, bytes: 2097152, buf: 0x2aaab481a010, type: 1 [D 20:01:02.245791] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301f90 (handle 2863311531, ret 0)) [D 20:01:02.245803] DBPF I/O ops in progress: 15 [D 20:01:02.245815] starting delayed I/O operation 0x2aaabd3020e0 (15 in progress) [D 20:01:02.245819] dbpf_open_cache_put: cached entry. [D 20:01:02.245824] lio_listio called with 1 following aiocbs: [D 20:01:02.245836] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.245851] dbpf_open_cache_put: cached entry. [D 20:01:02.245854] op_queue add: 0x2aaab1535f30 [D 20:01:02.245844] aiocb_ptr_array[0]: fd: 13, off: 327155712, bytes: 2097152, buf: 0x2aaab4a1b010, type: 1 [D 20:01:02.245877] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3020e0 (handle 2863311531, ret 0)) [D 20:01:02.245888] op_queue add: 0x2aaabd300b80 [D 20:01:02.245895] delayed I/O operation 0x2aaabd300b80 (16 already in progress) [D 20:01:02.245902] DBPF I/O ops in progress: 16 [D 20:01:02.245910] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.245918] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:02.245926] ========================================================= [D 20:01:02.245932] PINT_process_request [D 20:01:02.245938] tiling 2 copies [D 20:01:02.245944] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.245952] lvl 0 el 0 blk 0 by 18874368 [D 20:01:02.245958] to 18874368 ta 0 fi 33554432 [D 20:01:02.245964] basic type or contiguous data [D 20:01:02.245970] Request disp [D 20:01:02.245976] contig_offset = 18874368 contig_size = 48234496 lvl_flag = 1 [D 20:01:02.245983] calling distribute [D 20:01:02.245989] PINT_distribute [D 20:01:02.245994] of 18874368 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.246002] begin iteration loff: 18874368 [D 20:01:02.246008] segment exceeds byte limit [D 20:01:02.246014] process a segment [D 20:01:02.246019] of 18874368 sz 2097152 [D 20:01:02.246026] add a segment 0 [D 20:01:02.246032] end iteration [D 20:01:02.246037] done with segments or bytes [D 20:01:02.246043] of 20971520 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.246050] next loff: 20971520 next poff: 20971520 [D 20:01:02.246056] return value 2097152 [D 20:01:02.246063] finished [D 20:01:02.246068] retval = 2097152 [D 20:01:02.246074] exiting distribute returned less than expected [D 20:01:02.246081] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 20971520 fo 33554432 eof 0 [D 20:01:02.246088] ========================================================= [D 20:01:02.246095] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.246102] dbpf_bstream_rw_list: mem_offset: 0x2aaabca5e010, mem_size: 2097152 [D 20:01:02.246109] dbpf_bstream_rw_list: stream_offset: 18874368, stream_size: 2097152 [D 20:01:02.246115] dbpf_open_cache_get: called [D 20:01:02.246121] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.246127] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.246140] op_queue add: 0x2aaabd300f80 [D 20:01:02.246147] delayed I/O operation 0x2aaabd300f80 (16 already in progress) [D 20:01:02.246153] DBPF I/O ops in progress: 16 [D 20:01:02.246160] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.246167] (send callback time) ini posts: 0, pending: 2, last: 0, src_list emtpy: no [D 20:01:02.246174] ========================================================= [D 20:01:02.246180] PINT_process_request [D 20:01:02.246186] tiling 3 copies [D 20:01:02.246192] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.246200] lvl 0 el 0 blk 0 by 50331648 [D 20:01:02.246206] to 50331648 ta 33554432 fi 67108864 [D 20:01:02.246212] basic type or contiguous data [D 20:01:02.246218] Request disp [D 20:01:02.246224] contig_offset = 50331648 contig_size = 50331648 lvl_flag = 1 [D 20:01:02.246230] calling distribute [D 20:01:02.246236] PINT_distribute [D 20:01:02.246242] of 50331648 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.246249] begin iteration loff: 50331648 [D 20:01:02.246255] segment exceeds byte limit [D 20:01:02.246261] process a segment [D 20:01:02.246267] of 50331648 sz 2097152 [D 20:01:02.246273] add a segment 0 [D 20:01:02.246279] end iteration [D 20:01:02.246284] done with segments or bytes [D 20:01:02.246290] of 52428800 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.246297] next loff: 52428800 next poff: 52428800 [D 20:01:02.246303] return value 2097152 [D 20:01:02.246309] finished [D 20:01:02.246315] retval = 2097152 [D 20:01:02.246321] exiting distribute returned less than expected [D 20:01:02.246327] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 52428800 fo 67108864 eof 0 [D 20:01:02.246334] ========================================================= [D 20:01:02.246341] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.246348] dbpf_bstream_rw_list: mem_offset: 0x2aaabda03010, mem_size: 2097152 [D 20:01:02.246354] dbpf_bstream_rw_list: stream_offset: 50331648, stream_size: 2097152 [D 20:01:02.246361] dbpf_open_cache_get: called [D 20:01:02.246367] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.246373] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.246385] op_queue add: 0x2aaabd300d80 [D 20:01:02.246392] delayed I/O operation 0x2aaabd300d80 (16 already in progress) [D 20:01:02.246399] DBPF I/O ops in progress: 16 [D 20:01:02.246410] openib_check_cq: completed send wr_id 683620. [D 20:01:02.246418] openib_check_cq: completed send wr_id 683650. [D 20:01:02.252527] DBPF I/O ops in progress: 15 [D 20:01:02.252536] starting delayed I/O operation 0x2aaabd310ef0 (15 in progress) [D 20:01:02.252545] lio_listio called with 1 following aiocbs: [D 20:01:02.252551] aiocb_ptr_array[0]: fd: 13, off: 360710144, bytes: 2097152, buf: 0x2aaab4c1c010, type: 1 [D 20:01:02.252560] start_delayed_ops_if_any: lio_listio posted 0x2aaabd310ef0 (handle 2863311531, ret 0)) [D 20:01:02.252569] DBPF I/O ops in progress: 15 [D 20:01:02.252577] starting delayed I/O operation 0x2aaabd300a30 (15 in progress) [D 20:01:02.252585] lio_listio called with 1 following aiocbs: [D 20:01:02.252592] aiocb_ptr_array[0]: fd: 13, off: 394264576, bytes: 2097152, buf: 0x2aaab4e1d010, type: 1 [D 20:01:02.252600] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300a30 (handle 2863311531, ret 0)) [D 20:01:02.245863] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.339636] op_queue add: 0x2aaab1522e30 [D 20:01:02.339651] DBPF I/O ops in progress: 15 [D 20:01:02.339660] starting delayed I/O operation 0x2aaabd311110 (15 in progress) [D 20:01:02.339671] lio_listio called with 1 following aiocbs: [D 20:01:02.339678] aiocb_ptr_array[0]: fd: 13, off: 486539264, bytes: 2097152, buf: 0x2aaab521f010, type: 1 [D 20:01:02.339692] start_delayed_ops_if_any: lio_listio posted 0x2aaabd311110 (handle 2863311531, ret 0)) [D 20:01:02.339713] DBPF I/O ops in progress: 15 [D 20:01:02.339723] starting delayed I/O operation 0x2aaabd302230 (15 in progress) [D 20:01:02.339733] lio_listio called with 1 following aiocbs: [D 20:01:02.339741] aiocb_ptr_array[0]: fd: 13, off: 427819008, bytes: 2097152, buf: 0x2aaab501e010, type: 1 [D 20:01:02.339751] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302230 (handle 2863311531, ret 0)) [D 20:01:02.252636] dbpf_open_cache_put: cached entry. [D 20:01:02.339767] DBPF I/O ops in progress: 15 [D 20:01:02.339796] starting delayed I/O operation 0x2aaabd302380 (15 in progress) [D 20:01:02.339776] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.339813] lio_listio called with 1 following aiocbs: [D 20:01:02.254225] --- aio_progress_notification called with handle 2863311531 (0x2aaab1522b60) [D 20:01:02.254923] --- aio_progress_notification called with handle 2863311531 (0x2aaab152bb90) [D 20:01:02.339868] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.339878] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.339896] dbpf_open_cache_put: cached entry. [D 20:01:02.339924] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.264582] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301c40) [D 20:01:02.339932] dbpf_open_cache_put: cached entry. [D 20:01:02.264664] --- aio_progress_notification called with handle 2863311531 (0x2aaabd308af0) [D 20:01:02.267632] --- aio_progress_notification called with handle 2863311531 (0x2aaabd324d90) [D 20:01:02.339951] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.264683] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3112d0) -[D 20:01:02.339998] openib_check_cq: completed recv wr_id 684470. [D 20:01:02.340013] openib_post_rr: 10.1.4.56:39990 bh 48 wr_id 684470. [D 20:01:02.340023] openib_check_cq: completed recv wr_id 6844a0. [D 20:01:02.340036] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec660 to 10.1.4.56:39990 remote addr a064f000 rkey 108d27 nic_wr_credit 1. [D 20:01:02.340046] openib_post_rr: 10.1.4.56:39990 bh 49 wr_id 6844a0. [D 20:01:02.340054] openib_check_cq: completed recv wr_id 6844d0. [D 20:01:02.340061] openib_post_sr_rdmaw: ibv_post_send wr_id 6ecb00 to 10.1.4.56:39990 remote addr 88a37000 rkey 108a27 nic_wr_credit 2. [D 20:01:02.340069] openib_post_rr: 10.1.4.56:39990 bh 50 wr_id 6844d0. [D 20:01:02.340077] openib_check_cq: completed recv wr_id 684500. [D 20:01:02.340086] openib_post_sr_rdmaw: ibv_post_send wr_id 6ecf80 to 10.1.4.56:39990 remote addr a2851000 rkey 108c27 nic_wr_credit 3. [D 20:01:02.340094] openib_post_rr: 10.1.4.56:39990 bh 51 wr_id 684500. [D 20:01:02.340102] openib_check_cq: completed recv wr_id 684530. [D 20:01:02.340109] openib_post_rr: 10.1.4.56:39990 bh 52 wr_id 684530. [D 20:01:02.340116] openib_check_cq: completed recv wr_id 684560. [D 20:01:02.340124] openib_post_rr: 10.1.4.56:39990 bh 53 wr_id 684560. [D 20:01:02.340132] openib_check_cq: completed recv wr_id 684590. [D 20:01:02.340139] openib_post_rr: 10.1.4.56:39990 bh 54 wr_id 684590. [D 20:01:02.340146] openib_check_cq: completed recv wr_id 6845c0. [D 20:01:02.340154] openib_post_rr: 10.1.4.56:39990 bh 55 wr_id 6845c0. [D 20:01:02.340161] openib_check_cq: completed recv wr_id 6845f0. [D 20:01:02.340169] openib_post_rr: 10.1.4.56:39990 bh 56 wr_id 6845f0. [D 20:01:02.340176] openib_check_cq: completed recv wr_id 684620. [D 20:01:02.340184] openib_post_rr: 10.1.4.56:39990 bh 57 wr_id 684620. [D 20:01:02.340191] openib_check_cq: completed recv wr_id 684650. [D 20:01:02.340198] openib_post_rr: 10.1.4.56:39990 bh 58 wr_id 684650. [D 20:01:02.340205] openib_check_cq: completed recv wr_id 684680. [D 20:01:02.340214] openib_post_rr: 10.1.4.56:39990 bh 59 wr_id 684680. [D 20:01:02.340222] openib_check_cq: completed recv wr_id 6846b0. [D 20:01:02.340229] openib_post_rr: 10.1.4.56:39990 bh 60 wr_id 6846b0. [D 20:01:02.340237] openib_check_cq: completed recv wr_id 6846e0. [D 20:01:02.340245] openib_post_rr: 10.1.4.56:39990 bh 61 wr_id 6846e0. [D 20:01:02.340252] openib_check_cq: completed recv wr_id 684710. [D 20:01:02.340260] openib_post_rr: 10.1.4.56:39990 bh 62 wr_id 684710. [D 20:01:02.340267] openib_check_cq: completed recv wr_id 684740. [D 20:01:02.340275] openib_post_rr: 10.1.4.56:39990 bh 63 wr_id 684740. [D 20:01:02.340282] openib_check_cq: completed recv wr_id 684770. [D 20:01:02.340290] openib_post_rr: 10.1.4.56:39990 bh 64 wr_id 684770. [D 20:01:02.340297] openib_check_cq: completed recv wr_id 6847a0. [D 20:01:02.340306] openib_post_rr: 10.1.4.56:39990 bh 65 wr_id 6847a0. [D 20:01:02.340313] openib_check_cq: completed recv wr_id 6847d0. [D 20:01:02.340323] openib_post_rr: 10.1.4.56:39990 bh 66 wr_id 6847d0. [D 20:01:02.340330] openib_check_cq: completed recv wr_id 684800. [D 20:01:02.340338] openib_post_rr: 10.1.4.56:39990 bh 67 wr_id 684800. [D 20:01:02.340346] openib_check_cq: completed recv wr_id 684830. [D 20:01:02.340353] openib_post_rr: 10.1.4.56:39990 bh 68 wr_id 684830. [D 20:01:02.340361] openib_check_cq: completed recv wr_id 684860. [D 20:01:02.340369] openib_post_rr: 10.1.4.56:39990 bh 69 wr_id 684860. [D 20:01:02.340376] openib_check_cq: completed recv wr_id 684890. [D 20:01:02.340384] openib_post_rr: 10.1.4.56:39990 bh 70 wr_id 684890. [D 20:01:02.340392] openib_check_cq: completed recv wr_id 6848c0. [D 20:01:02.340400] openib_post_rr: 10.1.4.56:39990 bh 71 wr_id 6848c0. [D 20:01:02.340407] openib_check_cq: completed recv wr_id 6848f0. [D 20:01:02.340416] openib_post_rr: 10.1.4.56:39990 bh 72 wr_id 6848f0. [D 20:01:02.340423] openib_check_cq: completed recv wr_id 684920. [D 20:01:02.340431] openib_post_rr: 10.1.4.56:39990 bh 73 wr_id 684920. [D 20:01:02.340438] openib_check_cq: completed recv wr_id 684950. [D 20:01:02.340446] openib_post_rr: 10.1.4.56:39990 bh 74 wr_id 684950. [D 20:01:02.340453] openib_check_cq: completed recv wr_id 684980. [D 20:01:02.340461] openib_post_rr: 10.1.4.56:39990 bh 75 wr_id 684980. [D 20:01:02.340469] openib_check_cq: completed recv wr_id 6849b0. [D 20:01:02.340476] openib_post_rr: 10.1.4.56:39990 bh 76 wr_id 6849b0. [D 20:01:02.340484] openib_check_cq: completed recv wr_id 6849e0. [D 20:01:02.340491] openib_post_rr: 10.1.4.56:39990 bh 77 wr_id 6849e0. [D 20:01:02.340499] openib_check_cq: completed recv wr_id 684a10. [D 20:01:02.340507] openib_post_rr: 10.1.4.56:39990 bh 78 wr_id 684a10. [D 20:01:02.340514] openib_check_cq: completed recv wr_id 684a40. [D 20:01:02.340522] openib_post_rr: 10.1.4.56:39990 bh 79 wr_id 684a40. [D 20:01:02.340535] openib_check_cq: completed recv wr_id 684a70. [D 20:01:02.340542] openib_post_rr: 10.1.4.56:39990 bh 80 wr_id 684a70. [D 20:01:02.340565] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:01:02.340576] get_data_sync_mode returning 0 [D 20:01:02.340590] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.340597] dbpf_bstream_rw_list: mem_offset: 0x2aaab6027010, mem_size: 2097152 [D 20:01:02.340604] dbpf_bstream_rw_list: stream_offset: 161480704, stream_size: 2097152 [D 20:01:02.340612] dbpf_open_cache_get: called [D 20:01:02.340625] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.272583] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301e40) [D 20:01:02.340649] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.276639] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301f90) [D 20:01:02.340674] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.280561] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3020e0) [D 20:01:02.339817] op_queue add: 0x2aaab1522750 [D 20:01:02.339837] aiocb_ptr_array[0]: fd: 13, off: 461373440, bytes: 2097152, buf: 0x2aaab5420010, type: 1 [D 20:01:02.340710] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302380 (handle 2863311531, ret 0)) [D 20:01:02.340729] op_queue add: 0x2aaab152be30 [D 20:01:02.340735] op_queue add: 0x2aaab1522b60 [D 20:01:02.340745] delayed I/O operation 0x2aaab152be30 (16 already in progress) [D 20:01:02.340740] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.340752] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.340774] dbpf_open_cache_put: cached entry. [D 20:01:02.340696] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.340764] DBPF I/O ops in progress: 16 [D 20:01:02.340788] dbpf_open_cache_put: cached entry. [D 20:01:02.340803] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.340813] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.340821] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b630. [D 20:01:02.340830] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.340841] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.340786] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.340877] dbpf_open_cache_put: cached entry. [D 20:01:02.340886] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.340815] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.340906] dbpf_open_cache_put: cached entry. [D 20:01:02.340913] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.340824] DBPF I/O ops in progress: 15 [D 20:01:02.342184] dbpf_open_cache_put: cached entry. [D 20:01:02.342216] starting delayed I/O operation 0x2aaabd3024d0 (15 in progress) [D 20:01:02.342230] dbpf_open_cache_put: cached entry. [D 20:01:02.342237] lio_listio called with 1 following aiocbs: [D 20:01:02.342222] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.342249] aiocb_ptr_array[0]: fd: 13, off: 27262976, bytes: 2097152, buf: 0x2aaab5621010, type: 1 [D 20:01:02.342241] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.342263] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3024d0 (handle 2863311531, ret 0)) [D 20:01:02.342337] ========================================================= [D 20:01:02.342347] PINT_process_request [D 20:01:02.342354] tiling 4 copies [D 20:01:02.342361] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.342368] lvl 0 el 0 blk 0 by 79691776 [D 20:01:02.342375] to 79691776 ta 67108864 fi 100663296 [D 20:01:02.342382] basic type or contiguous data [D 20:01:02.342388] Request disp [D 20:01:02.342394] contig_offset = 79691776 contig_size = 54525952 lvl_flag = 1 [D 20:01:02.342401] calling distribute [D 20:01:02.342407] PINT_distribute [D 20:01:02.342413] of 79691776 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.342422] begin iteration loff: 79691776 [D 20:01:02.342429] segment exceeds byte limit [D 20:01:02.342435] process a segment [D 20:01:02.342441] of 79691776 sz 2097152 [D 20:01:02.342447] add a segment 0 [D 20:01:02.342453] end iteration [D 20:01:02.342459] done with segments or bytes [D 20:01:02.342465] of 81788928 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.342472] next loff: 81788928 next poff: 81788928 [D 20:01:02.342478] return value 2097152 [D 20:01:02.342485] finished [D 20:01:02.342490] retval = 2097152 [D 20:01:02.342496] exiting distribute returned less than expected [D 20:01:02.342503] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 81788928 fo 100663296 eof 0 [D 20:01:02.342510] ========================================================= [D 20:01:02.342521] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.342528] dbpf_bstream_rw_list: mem_offset: 0x2aaabf611010, mem_size: 2097152 [D 20:01:02.342535] dbpf_bstream_rw_list: stream_offset: 79691776, stream_size: 2097152 [D 20:01:02.342542] dbpf_open_cache_get: called [D 20:01:02.342548] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.342555] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.342569] op_queue add: 0x2aaab152b470 [D 20:01:02.342576] delayed I/O operation 0x2aaab152b470 (16 already in progress) [D 20:01:02.342583] DBPF I/O ops in progress: 16 [D 20:01:02.342589] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.342595] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b630. [D 20:01:02.342603] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.344089] DBPF I/O ops in progress: 15 [D 20:01:02.344119] starting delayed I/O operation 0x2aaabd3026f0 (15 in progress) [D 20:01:02.344129] lio_listio called with 1 following aiocbs: [D 20:01:02.344135] aiocb_ptr_array[0]: fd: 13, off: 60817408, bytes: 2097152, buf: 0x2aaab5822010, type: 1 [D 20:01:02.344145] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3026f0 (handle 2863311531, ret 0)) [D 20:01:02.344174] DBPF I/O ops in progress: 15 [D 20:01:02.344189] starting delayed I/O operation 0x2aaabd302840 (15 in progress) [D 20:01:02.344200] lio_listio called with 1 following aiocbs: [D 20:01:02.344206] aiocb_ptr_array[0]: fd: 13, off: 94371840, bytes: 2097152, buf: 0x2aaab5a23010, type: 1 [D 20:01:02.344217] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302840 (handle 2863311531, ret 0)) [D 20:01:02.340781] op_queue add: 0x2aaab152bb90 [D 20:01:02.344232] op_queue add: 0x2aaabd30b770 [D 20:01:02.344277] op_queue add: 0x2aaabd3112d0 [D 20:01:02.344218] ========================================================= [D 20:01:02.344246] delayed I/O operation 0x2aaabd30b770 (16 already in progress) [D 20:01:02.344298] PINT_process_request [D 20:01:02.344303] DBPF I/O ops in progress: 16 [D 20:01:02.344312] tiling 4 copies [D 20:01:02.344326] ========================================================= [D 20:01:02.344333] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.344338] PINT_process_request [D 20:01:02.344359] tiling 6 copies [D 20:01:02.344369] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.344343] DBPF I/O ops in progress: 15 [D 20:01:02.344331] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.344382] starting delayed I/O operation 0x2aaabd302990 (15 in progress) [D 20:01:02.344359] dbpf_open_cache_put: cached entry. [D 20:01:02.344377] lvl 0 el 0 blk 0 by 163577856 [D 20:01:02.344400] lio_listio called with 1 following aiocbs: [D 20:01:02.344389] lvl 0 el 0 blk 0 by 81788928 [D 20:01:02.344421] to 163577856 ta 134217728 fi 167772160 [D 20:01:02.344451] basic type or contiguous data [D 20:01:02.344461] Request disp [D 20:01:02.344470] contig_offset = 163577856 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.344480] calling distribute [D 20:01:02.344490] PINT_distribute [D 20:01:02.344499] of 163577856 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 163577856 exfl 1 [D 20:01:02.344512] begin iteration loff: 163577856 [D 20:01:02.344522] segment exceeds byte limit [D 20:01:02.344548] file being extended [D 20:01:02.344557] process a segment [D 20:01:02.344427] aiocb_ptr_array[0]: fd: 13, off: 127926272, bytes: 2097152, buf: 0x2aaab5e26010, type: 1 [D 20:01:02.344433] to 81788928 ta 67108864 fi 100663296 [D 20:01:02.344408] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.344564] of 163577856 sz 2097152 [D 20:01:02.344589] add a segment 0 [D 20:01:02.344596] end iteration [D 20:01:02.344602] done with segments or bytes [D 20:01:02.344608] of 165675008 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.344616] next loff: 165675008 next poff: 165675008 [D 20:01:02.344623] return value 2097152 [D 20:01:02.344630] finished [D 20:01:02.344636] retval = 2097152 [D 20:01:02.344643] exiting distribute returned less than expected [D 20:01:02.344649] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 165675008 fo 167772160 eof 0 [D 20:01:02.344657] ========================================================= [D 20:01:02.344666] BMI_post_recv: addr: 6843072, offset: 0x2aaab7c36010, size: 2097152, tag: 12 [D 20:01:02.344685] openib_check_cq: completed rdma wr_id 6ec660, nic_wr_credit 2. [D 20:01:02.344698] post_sr: 10.1.4.56:39990 bh 74 len 16 wr_id 683680. [D 20:01:02.344710] openib_check_cq: completed rdma wr_id 6ecb00, nic_wr_credit 1. [D 20:01:02.344718] post_sr: 10.1.4.56:39990 bh 75 len 16 wr_id 6836b0. [D 20:01:02.344754] post_sr: 10.1.4.56:39990 bh 76 len 48 wr_id 6836e0. [D 20:01:02.344767] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:01:02.344777] get_data_sync_mode returning 0 [D 20:01:02.344792] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.344799] dbpf_bstream_rw_list: mem_offset: 0x2aaab6228010, mem_size: 2097152 [D 20:01:02.344806] dbpf_bstream_rw_list: stream_offset: 195035136, stream_size: 2097152 [D 20:01:02.344814] dbpf_open_cache_get: called [D 20:01:02.344821] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.344829] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.344572] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302990 (handle 2863311531, ret 0)) [D 20:01:02.344582] basic type or contiguous data [D 20:01:02.344879] Request disp [D 20:01:02.344886] contig_offset = 81788928 contig_size = 52428800 lvl_flag = 1 [D 20:01:02.344341] op_queue add: 0x2aaabd301c40 [D 20:01:02.344895] calling distribute [D 20:01:02.344914] op_queue add: 0x2aaabd308af0 [D 20:01:02.344924] PINT_distribute [D 20:01:02.344941] op_queue add: 0x2aaabd3020e0 [D 20:01:02.344950] of 81788928 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.344967] op_queue add: 0x2aaabd301e40 [D 20:01:02.344978] begin iteration loff: 81788928 [D 20:01:02.344990] op_queue add: 0x2aaabd301f90 [D 20:01:02.344992] segment exceeds byte limit [D 20:01:02.345016] process a segment [D 20:01:02.345022] of 81788928 sz 2097152 [D 20:01:02.345029] add a segment 0 [D 20:01:02.345035] end iteration [D 20:01:02.345040] done with segments or bytes [D 20:01:02.345046] of 83886080 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.345053] next loff: 83886080 next poff: 83886080 [D 20:01:02.345059] return value 2097152 [D 20:01:02.345065] finished [D 20:01:02.345071] retval = 2097152 [D 20:01:02.345077] exiting distribute returned less than expected [D 20:01:02.345083] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 83886080 fo 100663296 eof 0 [D 20:01:02.345091] ========================================================= [D 20:01:02.345100] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.345107] dbpf_bstream_rw_list: mem_offset: 0x2aaabf812010, mem_size: 2097152 [D 20:01:02.345113] dbpf_bstream_rw_list: stream_offset: 81788928, stream_size: 2097152 [D 20:01:02.345120] dbpf_open_cache_get: called [D 20:01:02.345126] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.345132] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.344899] DBPF I/O ops in progress: 15 [D 20:01:02.345157] starting delayed I/O operation 0x2aaab152bce0 (15 in progress) [D 20:01:02.345167] lio_listio called with 1 following aiocbs: [D 20:01:02.345174] aiocb_ptr_array[0]: fd: 13, off: 75497472, bytes: 2097152, buf: 0x2aaabf20f010, type: 0 [D 20:01:02.345182] start_delayed_ops_if_any: lio_listio posted 0x2aaab152bce0 (handle 2863311531, ret 0)) [D 20:01:02.345193] op_queue add: 0x2aaabd305f70 [D 20:01:02.345205] delayed I/O operation 0x2aaabd305f70 (16 already in progress) [D 20:01:02.345208] op_queue add: 0x2aaabd324d90 [D 20:01:02.345212] DBPF I/O ops in progress: 16 [D 20:01:02.345231] ========================================================= [D 20:01:02.345238] PINT_process_request [D 20:01:02.345244] tiling 7 copies [D 20:01:02.345250] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.345258] lvl 0 el 0 blk 0 by 197132288 [D 20:01:02.345265] to 197132288 ta 167772160 fi 201326592 [D 20:01:02.345271] basic type or contiguous data [D 20:01:02.345277] Request disp [D 20:01:02.345283] contig_offset = 197132288 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.345290] calling distribute [D 20:01:02.345296] PINT_distribute [D 20:01:02.345302] of 197132288 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 197132288 exfl 1 [D 20:01:02.345310] begin iteration loff: 197132288 [D 20:01:02.345316] segment exceeds byte limit [D 20:01:02.345322] file being extended [D 20:01:02.345328] process a segment [D 20:01:02.345333] of 197132288 sz 2097152 [D 20:01:02.345340] add a segment 0 [D 20:01:02.345345] end iteration [D 20:01:02.345351] done with segments or bytes [D 20:01:02.345357] of 199229440 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.345364] next loff: 199229440 next poff: 199229440 [D 20:01:02.345371] return value 2097152 [D 20:01:02.345376] finished [D 20:01:02.345382] retval = 2097152 [D 20:01:02.345388] exiting distribute returned less than expected [D 20:01:02.345394] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 199229440 fo 201326592 eof 0 [D 20:01:02.345402] ========================================================= [D 20:01:02.345409] BMI_post_recv: addr: 6843072, offset: 0x2aaab7e37010, size: 2097152, tag: 13 [D 20:01:02.345422] post_sr: 10.1.4.56:39990 bh 77 len 48 wr_id 683710. [D 20:01:02.345432] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:01:02.345440] get_data_sync_mode returning 0 [D 20:01:02.345448] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.345454] dbpf_bstream_rw_list: mem_offset: 0x2aaab642a010, mem_size: 2097152 [D 20:01:02.345461] dbpf_bstream_rw_list: stream_offset: 228589568, stream_size: 2097152 [D 20:01:02.345468] dbpf_open_cache_get: called [D 20:01:02.345474] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.345480] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.345525] op_queue add: 0x2aaabd3013e0 [D 20:01:02.345534] delayed I/O operation 0x2aaabd3013e0 (16 already in progress) [D 20:01:02.345541] DBPF I/O ops in progress: 16 [D 20:01:02.345549] ========================================================= [D 20:01:02.345556] PINT_process_request [D 20:01:02.345562] tiling 8 copies [D 20:01:02.345568] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.345576] lvl 0 el 0 blk 0 by 230686720 [D 20:01:02.345583] to 230686720 ta 201326592 fi 234881024 [D 20:01:02.345591] basic type or contiguous data [D 20:01:02.345597] Request disp [D 20:01:02.345603] contig_offset = 230686720 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.345610] calling distribute [D 20:01:02.345616] PINT_distribute [D 20:01:02.345622] of 230686720 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 230686720 exfl 1 [D 20:01:02.345630] begin iteration loff: 230686720 [D 20:01:02.345636] segment exceeds byte limit [D 20:01:02.345642] file being extended [D 20:01:02.345648] process a segment [D 20:01:02.345654] of 230686720 sz 2097152 [D 20:01:02.345661] add a segment 0 [D 20:01:02.345667] end iteration [D 20:01:02.345673] done with segments or bytes [D 20:01:02.345679] of 232783872 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.345686] next loff: 232783872 next poff: 232783872 [D 20:01:02.345692] return value 2097152 [D 20:01:02.345699] finished [D 20:01:02.345704] retval = 2097152 [D 20:01:02.345711] exiting distribute returned less than expected [D 20:01:02.345717] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 232783872 fo 234881024 eof 0 [D 20:01:02.345725] ========================================================= [D 20:01:02.345732] BMI_post_recv: addr: 6843072, offset: 0x2aaab8038010, size: 2097152, tag: 14 [D 20:01:02.345744] post_sr: 10.1.4.56:39990 bh 78 len 48 wr_id 683740. [D 20:01:02.345754] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:01:02.345762] get_data_sync_mode returning 0 [D 20:01:02.345771] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.345778] dbpf_bstream_rw_list: mem_offset: 0x2aaab662b010, mem_size: 2097152 [D 20:01:02.345784] dbpf_bstream_rw_list: stream_offset: 262144000, stream_size: 2097152 [D 20:01:02.345791] dbpf_open_cache_get: called [D 20:01:02.345797] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.345804] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.345828] op_queue add: 0x2aaabd302ff0 [D 20:01:02.345837] delayed I/O operation 0x2aaabd302ff0 (16 already in progress) [D 20:01:02.345844] DBPF I/O ops in progress: 16 [D 20:01:02.345851] ========================================================= [D 20:01:02.345857] PINT_process_request [D 20:01:02.345863] tiling 9 copies [D 20:01:02.345869] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.345877] lvl 0 el 0 blk 0 by 264241152 [D 20:01:02.345883] to 264241152 ta 234881024 fi 268435456 [D 20:01:02.345890] basic type or contiguous data [D 20:01:02.345896] Request disp [D 20:01:02.345902] contig_offset = 264241152 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.345909] calling distribute [D 20:01:02.345916] PINT_distribute [D 20:01:02.345922] of 264241152 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 264241152 exfl 1 [D 20:01:02.345930] begin iteration loff: 264241152 [D 20:01:02.345936] segment exceeds byte limit [D 20:01:02.345942] file being extended [D 20:01:02.345948] process a segment [D 20:01:02.345954] of 264241152 sz 2097152 [D 20:01:02.345960] add a segment 0 [D 20:01:02.345966] end iteration [D 20:01:02.345972] done with segments or bytes [D 20:01:02.345978] of 266338304 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.345986] next loff: 266338304 next poff: 266338304 [D 20:01:02.345992] return value 2097152 [D 20:01:02.345998] finished [D 20:01:02.346004] retval = 2097152 [D 20:01:02.346010] exiting distribute returned less than expected [D 20:01:02.346016] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 266338304 fo 268435456 eof 0 [D 20:01:02.346024] ========================================================= [D 20:01:02.346030] BMI_post_recv: addr: 6843072, offset: 0x2aaab8239010, size: 2097152, tag: 15 [D 20:01:02.346042] post_sr: 10.1.4.56:39990 bh 79 len 48 wr_id 683770. [D 20:01:02.346052] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:01:02.346060] get_data_sync_mode returning 0 [D 20:01:02.346067] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.346074] dbpf_bstream_rw_list: mem_offset: 0x2aaab682c010, mem_size: 2097152 [D 20:01:02.346081] dbpf_bstream_rw_list: stream_offset: 295698432, stream_size: 2097152 [D 20:01:02.346087] dbpf_open_cache_get: called [D 20:01:02.346094] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.346101] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.346130] op_queue add: 0x2aaabd303140 [D 20:01:02.346139] delayed I/O operation 0x2aaabd303140 (16 already in progress) [D 20:01:02.346145] DBPF I/O ops in progress: 16 [D 20:01:02.346153] ========================================================= [D 20:01:02.346159] PINT_process_request [D 20:01:02.346166] tiling 10 copies [D 20:01:02.346172] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.346180] lvl 0 el 0 blk 0 by 297795584 [D 20:01:02.346186] to 297795584 ta 268435456 fi 301989888 [D 20:01:02.346193] basic type or contiguous data [D 20:01:02.346200] Request disp [D 20:01:02.346206] contig_offset = 297795584 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.346212] calling distribute [D 20:01:02.346218] PINT_distribute [D 20:01:02.346224] of 297795584 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 297795584 exfl 1 [D 20:01:02.346232] begin iteration loff: 297795584 [D 20:01:02.346238] segment exceeds byte limit [D 20:01:02.346244] file being extended [D 20:01:02.346250] process a segment [D 20:01:02.346256] of 297795584 sz 2097152 [D 20:01:02.346262] add a segment 0 [D 20:01:02.346269] end iteration [D 20:01:02.346275] done with segments or bytes [D 20:01:02.346281] of 299892736 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.346288] next loff: 299892736 next poff: 299892736 [D 20:01:02.346294] return value 2097152 [D 20:01:02.346301] finished [D 20:01:02.346306] retval = 2097152 [D 20:01:02.346313] exiting distribute returned less than expected [D 20:01:02.346319] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 299892736 fo 301989888 eof 0 [D 20:01:02.346326] ========================================================= [D 20:01:02.346332] BMI_post_recv: addr: 6843072, offset: 0x2aaab843a010, size: 2097152, tag: 16 [D 20:01:02.346345] post_sr: 10.1.4.56:39990 bh 80 len 48 wr_id 6837a0. [D 20:01:02.346377] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:01:02.346385] get_data_sync_mode returning 0 [D 20:01:02.346393] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.346400] dbpf_bstream_rw_list: mem_offset: 0x2aaab6a2d010, mem_size: 2097152 [D 20:01:02.346407] dbpf_bstream_rw_list: stream_offset: 329252864, stream_size: 2097152 [D 20:01:02.346414] dbpf_open_cache_get: called [D 20:01:02.346421] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.346428] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.346458] op_queue add: 0x2aaabd303340 [D 20:01:02.346467] delayed I/O operation 0x2aaabd303340 (16 already in progress) [D 20:01:02.346474] DBPF I/O ops in progress: 16 [D 20:01:02.346481] ========================================================= [D 20:01:02.346488] PINT_process_request [D 20:01:02.346494] tiling 11 copies [D 20:01:02.346500] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.346507] lvl 0 el 0 blk 0 by 331350016 [D 20:01:02.346514] to 331350016 ta 301989888 fi 335544320 [D 20:01:02.346521] basic type or contiguous data [D 20:01:02.346527] Request disp [D 20:01:02.346533] contig_offset = 331350016 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.346540] calling distribute [D 20:01:02.346546] PINT_distribute [D 20:01:02.346552] of 331350016 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 331350016 exfl 1 [D 20:01:02.346560] begin iteration loff: 331350016 [D 20:01:02.346566] segment exceeds byte limit [D 20:01:02.346572] file being extended [D 20:01:02.346578] process a segment [D 20:01:02.346584] of 331350016 sz 2097152 [D 20:01:02.346591] add a segment 0 [D 20:01:02.346597] end iteration [D 20:01:02.346603] done with segments or bytes [D 20:01:02.346609] of 333447168 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.346616] next loff: 333447168 next poff: 333447168 [D 20:01:02.346622] return value 2097152 [D 20:01:02.346628] finished [D 20:01:02.346634] retval = 2097152 [D 20:01:02.346640] exiting distribute returned less than expected [D 20:01:02.346646] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 333447168 fo 335544320 eof 0 [D 20:01:02.346654] ========================================================= [D 20:01:02.346660] BMI_post_recv: addr: 6843072, offset: 0x2aaab863b010, size: 2097152, tag: 17 [D 20:01:02.346673] post_sr: 10.1.4.56:39990 bh 81 len 48 wr_id 6837d0. [D 20:01:02.346683] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:01:02.346692] get_data_sync_mode returning 0 [D 20:01:02.346699] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.346705] dbpf_bstream_rw_list: mem_offset: 0x2aaab6c2e010, mem_size: 2097152 [D 20:01:02.346712] dbpf_bstream_rw_list: stream_offset: 362807296, stream_size: 2097152 [D 20:01:02.346719] dbpf_open_cache_get: called [D 20:01:02.346725] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.346732] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.346754] op_queue add: 0x2aaabd3015f0 [D 20:01:02.346763] delayed I/O operation 0x2aaabd3015f0 (16 already in progress) [D 20:01:02.346770] DBPF I/O ops in progress: 16 [D 20:01:02.346777] ========================================================= [D 20:01:02.346784] PINT_process_request [D 20:01:02.346790] tiling 12 copies [D 20:01:02.346797] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.346804] lvl 0 el 0 blk 0 by 364904448 [D 20:01:02.346811] to 364904448 ta 335544320 fi 369098752 [D 20:01:02.346818] basic type or contiguous data [D 20:01:02.346824] Request disp [D 20:01:02.346830] contig_offset = 364904448 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.346837] calling distribute [D 20:01:02.346843] PINT_distribute [D 20:01:02.346849] of 364904448 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 364904448 exfl 1 [D 20:01:02.346856] begin iteration loff: 364904448 [D 20:01:02.346863] segment exceeds byte limit [D 20:01:02.346869] file being extended [D 20:01:02.346874] process a segment [D 20:01:02.346880] of 364904448 sz 2097152 [D 20:01:02.346886] add a segment 0 [D 20:01:02.346892] end iteration [D 20:01:02.346898] done with segments or bytes [D 20:01:02.346904] of 367001600 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.346912] next loff: 367001600 next poff: 367001600 [D 20:01:02.346918] return value 2097152 [D 20:01:02.346924] finished [D 20:01:02.346930] retval = 2097152 [D 20:01:02.346936] exiting distribute returned less than expected [D 20:01:02.346942] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 367001600 fo 369098752 eof 0 [D 20:01:02.346949] ========================================================= [D 20:01:02.346955] BMI_post_recv: addr: 6843072, offset: 0x2aaab883c010, size: 2097152, tag: 18 [D 20:01:02.346966] openib_check_cq: completed rdma wr_id 6ecf80, nic_wr_credit 0. [D 20:01:02.346975] post_sr: 10.1.4.56:39990 bh 82 len 16 wr_id 683800. [D 20:01:02.346983] openib_check_cq: completed send wr_id 683680. [D 20:01:02.346991] openib_check_cq: completed send wr_id 6836b0. [D 20:01:02.346998] openib_check_cq: completed send wr_id 6836e0. [D 20:01:02.347005] openib_check_cq: completed send wr_id 683710. [D 20:01:02.347013] openib_check_cq: completed send wr_id 683740. [D 20:01:02.347019] openib_check_cq: completed send wr_id 683770. [D 20:01:02.347026] openib_check_cq: completed send wr_id 6837a0. [D 20:01:02.347032] openib_check_cq: completed send wr_id 6837d0. [D 20:01:02.347039] openib_check_cq: completed send wr_id 683800. [D 20:01:02.347051] post_sr: 10.1.4.56:39990 bh 83 len 48 wr_id 683830. [D 20:01:02.347060] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:01:02.347068] get_data_sync_mode returning 0 [D 20:01:02.347075] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.347082] dbpf_bstream_rw_list: mem_offset: 0x2aaab6e2f010, mem_size: 2097152 [D 20:01:02.347089] dbpf_bstream_rw_list: stream_offset: 396361728, stream_size: 2097152 [D 20:01:02.347096] dbpf_open_cache_get: called [D 20:01:02.347102] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.347109] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.347143] op_queue add: 0x2aaabd301740 [D 20:01:02.347153] delayed I/O operation 0x2aaabd301740 (16 already in progress) [D 20:01:02.347160] DBPF I/O ops in progress: 16 [D 20:01:02.347167] ========================================================= [D 20:01:02.347174] PINT_process_request [D 20:01:02.347181] tiling 13 copies [D 20:01:02.347187] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.347195] lvl 0 el 0 blk 0 by 398458880 [D 20:01:02.347201] to 398458880 ta 369098752 fi 402653184 [D 20:01:02.347208] basic type or contiguous data [D 20:01:02.347214] Request disp [D 20:01:02.347221] contig_offset = 398458880 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.347228] calling distribute [D 20:01:02.347234] PINT_distribute [D 20:01:02.347239] of 398458880 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 398458880 exfl 1 [D 20:01:02.347247] begin iteration loff: 398458880 [D 20:01:02.347254] segment exceeds byte limit [D 20:01:02.347259] file being extended [D 20:01:02.347265] process a segment [D 20:01:02.347271] of 398458880 sz 2097152 [D 20:01:02.347278] add a segment 0 [D 20:01:02.347284] end iteration [D 20:01:02.347290] done with segments or bytes [D 20:01:02.347296] of 400556032 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.347303] next loff: 400556032 next poff: 400556032 [D 20:01:02.347309] return value 2097152 [D 20:01:02.347315] finished [D 20:01:02.347321] retval = 2097152 [D 20:01:02.347327] exiting distribute returned less than expected [D 20:01:02.347333] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 400556032 fo 402653184 eof 0 [D 20:01:02.347341] ========================================================= [D 20:01:02.347347] BMI_post_recv: addr: 6843072, offset: 0x2aaab8a3d010, size: 2097152, tag: 19 [D 20:01:02.347356] openib_check_cq: completed send wr_id 683830. [D 20:01:02.347368] post_sr: 10.1.4.56:39990 bh 84 len 48 wr_id 683860. [D 20:01:02.347378] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:02.347386] get_data_sync_mode returning 0 [D 20:01:02.347395] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.347402] dbpf_bstream_rw_list: mem_offset: 0x2aaab5c25010, mem_size: 2097152 [D 20:01:02.347409] dbpf_bstream_rw_list: stream_offset: 488636416, stream_size: 2097152 [D 20:01:02.347416] dbpf_open_cache_get: called [D 20:01:02.347422] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.347428] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.347457] op_queue add: 0x2aaabd321080 [D 20:01:02.347466] delayed I/O operation 0x2aaabd321080 (16 already in progress) [D 20:01:02.347473] DBPF I/O ops in progress: 16 [D 20:01:02.347480] ========================================================= [D 20:01:02.347486] PINT_process_request [D 20:01:02.347493] tiling 16 copies [D 20:01:02.347499] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.347506] lvl 0 el 0 blk 0 by 490733568 [D 20:01:02.347513] to 490733568 ta 469762048 fi 503316480 [D 20:01:02.347520] basic type or contiguous data [D 20:01:02.347526] Request disp [D 20:01:02.347532] contig_offset = 490733568 contig_size = 46137344 lvl_flag = 1 [D 20:01:02.347539] calling distribute [D 20:01:02.347545] PINT_distribute [D 20:01:02.347550] of 490733568 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 490733568 exfl 1 [D 20:01:02.347559] begin iteration loff: 490733568 [D 20:01:02.347565] segment exceeds byte limit [D 20:01:02.347571] file being extended [D 20:01:02.347577] process a segment [D 20:01:02.347583] of 490733568 sz 2097152 [D 20:01:02.347589] add a segment 0 [D 20:01:02.347595] end iteration [D 20:01:02.347601] done with segments or bytes [D 20:01:02.347607] of 492830720 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.347614] next loff: 492830720 next poff: 492830720 [D 20:01:02.347621] return value 2097152 [D 20:01:02.347627] finished [D 20:01:02.347633] retval = 2097152 [D 20:01:02.347639] exiting distribute returned less than expected [D 20:01:02.347645] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 492830720 fo 503316480 eof 0 [D 20:01:02.347652] ========================================================= [D 20:01:02.347658] BMI_post_recv: addr: 6843072, offset: 0x2aaab8e3f010, size: 2097152, tag: 22 [D 20:01:02.347667] openib_check_cq: completed send wr_id 683860. [D 20:01:02.347678] post_sr: 10.1.4.56:39990 bh 85 len 48 wr_id 683890. [D 20:01:02.347687] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:02.347695] get_data_sync_mode returning 0 [D 20:01:02.347707] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.347714] dbpf_bstream_rw_list: mem_offset: 0x2aaab7030010, mem_size: 2097152 [D 20:01:02.347721] dbpf_bstream_rw_list: stream_offset: 429916160, stream_size: 2097152 [D 20:01:02.347728] dbpf_open_cache_get: called [D 20:01:02.347734] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.347740] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.347764] op_queue add: 0x2aaabd3211d0 [D 20:01:02.347773] delayed I/O operation 0x2aaabd3211d0 (16 already in progress) [D 20:01:02.347779] DBPF I/O ops in progress: 16 [D 20:01:02.347787] ========================================================= [D 20:01:02.347793] PINT_process_request [D 20:01:02.347800] tiling 14 copies [D 20:01:02.347806] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.347814] lvl 0 el 0 blk 0 by 432013312 [D 20:01:02.347821] to 432013312 ta 402653184 fi 436207616 [D 20:01:02.347828] basic type or contiguous data [D 20:01:02.347834] Request disp [D 20:01:02.347840] contig_offset = 432013312 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.347846] calling distribute [D 20:01:02.347852] PINT_distribute [D 20:01:02.347858] of 432013312 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 432013312 exfl 1 [D 20:01:02.347866] begin iteration loff: 432013312 [D 20:01:02.347873] segment exceeds byte limit [D 20:01:02.347879] file being extended [D 20:01:02.347886] process a segment [D 20:01:02.347891] of 432013312 sz 2097152 [D 20:01:02.347898] add a segment 0 [D 20:01:02.347904] end iteration [D 20:01:02.347909] done with segments or bytes [D 20:01:02.347915] of 434110464 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.347922] next loff: 434110464 next poff: 434110464 [D 20:01:02.347929] return value 2097152 [D 20:01:02.347935] finished [D 20:01:02.347941] retval = 2097152 [D 20:01:02.347947] exiting distribute returned less than expected [D 20:01:02.347953] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 434110464 fo 436207616 eof 0 [D 20:01:02.347961] ========================================================= [D 20:01:02.347967] BMI_post_recv: addr: 6843072, offset: 0x2aaab8c3e010, size: 2097152, tag: 20 [D 20:01:02.347975] openib_check_cq: completed send wr_id 683890. [D 20:01:02.347988] post_sr: 10.1.4.56:39990 bh 86 len 48 wr_id 6838c0. [D 20:01:02.348015] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.348028] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:02.348036] ========================================================= [D 20:01:02.348042] PINT_process_request [D 20:01:02.348050] tiling 15 copies [D 20:01:02.348057] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.348064] lvl 0 el 0 blk 0 by 459276288 [D 20:01:02.348070] to 459276288 ta 436207616 fi 469762048 [D 20:01:02.348077] basic type or contiguous data [D 20:01:02.348083] Request disp [D 20:01:02.348089] contig_offset = 459276288 contig_size = 44040192 lvl_flag = 1 [D 20:01:02.348096] calling distribute [D 20:01:02.348103] PINT_distribute [D 20:01:02.348108] of 459276288 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:02.348116] begin iteration loff: 459276288 [D 20:01:02.348123] segment exceeds byte limit [D 20:01:02.348129] process a segment [D 20:01:02.348134] of 459276288 sz 2097152 [D 20:01:02.348141] add a segment 0 [D 20:01:02.348147] end iteration [D 20:01:02.348153] done with segments or bytes [D 20:01:02.348159] of 461373440 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.348166] next loff: 461373440 next poff: 461373440 [D 20:01:02.348172] return value 2097152 [D 20:01:02.348178] finished [D 20:01:02.348184] retval = 2097152 [D 20:01:02.348190] exiting distribute returned less than expected [D 20:01:02.348197] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 461373440 fo 469762048 eof 0 [D 20:01:02.348204] ========================================================= [D 20:01:02.348213] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.348220] dbpf_bstream_rw_list: mem_offset: 0x2aaabbc57010, mem_size: 2097152 [D 20:01:02.348227] dbpf_bstream_rw_list: stream_offset: 459276288, stream_size: 2097152 [D 20:01:02.348234] dbpf_open_cache_get: called [D 20:01:02.348240] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.348246] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.348273] op_queue add: 0x2aaabd321320 [D 20:01:02.348282] delayed I/O operation 0x2aaabd321320 (16 already in progress) [D 20:01:02.348289] DBPF I/O ops in progress: 16 [D 20:01:02.348297] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.348306] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:02.348314] ========================================================= [D 20:01:02.348320] PINT_process_request [D 20:01:02.348326] tiling 2 copies [D 20:01:02.348333] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.348340] lvl 0 el 0 blk 0 by 20971520 [D 20:01:02.348346] to 20971520 ta 0 fi 33554432 [D 20:01:02.348353] basic type or contiguous data [D 20:01:02.348359] Request disp [D 20:01:02.348366] contig_offset = 20971520 contig_size = 46137344 lvl_flag = 1 [D 20:01:02.348373] calling distribute [D 20:01:02.348379] PINT_distribute [D 20:01:02.348385] of 20971520 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.348393] begin iteration loff: 20971520 [D 20:01:02.348400] segment exceeds byte limit [D 20:01:02.348406] process a segment [D 20:01:02.348411] of 20971520 sz 2097152 [D 20:01:02.348418] add a segment 0 [D 20:01:02.348423] end iteration [D 20:01:02.348429] done with segments or bytes [D 20:01:02.348436] of 23068672 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.348443] next loff: 23068672 next poff: 23068672 [D 20:01:02.348450] return value 2097152 [D 20:01:02.348456] finished [D 20:01:02.348462] retval = 2097152 [D 20:01:02.348468] exiting distribute returned less than expected [D 20:01:02.348474] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 23068672 fo 33554432 eof 0 [D 20:01:02.348481] ========================================================= [D 20:01:02.348489] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.348495] dbpf_bstream_rw_list: mem_offset: 0x2aaabcc5f010, mem_size: 2097152 [D 20:01:02.348502] dbpf_bstream_rw_list: stream_offset: 20971520, stream_size: 2097152 [D 20:01:02.348509] dbpf_open_cache_get: called [D 20:01:02.348515] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.348522] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.348566] op_queue add: 0x2aaabd321470 [D 20:01:02.348576] delayed I/O operation 0x2aaabd321470 (16 already in progress) [D 20:01:02.348583] DBPF I/O ops in progress: 16 [D 20:01:02.348590] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.348598] (send callback time) ini posts: 0, pending: 1, last: 0, src_list emtpy: no [D 20:01:02.348606] ========================================================= [D 20:01:02.348612] PINT_process_request [D 20:01:02.348619] tiling 3 copies [D 20:01:02.348625] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.348633] lvl 0 el 0 blk 0 by 52428800 [D 20:01:02.348639] to 52428800 ta 33554432 fi 67108864 [D 20:01:02.348646] basic type or contiguous data [D 20:01:02.348652] Request disp [D 20:01:02.348658] contig_offset = 52428800 contig_size = 48234496 lvl_flag = 1 [D 20:01:02.348665] calling distribute [D 20:01:02.348670] PINT_distribute [D 20:01:02.348676] of 52428800 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.348684] begin iteration loff: 52428800 [D 20:01:02.348690] segment exceeds byte limit [D 20:01:02.348696] process a segment [D 20:01:02.348702] of 52428800 sz 2097152 [D 20:01:02.348708] add a segment 0 [D 20:01:02.348714] end iteration [D 20:01:02.348720] done with segments or bytes [D 20:01:02.348726] of 54525952 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.348733] next loff: 54525952 next poff: 54525952 [D 20:01:02.348740] return value 2097152 [D 20:01:02.348746] finished [D 20:01:02.348752] retval = 2097152 [D 20:01:02.348757] exiting distribute returned less than expected [D 20:01:02.348764] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 54525952 fo 67108864 eof 0 [D 20:01:02.348771] ========================================================= [D 20:01:02.348778] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.348785] dbpf_bstream_rw_list: mem_offset: 0x2aaabdc04010, mem_size: 2097152 [D 20:01:02.348791] dbpf_bstream_rw_list: stream_offset: 52428800, stream_size: 2097152 [D 20:01:02.348798] dbpf_open_cache_get: called [D 20:01:02.348804] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.348811] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.348842] op_queue add: 0x2aaabd3215c0 [D 20:01:02.348851] delayed I/O operation 0x2aaabd3215c0 (16 already in progress) [D 20:01:02.348858] DBPF I/O ops in progress: 16 [D 20:01:02.348865] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:02.348873] get_data_sync_mode returning 0 [D 20:01:02.348881] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.348888] dbpf_bstream_rw_list: mem_offset: 0x2aaab7231010, mem_size: 2097152 [D 20:01:02.348895] dbpf_bstream_rw_list: stream_offset: 463470592, stream_size: 2097152 [D 20:01:02.348901] dbpf_open_cache_get: called [D 20:01:02.348907] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.348914] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.348942] op_queue add: 0x2aaabd321710 [D 20:01:02.348951] delayed I/O operation 0x2aaabd321710 (16 already in progress) [D 20:01:02.348957] DBPF I/O ops in progress: 16 [D 20:01:02.348965] ========================================================= [D 20:01:02.348972] PINT_process_request [D 20:01:02.348978] tiling 15 copies [D 20:01:02.348985] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.348992] lvl 0 el 0 blk 0 by 465567744 [D 20:01:02.348999] to 465567744 ta 436207616 fi 469762048 [D 20:01:02.349007] basic type or contiguous data [D 20:01:02.349014] Request disp [D 20:01:02.349020] contig_offset = 465567744 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.349026] calling distribute [D 20:01:02.349033] PINT_distribute [D 20:01:02.349038] of 465567744 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 465567744 exfl 1 [D 20:01:02.349046] begin iteration loff: 465567744 [D 20:01:02.349053] segment exceeds byte limit [D 20:01:02.349059] file being extended [D 20:01:02.349065] process a segment [D 20:01:02.349070] of 465567744 sz 2097152 [D 20:01:02.349076] add a segment 0 [D 20:01:02.349083] end iteration [D 20:01:02.349089] done with segments or bytes [D 20:01:02.349095] of 467664896 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.349102] next loff: 467664896 next poff: 467664896 [D 20:01:02.349108] return value 2097152 [D 20:01:02.349114] finished [D 20:01:02.349120] retval = 2097152 [D 20:01:02.349126] exiting distribute returned less than expected [D 20:01:02.349132] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 467664896 fo 469762048 eof 0 [D 20:01:02.349140] ========================================================= [D 20:01:02.349146] BMI_post_recv: addr: 6843072, offset: 0x2aaab9040010, size: 2097152, tag: 21 [D 20:01:02.349157] openib_check_cq: completed send wr_id 6838c0. [D 20:01:02.349180] post_sr: 10.1.4.56:39990 bh 87 len 48 wr_id 6838f0. [D 20:01:02.349191] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:01:02.349199] get_data_sync_mode returning 0 [D 20:01:02.349207] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.349214] dbpf_bstream_rw_list: mem_offset: 0x2aaab7432010, mem_size: 2097152 [D 20:01:02.349221] dbpf_bstream_rw_list: stream_offset: 29360128, stream_size: 2097152 [D 20:01:02.349228] dbpf_open_cache_get: called [D 20:01:02.349234] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.349240] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.349266] op_queue add: 0x2aaabd321860 [D 20:01:02.349275] delayed I/O operation 0x2aaabd321860 (16 already in progress) [D 20:01:02.349281] DBPF I/O ops in progress: 16 [D 20:01:02.349288] ========================================================= [D 20:01:02.349295] PINT_process_request [D 20:01:02.349301] tiling 2 copies [D 20:01:02.349307] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.349315] lvl 0 el 0 blk 0 by 31457280 [D 20:01:02.349321] to 31457280 ta 0 fi 33554432 [D 20:01:02.349328] basic type or contiguous data [D 20:01:02.349334] Request disp [D 20:01:02.349340] contig_offset = 31457280 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.349347] calling distribute [D 20:01:02.349353] PINT_distribute [D 20:01:02.349359] of 31457280 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 31457280 exfl 1 [D 20:01:02.349367] begin iteration loff: 31457280 [D 20:01:02.349373] file being extended [D 20:01:02.349381] process a segment [D 20:01:02.349387] of 31457280 sz 2097152 [D 20:01:02.349393] add a segment 0 [D 20:01:02.349400] end iteration [D 20:01:02.349406] done with segments or bytes [D 20:01:02.349411] of 33554432 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.349418] next loff: 33554432 next poff: 33554432 [D 20:01:02.349425] (return value) 2097152 [D 20:01:02.349431] finished [D 20:01:02.349437] retval = 2097152 [D 20:01:02.349443] exiting distribute returned less than expected [D 20:01:02.349449] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 33554432 fo 33554432 eof 0 [D 20:01:02.349456] ========================================================= [D 20:01:02.349462] BMI_post_recv: addr: 6843072, offset: 0x2aaab9241010, size: 2097152, tag: 8 [D 20:01:02.349472] openib_check_cq: completed send wr_id 6838f0. [D 20:01:02.349484] post_sr: 10.1.4.56:39990 bh 88 len 48 wr_id 683920. [D 20:01:02.349507] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:01:02.349516] get_data_sync_mode returning 0 [D 20:01:02.349523] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.349530] dbpf_bstream_rw_list: mem_offset: 0x2aaab7633010, mem_size: 2097152 [D 20:01:02.349538] dbpf_bstream_rw_list: stream_offset: 62914560, stream_size: 2097152 [D 20:01:02.349544] dbpf_open_cache_get: called [D 20:01:02.349550] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.349557] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.349585] op_queue add: 0x2aaabd301990 [D 20:01:02.349594] delayed I/O operation 0x2aaabd301990 (16 already in progress) [D 20:01:02.349601] DBPF I/O ops in progress: 16 [D 20:01:02.349608] ========================================================= [D 20:01:02.349615] PINT_process_request [D 20:01:02.349621] tiling 3 copies [D 20:01:02.349627] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.349635] lvl 0 el 0 blk 0 by 65011712 [D 20:01:02.349641] to 65011712 ta 33554432 fi 67108864 [D 20:01:02.349648] basic type or contiguous data [D 20:01:02.349654] Request disp [D 20:01:02.349660] contig_offset = 65011712 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.349667] calling distribute [D 20:01:02.349673] PINT_distribute [D 20:01:02.349679] of 65011712 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 65011712 exfl 1 [D 20:01:02.349686] begin iteration loff: 65011712 [D 20:01:02.349693] file being extended [D 20:01:02.349699] process a segment [D 20:01:02.349705] of 65011712 sz 2097152 [D 20:01:02.349711] add a segment 0 [D 20:01:02.349717] end iteration [D 20:01:02.349723] done with segments or bytes [D 20:01:02.349729] of 67108864 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.349736] next loff: 67108864 next poff: 67108864 [D 20:01:02.349742] (return value) 2097152 [D 20:01:02.349748] finished [D 20:01:02.349754] retval = 2097152 [D 20:01:02.349760] exiting distribute returned less than expected [D 20:01:02.349766] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 67108864 fo 67108864 eof 0 [D 20:01:02.349774] ========================================================= [D 20:01:02.349780] BMI_post_recv: addr: 6843072, offset: 0x2aaab9442010, size: 2097152, tag: 9 [D 20:01:02.349789] openib_check_cq: completed send wr_id 683920. [D 20:01:02.349801] post_sr: 10.1.4.56:39990 bh 89 len 48 wr_id 683950. [D 20:01:02.349811] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:01:02.349819] get_data_sync_mode returning 0 [D 20:01:02.349827] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.349833] dbpf_bstream_rw_list: mem_offset: 0x2aaab7834010, mem_size: 2097152 [D 20:01:02.349840] dbpf_bstream_rw_list: stream_offset: 96468992, stream_size: 2097152 [D 20:01:02.349847] dbpf_open_cache_get: called [D 20:01:02.349853] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.349859] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.349886] op_queue add: 0x2aaabd3219b0 [D 20:01:02.349895] delayed I/O operation 0x2aaabd3219b0 (16 already in progress) [D 20:01:02.349902] DBPF I/O ops in progress: 16 [D 20:01:02.349909] ========================================================= [D 20:01:02.349915] PINT_process_request [D 20:01:02.349922] tiling 4 copies [D 20:01:02.349928] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.349935] lvl 0 el 0 blk 0 by 98566144 [D 20:01:02.349942] to 98566144 ta 67108864 fi 100663296 [D 20:01:02.349949] basic type or contiguous data [D 20:01:02.349955] Request disp [D 20:01:02.349961] contig_offset = 98566144 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.349968] calling distribute [D 20:01:02.349974] PINT_distribute [D 20:01:02.349980] of 98566144 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 98566144 exfl 1 [D 20:01:02.349988] begin iteration loff: 98566144 [D 20:01:02.349994] file being extended [D 20:01:02.350000] process a segment [D 20:01:02.350006] of 98566144 sz 2097152 [D 20:01:02.350013] add a segment 0 [D 20:01:02.350019] end iteration [D 20:01:02.350025] done with segments or bytes [D 20:01:02.350031] of 100663296 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.350038] next loff: 100663296 next poff: 100663296 [D 20:01:02.350045] (return value) 2097152 [D 20:01:02.350051] finished [D 20:01:02.350057] retval = 2097152 [D 20:01:02.350063] exiting distribute returned less than expected [D 20:01:02.350069] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 100663296 fo 100663296 eof 0 [D 20:01:02.350076] ========================================================= [D 20:01:02.350083] BMI_post_recv: addr: 6843072, offset: 0x2aaab9643010, size: 2097152, tag: 10 [D 20:01:02.350091] openib_check_cq: completed send wr_id 683950. [D 20:01:02.350102] post_sr: 10.1.4.56:39990 bh 90 len 48 wr_id 683980. [D 20:01:02.350112] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:01:02.350120] get_data_sync_mode returning 0 [D 20:01:02.350128] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.350135] dbpf_bstream_rw_list: mem_offset: 0x2aaab7a35010, mem_size: 2097152 [D 20:01:02.350142] dbpf_bstream_rw_list: stream_offset: 130023424, stream_size: 2097152 [D 20:01:02.350149] dbpf_open_cache_get: called [D 20:01:02.350155] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.350161] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.350186] op_queue add: 0x2aaabd321b00 [D 20:01:02.350195] delayed I/O operation 0x2aaabd321b00 (16 already in progress) [D 20:01:02.350202] DBPF I/O ops in progress: 16 [D 20:01:02.350209] ========================================================= [D 20:01:02.350216] PINT_process_request [D 20:01:02.350222] tiling 5 copies [D 20:01:02.350228] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.350235] lvl 0 el 0 blk 0 by 132120576 [D 20:01:02.350242] to 132120576 ta 100663296 fi 134217728 [D 20:01:02.350249] basic type or contiguous data [D 20:01:02.350255] Request disp [D 20:01:02.350261] contig_offset = 132120576 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.350268] calling distribute [D 20:01:02.350274] PINT_distribute [D 20:01:02.350279] of 132120576 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 132120576 exfl 1 [D 20:01:02.350287] begin iteration loff: 132120576 [D 20:01:02.350293] file being extended [D 20:01:02.350299] process a segment [D 20:01:02.350305] of 132120576 sz 2097152 [D 20:01:02.350311] add a segment 0 [D 20:01:02.350318] end iteration [D 20:01:02.350324] done with segments or bytes [D 20:01:02.350330] of 134217728 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.350337] next loff: 134217728 next poff: 134217728 [D 20:01:02.350343] (return value) 2097152 [D 20:01:02.350349] finished [D 20:01:02.350355] retval = 2097152 [D 20:01:02.350361] exiting distribute returned less than expected [D 20:01:02.350367] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 134217728 fo 134217728 eof 0 [D 20:01:02.350376] ========================================================= [D 20:01:02.350382] BMI_post_recv: addr: 6843072, offset: 0x2aaab9a45010, size: 2097152, tag: 11 [D 20:01:02.350391] openib_check_cq: completed send wr_id 683980. [D 20:01:02.350402] post_sr: 10.1.4.56:39990 bh 91 len 48 wr_id 6839b0. [D 20:01:02.350420] openib_check_cq: completed send wr_id 6839b0. [D 20:01:02.280826] --- aio_progress_notification called with handle 2863311531 (0x2aaabd310ef0) [D 20:01:02.354572] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.354624] dbpf_open_cache_put: cached entry. [D 20:01:02.354632] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.354639] op_queue add: 0x2aaabd310ef0 [D 20:01:02.354647] DBPF I/O ops in progress: 15 [D 20:01:02.354654] starting delayed I/O operation 0x2aaabd300b80 (15 in progress) [D 20:01:02.354663] lio_listio called with 1 following aiocbs: [D 20:01:02.354670] aiocb_ptr_array[0]: fd: 13, off: 457179136, bytes: 2097152, buf: 0x2aaabba56010, type: 0 [D 20:01:02.283783] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300a30) [D 20:01:02.354686] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.354703] dbpf_open_cache_put: cached entry. [D 20:01:02.354710] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.354716] op_queue add: 0x2aaabd300a30 [D 20:01:02.354731] --- aio_progress_notification called with handle 2863311531 (0x2aaabd311110) [D 20:01:02.354741] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.354750] dbpf_open_cache_put: cached entry. [D 20:01:02.354757] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.354763] op_queue add: 0x2aaabd311110 [D 20:01:02.354774] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302230) [D 20:01:02.354782] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.354792] dbpf_open_cache_put: cached entry. [D 20:01:02.354798] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.354804] op_queue add: 0x2aaabd302230 [D 20:01:02.354814] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302380) [D 20:01:02.354823] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.354832] dbpf_open_cache_put: cached entry. [D 20:01:02.354838] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.354844] op_queue add: 0x2aaabd302380 [D 20:01:02.360579] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300b80 (handle 2863311531, ret 0)) [D 20:01:02.360673] DBPF I/O ops in progress: 15 [D 20:01:02.360686] starting delayed I/O operation 0x2aaabd300f80 (15 in progress) [D 20:01:02.360699] lio_listio called with 1 following aiocbs: [D 20:01:02.360706] aiocb_ptr_array[0]: fd: 13, off: 18874368, bytes: 2097152, buf: 0x2aaabca5e010, type: 0 [D 20:01:02.360720] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300f80 (handle 2863311531, ret 0)) [D 20:01:02.360782] DBPF I/O ops in progress: 15 [D 20:01:02.360793] starting delayed I/O operation 0x2aaabd300d80 (15 in progress) [D 20:01:02.360807] lio_listio called with 1 following aiocbs: [D 20:01:02.360813] aiocb_ptr_array[0]: fd: 13, off: 50331648, bytes: 2097152, buf: 0x2aaabda03010, type: 0 [D 20:01:02.360822] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300d80 (handle 2863311531, ret 0)) [D 20:01:02.360846] DBPF I/O ops in progress: 15 [D 20:01:02.360857] starting delayed I/O operation 0x2aaab152be30 (15 in progress) [D 20:01:02.360868] lio_listio called with 1 following aiocbs: [D 20:01:02.360874] aiocb_ptr_array[0]: fd: 13, off: 77594624, bytes: 2097152, buf: 0x2aaabf410010, type: 0 [D 20:01:02.360885] start_delayed_ops_if_any: lio_listio posted 0x2aaab152be30 (handle 2863311531, ret 0)) [D 20:01:02.431071] DBPF I/O ops in progress: 15 [D 20:01:02.431083] starting delayed I/O operation 0x2aaab152b470 (15 in progress) [D 20:01:02.431092] lio_listio called with 1 following aiocbs: [D 20:01:02.431232] aiocb_ptr_array[0]: fd: 13, off: 79691776, bytes: 2097152, buf: 0x2aaabf611010, type: 0 [D 20:01:02.431247] start_delayed_ops_if_any: lio_listio posted 0x2aaab152b470 (handle 2863311531, ret 0)) [D 20:01:02.364557] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3026f0) [D 20:01:02.431352] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.431420] dbpf_open_cache_put: cached entry. [D 20:01:02.431428] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.364623] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3024d0) [D 20:01:02.431543] op_queue add: 0x2aaab152bf80 [D 20:01:02.431579] op_queue add: 0x2aaabd3026f0 [D 20:01:02.431597] delayed I/O operation 0x2aaab152bf80 (16 already in progress) [D 20:01:02.431588] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.431607] DBPF I/O ops in progress: 16 [D 20:01:02.431621] dbpf_open_cache_put: cached entry. [D 20:01:02.431636] flowproto posted 0x2aaab152b630 [D 20:01:02.431637] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.431653] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.431657] op_queue add: 0x2aaabd3024d0 [D 20:01:02.431648] Job flows in progress (post time): 19 [D 20:01:02.431674] DBPF I/O ops in progress: 15 [D 20:01:02.431687] [SM Exiting]: (0x6619d0) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:02.431690] starting delayed I/O operation 0x2aaabd30b770 (15 in progress) [D 20:01:02.431708] lio_listio called with 1 following aiocbs: [D 20:01:02.431715] aiocb_ptr_array[0]: fd: 13, off: 161480704, bytes: 2097152, buf: 0x2aaab6027010, type: 1 [D 20:01:02.431723] start_delayed_ops_if_any: lio_listio posted 0x2aaabd30b770 (handle 2863311531, ret 0)) [D 20:01:02.431735] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.431746] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.431754] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.431761] BMI_post_send: addr: 6843072, offset: 0x2aaabe006010, size: 2097152, tag: 24 [D 20:01:02.431774] post_sr: 10.1.4.56:39990 bh 92 len 32 wr_id 6839e0. [D 20:01:02.431802] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.431809] BMI_post_send: addr: 6843072, offset: 0x2aaabe207010, size: 2097152, tag: 24 [D 20:01:02.431818] post_sr: 10.1.4.56:39990 bh 93 len 32 wr_id 683a10. [D 20:01:02.431828] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:02.431836] BMI_post_send: addr: 6843072, offset: 0x2aaabe408010, size: 2097152, tag: 24 [D 20:01:02.431844] post_sr: 10.1.4.56:39990 bh 94 len 32 wr_id 683a40. [D 20:01:02.431853] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.431861] BMI_post_send: addr: 6843072, offset: 0x2aaabe609010, size: 2097152, tag: 24 [D 20:01:02.431869] post_sr: 10.1.4.56:39990 bh 95 len 32 wr_id 683a70. [D 20:01:02.431878] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:02.431885] BMI_post_send: addr: 6843072, offset: 0x2aaabe80a010, size: 2097152, tag: 24 [D 20:01:02.431894] post_sr: 10.1.4.56:39990 bh 96 len 32 wr_id 683aa0. [D 20:01:02.431903] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:02.431911] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.431917] BMI_post_send: addr: 6843072, offset: 0x2aaabea0b010, size: 2097152, tag: 25 [D 20:01:02.431925] post_sr: 10.1.4.56:39990 bh 97 len 32 wr_id 683ad0. [D 20:01:02.431935] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.431942] BMI_post_send: addr: 6843072, offset: 0x2aaabec0c010, size: 2097152, tag: 25 [D 20:01:02.431950] post_sr: 10.1.4.56:39990 bh 98 len 32 wr_id 683b00. [D 20:01:02.431960] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.431967] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.431974] BMI_post_send: addr: 6843072, offset: 0x2aaabee0d010, size: 2097152, tag: 25 [D 20:01:02.431982] post_sr: 10.1.4.56:39990 bh 99 len 32 wr_id 683b30. [D 20:01:02.431992] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:02.432004] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.432012] BMI_post_send: addr: 6843072, offset: 0x2aaabf00e010, size: 2097152, tag: 25 [D 20:01:02.432020] post_sr: 10.1.4.56:39990 bh 0 len 32 wr_id 6828a0. [D 20:01:02.432029] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.432038] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:02.432045] BMI_post_send: addr: 6843072, offset: 0x2aaabb855010, size: 2097152, tag: 36 [D 20:01:02.432053] post_sr: 10.1.4.56:39990 bh 1 len 32 wr_id 6828d0. [D 20:01:02.432063] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:02.432071] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:02.432081] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:01:02.432088] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:02.432100] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:02.432109] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:02.432116] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:02.432124] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:02.432131] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:02.432142] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:02.432150] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:02.432158] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:02.432165] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:02.432172] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:01:02.431726] [SM Entering]: (0x662c90) io_sm:start_flow (status: 0) [D 20:01:02.432191] [SM frame get]: (0x662c90) op-id: 3 index: 0 base-frm: 0 [D 20:01:02.432201] io_start_flow() issuing flow to read data. [D 20:01:02.432208] flow: fsize: 442499072, server_nr: 0, server_ct: 1 [D 20:01:02.432215] file_req_offset: 100663296, aggregate_size: 33554432, handle: 2863311531 [D 20:01:02.432224] PINT_new_request_states n=1 [D 20:01:02.432232] flowproto posting 0x2aaab1522750 [D 20:01:02.432244] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.432251] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1522750. [D 20:01:02.432259] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:02.432536] DBPF I/O ops in progress: 15 [D 20:01:02.432547] starting delayed I/O operation 0x2aaabd305f70 (15 in progress) [D 20:01:02.432556] lio_listio called with 1 following aiocbs: [D 20:01:02.432563] aiocb_ptr_array[0]: fd: 13, off: 195035136, bytes: 2097152, buf: 0x2aaab6228010, type: 1 [D 20:01:02.432572] start_delayed_ops_if_any: lio_listio posted 0x2aaabd305f70 (handle 2863311531, ret 0)) [D 20:01:02.368554] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302840) [D 20:01:02.434540] DBPF I/O ops in progress: 15 [D 20:01:02.434703] starting delayed I/O operation 0x2aaabd3013e0 (15 in progress) [D 20:01:02.434718] lio_listio called with 1 following aiocbs: [D 20:01:02.434726] aiocb_ptr_array[0]: fd: 13, off: 228589568, bytes: 2097152, buf: 0x2aaab642a010, type: 1 [D 20:01:02.434743] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3013e0 (handle 2863311531, ret 0)) [D 20:01:02.434770] DBPF I/O ops in progress: 15 [D 20:01:02.434782] starting delayed I/O operation 0x2aaabd302ff0 (15 in progress) [D 20:01:02.434793] lio_listio called with 1 following aiocbs: [D 20:01:02.434799] aiocb_ptr_array[0]: fd: 13, off: 262144000, bytes: 2097152, buf: 0x2aaab662b010, type: 1 [D 20:01:02.434809] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302ff0 (handle 2863311531, ret 0)) -[D 20:01:02.435340] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.435385] dbpf_open_cache_put: cached entry. [D 20:01:02.435392] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.435399] op_queue add: 0x2aaabd302840 [D 20:01:02.368585] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302990) [D 20:01:02.372602] --- aio_progress_notification called with handle 2863311531 (0x2aaab152bce0) [D 20:01:02.435369] openib_check_cq: completed recv wr_id 684aa0. [D 20:01:02.435373] DBPF I/O ops in progress: 15 [D 20:01:02.435461] starting delayed I/O operation 0x2aaabd303140 (15 in progress) [D 20:01:02.435434] openib_post_sr_rdmaw: ibv_post_send wr_id 6fed90 to 10.1.4.56:39990 remote addr 88c37000 rkey 108a27 nic_wr_credit 1. [D 20:01:02.435451] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:02.435485] openib_post_rr: 10.1.4.56:39990 bh 81 wr_id 684aa0. [D 20:01:02.435472] lio_listio called with 1 following aiocbs: [D 20:01:02.435510] aiocb_ptr_array[0]: fd: 13, off: 295698432, bytes: 2097152, buf: 0x2aaab682c010, type: 1 [D 20:01:02.435520] start_delayed_ops_if_any: lio_listio posted 0x2aaabd303140 (handle 2863311531, ret 0)) [D 20:01:02.435542] DBPF I/O ops in progress: 15 [D 20:01:02.435497] openib_check_cq: completed recv wr_id 684ad0. [D 20:01:02.435556] starting delayed I/O operation 0x2aaabd303340 (15 in progress) [D 20:01:02.435573] openib_post_rr: 10.1.4.56:39990 bh 82 wr_id 684ad0. [D 20:01:02.435581] lio_listio called with 1 following aiocbs: [D 20:01:02.435591] aiocb_ptr_array[0]: fd: 13, off: 329252864, bytes: 2097152, buf: 0x2aaab6a2d010, type: 1 [D 20:01:02.435585] openib_check_cq: completed recv wr_id 684b00. [D 20:01:02.435601] start_delayed_ops_if_any: lio_listio posted 0x2aaabd303340 (handle 2863311531, ret 0)) [D 20:01:02.435628] DBPF I/O ops in progress: 15 [D 20:01:02.435638] starting delayed I/O operation 0x2aaabd3015f0 (15 in progress) [D 20:01:02.435648] lio_listio called with 1 following aiocbs: [D 20:01:02.435654] aiocb_ptr_array[0]: fd: 13, off: 362807296, bytes: 2097152, buf: 0x2aaab6c2e010, type: 1 [D 20:01:02.435664] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3015f0 (handle 2863311531, ret 0)) [D 20:01:02.438905] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.435611] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec3a0 to 10.1.4.56:39990 remote addr a2a51000 rkey 108c27 nic_wr_credit 2. [D 20:01:02.438962] dbpf_open_cache_put: cached entry. [D 20:01:02.376574] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300f80) [D 20:01:02.438965] openib_post_rr: 10.1.4.56:39990 bh 83 wr_id 684b00. [D 20:01:02.438985] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.438993] openib_check_cq: completed recv wr_id 684b30. [D 20:01:02.439004] dbpf_open_cache_put: cached entry. [D 20:01:02.439017] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.439025] op_queue add: 0x2aaabd300f80 [D 20:01:02.376691] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300b80) [D 20:01:02.439009] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec210 to 10.1.4.56:39990 remote addr a084f000 rkey 108d27 nic_wr_credit 3. [D 20:01:02.439057] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.378571] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300d80) [D 20:01:02.434678] ========================================================= [D 20:01:02.438976] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.439094] PINT_process_request [D 20:01:02.435444] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.439061] openib_post_rr: 10.1.4.56:39990 bh 84 wr_id 684b30. [D 20:01:02.439097] op_queue add: 0x2aaab152bce0 [D 20:01:02.439080] dbpf_open_cache_put: cached entry. [D 20:01:02.439050] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:02.439146] dbpf_open_cache_put: cached entry. [D 20:01:02.439153] BMI_post_send: addr: 6843072, offset: 0x2aaabca5e010, size: 2097152, tag: 23 [D 20:01:02.439140] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.439125] openib_check_cq: completed recv wr_id 684b60. [D 20:01:02.439157] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.439184] openib_post_rr: 10.1.4.56:39990 bh 85 wr_id 684b60. [D 20:01:02.439129] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.439046] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b470) [D 20:01:02.439207] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.439196] openib_check_cq: completed recv wr_id 684b90. [D 20:01:02.439230] openib_post_rr: 10.1.4.56:39990 bh 86 wr_id 684b90. [D 20:01:02.439238] openib_check_cq: completed recv wr_id 684bc0. [D 20:01:02.439246] openib_post_rr: 10.1.4.56:39990 bh 87 wr_id 684bc0. [D 20:01:02.439253] openib_check_cq: completed recv wr_id 684bf0. [D 20:01:02.439260] openib_post_rr: 10.1.4.56:39990 bh 88 wr_id 684bf0. [D 20:01:02.439267] openib_check_cq: completed recv wr_id 684c20. [D 20:01:02.439275] openib_post_rr: 10.1.4.56:39990 bh 89 wr_id 684c20. [D 20:01:02.439282] openib_check_cq: completed recv wr_id 684c50. [D 20:01:02.439289] openib_post_rr: 10.1.4.56:39990 bh 90 wr_id 684c50. [D 20:01:02.439297] openib_check_cq: completed recv wr_id 684c80. [D 20:01:02.439304] openib_post_rr: 10.1.4.56:39990 bh 91 wr_id 684c80. [D 20:01:02.439311] openib_check_cq: completed recv wr_id 684cb0. [D 20:01:02.439318] openib_post_rr: 10.1.4.56:39990 bh 92 wr_id 684cb0. [D 20:01:02.439325] openib_check_cq: completed recv wr_id 684ce0. [D 20:01:02.439333] openib_post_rr: 10.1.4.56:39990 bh 93 wr_id 684ce0. [D 20:01:02.439340] openib_check_cq: completed recv wr_id 684d10. [D 20:01:02.439347] openib_post_rr: 10.1.4.56:39990 bh 94 wr_id 684d10. [D 20:01:02.439354] openib_check_cq: completed recv wr_id 684d40. [D 20:01:02.439364] openib_post_rr: 10.1.4.56:39990 bh 95 wr_id 684d40. [D 20:01:02.439372] openib_check_cq: completed recv wr_id 684d70. [D 20:01:02.439379] openib_post_rr: 10.1.4.56:39990 bh 96 wr_id 684d70. [D 20:01:02.439386] openib_check_cq: completed recv wr_id 684da0. [D 20:01:02.439394] openib_post_rr: 10.1.4.56:39990 bh 97 wr_id 684da0. [D 20:01:02.439401] openib_check_cq: completed recv wr_id 684dd0. [D 20:01:02.439408] openib_post_rr: 10.1.4.56:39990 bh 98 wr_id 684dd0. [D 20:01:02.439415] openib_check_cq: completed recv wr_id 684e00. [D 20:01:02.439422] openib_post_rr: 10.1.4.56:39990 bh 99 wr_id 684e00. [D 20:01:02.439429] openib_check_cq: completed recv wr_id 683b70. [D 20:01:02.439437] openib_post_rr: 10.1.4.56:39990 bh 0 wr_id 683b70. [D 20:01:02.439444] openib_check_cq: completed recv wr_id 683ba0. [D 20:01:02.439452] openib_post_rr: 10.1.4.56:39990 bh 1 wr_id 683ba0. [D 20:01:02.439459] openib_check_cq: completed recv wr_id 683bd0. [D 20:01:02.439466] openib_post_rr: 10.1.4.56:39990 bh 2 wr_id 683bd0. [D 20:01:02.439473] openib_check_cq: completed recv wr_id 683c00. [D 20:01:02.439482] openib_post_rr: 10.1.4.56:39990 bh 3 wr_id 683c00. [D 20:01:02.439489] openib_check_cq: completed recv wr_id 683c30. [D 20:01:02.439496] openib_post_rr: 10.1.4.56:39990 bh 4 wr_id 683c30. [D 20:01:02.439504] openib_check_cq: completed recv wr_id 683c60. [D 20:01:02.439511] openib_post_rr: 10.1.4.56:39990 bh 5 wr_id 683c60. [D 20:01:02.439518] openib_check_cq: completed recv wr_id 683c90. [D 20:01:02.439525] openib_post_rr: 10.1.4.56:39990 bh 6 wr_id 683c90. [D 20:01:02.439533] openib_check_cq: completed recv wr_id 683cc0. [D 20:01:02.439540] openib_post_rr: 10.1.4.56:39990 bh 7 wr_id 683cc0. [D 20:01:02.439547] openib_check_cq: completed recv wr_id 683cf0. [D 20:01:02.439554] openib_post_rr: 10.1.4.56:39990 bh 8 wr_id 683cf0. [D 20:01:02.439562] openib_check_cq: completed recv wr_id 683d20. [D 20:01:02.439569] openib_post_rr: 10.1.4.56:39990 bh 9 wr_id 683d20. [D 20:01:02.439577] openib_check_cq: completed send wr_id 6839e0. [D 20:01:02.439584] openib_check_cq: completed send wr_id 683a10. [D 20:01:02.439591] openib_check_cq: completed send wr_id 683a40. [D 20:01:02.439597] openib_check_cq: completed send wr_id 683a70. [D 20:01:02.439604] openib_check_cq: completed send wr_id 683aa0. [D 20:01:02.439611] openib_check_cq: completed send wr_id 683ad0. [D 20:01:02.439618] openib_check_cq: completed send wr_id 683b00. [D 20:01:02.439625] openib_check_cq: completed send wr_id 683b30. [D 20:01:02.439631] openib_check_cq: completed send wr_id 6828a0. [D 20:01:02.439638] openib_check_cq: completed send wr_id 6828d0. [D 20:01:02.439645] openib_check_cq: completed recv wr_id 683d50. [D 20:01:02.439652] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea650 to 10.1.4.56:39990 remote addr 9e44d000 rkey 108e27 nic_wr_credit 4. [D 20:01:02.439660] openib_post_rr: 10.1.4.56:39990 bh 10 wr_id 683d50. [D 20:01:02.439668] openib_check_cq: completed rdma wr_id 6fed90, nic_wr_credit 3. [D 20:01:02.439676] post_sr: 10.1.4.56:39990 bh 2 len 16 wr_id 682900. [D 20:01:02.439694] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:01:02.439709] get_data_sync_mode returning 0 [D 20:01:02.439720] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.439727] dbpf_bstream_rw_list: mem_offset: 0x2aaab7c36010, mem_size: 2097152 [D 20:01:02.439734] dbpf_bstream_rw_list: stream_offset: 163577856, stream_size: 2097152 [D 20:01:02.439741] dbpf_open_cache_get: called [D 20:01:02.439108] tiling 5 copies [D 20:01:02.439173] op_queue add: 0x2aaabd300b80 [D 20:01:02.439764] post_sr: 10.1.4.56:39990 bh 3 len 32 wr_id 682930. [D 20:01:02.439037] DBPF I/O ops in progress: 15 [D 20:01:02.439783] starting delayed I/O operation 0x2aaabd301740 (15 in progress) [D 20:01:02.381098] --- aio_progress_notification called with handle 2863311531 (0x2aaab152be30) [D 20:01:02.439804] op_queue add: 0x2aaabd302990 [D 20:01:02.439794] lio_listio called with 1 following aiocbs: [D 20:01:02.439759] skipping ahead to target_offset [D 20:01:02.439795] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:02.439828] aiocb_ptr_array[0]: fd: 13, off: 396361728, bytes: 2097152, buf: 0x2aaab6e2f010, type: 1 [D 20:01:02.439838] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.439848] BMI_post_send: addr: 6843072, offset: 0x2aaabf20f010, size: 2097152, tag: 25 [D 20:01:02.439857] post_sr: 10.1.4.56:39990 bh 4 len 32 wr_id 682960. [D 20:01:02.439866] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:02.439875] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:02.439883] BMI_post_send: addr: 6843072, offset: 0x2aaabba56010, size: 2097152, tag: 36 [D 20:01:02.439891] post_sr: 10.1.4.56:39990 bh 5 len 32 wr_id 682990. [D 20:01:02.439901] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:02.439909] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:01:02.439830] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.439932] lvl 0 el 0 blk 0 by 0 [D 20:01:02.439939] to 0 ta 100663296 fi 134217728 [D 20:01:02.439945] basic type or contiguous data [D 20:01:02.439952] Request disp [D 20:01:02.439958] contig_offset = 0 contig_size = 167772160 lvl_flag = 1 [D 20:01:02.439964] process logical skip [D 20:01:02.439970] retval = 100663296 [D 20:01:02.439976] exiting logical skip because distribute indicates done [D 20:01:02.439982] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.439989] lvl 0 el 0 blk 0 by 100663296 [D 20:01:02.439995] to 100663296 ta 100663296 fi 134217728 [D 20:01:02.440002] basic type or contiguous data [D 20:01:02.440008] Request disp [D 20:01:02.440013] contig_offset = 100663296 contig_size = 67108864 lvl_flag = 1 [D 20:01:02.440020] calling distribute [D 20:01:02.440026] PINT_distribute [D 20:01:02.440032] of 100663296 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.440040] begin iteration loff: 100663296 [D 20:01:02.440047] segment exceeds byte limit [D 20:01:02.440054] process a segment [D 20:01:02.440059] of 100663296 sz 2097152 [D 20:01:02.440066] add a segment 0 [D 20:01:02.440072] end iteration [D 20:01:02.440078] done with segments or bytes [D 20:01:02.440083] of 102760448 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.440090] next loff: 102760448 next poff: 102760448 [D 20:01:02.440097] return value 2097152 [D 20:01:02.440103] finished [D 20:01:02.440108] retval = 2097152 [D 20:01:02.440114] exiting distribute returned less than expected [D 20:01:02.440120] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 102760448 fo 134217728 eof 0 [D 20:01:02.440127] ========================================================= [D 20:01:02.440137] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.440144] dbpf_bstream_rw_list: mem_offset: 0x2aaabfa13010, mem_size: 2097152 [D 20:01:02.440150] dbpf_bstream_rw_list: stream_offset: 100663296, stream_size: 2097152 [D 20:01:02.440157] dbpf_open_cache_get: called [D 20:01:02.440166] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.440202] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301740 (handle 2863311531, ret 0)) [D 20:01:02.440225] DBPF I/O ops in progress: 15 [D 20:01:02.440235] starting delayed I/O operation 0x2aaabd321080 (15 in progress) [D 20:01:02.440245] lio_listio called with 1 following aiocbs: [D 20:01:02.440251] aiocb_ptr_array[0]: fd: 13, off: 488636416, bytes: 2097152, buf: 0x2aaab5c25010, type: 1 [D 20:01:02.440261] start_delayed_ops_if_any: lio_listio posted 0x2aaabd321080 (handle 2863311531, ret 0)) [D 20:01:02.440537] DBPF I/O ops in progress: 15 [D 20:01:02.440550] starting delayed I/O operation 0x2aaabd3211d0 (15 in progress) [D 20:01:02.440561] lio_listio called with 1 following aiocbs: [D 20:01:02.440567] aiocb_ptr_array[0]: fd: 13, off: 429916160, bytes: 2097152, buf: 0x2aaab7030010, type: 1 [D 20:01:02.440577] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3211d0 (handle 2863311531, ret 0)) [D 20:01:02.440608] --- aio_progress_notification called with handle 2863311531 (0x2aaabd30b770) [D 20:01:02.440621] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.440636] DBPF I/O ops in progress: 15 [D 20:01:02.440644] starting delayed I/O operation 0x2aaabd321320 (15 in progress) [D 20:01:02.440655] lio_listio called with 1 following aiocbs: [D 20:01:02.440662] aiocb_ptr_array[0]: fd: 13, off: 459276288, bytes: 2097152, buf: 0x2aaabbc57010, type: 0 [D 20:01:02.440672] start_delayed_ops_if_any: lio_listio posted 0x2aaabd321320 (handle 2863311531, ret 0)) [D 20:01:02.440700] DBPF I/O ops in progress: 15 [D 20:01:02.440711] starting delayed I/O operation 0x2aaabd321470 (15 in progress) [D 20:01:02.440720] lio_listio called with 1 following aiocbs: [D 20:01:02.440726] aiocb_ptr_array[0]: fd: 13, off: 20971520, bytes: 2097152, buf: 0x2aaabcc5f010, type: 0 [D 20:01:02.440737] start_delayed_ops_if_any: lio_listio posted 0x2aaabd321470 (handle 2863311531, ret 0)) [D 20:01:02.440764] DBPF I/O ops in progress: 15 [D 20:01:02.440774] starting delayed I/O operation 0x2aaabd3215c0 (15 in progress) [D 20:01:02.440785] lio_listio called with 1 following aiocbs: [D 20:01:02.440792] aiocb_ptr_array[0]: fd: 13, off: 52428800, bytes: 2097152, buf: 0x2aaabdc04010, type: 0 [D 20:01:02.440801] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3215c0 (handle 2863311531, ret 0)) [D 20:01:02.439210] dbpf_open_cache_put: cached entry. [D 20:01:02.443549] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.443551] dbpf_open_cache_put: cached entry. [D 20:01:02.443567] --- aio_progress_notification called with handle 2863311531 (0x2aaabd305f70) [D 20:01:02.443575] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.443579] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.443607] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.443560] op_queue add: 0x2aaabd300d80 [D 20:01:02.443646] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.443648] op_queue add: 0x2aaabd3011a0 [D 20:01:02.443661] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.443674] delayed I/O operation 0x2aaabd3011a0 (16 already in progress) [D 20:01:02.443695] DBPF I/O ops in progress: 16 [D 20:01:02.443708] ========================================================= [D 20:01:02.443716] PINT_process_request [D 20:01:02.443723] tiling 6 copies [D 20:01:02.443730] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.443738] lvl 0 el 0 blk 0 by 165675008 [D 20:01:02.443745] to 165675008 ta 134217728 fi 167772160 [D 20:01:02.443753] basic type or contiguous data [D 20:01:02.443760] Request disp [D 20:01:02.443766] contig_offset = 165675008 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.443774] calling distribute [D 20:01:02.443780] PINT_distribute [D 20:01:02.443786] of 165675008 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 165675008 exfl 1 [D 20:01:02.443794] begin iteration loff: 165675008 [D 20:01:02.443801] file being extended [D 20:01:02.443807] process a segment [D 20:01:02.443813] of 165675008 sz 2097152 [D 20:01:02.443820] add a segment 0 [D 20:01:02.443826] end iteration [D 20:01:02.443832] done with segments or bytes [D 20:01:02.443839] of 167772160 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.443846] next loff: 167772160 next poff: 167772160 [D 20:01:02.443853] (return value) 2097152 [D 20:01:02.443859] finished [D 20:01:02.443866] retval = 2097152 [D 20:01:02.443872] exiting distribute returned less than expected [D 20:01:02.443878] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 167772160 fo 167772160 eof 0 [D 20:01:02.443885] ========================================================= [D 20:01:02.443893] BMI_post_recv: addr: 6843072, offset: 0x2aaab9c46010, size: 2097152, tag: 12 [D 20:01:02.443910] openib_check_cq: completed rdma wr_id 6ec3a0, nic_wr_credit 2. [D 20:01:02.443923] post_sr: 10.1.4.56:39990 bh 6 len 16 wr_id 6829c0. [D 20:01:02.443936] openib_check_cq: completed rdma wr_id 6ec210, nic_wr_credit 1. [D 20:01:02.443944] post_sr: 10.1.4.56:39990 bh 7 len 16 wr_id 6829f0. [D 20:01:02.443985] post_sr: 10.1.4.56:39990 bh 8 len 48 wr_id 682a20. [D 20:01:02.443996] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:01:02.444005] get_data_sync_mode returning 0 [D 20:01:02.444017] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.444024] dbpf_bstream_rw_list: mem_offset: 0x2aaab7e37010, mem_size: 2097152 [D 20:01:02.444031] dbpf_bstream_rw_list: stream_offset: 197132288, stream_size: 2097152 [D 20:01:02.444038] dbpf_open_cache_get: called [D 20:01:02.444045] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.444051] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.444069] op_queue add: 0x2aaabd300b50 [D 20:01:02.444077] delayed I/O operation 0x2aaabd300b50 (16 already in progress) [D 20:01:02.444083] DBPF I/O ops in progress: 16 [D 20:01:02.444090] ========================================================= [D 20:01:02.444097] PINT_process_request [D 20:01:02.444104] tiling 7 copies [D 20:01:02.444110] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.444118] lvl 0 el 0 blk 0 by 199229440 [D 20:01:02.444124] to 199229440 ta 167772160 fi 201326592 [D 20:01:02.444131] basic type or contiguous data [D 20:01:02.444137] Request disp [D 20:01:02.444143] contig_offset = 199229440 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.444150] calling distribute [D 20:01:02.444156] PINT_distribute [D 20:01:02.444162] of 199229440 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 199229440 exfl 1 [D 20:01:02.444169] begin iteration loff: 199229440 [D 20:01:02.444175] file being extended [D 20:01:02.444181] process a segment [D 20:01:02.444187] of 199229440 sz 2097152 [D 20:01:02.444193] add a segment 0 [D 20:01:02.444199] end iteration [D 20:01:02.444205] done with segments or bytes [D 20:01:02.444212] of 201326592 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.444218] next loff: 201326592 next poff: 201326592 [D 20:01:02.444225] (return value) 2097152 [D 20:01:02.444231] finished [D 20:01:02.444237] retval = 2097152 [D 20:01:02.444243] exiting distribute returned less than expected [D 20:01:02.444249] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 201326592 fo 201326592 eof 0 [D 20:01:02.444257] ========================================================= [D 20:01:02.444263] BMI_post_recv: addr: 6843072, offset: 0x2aaab9e47010, size: 2097152, tag: 13 [D 20:01:02.444276] post_sr: 10.1.4.56:39990 bh 9 len 48 wr_id 682a50. [D 20:01:02.444284] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:01:02.444293] get_data_sync_mode returning 0 [D 20:01:02.444302] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.444308] dbpf_bstream_rw_list: mem_offset: 0x2aaab8038010, mem_size: 2097152 [D 20:01:02.444315] dbpf_bstream_rw_list: stream_offset: 230686720, stream_size: 2097152 [D 20:01:02.444322] dbpf_open_cache_get: called [D 20:01:02.444328] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.444334] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.444353] op_queue add: 0x2aaabd3026d0 [D 20:01:02.444361] delayed I/O operation 0x2aaabd3026d0 (16 already in progress) [D 20:01:02.444367] DBPF I/O ops in progress: 16 [D 20:01:02.444374] ========================================================= [D 20:01:02.444381] PINT_process_request [D 20:01:02.444387] tiling 8 copies [D 20:01:02.444393] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.444401] lvl 0 el 0 blk 0 by 232783872 [D 20:01:02.444407] to 232783872 ta 201326592 fi 234881024 [D 20:01:02.444414] basic type or contiguous data [D 20:01:02.444420] Request disp [D 20:01:02.444426] contig_offset = 232783872 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.444433] calling distribute [D 20:01:02.444439] PINT_distribute [D 20:01:02.444445] of 232783872 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 232783872 exfl 1 [D 20:01:02.444453] begin iteration loff: 232783872 [D 20:01:02.444459] file being extended [D 20:01:02.444465] process a segment [D 20:01:02.444471] of 232783872 sz 2097152 [D 20:01:02.444477] add a segment 0 [D 20:01:02.444483] end iteration [D 20:01:02.444489] done with segments or bytes [D 20:01:02.444495] of 234881024 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.444502] next loff: 234881024 next poff: 234881024 [D 20:01:02.444508] (return value) 2097152 [D 20:01:02.444514] finished [D 20:01:02.444520] retval = 2097152 [D 20:01:02.444526] exiting distribute returned less than expected [D 20:01:02.444552] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 234881024 fo 234881024 eof 0 [D 20:01:02.444560] ========================================================= [D 20:01:02.444567] BMI_post_recv: addr: 6843072, offset: 0x2aaaba048010, size: 2097152, tag: 14 [D 20:01:02.444580] post_sr: 10.1.4.56:39990 bh 10 len 48 wr_id 682a80. [D 20:01:02.444590] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:01:02.444597] get_data_sync_mode returning 0 [D 20:01:02.444604] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.444611] dbpf_bstream_rw_list: mem_offset: 0x2aaab8239010, mem_size: 2097152 [D 20:01:02.444618] dbpf_bstream_rw_list: stream_offset: 264241152, stream_size: 2097152 [D 20:01:02.444625] dbpf_open_cache_get: called [D 20:01:02.444631] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.444638] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.444656] op_queue add: 0x2aaabd308ad0 [D 20:01:02.444663] delayed I/O operation 0x2aaabd308ad0 (16 already in progress) [D 20:01:02.444670] DBPF I/O ops in progress: 16 [D 20:01:02.444677] ========================================================= [D 20:01:02.444683] PINT_process_request [D 20:01:02.444689] tiling 9 copies [D 20:01:02.444696] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.444703] lvl 0 el 0 blk 0 by 266338304 [D 20:01:02.444710] to 266338304 ta 234881024 fi 268435456 [D 20:01:02.444717] basic type or contiguous data [D 20:01:02.444723] Request disp [D 20:01:02.444728] contig_offset = 266338304 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.444735] calling distribute [D 20:01:02.444741] PINT_distribute [D 20:01:02.444747] of 266338304 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 266338304 exfl 1 [D 20:01:02.444755] begin iteration loff: 266338304 [D 20:01:02.444761] file being extended [D 20:01:02.444767] process a segment [D 20:01:02.444773] of 266338304 sz 2097152 [D 20:01:02.444779] add a segment 0 [D 20:01:02.444785] end iteration [D 20:01:02.444791] done with segments or bytes [D 20:01:02.444797] of 268435456 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.444804] next loff: 268435456 next poff: 268435456 [D 20:01:02.444811] (return value) 2097152 [D 20:01:02.444817] finished [D 20:01:02.444823] retval = 2097152 [D 20:01:02.444829] exiting distribute returned less than expected [D 20:01:02.444835] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 268435456 fo 268435456 eof 0 [D 20:01:02.444842] ========================================================= [D 20:01:02.444849] BMI_post_recv: addr: 6843072, offset: 0x2aaaba249010, size: 2097152, tag: 15 [D 20:01:02.444861] post_sr: 10.1.4.56:39990 bh 11 len 48 wr_id 682ab0. [D 20:01:02.444870] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:01:02.444879] get_data_sync_mode returning 0 [D 20:01:02.444887] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.444893] dbpf_bstream_rw_list: mem_offset: 0x2aaab843a010, mem_size: 2097152 [D 20:01:02.444900] dbpf_bstream_rw_list: stream_offset: 297795584, stream_size: 2097152 [D 20:01:02.444907] dbpf_open_cache_get: called [D 20:01:02.444913] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.444920] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.444938] op_queue add: 0x2aaabd302820 [D 20:01:02.444945] delayed I/O operation 0x2aaabd302820 (16 already in progress) [D 20:01:02.444951] DBPF I/O ops in progress: 16 [D 20:01:02.444958] ========================================================= [D 20:01:02.444965] PINT_process_request [D 20:01:02.444971] tiling 10 copies [D 20:01:02.444977] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.444985] lvl 0 el 0 blk 0 by 299892736 [D 20:01:02.444991] to 299892736 ta 268435456 fi 301989888 [D 20:01:02.444998] basic type or contiguous data [D 20:01:02.445004] Request disp [D 20:01:02.445010] contig_offset = 299892736 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.445017] calling distribute [D 20:01:02.445023] PINT_distribute [D 20:01:02.445029] of 299892736 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 299892736 exfl 1 [D 20:01:02.445037] begin iteration loff: 299892736 [D 20:01:02.445043] file being extended [D 20:01:02.445049] process a segment [D 20:01:02.445055] of 299892736 sz 2097152 [D 20:01:02.445061] add a segment 0 [D 20:01:02.445067] end iteration [D 20:01:02.445073] done with segments or bytes [D 20:01:02.445079] of 301989888 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.445086] next loff: 301989888 next poff: 301989888 [D 20:01:02.445092] (return value) 2097152 [D 20:01:02.445098] finished [D 20:01:02.445104] retval = 2097152 [D 20:01:02.445110] exiting distribute returned less than expected [D 20:01:02.445116] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 301989888 fo 301989888 eof 0 [D 20:01:02.445123] ========================================================= [D 20:01:02.445129] BMI_post_recv: addr: 6843072, offset: 0x2aaaba44a010, size: 2097152, tag: 16 [D 20:01:02.445141] post_sr: 10.1.4.56:39990 bh 12 len 48 wr_id 682ae0. [D 20:01:02.445175] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:01:02.445185] get_data_sync_mode returning 0 [D 20:01:02.445192] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.445199] dbpf_bstream_rw_list: mem_offset: 0x2aaab863b010, mem_size: 2097152 [D 20:01:02.445206] dbpf_bstream_rw_list: stream_offset: 331350016, stream_size: 2097152 [D 20:01:02.445212] dbpf_open_cache_get: called [D 20:01:02.445218] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.445225] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.445242] op_queue add: 0x2aaabd302970 [D 20:01:02.445249] delayed I/O operation 0x2aaabd302970 (16 already in progress) [D 20:01:02.445256] DBPF I/O ops in progress: 16 [D 20:01:02.445263] ========================================================= [D 20:01:02.445270] PINT_process_request [D 20:01:02.445276] tiling 11 copies [D 20:01:02.445282] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.445289] lvl 0 el 0 blk 0 by 333447168 [D 20:01:02.445296] to 333447168 ta 301989888 fi 335544320 [D 20:01:02.445302] basic type or contiguous data [D 20:01:02.445308] Request disp [D 20:01:02.445314] contig_offset = 333447168 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.445321] calling distribute [D 20:01:02.445327] PINT_distribute [D 20:01:02.445333] of 333447168 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 333447168 exfl 1 [D 20:01:02.445340] begin iteration loff: 333447168 [D 20:01:02.445346] file being extended [D 20:01:02.445352] process a segment [D 20:01:02.445358] of 333447168 sz 2097152 [D 20:01:02.445364] add a segment 0 [D 20:01:02.445370] end iteration [D 20:01:02.445376] done with segments or bytes [D 20:01:02.445381] of 335544320 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.445388] next loff: 335544320 next poff: 335544320 [D 20:01:02.445395] (return value) 2097152 [D 20:01:02.445401] finished [D 20:01:02.445407] retval = 2097152 [D 20:01:02.445413] exiting distribute returned less than expected [D 20:01:02.445419] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 335544320 fo 335544320 eof 0 [D 20:01:02.445426] ========================================================= [D 20:01:02.445433] BMI_post_recv: addr: 6843072, offset: 0x2aaaba64b010, size: 2097152, tag: 17 [D 20:01:02.445444] post_sr: 10.1.4.56:39990 bh 13 len 48 wr_id 682b10. [D 20:01:02.445453] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:01:02.445461] get_data_sync_mode returning 0 [D 20:01:02.445468] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.445474] dbpf_bstream_rw_list: mem_offset: 0x2aaab883c010, mem_size: 2097152 [D 20:01:02.445481] dbpf_bstream_rw_list: stream_offset: 364904448, stream_size: 2097152 [D 20:01:02.445488] dbpf_open_cache_get: called [D 20:01:02.445495] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.445502] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.445522] op_queue add: 0x2aaabd302ac0 [D 20:01:02.445529] delayed I/O operation 0x2aaabd302ac0 (16 already in progress) [D 20:01:02.445535] DBPF I/O ops in progress: 16 [D 20:01:02.445543] ========================================================= [D 20:01:02.445549] PINT_process_request [D 20:01:02.445556] tiling 12 copies [D 20:01:02.445562] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.445570] lvl 0 el 0 blk 0 by 367001600 [D 20:01:02.445576] to 367001600 ta 335544320 fi 369098752 [D 20:01:02.445584] basic type or contiguous data [D 20:01:02.445590] Request disp [D 20:01:02.445596] contig_offset = 367001600 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.445603] calling distribute [D 20:01:02.445609] PINT_distribute [D 20:01:02.445614] of 367001600 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 367001600 exfl 1 [D 20:01:02.445622] begin iteration loff: 367001600 [D 20:01:02.445628] file being extended [D 20:01:02.445634] process a segment [D 20:01:02.445640] of 367001600 sz 2097152 [D 20:01:02.445646] add a segment 0 [D 20:01:02.445653] end iteration [D 20:01:02.445659] done with segments or bytes [D 20:01:02.445665] of 369098752 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.445672] next loff: 369098752 next poff: 369098752 [D 20:01:02.445678] (return value) 2097152 [D 20:01:02.445684] finished [D 20:01:02.445690] retval = 2097152 [D 20:01:02.445696] exiting distribute returned less than expected [D 20:01:02.445702] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 369098752 fo 369098752 eof 0 [D 20:01:02.445709] ========================================================= [D 20:01:02.445716] BMI_post_recv: addr: 6843072, offset: 0x2aaaba84c010, size: 2097152, tag: 18 [D 20:01:02.445724] openib_check_cq: completed rdma wr_id 6ea650, nic_wr_credit 0. [D 20:01:02.445733] post_sr: 10.1.4.56:39990 bh 14 len 16 wr_id 682b40. [D 20:01:02.445741] openib_check_cq: completed send wr_id 682900. [D 20:01:02.445749] openib_check_cq: completed send wr_id 682930. [D 20:01:02.445756] openib_check_cq: completed send wr_id 682960. [D 20:01:02.445762] openib_check_cq: completed send wr_id 682990. [D 20:01:02.445769] openib_check_cq: completed send wr_id 6829c0. [D 20:01:02.445776] openib_check_cq: completed send wr_id 6829f0. [D 20:01:02.445783] openib_check_cq: completed send wr_id 682a20. [D 20:01:02.445790] openib_check_cq: completed send wr_id 682a50. [D 20:01:02.445797] openib_check_cq: completed send wr_id 682a80. [D 20:01:02.445804] openib_check_cq: completed send wr_id 682ab0. [D 20:01:02.445810] openib_check_cq: completed send wr_id 682ae0. [D 20:01:02.445817] openib_check_cq: completed send wr_id 682b10. [D 20:01:02.445824] openib_check_cq: completed send wr_id 682b40. [D 20:01:02.445834] post_sr: 10.1.4.56:39990 bh 15 len 48 wr_id 682b70. [D 20:01:02.445844] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:01:02.445853] get_data_sync_mode returning 0 [D 20:01:02.445860] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.445866] dbpf_bstream_rw_list: mem_offset: 0x2aaab8a3d010, mem_size: 2097152 [D 20:01:02.445873] dbpf_bstream_rw_list: stream_offset: 398458880, stream_size: 2097152 [D 20:01:02.445880] dbpf_open_cache_get: called [D 20:01:02.445886] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.445893] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.445911] op_queue add: 0x2aaabd301d60 [D 20:01:02.445918] delayed I/O operation 0x2aaabd301d60 (16 already in progress) [D 20:01:02.445924] DBPF I/O ops in progress: 16 [D 20:01:02.445932] ========================================================= [D 20:01:02.445938] PINT_process_request [D 20:01:02.445944] tiling 13 copies [D 20:01:02.445950] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.445958] lvl 0 el 0 blk 0 by 400556032 [D 20:01:02.445965] to 400556032 ta 369098752 fi 402653184 [D 20:01:02.445972] basic type or contiguous data [D 20:01:02.445978] Request disp [D 20:01:02.445984] contig_offset = 400556032 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.445990] calling distribute [D 20:01:02.445996] PINT_distribute [D 20:01:02.446002] of 400556032 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 400556032 exfl 1 [D 20:01:02.446010] begin iteration loff: 400556032 [D 20:01:02.446016] file being extended [D 20:01:02.446022] process a segment [D 20:01:02.446027] of 400556032 sz 2097152 [D 20:01:02.446033] add a segment 0 [D 20:01:02.446039] end iteration [D 20:01:02.446045] done with segments or bytes [D 20:01:02.446052] of 402653184 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.446058] next loff: 402653184 next poff: 402653184 [D 20:01:02.446065] (return value) 2097152 [D 20:01:02.446071] finished [D 20:01:02.446077] retval = 2097152 [D 20:01:02.446083] exiting distribute returned less than expected [D 20:01:02.446089] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 402653184 fo 402653184 eof 0 [D 20:01:02.446096] ========================================================= [D 20:01:02.446103] BMI_post_recv: addr: 6843072, offset: 0x2aaabaa4d010, size: 2097152, tag: 19 [D 20:01:02.446110] openib_check_cq: completed send wr_id 682b70. [D 20:01:02.446122] post_sr: 10.1.4.56:39990 bh 16 len 48 wr_id 682ba0. [D 20:01:02.446132] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:02.446141] get_data_sync_mode returning 0 [D 20:01:02.446148] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.446155] dbpf_bstream_rw_list: mem_offset: 0x2aaab8e3f010, mem_size: 2097152 [D 20:01:02.446161] dbpf_bstream_rw_list: stream_offset: 490733568, stream_size: 2097152 [D 20:01:02.446168] dbpf_open_cache_get: called [D 20:01:02.446174] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.446180] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.446200] op_queue add: 0x2aaabd301eb0 [D 20:01:02.446207] delayed I/O operation 0x2aaabd301eb0 (16 already in progress) [D 20:01:02.446214] DBPF I/O ops in progress: 16 [D 20:01:02.446221] ========================================================= [D 20:01:02.446228] PINT_process_request [D 20:01:02.446234] tiling 16 copies [D 20:01:02.446240] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.446248] lvl 0 el 0 blk 0 by 492830720 [D 20:01:02.446255] to 492830720 ta 469762048 fi 503316480 [D 20:01:02.446262] basic type or contiguous data [D 20:01:02.446269] Request disp [D 20:01:02.446274] contig_offset = 492830720 contig_size = 44040192 lvl_flag = 1 [D 20:01:02.446281] calling distribute [D 20:01:02.446287] PINT_distribute [D 20:01:02.446293] of 492830720 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 492830720 exfl 1 [D 20:01:02.446300] begin iteration loff: 492830720 [D 20:01:02.446306] segment exceeds byte limit [D 20:01:02.446312] file being extended [D 20:01:02.446318] process a segment [D 20:01:02.446324] of 492830720 sz 2097152 [D 20:01:02.446330] add a segment 0 [D 20:01:02.446336] end iteration [D 20:01:02.446342] done with segments or bytes [D 20:01:02.446348] of 494927872 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.446354] next loff: 494927872 next poff: 494927872 [D 20:01:02.446361] return value 2097152 [D 20:01:02.446367] finished [D 20:01:02.446373] retval = 2097152 [D 20:01:02.446379] exiting distribute returned less than expected [D 20:01:02.446385] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 494927872 fo 503316480 eof 0 [D 20:01:02.446393] ========================================================= [D 20:01:02.446399] BMI_post_recv: addr: 6843072, offset: 0x2aaab9844010, size: 2097152, tag: 22 [D 20:01:02.446406] openib_check_cq: completed send wr_id 682ba0. [D 20:01:02.446417] post_sr: 10.1.4.56:39990 bh 17 len 48 wr_id 682bd0. [D 20:01:02.446427] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:02.446436] get_data_sync_mode returning 0 [D 20:01:02.446444] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.446450] dbpf_bstream_rw_list: mem_offset: 0x2aaab8c3e010, mem_size: 2097152 [D 20:01:02.446457] dbpf_bstream_rw_list: stream_offset: 432013312, stream_size: 2097152 [D 20:01:02.446464] dbpf_open_cache_get: called [D 20:01:02.446470] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.446476] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.446494] op_queue add: 0x2aaabd302000 [D 20:01:02.446501] delayed I/O operation 0x2aaabd302000 (16 already in progress) [D 20:01:02.446507] DBPF I/O ops in progress: 16 [D 20:01:02.446514] ========================================================= [D 20:01:02.446521] PINT_process_request [D 20:01:02.446527] tiling 14 copies [D 20:01:02.446533] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.446540] lvl 0 el 0 blk 0 by 434110464 [D 20:01:02.446547] to 434110464 ta 402653184 fi 436207616 [D 20:01:02.446554] basic type or contiguous data [D 20:01:02.446560] Request disp [D 20:01:02.446566] contig_offset = 434110464 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.446573] calling distribute [D 20:01:02.446579] PINT_distribute [D 20:01:02.446585] of 434110464 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 434110464 exfl 1 [D 20:01:02.446593] begin iteration loff: 434110464 [D 20:01:02.446599] file being extended [D 20:01:02.446605] process a segment [D 20:01:02.446611] of 434110464 sz 2097152 [D 20:01:02.446617] add a segment 0 [D 20:01:02.446623] end iteration [D 20:01:02.446628] done with segments or bytes [D 20:01:02.446634] of 436207616 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.446641] next loff: 436207616 next poff: 436207616 [D 20:01:02.446647] (return value) 2097152 [D 20:01:02.446653] finished [D 20:01:02.446659] retval = 2097152 [D 20:01:02.446665] exiting distribute returned less than expected [D 20:01:02.446671] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 436207616 fo 436207616 eof 0 [D 20:01:02.446679] ========================================================= [D 20:01:02.446686] BMI_post_recv: addr: 6843072, offset: 0x2aaabac4f010, size: 2097152, tag: 20 [D 20:01:02.446694] openib_check_cq: completed send wr_id 682bd0. [D 20:01:02.446705] post_sr: 10.1.4.56:39990 bh 18 len 48 wr_id 682c00. [D 20:01:02.446734] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.446746] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:02.446754] ========================================================= [D 20:01:02.446760] PINT_process_request [D 20:01:02.446766] tiling 15 copies [D 20:01:02.446773] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.446780] lvl 0 el 0 blk 0 by 461373440 [D 20:01:02.446786] to 461373440 ta 436207616 fi 469762048 [D 20:01:02.446793] basic type or contiguous data [D 20:01:02.446799] Request disp [D 20:01:02.446805] contig_offset = 461373440 contig_size = 41943040 lvl_flag = 1 [D 20:01:02.446811] calling distribute [D 20:01:02.446817] PINT_distribute [D 20:01:02.446823] of 461373440 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:02.446831] begin iteration loff: 461373440 [D 20:01:02.446837] segment exceeds byte limit [D 20:01:02.446843] process a segment [D 20:01:02.446849] of 461373440 sz 2097152 [D 20:01:02.446855] add a segment 0 [D 20:01:02.446861] end iteration [D 20:01:02.446867] done with segments or bytes [D 20:01:02.446873] of 463470592 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.446879] next loff: 463470592 next poff: 463470592 [D 20:01:02.446886] return value 2097152 [D 20:01:02.446891] finished [D 20:01:02.446897] retval = 2097152 [D 20:01:02.446903] exiting distribute returned less than expected [D 20:01:02.446910] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 463470592 fo 469762048 eof 0 [D 20:01:02.446918] ========================================================= [D 20:01:02.446928] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.446935] dbpf_bstream_rw_list: mem_offset: 0x2aaabbe58010, mem_size: 2097152 [D 20:01:02.446941] dbpf_bstream_rw_list: stream_offset: 461373440, stream_size: 2097152 [D 20:01:02.446948] dbpf_open_cache_get: called [D 20:01:02.446954] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.446960] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.446981] op_queue add: 0x2aaabd302c10 [D 20:01:02.446988] delayed I/O operation 0x2aaabd302c10 (16 already in progress) [D 20:01:02.446994] DBPF I/O ops in progress: 16 [D 20:01:02.447002] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.447010] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:02.447017] ========================================================= [D 20:01:02.447023] PINT_process_request [D 20:01:02.447029] tiling 2 copies [D 20:01:02.447035] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.447043] lvl 0 el 0 blk 0 by 23068672 [D 20:01:02.447049] to 23068672 ta 0 fi 33554432 [D 20:01:02.447055] basic type or contiguous data [D 20:01:02.447061] Request disp [D 20:01:02.447067] contig_offset = 23068672 contig_size = 44040192 lvl_flag = 1 [D 20:01:02.447073] calling distribute [D 20:01:02.447079] PINT_distribute [D 20:01:02.447085] of 23068672 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.447093] begin iteration loff: 23068672 [D 20:01:02.447099] segment exceeds byte limit [D 20:01:02.447105] process a segment [D 20:01:02.447111] of 23068672 sz 2097152 [D 20:01:02.447117] add a segment 0 [D 20:01:02.447123] end iteration [D 20:01:02.447128] done with segments or bytes [D 20:01:02.447134] of 25165824 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.447141] next loff: 25165824 next poff: 25165824 [D 20:01:02.447148] return value 2097152 [D 20:01:02.447154] finished [D 20:01:02.447160] retval = 2097152 [D 20:01:02.447165] exiting distribute returned less than expected [D 20:01:02.447171] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 25165824 fo 33554432 eof 0 [D 20:01:02.447179] ========================================================= [D 20:01:02.447186] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.447193] dbpf_bstream_rw_list: mem_offset: 0x2aaabce60010, mem_size: 2097152 [D 20:01:02.447199] dbpf_bstream_rw_list: stream_offset: 23068672, stream_size: 2097152 [D 20:01:02.447206] dbpf_open_cache_get: called [D 20:01:02.447212] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.447218] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.447236] op_queue add: 0x2aaabd302150 [D 20:01:02.447243] delayed I/O operation 0x2aaabd302150 (16 already in progress) [D 20:01:02.447249] DBPF I/O ops in progress: 16 [D 20:01:02.447256] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.447264] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:02.447271] ========================================================= [D 20:01:02.447278] PINT_process_request [D 20:01:02.447284] tiling 3 copies [D 20:01:02.447291] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.447298] lvl 0 el 0 blk 0 by 54525952 [D 20:01:02.447304] to 54525952 ta 33554432 fi 67108864 [D 20:01:02.447311] basic type or contiguous data [D 20:01:02.447316] Request disp [D 20:01:02.447322] contig_offset = 54525952 contig_size = 46137344 lvl_flag = 1 [D 20:01:02.447329] calling distribute [D 20:01:02.447334] PINT_distribute [D 20:01:02.447340] of 54525952 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.447349] begin iteration loff: 54525952 [D 20:01:02.447355] segment exceeds byte limit [D 20:01:02.447361] process a segment [D 20:01:02.447367] of 54525952 sz 2097152 [D 20:01:02.447373] add a segment 0 [D 20:01:02.447379] end iteration [D 20:01:02.447385] done with segments or bytes [D 20:01:02.447390] of 56623104 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.447397] next loff: 56623104 next poff: 56623104 [D 20:01:02.447404] return value 2097152 [D 20:01:02.447410] finished [D 20:01:02.447415] retval = 2097152 [D 20:01:02.447421] exiting distribute returned less than expected [D 20:01:02.447427] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 56623104 fo 67108864 eof 0 [D 20:01:02.447435] ========================================================= [D 20:01:02.447442] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.447449] dbpf_bstream_rw_list: mem_offset: 0x2aaabde05010, mem_size: 2097152 [D 20:01:02.447455] dbpf_bstream_rw_list: stream_offset: 54525952, stream_size: 2097152 [D 20:01:02.447462] dbpf_open_cache_get: called [D 20:01:02.447468] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.447474] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.447494] op_queue add: 0x2aaabd3022a0 [D 20:01:02.447501] delayed I/O operation 0x2aaabd3022a0 (16 already in progress) [D 20:01:02.447508] DBPF I/O ops in progress: 16 [D 20:01:02.447515] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.447523] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:02.447529] ========================================================= [D 20:01:02.447536] PINT_process_request [D 20:01:02.447542] tiling 4 copies [D 20:01:02.447548] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.447555] lvl 0 el 0 blk 0 by 83886080 [D 20:01:02.447561] to 83886080 ta 67108864 fi 100663296 [D 20:01:02.447568] basic type or contiguous data [D 20:01:02.447574] Request disp [D 20:01:02.447580] contig_offset = 83886080 contig_size = 50331648 lvl_flag = 1 [D 20:01:02.447586] calling distribute [D 20:01:02.447592] PINT_distribute [D 20:01:02.447598] of 83886080 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.447606] begin iteration loff: 83886080 [D 20:01:02.447612] segment exceeds byte limit [D 20:01:02.447618] process a segment [D 20:01:02.447623] of 83886080 sz 2097152 [D 20:01:02.447629] add a segment 0 [D 20:01:02.447635] end iteration [D 20:01:02.447641] done with segments or bytes [D 20:01:02.447647] of 85983232 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.447654] next loff: 85983232 next poff: 85983232 [D 20:01:02.447660] return value 2097152 [D 20:01:02.447666] finished [D 20:01:02.447672] retval = 2097152 [D 20:01:02.447679] exiting distribute returned less than expected [D 20:01:02.447685] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 85983232 fo 100663296 eof 0 [D 20:01:02.447692] ========================================================= [D 20:01:02.447700] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.447709] dbpf_bstream_rw_list: mem_offset: 0x2aaabea0b010, mem_size: 2097152 [D 20:01:02.447715] dbpf_bstream_rw_list: stream_offset: 83886080, stream_size: 2097152 [D 20:01:02.447722] dbpf_open_cache_get: called [D 20:01:02.447728] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.447734] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.447754] op_queue add: 0x2aaabd3023f0 [D 20:01:02.447761] delayed I/O operation 0x2aaabd3023f0 (16 already in progress) [D 20:01:02.447768] DBPF I/O ops in progress: 16 [D 20:01:02.447775] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:02.447784] get_data_sync_mode returning 0 [D 20:01:02.447791] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.447799] dbpf_bstream_rw_list: mem_offset: 0x2aaab9040010, mem_size: 2097152 [D 20:01:02.447806] dbpf_bstream_rw_list: stream_offset: 465567744, stream_size: 2097152 [D 20:01:02.447813] dbpf_open_cache_get: called [D 20:01:02.447819] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.447826] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.447848] op_queue add: 0x2aaabd302540 [D 20:01:02.447855] delayed I/O operation 0x2aaabd302540 (16 already in progress) [D 20:01:02.447862] DBPF I/O ops in progress: 16 [D 20:01:02.447870] ========================================================= [D 20:01:02.447876] PINT_process_request [D 20:01:02.447883] tiling 15 copies [D 20:01:02.447889] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.447896] lvl 0 el 0 blk 0 by 467664896 [D 20:01:02.447903] to 467664896 ta 436207616 fi 469762048 [D 20:01:02.447910] basic type or contiguous data [D 20:01:02.447916] Request disp [D 20:01:02.447922] contig_offset = 467664896 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.447929] calling distribute [D 20:01:02.447934] PINT_distribute [D 20:01:02.447940] of 467664896 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 467664896 exfl 1 [D 20:01:02.447949] begin iteration loff: 467664896 [D 20:01:02.447955] file being extended [D 20:01:02.447961] process a segment [D 20:01:02.447967] of 467664896 sz 2097152 [D 20:01:02.447973] add a segment 0 [D 20:01:02.447979] end iteration [D 20:01:02.447985] done with segments or bytes [D 20:01:02.447991] of 469762048 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.447998] next loff: 469762048 next poff: 469762048 [D 20:01:02.448004] (return value) 2097152 [D 20:01:02.448010] finished [D 20:01:02.448016] retval = 2097152 [D 20:01:02.448022] exiting distribute returned less than expected [D 20:01:02.448028] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 469762048 fo 469762048 eof 0 [D 20:01:02.448036] ========================================================= [D 20:01:02.448043] BMI_post_recv: addr: 6843072, offset: 0x2aaabae50010, size: 2097152, tag: 21 [D 20:01:02.448054] openib_check_cq: completed send wr_id 682c00. [D 20:01:02.448084] post_sr: 10.1.4.56:39990 bh 19 len 48 wr_id 682c30. [D 20:01:02.448112] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aadd0. [D 20:01:02.448122] get_data_sync_mode returning 0 [D 20:01:02.448130] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.448136] dbpf_bstream_rw_list: mem_offset: 0x2aaab9241010, mem_size: 2097152 [D 20:01:02.448143] dbpf_bstream_rw_list: stream_offset: 31457280, stream_size: 2097152 [D 20:01:02.448150] dbpf_open_cache_get: called [D 20:01:02.448156] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.448162] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.448183] op_queue add: 0x2aaabd302d60 [D 20:01:02.448189] delayed I/O operation 0x2aaabd302d60 (16 already in progress) [D 20:01:02.448196] DBPF I/O ops in progress: 16 [D 20:01:02.448203] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6aecb0. [D 20:01:02.448211] get_data_sync_mode returning 0 [D 20:01:02.448218] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.448225] dbpf_bstream_rw_list: mem_offset: 0x2aaab9442010, mem_size: 2097152 [D 20:01:02.448232] dbpf_bstream_rw_list: stream_offset: 65011712, stream_size: 2097152 [D 20:01:02.448239] dbpf_open_cache_get: called [D 20:01:02.448245] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.448251] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.448271] op_queue add: 0x2aaabd32c490 [D 20:01:02.448277] delayed I/O operation 0x2aaabd32c490 (16 already in progress) [D 20:01:02.448284] DBPF I/O ops in progress: 16 [D 20:01:02.448291] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b2df0. [D 20:01:02.448299] get_data_sync_mode returning 0 [D 20:01:02.448306] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.508188] dbpf_bstream_rw_list: mem_offset: 0x2aaab9643010, mem_size: 2097152 [D 20:01:02.508230] dbpf_bstream_rw_list: stream_offset: 98566144, stream_size: 2097152 [D 20:01:02.508238] dbpf_open_cache_get: called [D 20:01:02.448327] op_queue add: 0x2aaab152b470 [D 20:01:02.508264] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.508275] BMI_post_send: addr: 6843072, offset: 0x2aaabda03010, size: 2097152, tag: 24 [D 20:01:02.508289] post_sr: 10.1.4.56:39990 bh 20 len 32 wr_id 682c60. [D 20:01:02.508320] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:02.508382] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.448348] DBPF I/O ops in progress: 15 [D 20:01:02.508403] starting delayed I/O operation 0x2aaabd321710 (15 in progress) [D 20:01:02.508414] lio_listio called with 1 following aiocbs: [D 20:01:02.508420] aiocb_ptr_array[0]: fd: 13, off: 463470592, bytes: 2097152, buf: 0x2aaab7231010, type: 1 [D 20:01:02.508433] start_delayed_ops_if_any: lio_listio posted 0x2aaabd321710 (handle 2863311531, ret 0)) [D 20:01:02.448548] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3013e0) [D 20:01:02.448581] dbpf_open_cache_put: cached entry. [D 20:01:02.508466] op_queue add: 0x2aaab1522b60 [D 20:01:02.508472] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.508510] op_queue add: 0x2aaab152be30 [D 20:01:02.508538] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.508569] BMI_post_send: addr: 6843072, offset: 0x2aaabf410010, size: 2097152, tag: 25 [D 20:01:02.508577] post_sr: 10.1.4.56:39990 bh 21 len 32 wr_id 682c90. [D 20:01:02.508588] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:02.508595] BMI_post_send: addr: 6843072, offset: 0x2aaabf611010, size: 2097152, tag: 25 [D 20:01:02.508603] post_sr: 10.1.4.56:39990 bh 22 len 32 wr_id 682cc0. [D 20:01:02.508613] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:02.452552] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302ff0) [D 20:01:02.508632] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.508713] dbpf_open_cache_put: cached entry. [D 20:01:02.508720] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.508726] op_queue add: 0x2aaabd302ff0 [D 20:01:02.508742] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:02.508757] dbpf_open_cache_put: cached entry. [D 20:01:02.508768] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.508775] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.443589] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.508485] delayed I/O operation 0x2aaab1522b60 (16 already in progress) [D 20:01:02.508804] dbpf_open_cache_put: cached entry. [D 20:01:02.508846] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.508895] op_queue add: 0x2aaabd305f70 [D 20:01:02.508911] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:02.464478] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3015f0) [D 20:01:02.508928] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.508944] dbpf_open_cache_put: cached entry. [D 20:01:02.508951] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.508958] op_queue add: 0x2aaabd3015f0 [D 20:01:02.508973] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:02.464605] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301740) [D 20:01:02.508992] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.508804] DBPF I/O ops in progress: 16 [D 20:01:02.509079] dbpf_open_cache_put: cached entry. [D 20:01:02.509095] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.509102] op_queue add: 0x2aaabd301740 [D 20:01:02.509116] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:02.509086] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.509088] DBPF I/O ops in progress: 15 [D 20:01:02.509197] starting delayed I/O operation 0x2aaabd321860 (15 in progress) [D 20:01:02.509230] lio_listio called with 1 following aiocbs: [D 20:01:02.509242] aiocb_ptr_array[0]: fd: 13, off: 29360128, bytes: 2097152, buf: 0x2aaab7432010, type: 1 [D 20:01:02.509305] start_delayed_ops_if_any: lio_listio posted 0x2aaabd321860 (handle 2863311531, ret 0)) [D 20:01:02.509320] DBPF I/O ops in progress: 15 [D 20:01:02.509329] starting delayed I/O operation 0x2aaabd301990 (15 in progress) [D 20:01:02.509337] lio_listio called with 1 following aiocbs: [D 20:01:02.509343] aiocb_ptr_array[0]: fd: 13, off: 62914560, bytes: 2097152, buf: 0x2aaab7633010, type: 1 [D 20:01:02.509351] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301990 (handle 2863311531, ret 0)) [D 20:01:02.509366] DBPF I/O ops in progress: 15 [D 20:01:02.509374] starting delayed I/O operation 0x2aaabd3219b0 (15 in progress) [D 20:01:02.509189] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1522750. [D 20:01:02.509383] lio_listio called with 1 following aiocbs: [D 20:01:02.509396] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.509400] aiocb_ptr_array[0]: fd: 13, off: 96468992, bytes: 2097152, buf: 0x2aaab7834010, type: 1 [D 20:01:02.509416] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3219b0 (handle 2863311531, ret 0)) [D 20:01:02.509426] op_queue add: 0x2aaabd32c5e0 [D 20:01:02.509433] delayed I/O operation 0x2aaabd32c5e0 (16 already in progress) [D 20:01:02.509440] DBPF I/O ops in progress: 16 [D 20:01:02.510576] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6b7070. [D 20:01:02.510602] get_data_sync_mode returning 0 [D 20:01:02.510697] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.510708] dbpf_bstream_rw_list: mem_offset: 0x2aaab9a45010, mem_size: 2097152 [D 20:01:02.510715] dbpf_bstream_rw_list: stream_offset: 132120576, stream_size: 2097152 [D 20:01:02.510722] dbpf_open_cache_get: called [D 20:01:02.510729] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.510736] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.510752] op_queue add: 0x2aaabd305f70 [D 20:01:02.510761] delayed I/O operation 0x2aaabd305f70 (16 already in progress) [D 20:01:02.510768] DBPF I/O ops in progress: 16 [D 20:01:02.510780] openib_check_cq: completed send wr_id 682c30. [D 20:01:02.510790] openib_check_cq: completed recv wr_id 683d80. [D 20:01:02.510803] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea790 to 10.1.4.56:39990 remote addr 9e64d000 rkey 108e27 nic_wr_credit 1. [D 20:01:02.510872] openib_post_rr: 10.1.4.56:39990 bh 11 wr_id 683d80. [D 20:01:02.510881] openib_check_cq: completed recv wr_id 683db0. [D 20:01:02.510911] DBPF I/O ops in progress: 15 [D 20:01:02.510959] starting delayed I/O operation 0x2aaabd321b00 (15 in progress) [D 20:01:02.510971] lio_listio called with 1 following aiocbs: [D 20:01:02.510977] aiocb_ptr_array[0]: fd: 13, off: 130023424, bytes: 2097152, buf: 0x2aaab7a35010, type: 1 [D 20:01:02.510988] start_delayed_ops_if_any: lio_listio posted 0x2aaabd321b00 (handle 2863311531, ret 0)) [D 20:01:02.511023] DBPF I/O ops in progress: 15 [D 20:01:02.511035] starting delayed I/O operation 0x2aaab152bf80 (15 in progress) [D 20:01:02.511045] lio_listio called with 1 following aiocbs: [D 20:01:02.511140] aiocb_ptr_array[0]: fd: 13, off: 81788928, bytes: 2097152, buf: 0x2aaabf812010, type: 0 [D 20:01:02.511148] start_delayed_ops_if_any: lio_listio posted 0x2aaab152bf80 (handle 2863311531, ret 0)) [D 20:01:02.511174] DBPF I/O ops in progress: 15 [D 20:01:02.511217] starting delayed I/O operation 0x2aaabd3011a0 (15 in progress) [D 20:01:02.511232] lio_listio called with 1 following aiocbs: [D 20:01:02.511254] aiocb_ptr_array[0]: fd: 13, off: 163577856, bytes: 2097152, buf: 0x2aaab7c36010, type: 1 [D 20:01:02.511274] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3011a0 (handle 2863311531, ret 0)) [D 20:01:02.511260] ========================================================= [D 20:01:02.511262] openib_post_rr: 10.1.4.56:39990 bh 12 wr_id 683db0. [D 20:01:02.511320] PINT_process_request [D 20:01:02.511332] openib_check_cq: completed recv wr_id 683de0. [D 20:01:02.511354] openib_post_sr_rdmaw: ibv_post_send wr_id 6ff180 to 10.1.4.56:39990 remote addr 88e37000 rkey 108a27 nic_wr_credit 2. [D 20:01:02.511365] openib_post_rr: 10.1.4.56:39990 bh 13 wr_id 683de0. [D 20:01:02.511343] tiling 5 copies [D 20:01:02.511376] openib_check_cq: completed recv wr_id 683e10. [D 20:01:02.511390] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.511395] openib_post_sr_rdmaw: ibv_post_send wr_id 6ecdf0 to 10.1.4.56:39990 remote addr a2c51000 rkey 108c27 nic_wr_credit 3. [D 20:01:02.511413] openib_post_rr: 10.1.4.56:39990 bh 14 wr_id 683e10. [D 20:01:02.511423] openib_check_cq: completed recv wr_id 683e40. [D 20:01:02.511403] lvl 0 el 0 blk 0 by 102760448 [D 20:01:02.511435] openib_post_rr: 10.1.4.56:39990 bh 15 wr_id 683e40. [D 20:01:02.511458] openib_check_cq: completed recv wr_id 683e70. [D 20:01:02.511450] to 102760448 ta 100663296 fi 134217728 [D 20:01:02.511466] openib_post_sr_rdmaw: ibv_post_send wr_id 6edb10 to 10.1.4.56:39990 remote addr a0a4f000 rkey 108d27 nic_wr_credit 4. [D 20:01:02.511485] openib_post_rr: 10.1.4.56:39990 bh 16 wr_id 683e70. [D 20:01:02.511495] openib_check_cq: completed recv wr_id 683ea0. [D 20:01:02.511507] openib_post_rr: 10.1.4.56:39990 bh 17 wr_id 683ea0. [D 20:01:02.511517] openib_check_cq: completed recv wr_id 683ed0. [D 20:01:02.511527] openib_post_rr: 10.1.4.56:39990 bh 18 wr_id 683ed0. [D 20:01:02.511537] openib_check_cq: completed recv wr_id 683f00. [D 20:01:02.511547] openib_post_rr: 10.1.4.56:39990 bh 19 wr_id 683f00. [D 20:01:02.511475] basic type or contiguous data [D 20:01:02.511558] openib_check_cq: completed recv wr_id 683f30. [D 20:01:02.511582] openib_post_rr: 10.1.4.56:39990 bh 20 wr_id 683f30. [D 20:01:02.511592] openib_check_cq: completed recv wr_id 683f60. [D 20:01:02.511603] openib_post_rr: 10.1.4.56:39990 bh 21 wr_id 683f60. [D 20:01:02.511572] Request disp [D 20:01:02.511614] openib_check_cq: completed recv wr_id 683f90. [D 20:01:02.511637] openib_post_rr: 10.1.4.56:39990 bh 22 wr_id 683f90. [D 20:01:02.511647] openib_check_cq: completed recv wr_id 683fc0. [D 20:01:02.511627] contig_offset = 102760448 contig_size = 65011712 lvl_flag = 1 [D 20:01:02.511659] openib_post_rr: 10.1.4.56:39990 bh 23 wr_id 683fc0. [D 20:01:02.511679] openib_check_cq: completed recv wr_id 683ff0. [D 20:01:02.511672] calling distribute [D 20:01:02.511687] openib_post_rr: 10.1.4.56:39990 bh 24 wr_id 683ff0. [D 20:01:02.511708] openib_check_cq: completed recv wr_id 684020. [D 20:01:02.511696] PINT_distribute [D 20:01:02.511717] openib_post_rr: 10.1.4.56:39990 bh 25 wr_id 684020. [D 20:01:02.511735] openib_check_cq: completed recv wr_id 684050. [D 20:01:02.511747] openib_post_rr: 10.1.4.56:39990 bh 26 wr_id 684050. [D 20:01:02.511725] of 102760448 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.511758] openib_check_cq: completed send wr_id 682c60. [D 20:01:02.511773] begin iteration loff: 102760448 [D 20:01:02.511779] openib_check_cq: completed send wr_id 682c90. [D 20:01:02.511795] openib_check_cq: completed send wr_id 682cc0. [D 20:01:02.511786] segment exceeds byte limit [D 20:01:02.511814] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bb330. [D 20:01:02.511815] process a segment [D 20:01:02.511827] get_data_sync_mode returning 0 [D 20:01:02.511848] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.511858] dbpf_bstream_rw_list: mem_offset: 0x2aaab9c46010, mem_size: 2097152 [D 20:01:02.511839] of 102760448 sz 2097152 [D 20:01:02.511868] dbpf_bstream_rw_list: stream_offset: 165675008, stream_size: 2097152 [D 20:01:02.511890] dbpf_open_cache_get: called [D 20:01:02.511883] add a segment 0 [D 20:01:02.511896] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.511915] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.511907] end iteration [D 20:01:02.511936] op_queue add: 0x2aaabd300d80 [D 20:01:02.511953] delayed I/O operation 0x2aaabd300d80 (16 already in progress) [D 20:01:02.511962] DBPF I/O ops in progress: 16 [D 20:01:02.511972] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6bf6b0. [D 20:01:02.511983] get_data_sync_mode returning 0 [D 20:01:02.511994] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.512003] dbpf_bstream_rw_list: mem_offset: 0x2aaab9e47010, mem_size: 2097152 [D 20:01:02.512012] dbpf_bstream_rw_list: stream_offset: 199229440, stream_size: 2097152 [D 20:01:02.512021] dbpf_open_cache_get: called [D 20:01:02.512030] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.512039] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.511943] done with segments or bytes [D 20:01:02.512057] op_queue add: 0x2aaabd310e70 [D 20:01:02.512075] delayed I/O operation 0x2aaabd310e70 (16 already in progress) [D 20:01:02.512085] DBPF I/O ops in progress: 16 [D 20:01:02.512065] of 104857600 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.512096] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c0290. [D 20:01:02.512117] get_data_sync_mode returning 0 [D 20:01:02.512128] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.512137] dbpf_bstream_rw_list: mem_offset: 0x2aaaba048010, mem_size: 2097152 [D 20:01:02.512147] dbpf_bstream_rw_list: stream_offset: 232783872, stream_size: 2097152 [D 20:01:02.512156] dbpf_open_cache_get: called [D 20:01:02.512165] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.512174] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.512108] next loff: 104857600 next poff: 104857600 [D 20:01:02.512194] op_queue add: 0x2aaabd320e90 [D 20:01:02.512211] delayed I/O operation 0x2aaabd320e90 (16 already in progress) [D 20:01:02.512220] DBPF I/O ops in progress: 16 [D 20:01:02.512230] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6c7f30. [D 20:01:02.512241] get_data_sync_mode returning 0 [D 20:01:02.512202] return value 2097152 [D 20:01:02.512251] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.512273] dbpf_bstream_rw_list: mem_offset: 0x2aaaba249010, mem_size: 2097152 [D 20:01:02.512264] finished [D 20:01:02.512280] dbpf_bstream_rw_list: stream_offset: 266338304, stream_size: 2097152 [D 20:01:02.512300] dbpf_open_cache_get: called [D 20:01:02.512309] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.512317] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.512290] retval = 2097152 [D 20:01:02.512335] op_queue add: 0x2aaabd301ae0 [D 20:01:02.512354] delayed I/O operation 0x2aaabd301ae0 (16 already in progress) [D 20:01:02.512344] exiting distribute returned less than expected [D 20:01:02.512361] DBPF I/O ops in progress: 16 [D 20:01:02.512385] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6cc2b0. [D 20:01:02.512395] get_data_sync_mode returning 0 [D 20:01:02.512406] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.512415] dbpf_bstream_rw_list: mem_offset: 0x2aaaba44a010, mem_size: 2097152 [D 20:01:02.512375] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 104857600 fo 134217728 eof 0 [D 20:01:02.512424] dbpf_bstream_rw_list: stream_offset: 299892736, stream_size: 2097152 [D 20:01:02.512440] ========================================================= [D 20:01:02.512445] dbpf_open_cache_get: called [D 20:01:02.512454] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.512458] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.512476] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.512466] dbpf_bstream_rw_list: mem_offset: 0x2aaabfc14010, mem_size: 2097152 [D 20:01:02.512515] op_queue add: 0x2aaabd302eb0 [D 20:01:02.512516] dbpf_bstream_rw_list: stream_offset: 102760448, stream_size: 2097152 [D 20:01:02.512549] dbpf_open_cache_get: called [D 20:01:02.512556] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.512563] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.512528] delayed I/O operation 0x2aaabd302eb0 (16 already in progress) [D 20:01:02.512594] DBPF I/O ops in progress: 16 [D 20:01:02.512602] op_queue add: 0x2aaab1530a60 [D 20:01:02.512610] delayed I/O operation 0x2aaab1530a60 (16 already in progress) [D 20:01:02.512616] DBPF I/O ops in progress: 16 [D 20:01:02.512624] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.512631] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1522750. [D 20:01:02.512639] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.515274] openib_check_cq: completed rdma wr_id 6ea790, nic_wr_credit 3. [D 20:01:02.515430] ========================================================= [D 20:01:02.515440] PINT_process_request [D 20:01:02.515447] tiling 5 copies [D 20:01:02.515456] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.515464] lvl 0 el 0 blk 0 by 104857600 [D 20:01:02.515470] to 104857600 ta 100663296 fi 134217728 [D 20:01:02.515477] basic type or contiguous data [D 20:01:02.515484] Request disp [D 20:01:02.515491] contig_offset = 104857600 contig_size = 62914560 lvl_flag = 1 [D 20:01:02.515500] calling distribute [D 20:01:02.515506] PINT_distribute [D 20:01:02.515512] of 104857600 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.515522] begin iteration loff: 104857600 [D 20:01:02.515530] segment exceeds byte limit [D 20:01:02.515538] process a segment [D 20:01:02.515544] of 104857600 sz 2097152 [D 20:01:02.515551] add a segment 0 [D 20:01:02.515558] end iteration [D 20:01:02.515564] done with segments or bytes [D 20:01:02.515570] of 106954752 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.515578] next loff: 106954752 next poff: 106954752 [D 20:01:02.515584] return value 2097152 [D 20:01:02.515591] finished [D 20:01:02.515597] retval = 2097152 [D 20:01:02.515604] exiting distribute returned less than expected [D 20:01:02.515611] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 106954752 fo 134217728 eof 0 [D 20:01:02.515618] ========================================================= [D 20:01:02.515636] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.515644] dbpf_bstream_rw_list: mem_offset: 0x2aaabfe15010, mem_size: 2097152 [D 20:01:02.515651] dbpf_bstream_rw_list: stream_offset: 104857600, stream_size: 2097152 [D 20:01:02.515660] dbpf_open_cache_get: called [D 20:01:02.515668] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.515676] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.515703] op_queue add: 0x2aaab152b470 [D 20:01:02.515711] delayed I/O operation 0x2aaab152b470 (16 already in progress) [D 20:01:02.515718] DBPF I/O ops in progress: 16 [D 20:01:02.515725] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.515732] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1522750. [D 20:01:02.515742] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.457222] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303340) [D 20:01:02.470940] --- aio_progress_notification called with handle 2863311531 (0x2aaabd321080) [D 20:01:02.516727] --- aio_progress_notification called with handle 2863311531 (0x2aaabd321710) [D 20:01:02.516770] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.516785] dbpf_open_cache_put: cached entry. [D 20:01:02.516793] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.516800] op_queue add: 0x2aaabd321710 [D 20:01:02.518315] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.518434] ========================================================= [D 20:01:02.518446] PINT_process_request [D 20:01:02.518453] tiling 5 copies [D 20:01:02.518462] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.518470] lvl 0 el 0 blk 0 by 106954752 [D 20:01:02.518477] to 106954752 ta 100663296 fi 134217728 [D 20:01:02.518484] basic type or contiguous data [D 20:01:02.518491] Request disp [D 20:01:02.518498] contig_offset = 106954752 contig_size = 60817408 lvl_flag = 1 [D 20:01:02.518506] calling distribute [D 20:01:02.518514] PINT_distribute [D 20:01:02.518521] of 106954752 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.518529] begin iteration loff: 106954752 [D 20:01:02.518537] segment exceeds byte limit [D 20:01:02.518543] process a segment [D 20:01:02.518549] of 106954752 sz 2097152 [D 20:01:02.518556] add a segment 0 [D 20:01:02.518563] end iteration [D 20:01:02.518569] done with segments or bytes [D 20:01:02.518575] of 109051904 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.518583] next loff: 109051904 next poff: 109051904 [D 20:01:02.518590] return value 2097152 [D 20:01:02.518596] finished [D 20:01:02.518603] retval = 2097152 [D 20:01:02.518610] exiting distribute returned less than expected [D 20:01:02.518616] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 109051904 fo 134217728 eof 0 [D 20:01:02.518624] ========================================================= [D 20:01:02.518642] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.518650] dbpf_bstream_rw_list: mem_offset: 0x2aaac0016010, mem_size: 2097152 [D 20:01:02.518658] dbpf_bstream_rw_list: stream_offset: 106954752, stream_size: 2097152 [D 20:01:02.518666] dbpf_open_cache_get: called [D 20:01:02.518684] post_sr: 10.1.4.56:39990 bh 23 len 16 wr_id 682cf0. [D 20:01:02.518707] openib_check_cq: completed rdma wr_id 6ff180, nic_wr_credit 2. [D 20:01:02.518717] post_sr: 10.1.4.56:39990 bh 24 len 16 wr_id 682d20. [D 20:01:02.518724] openib_check_cq: completed rdma wr_id 6ecdf0, nic_wr_credit 1. [D 20:01:02.518734] post_sr: 10.1.4.56:39990 bh 25 len 16 wr_id 682d50. [D 20:01:02.518773] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d01a0. [D 20:01:02.518786] get_data_sync_mode returning 0 [D 20:01:02.518796] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.518804] dbpf_bstream_rw_list: mem_offset: 0x2aaaba64b010, mem_size: 2097152 [D 20:01:02.518810] dbpf_bstream_rw_list: stream_offset: 333447168, stream_size: 2097152 [D 20:01:02.518817] dbpf_open_cache_get: called [D 20:01:02.518825] dbpf_open_cache_put: cached entry. [D 20:01:02.518836] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.518844] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.518873] op_queue add: 0x2aaab1525840 [D 20:01:02.518882] delayed I/O operation 0x2aaab1525840 (16 already in progress) [D 20:01:02.518889] DBPF I/O ops in progress: 16 [D 20:01:02.518898] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.518906] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1522750. [D 20:01:02.518915] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.472759] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3211d0) [D 20:01:02.479713] --- aio_progress_notification called with handle 2863311531 (0x2aaabd321320) [D 20:01:02.479761] --- aio_progress_notification called with handle 2863311531 (0x2aaabd321470) [D 20:01:02.518927] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.529195] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.520606] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:02.527564] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.529357] ========================================================= [D 20:01:02.529369] PINT_process_request [D 20:01:02.529377] tiling 5 copies [D 20:01:02.529385] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.529393] lvl 0 el 0 blk 0 by 109051904 [D 20:01:02.529399] to 109051904 ta 100663296 fi 134217728 [D 20:01:02.529408] basic type or contiguous data [D 20:01:02.529425] Request disp [D 20:01:02.529433] contig_offset = 109051904 contig_size = 58720256 lvl_flag = 1 [D 20:01:02.529441] calling distribute [D 20:01:02.529448] PINT_distribute [D 20:01:02.529455] of 109051904 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.529466] begin iteration loff: 109051904 [D 20:01:02.529475] segment exceeds byte limit [D 20:01:02.529483] process a segment [D 20:01:02.529499] of 109051904 sz 2097152 [D 20:01:02.529507] add a segment 0 [D 20:01:02.529514] end iteration [D 20:01:02.529520] done with segments or bytes [D 20:01:02.529526] of 111149056 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.529534] next loff: 111149056 next poff: 111149056 [D 20:01:02.529542] return value 2097152 [D 20:01:02.529548] finished [D 20:01:02.529555] retval = 2097152 [D 20:01:02.529561] exiting distribute returned less than expected [D 20:01:02.529568] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 111149056 fo 134217728 eof 0 [D 20:01:02.529575] ========================================================= [D 20:01:02.529590] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.529600] dbpf_bstream_rw_list: mem_offset: 0x2aaac0217010, mem_size: 2097152 [D 20:01:02.529607] dbpf_bstream_rw_list: stream_offset: 109051904, stream_size: 2097152 [D 20:01:02.529615] dbpf_open_cache_get: called [D 20:01:02.529622] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.529629] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.529664] dbpf_open_cache_put: cached entry. [D 20:01:02.529675] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.529684] op_queue add: 0x2aaabd3211d0 [D 20:01:02.529710] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:02.518930] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.529742] op_queue add: 0x2aaabd321080 [D 20:01:02.529760] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:02.519526] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.529817] dbpf_open_cache_put: cached entry. [D 20:01:02.529825] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.529832] op_queue add: 0x2aaabd303340 [D 20:01:02.532568] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3219b0) [D 20:01:02.532623] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.532636] dbpf_open_cache_put: cached entry. [D 20:01:02.532643] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.532651] op_queue add: 0x2aaabd3219b0 [D 20:01:02.519534] DBPF I/O ops in progress: 15 [D 20:01:02.532676] starting delayed I/O operation 0x2aaabd300b50 (15 in progress) [D 20:01:02.532685] lio_listio called with 1 following aiocbs: [D 20:01:02.532692] aiocb_ptr_array[0]: fd: 13, off: 197132288, bytes: 2097152, buf: 0x2aaab7e37010, type: 1 [D 20:01:02.532704] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300b50 (handle 2863311531, ret 0)) [D 20:01:02.532716] op_queue add: 0x2aaabd310fc0 [D 20:01:02.532725] delayed I/O operation 0x2aaabd310fc0 (16 already in progress) [D 20:01:02.532731] DBPF I/O ops in progress: 16 [D 20:01:02.532746] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d4400. [D 20:01:02.532755] get_data_sync_mode returning 0 [D 20:01:02.532765] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.532773] dbpf_bstream_rw_list: mem_offset: 0x2aaaba84c010, mem_size: 2097152 [D 20:01:02.532779] dbpf_bstream_rw_list: stream_offset: 367001600, stream_size: 2097152 [D 20:01:02.532787] dbpf_open_cache_get: called [D 20:01:02.532793] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.532800] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.532817] op_queue add: 0x2aaabd321710 [D 20:01:02.532824] delayed I/O operation 0x2aaabd321710 (16 already in progress) [D 20:01:02.532830] DBPF I/O ops in progress: 16 [D 20:01:02.532837] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6d86a0. [D 20:01:02.532845] get_data_sync_mode returning 0 [D 20:01:02.532852] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.532858] dbpf_bstream_rw_list: mem_offset: 0x2aaabaa4d010, mem_size: 2097152 [D 20:01:02.532865] dbpf_bstream_rw_list: stream_offset: 400556032, stream_size: 2097152 [D 20:01:02.532872] dbpf_open_cache_get: called [D 20:01:02.532878] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.532884] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.532898] op_queue add: 0x2aaabd3008d0 [D 20:01:02.532905] delayed I/O operation 0x2aaabd3008d0 (16 already in progress) [D 20:01:02.532911] DBPF I/O ops in progress: 16 [D 20:01:02.532918] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:02.532926] get_data_sync_mode returning 0 [D 20:01:02.532933] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.532939] dbpf_bstream_rw_list: mem_offset: 0x2aaab9844010, mem_size: 2097152 [D 20:01:02.532946] dbpf_bstream_rw_list: stream_offset: 492830720, stream_size: 2097152 [D 20:01:02.532953] dbpf_open_cache_get: called [D 20:01:02.532959] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.532965] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.532978] op_queue add: 0x2aaabd303290 [D 20:01:02.532984] delayed I/O operation 0x2aaabd303290 (16 already in progress) [D 20:01:02.532991] DBPF I/O ops in progress: 16 [D 20:01:02.532998] ========================================================= [D 20:01:02.533005] PINT_process_request [D 20:01:02.533012] tiling 16 copies [D 20:01:02.533018] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.533026] lvl 0 el 0 blk 0 by 494927872 [D 20:01:02.533032] to 494927872 ta 469762048 fi 503316480 [D 20:01:02.533039] basic type or contiguous data [D 20:01:02.533046] Request disp [D 20:01:02.533052] contig_offset = 494927872 contig_size = 41943040 lvl_flag = 1 [D 20:01:02.533059] calling distribute [D 20:01:02.533065] PINT_distribute [D 20:01:02.533071] of 494927872 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 494927872 exfl 1 [D 20:01:02.533079] begin iteration loff: 494927872 [D 20:01:02.533086] segment exceeds byte limit [D 20:01:02.533092] file being extended [D 20:01:02.533098] process a segment [D 20:01:02.533104] of 494927872 sz 2097152 [D 20:01:02.533110] add a segment 0 [D 20:01:02.533116] end iteration [D 20:01:02.533122] done with segments or bytes [D 20:01:02.533128] of 497025024 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.533135] next loff: 497025024 next poff: 497025024 [D 20:01:02.533141] return value 2097152 [D 20:01:02.533147] finished [D 20:01:02.533153] retval = 2097152 [D 20:01:02.533158] exiting distribute returned less than expected [D 20:01:02.533165] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 497025024 fo 503316480 eof 0 [D 20:01:02.533172] ========================================================= [D 20:01:02.533179] BMI_post_recv: addr: 6843072, offset: 0x2aaabb051010, size: 2097152, tag: 22 [D 20:01:02.533193] openib_check_cq: completed rdma wr_id 6edb10, nic_wr_credit 0. [D 20:01:02.533204] post_sr: 10.1.4.56:39990 bh 26 len 16 wr_id 682d80. [D 20:01:02.533214] openib_check_cq: completed send wr_id 682cf0. [D 20:01:02.533222] openib_check_cq: completed send wr_id 682d20. [D 20:01:02.533229] openib_check_cq: completed send wr_id 682d50. [D 20:01:02.533236] openib_check_cq: completed recv wr_id 684080. [D 20:01:02.533245] openib_post_sr_rdmaw: ibv_post_send wr_id 723840 to 10.1.4.56:39990 remote addr 89037000 rkey 108a27 nic_wr_credit 1. [D 20:01:02.533253] openib_post_rr: 10.1.4.56:39990 bh 27 wr_id 684080. [D 20:01:02.533261] openib_check_cq: completed recv wr_id 6840b0. [D 20:01:02.533270] openib_post_sr_rdmaw: ibv_post_send wr_id 6676a0 to 10.1.4.56:39990 remote addr a2e51000 rkey 108c27 nic_wr_credit 2. [D 20:01:02.533278] openib_post_rr: 10.1.4.56:39990 bh 28 wr_id 6840b0. [D 20:01:02.533285] openib_check_cq: completed recv wr_id 6840e0. [D 20:01:02.533294] openib_post_sr_rdmaw: ibv_post_send wr_id 6eacb0 to 10.1.4.56:39990 remote addr 9e84d000 rkey 108e27 nic_wr_credit 3. [D 20:01:02.533302] openib_post_rr: 10.1.4.56:39990 bh 29 wr_id 6840e0. [D 20:01:02.533309] openib_check_cq: completed send wr_id 682d80. [D 20:01:02.533339] post_sr: 10.1.4.56:39990 bh 27 len 48 wr_id 682db0. [D 20:01:02.533348] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6dc960. [D 20:01:02.533356] get_data_sync_mode returning 0 [D 20:01:02.533363] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.533369] dbpf_bstream_rw_list: mem_offset: 0x2aaabac4f010, mem_size: 2097152 [D 20:01:02.533376] dbpf_bstream_rw_list: stream_offset: 434110464, stream_size: 2097152 [D 20:01:02.533383] dbpf_open_cache_get: called [D 20:01:02.533389] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.533395] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.508478] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.479779] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3215c0) [D 20:01:02.533407] op_queue add: 0x2aaab152b210 [D 20:01:02.535601] delayed I/O operation 0x2aaab152b210 (16 already in progress) [D 20:01:02.535628] DBPF I/O ops in progress: 16 [D 20:01:02.535658] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.535671] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1522750. [D 20:01:02.535684] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.456550] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303140) [D 20:01:02.508793] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.535747] dbpf_open_cache_put: cached entry. [D 20:01:02.543051] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.543075] ========================================================= [D 20:01:02.543091] PINT_process_request [D 20:01:02.543100] tiling 5 copies [D 20:01:02.543107] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.543115] lvl 0 el 0 blk 0 by 111149056 [D 20:01:02.543122] to 111149056 ta 100663296 fi 134217728 [D 20:01:02.543130] basic type or contiguous data [D 20:01:02.543137] Request disp [D 20:01:02.543144] contig_offset = 111149056 contig_size = 56623104 lvl_flag = 1 [D 20:01:02.543151] calling distribute [D 20:01:02.543157] PINT_distribute [D 20:01:02.543164] of 111149056 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.543173] begin iteration loff: 111149056 [D 20:01:02.543180] segment exceeds byte limit [D 20:01:02.543187] process a segment [D 20:01:02.543193] of 111149056 sz 2097152 [D 20:01:02.543200] add a segment 0 [D 20:01:02.543206] end iteration [D 20:01:02.543213] done with segments or bytes [D 20:01:02.543219] of 113246208 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.543226] next loff: 113246208 next poff: 113246208 [D 20:01:02.543232] return value 2097152 [D 20:01:02.543239] finished [D 20:01:02.543245] retval = 2097152 [D 20:01:02.543252] exiting distribute returned less than expected [D 20:01:02.543258] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 113246208 fo 134217728 eof 0 [D 20:01:02.543266] ========================================================= [D 20:01:02.543285] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.543293] dbpf_bstream_rw_list: mem_offset: 0x2aaac0418010, mem_size: 2097152 [D 20:01:02.543300] dbpf_bstream_rw_list: stream_offset: 111149056, stream_size: 2097152 [D 20:01:02.543308] dbpf_open_cache_get: called [D 20:01:02.543316] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.543323] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.535406] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301990) [D 20:01:02.543371] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.543384] dbpf_open_cache_put: cached entry. [D 20:01:02.543391] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.542983] --- aio_progress_notification called with handle 2863311531 (0x2aaab152bf80) [D 20:01:02.543412] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.543430] dbpf_open_cache_put: cached entry. [D 20:01:02.543437] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.524575] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.543472] dbpf_open_cache_put: cached entry. [D 20:01:02.543479] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.535759] DBPF I/O ops in progress: 15 [D 20:01:02.543505] starting delayed I/O operation 0x2aaabd3026d0 (15 in progress) [D 20:01:02.543517] lio_listio called with 1 following aiocbs: [D 20:01:02.543523] aiocb_ptr_array[0]: fd: 13, off: 230686720, bytes: 2097152, buf: 0x2aaab8038010, type: 1 [D 20:01:02.543536] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3026d0 (handle 2863311531, ret 0)) [D 20:01:02.538213] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.543602] dbpf_open_cache_put: cached entry. [D 20:01:02.543611] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.543623] DBPF I/O ops in progress: 15 [D 20:01:02.543633] starting delayed I/O operation 0x2aaabd308ad0 (15 in progress) [D 20:01:02.543643] lio_listio called with 1 following aiocbs: [D 20:01:02.543649] aiocb_ptr_array[0]: fd: 13, off: 264241152, bytes: 2097152, buf: 0x2aaab8239010, type: 1 [D 20:01:02.543659] start_delayed_ops_if_any: lio_listio posted 0x2aaabd308ad0 (handle 2863311531, ret 0)) [D 20:01:02.543679] DBPF I/O ops in progress: 15 [D 20:01:02.543689] starting delayed I/O operation 0x2aaabd302820 (15 in progress) [D 20:01:02.543699] lio_listio called with 1 following aiocbs: [D 20:01:02.543713] aiocb_ptr_array[0]: fd: 13, off: 297795584, bytes: 2097152, buf: 0x2aaab843a010, type: 1 [D 20:01:02.543722] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302820 (handle 2863311531, ret 0)) [D 20:01:02.543745] DBPF I/O ops in progress: 15 [D 20:01:02.543756] starting delayed I/O operation 0x2aaabd302970 (15 in progress) [D 20:01:02.543766] lio_listio called with 1 following aiocbs: [D 20:01:02.543772] aiocb_ptr_array[0]: fd: 13, off: 331350016, bytes: 2097152, buf: 0x2aaab863b010, type: 1 [D 20:01:02.543781] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302970 (handle 2863311531, ret 0)) [D 20:01:02.543796] op_queue add: 0x2aaabd3033e0 [D 20:01:02.543806] delayed I/O operation 0x2aaabd3033e0 (16 already in progress) [D 20:01:02.543813] DBPF I/O ops in progress: 16 [D 20:01:02.543837] openib_check_cq: completed recv wr_id 684110. [D 20:01:02.543854] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea920 to 10.1.4.56:39990 remote addr a0c4f000 rkey 108d27 nic_wr_credit 4. [D 20:01:02.543867] openib_post_rr: 10.1.4.56:39990 bh 30 wr_id 684110. [D 20:01:02.543879] openib_check_cq: completed rdma wr_id 723840, nic_wr_credit 3. [D 20:01:02.543890] post_sr: 10.1.4.56:39990 bh 28 len 16 wr_id 682de0. [D 20:01:02.543898] openib_check_cq: completed rdma wr_id 6676a0, nic_wr_credit 2. [D 20:01:02.543906] post_sr: 10.1.4.56:39990 bh 29 len 16 wr_id 682e10. [D 20:01:02.543914] openib_check_cq: completed rdma wr_id 6eacb0, nic_wr_credit 1. [D 20:01:02.543923] post_sr: 10.1.4.56:39990 bh 30 len 16 wr_id 682e40. [D 20:01:02.543930] openib_check_cq: completed send wr_id 682db0. [D 20:01:02.543937] openib_check_cq: completed recv wr_id 684140. [D 20:01:02.543946] openib_post_rr: 10.1.4.56:39990 bh 31 wr_id 684140. [D 20:01:02.543953] openib_check_cq: completed recv wr_id 684170. [D 20:01:02.543965] openib_post_rr: 10.1.4.56:39990 bh 32 wr_id 684170. [D 20:01:02.543997] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.544011] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:02.544019] ========================================================= [D 20:01:02.544026] PINT_process_request [D 20:01:02.544033] tiling 15 copies [D 20:01:02.544040] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.544047] lvl 0 el 0 blk 0 by 463470592 [D 20:01:02.544053] to 463470592 ta 436207616 fi 469762048 [D 20:01:02.544060] basic type or contiguous data [D 20:01:02.544066] Request disp [D 20:01:02.544072] contig_offset = 463470592 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.544079] calling distribute [D 20:01:02.544085] PINT_distribute [D 20:01:02.544091] of 463470592 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:02.544100] begin iteration loff: 463470592 [D 20:01:02.544106] segment exceeds byte limit [D 20:01:02.544112] process a segment [D 20:01:02.544118] of 463470592 sz 2097152 [D 20:01:02.544125] add a segment 0 [D 20:01:02.544130] end iteration [D 20:01:02.544136] done with segments or bytes [D 20:01:02.544142] of 465567744 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.544149] next loff: 465567744 next poff: 465567744 [D 20:01:02.544155] return value 2097152 [D 20:01:02.544161] finished [D 20:01:02.544167] retval = 2097152 [D 20:01:02.544173] exiting distribute returned less than expected [D 20:01:02.544179] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 465567744 fo 469762048 eof 0 [D 20:01:02.544187] ========================================================= [D 20:01:02.544197] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.544205] dbpf_bstream_rw_list: mem_offset: 0x2aaabc059010, mem_size: 2097152 [D 20:01:02.544212] dbpf_bstream_rw_list: stream_offset: 463470592, stream_size: 2097152 [D 20:01:02.544218] dbpf_open_cache_get: called [D 20:01:02.544225] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.544231] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.544270] op_queue add: 0x2aaabd300ed0 [D 20:01:02.544281] delayed I/O operation 0x2aaabd300ed0 (16 already in progress) [D 20:01:02.544288] DBPF I/O ops in progress: 16 [D 20:01:02.544295] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.544303] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:02.544310] ========================================================= [D 20:01:02.544316] PINT_process_request [D 20:01:02.544323] tiling 2 copies [D 20:01:02.544329] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.544336] lvl 0 el 0 blk 0 by 25165824 [D 20:01:02.544342] to 25165824 ta 0 fi 33554432 [D 20:01:02.544349] basic type or contiguous data [D 20:01:02.544355] Request disp [D 20:01:02.544360] contig_offset = 25165824 contig_size = 41943040 lvl_flag = 1 [D 20:01:02.544367] calling distribute [D 20:01:02.544373] PINT_distribute [D 20:01:02.544379] of 25165824 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.544387] begin iteration loff: 25165824 [D 20:01:02.544394] segment exceeds byte limit [D 20:01:02.544400] process a segment [D 20:01:02.544406] of 25165824 sz 2097152 [D 20:01:02.544412] add a segment 0 [D 20:01:02.544418] end iteration [D 20:01:02.544423] done with segments or bytes [D 20:01:02.544429] of 27262976 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.544437] next loff: 27262976 next poff: 27262976 [D 20:01:02.544443] return value 2097152 [D 20:01:02.544450] finished [D 20:01:02.544456] retval = 2097152 [D 20:01:02.544462] exiting distribute returned less than expected [D 20:01:02.544468] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 27262976 fo 33554432 eof 0 [D 20:01:02.544476] ========================================================= [D 20:01:02.544484] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.544491] dbpf_bstream_rw_list: mem_offset: 0x2aaabd061010, mem_size: 2097152 [D 20:01:02.544498] dbpf_bstream_rw_list: stream_offset: 25165824, stream_size: 2097152 [D 20:01:02.544504] dbpf_open_cache_get: called [D 20:01:02.544511] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.544518] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.544559] op_queue add: 0x2aaabd301020 [D 20:01:02.544570] delayed I/O operation 0x2aaabd301020 (16 already in progress) [D 20:01:02.544576] DBPF I/O ops in progress: 16 [D 20:01:02.544584] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.544592] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:02.544600] ========================================================= [D 20:01:02.544606] PINT_process_request [D 20:01:02.544613] tiling 3 copies [D 20:01:02.544619] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.544627] lvl 0 el 0 blk 0 by 56623104 [D 20:01:02.544633] to 56623104 ta 33554432 fi 67108864 [D 20:01:02.544639] basic type or contiguous data [D 20:01:02.544645] Request disp [D 20:01:02.544651] contig_offset = 56623104 contig_size = 44040192 lvl_flag = 1 [D 20:01:02.544658] calling distribute [D 20:01:02.544664] PINT_distribute [D 20:01:02.544670] of 56623104 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.544677] begin iteration loff: 56623104 [D 20:01:02.544684] segment exceeds byte limit [D 20:01:02.544689] process a segment [D 20:01:02.544695] of 56623104 sz 2097152 [D 20:01:02.544702] add a segment 0 [D 20:01:02.544708] end iteration [D 20:01:02.544714] done with segments or bytes [D 20:01:02.544720] of 58720256 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.544726] next loff: 58720256 next poff: 58720256 [D 20:01:02.544733] return value 2097152 [D 20:01:02.544739] finished [D 20:01:02.544745] retval = 2097152 [D 20:01:02.544750] exiting distribute returned less than expected [D 20:01:02.544757] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 58720256 fo 67108864 eof 0 [D 20:01:02.544764] ========================================================= [D 20:01:02.544772] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.544778] dbpf_bstream_rw_list: mem_offset: 0x2aaabe006010, mem_size: 2097152 [D 20:01:02.544785] dbpf_bstream_rw_list: stream_offset: 56623104, stream_size: 2097152 [D 20:01:02.544792] dbpf_open_cache_get: called [D 20:01:02.544798] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.544804] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.544829] op_queue add: 0x2aaabd320fe0 [D 20:01:02.544838] delayed I/O operation 0x2aaabd320fe0 (16 already in progress) [D 20:01:02.544845] DBPF I/O ops in progress: 16 [D 20:01:02.544852] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.544860] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:02.544875] op_queue add: 0x2aaab1517400 [D 20:01:02.596897] delayed I/O operation 0x2aaab1517400 (16 already in progress) [D 20:01:02.596911] DBPF I/O ops in progress: 16 [D 20:01:02.596921] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.596929] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1522750. [D 20:01:02.596937] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.598109] ========================================================= [D 20:01:02.598136] PINT_process_request [D 20:01:02.598143] tiling 4 copies [D 20:01:02.598150] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.598158] lvl 0 el 0 blk 0 by 85983232 [D 20:01:02.598164] to 85983232 ta 67108864 fi 100663296 [D 20:01:02.598171] basic type or contiguous data [D 20:01:02.598178] Request disp [D 20:01:02.598184] contig_offset = 85983232 contig_size = 48234496 lvl_flag = 1 [D 20:01:02.598191] calling distribute [D 20:01:02.598197] PINT_distribute [D 20:01:02.598203] of 85983232 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.598212] begin iteration loff: 85983232 [D 20:01:02.598218] segment exceeds byte limit [D 20:01:02.598225] process a segment [D 20:01:02.598231] of 85983232 sz 2097152 [D 20:01:02.598237] add a segment 0 [D 20:01:02.598243] end iteration [D 20:01:02.598249] done with segments or bytes [D 20:01:02.598255] of 88080384 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.598262] next loff: 88080384 next poff: 88080384 [D 20:01:02.598268] return value 2097152 [D 20:01:02.598275] finished [D 20:01:02.598280] retval = 2097152 [D 20:01:02.598286] exiting distribute returned less than expected [D 20:01:02.598292] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 88080384 fo 100663296 eof 0 [D 20:01:02.598300] ========================================================= [D 20:01:02.598312] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.598320] dbpf_bstream_rw_list: mem_offset: 0x2aaabec0c010, mem_size: 2097152 [D 20:01:02.598326] dbpf_bstream_rw_list: stream_offset: 85983232, stream_size: 2097152 [D 20:01:02.598334] dbpf_open_cache_get: called [D 20:01:02.598340] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.598347] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.598377] op_queue add: 0x2aaabd321130 [D 20:01:02.598385] delayed I/O operation 0x2aaabd321130 (16 already in progress) [D 20:01:02.598392] DBPF I/O ops in progress: 16 [D 20:01:02.598401] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x6e0ce0. [D 20:01:02.598409] get_data_sync_mode returning 0 [D 20:01:02.598417] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.598423] dbpf_bstream_rw_list: mem_offset: 0x2aaabae50010, mem_size: 2097152 [D 20:01:02.598430] dbpf_bstream_rw_list: stream_offset: 467664896, stream_size: 2097152 [D 20:01:02.598437] dbpf_open_cache_get: called [D 20:01:02.598443] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.598449] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.598470] op_queue add: 0x2aaabd301530 [D 20:01:02.598478] delayed I/O operation 0x2aaabd301530 (16 already in progress) [D 20:01:02.598485] DBPF I/O ops in progress: 16 [D 20:01:02.598499] openib_check_cq: completed rdma wr_id 6ea920, nic_wr_credit 0. [D 20:01:02.599322] post_sr: 10.1.4.56:39990 bh 31 len 16 wr_id 682e70. [D 20:01:02.599336] openib_check_cq: completed send wr_id 682de0. [D 20:01:02.599344] openib_check_cq: completed send wr_id 682e10. [D 20:01:02.599351] openib_check_cq: completed send wr_id 682e40. [D 20:01:02.599364] openib_check_cq: completed recv wr_id 6841a0. [D 20:01:02.599373] openib_post_sr_rdmaw: ibv_post_send wr_id 7239a0 to 10.1.4.56:39990 remote addr 89237000 rkey 108a27 nic_wr_credit 1. [D 20:01:02.599381] openib_post_rr: 10.1.4.56:39990 bh 33 wr_id 6841a0. [D 20:01:02.599390] openib_check_cq: completed recv wr_id 6841d0. [D 20:01:02.599398] openib_post_sr_rdmaw: ibv_post_send wr_id 6eaa40 to 10.1.4.56:39990 remote addr a3051000 rkey 108c27 nic_wr_credit 2. [D 20:01:02.599406] openib_post_rr: 10.1.4.56:39990 bh 34 wr_id 6841d0. [D 20:01:02.599413] openib_check_cq: completed recv wr_id 684200. [D 20:01:02.599421] openib_post_sr_rdmaw: ibv_post_send wr_id 6eadf0 to 10.1.4.56:39990 remote addr 9ea4d000 rkey 108e27 nic_wr_credit 3. [D 20:01:02.599429] openib_post_rr: 10.1.4.56:39990 bh 35 wr_id 684200. [D 20:01:02.599436] openib_check_cq: completed send wr_id 682e70. [D 20:01:02.599455] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.599465] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:02.599472] ========================================================= [D 20:01:02.599478] PINT_process_request [D 20:01:02.599484] tiling 15 copies [D 20:01:02.599491] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.599498] lvl 0 el 0 blk 0 by 465567744 [D 20:01:02.599504] to 465567744 ta 436207616 fi 469762048 [D 20:01:02.599511] basic type or contiguous data [D 20:01:02.599516] Request disp [D 20:01:02.599522] contig_offset = 465567744 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.599529] calling distribute [D 20:01:02.599535] PINT_distribute [D 20:01:02.599540] of 465567744 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:02.599548] begin iteration loff: 465567744 [D 20:01:02.599554] segment exceeds byte limit [D 20:01:02.599560] process a segment [D 20:01:02.599566] of 465567744 sz 2097152 [D 20:01:02.599572] add a segment 0 [D 20:01:02.599578] end iteration [D 20:01:02.599583] done with segments or bytes [D 20:01:02.599589] of 467664896 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.599596] next loff: 467664896 next poff: 467664896 [D 20:01:02.599602] return value 2097152 [D 20:01:02.599608] finished [D 20:01:02.599614] retval = 2097152 [D 20:01:02.599620] exiting distribute returned less than expected [D 20:01:02.599626] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 467664896 fo 469762048 eof 0 [D 20:01:02.599633] ========================================================= [D 20:01:02.599640] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.599647] dbpf_bstream_rw_list: mem_offset: 0x2aaabc25a010, mem_size: 2097152 [D 20:01:02.599653] dbpf_bstream_rw_list: stream_offset: 465567744, stream_size: 2097152 [D 20:01:02.599660] dbpf_open_cache_get: called [D 20:01:02.599666] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.599672] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.599688] op_queue add: 0x2aaabd301680 [D 20:01:02.599696] delayed I/O operation 0x2aaabd301680 (16 already in progress) [D 20:01:02.599703] DBPF I/O ops in progress: 16 [D 20:01:02.599714] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.599722] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:02.599730] ========================================================= [D 20:01:02.599736] PINT_process_request [D 20:01:02.599742] tiling 2 copies [D 20:01:02.599748] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.599755] lvl 0 el 0 blk 0 by 27262976 [D 20:01:02.599762] to 27262976 ta 0 fi 33554432 [D 20:01:02.599768] basic type or contiguous data [D 20:01:02.599774] Request disp [D 20:01:02.599780] contig_offset = 27262976 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.599787] calling distribute [D 20:01:02.599792] PINT_distribute [D 20:01:02.599798] of 27262976 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.599806] begin iteration loff: 27262976 [D 20:01:02.599812] segment exceeds byte limit [D 20:01:02.599818] process a segment [D 20:01:02.599824] of 27262976 sz 2097152 [D 20:01:02.599830] add a segment 0 [D 20:01:02.599836] end iteration [D 20:01:02.599842] done with segments or bytes [D 20:01:02.599848] of 29360128 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.599855] next loff: 29360128 next poff: 29360128 [D 20:01:02.599861] return value 2097152 [D 20:01:02.599867] finished [D 20:01:02.599873] retval = 2097152 [D 20:01:02.599878] exiting distribute returned less than expected [D 20:01:02.599884] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 29360128 fo 33554432 eof 0 [D 20:01:02.599892] ========================================================= [D 20:01:02.599899] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.599905] dbpf_bstream_rw_list: mem_offset: 0x2aaabd400010, mem_size: 2097152 [D 20:01:02.599912] dbpf_bstream_rw_list: stream_offset: 27262976, stream_size: 2097152 [D 20:01:02.599918] dbpf_open_cache_get: called [D 20:01:02.599924] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.599930] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.599950] op_queue add: 0x2aaabd310cf0 [D 20:01:02.599959] delayed I/O operation 0x2aaabd310cf0 (16 already in progress) [D 20:01:02.599965] DBPF I/O ops in progress: 16 [D 20:01:02.599972] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.599980] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:02.599987] ========================================================= [D 20:01:02.599993] PINT_process_request [D 20:01:02.599999] tiling 3 copies [D 20:01:02.600005] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.600012] lvl 0 el 0 blk 0 by 58720256 [D 20:01:02.600019] to 58720256 ta 33554432 fi 67108864 [D 20:01:02.600025] basic type or contiguous data [D 20:01:02.600031] Request disp [D 20:01:02.600037] contig_offset = 58720256 contig_size = 41943040 lvl_flag = 1 [D 20:01:02.600043] calling distribute [D 20:01:02.600049] PINT_distribute [D 20:01:02.600055] of 58720256 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.600062] begin iteration loff: 58720256 [D 20:01:02.600068] segment exceeds byte limit [D 20:01:02.600074] process a segment [D 20:01:02.600080] of 58720256 sz 2097152 [D 20:01:02.600086] add a segment 0 [D 20:01:02.600092] end iteration [D 20:01:02.600097] done with segments or bytes [D 20:01:02.600103] of 60817408 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.600110] next loff: 60817408 next poff: 60817408 [D 20:01:02.600116] return value 2097152 [D 20:01:02.600122] finished [D 20:01:02.600128] retval = 2097152 [D 20:01:02.600133] exiting distribute returned less than expected [D 20:01:02.600139] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 60817408 fo 67108864 eof 0 [D 20:01:02.600147] ========================================================= [D 20:01:02.600154] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.600160] dbpf_bstream_rw_list: mem_offset: 0x2aaabe207010, mem_size: 2097152 [D 20:01:02.600167] dbpf_bstream_rw_list: stream_offset: 58720256, stream_size: 2097152 [D 20:01:02.600173] dbpf_open_cache_get: called [D 20:01:02.600179] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.600185] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.600207] op_queue add: 0x2aaabd3017d0 [D 20:01:02.600215] delayed I/O operation 0x2aaabd3017d0 (16 already in progress) [D 20:01:02.600222] DBPF I/O ops in progress: 16 [D 20:01:02.600228] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.600236] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:02.600243] ========================================================= [D 20:01:02.600249] PINT_process_request [D 20:01:02.600254] tiling 4 copies [D 20:01:02.600260] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.600268] lvl 0 el 0 blk 0 by 88080384 [D 20:01:02.600274] to 88080384 ta 67108864 fi 100663296 [D 20:01:02.600281] basic type or contiguous data [D 20:01:02.600287] Request disp [D 20:01:02.600292] contig_offset = 88080384 contig_size = 46137344 lvl_flag = 1 [D 20:01:02.600299] calling distribute [D 20:01:02.600305] PINT_distribute [D 20:01:02.600310] of 88080384 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.600318] begin iteration loff: 88080384 [D 20:01:02.600324] segment exceeds byte limit [D 20:01:02.600330] process a segment [D 20:01:02.600335] of 88080384 sz 2097152 [D 20:01:02.600342] add a segment 0 [D 20:01:02.600347] end iteration [D 20:01:02.600353] done with segments or bytes [D 20:01:02.600359] of 90177536 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.600366] next loff: 90177536 next poff: 90177536 [D 20:01:02.600372] return value 2097152 [D 20:01:02.600378] finished [D 20:01:02.600384] retval = 2097152 [D 20:01:02.600390] exiting distribute returned less than expected [D 20:01:02.600396] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 90177536 fo 100663296 eof 0 [D 20:01:02.600403] ========================================================= [D 20:01:02.600411] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.600417] dbpf_bstream_rw_list: mem_offset: 0x2aaabee0d010, mem_size: 2097152 [D 20:01:02.600424] dbpf_bstream_rw_list: stream_offset: 88080384, stream_size: 2097152 [D 20:01:02.600430] dbpf_open_cache_get: called [D 20:01:02.600436] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.600442] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.600461] op_queue add: 0x2aaabd32f140 [D 20:01:02.600469] delayed I/O operation 0x2aaabd32f140 (16 already in progress) [D 20:01:02.600475] DBPF I/O ops in progress: 16 [D 20:01:02.600482] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:02.600490] get_data_sync_mode returning 0 [D 20:01:02.600497] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.600503] dbpf_bstream_rw_list: mem_offset: 0x2aaabb051010, mem_size: 2097152 [D 20:01:02.600510] dbpf_bstream_rw_list: stream_offset: 494927872, stream_size: 2097152 [D 20:01:02.600517] dbpf_open_cache_get: called [D 20:01:02.600522] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.600529] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.600538] ========================================================= [D 20:01:02.600556] PINT_process_request [D 20:01:02.548586] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300b50) [D 20:01:02.533412] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:02.600590] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.524555] --- aio_progress_notification called with handle 2863311531 (0x2aaabd321860) [D 20:01:02.533426] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.600605] dbpf_open_cache_put: cached entry. [D 20:01:02.538289] op_queue add: 0x2aaabd30b770 [D 20:01:02.538257] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.600627] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.550998] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3011a0) [D 20:01:02.600631] dbpf_open_cache_put: cached entry. [D 20:01:02.551041] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3026d0) [D 20:01:02.600645] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.600560] op_queue add: 0x2aaabd32f290 [D 20:01:02.600565] tiling 5 copies [D 20:01:02.600647] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.600660] delayed I/O operation 0x2aaabd32f290 (16 already in progress) [D 20:01:02.535416] --- aio_progress_notification called with handle 2863311531 (0x2aaabd321b00) [D 20:01:02.600692] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.562505] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302970) [D 20:01:02.600704] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.600667] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.600696] op_queue add: 0x2aaabd3013e0 [D 20:01:02.600735] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.600694] DBPF I/O ops in progress: 16 [D 20:01:02.556561] --- aio_progress_notification called with handle 2863311531 (0x2aaabd308ad0) [D 20:01:02.600731] lvl 0 el 0 blk 0 by 113246208 [D 20:01:02.600762] ========================================================= [D 20:01:02.600772] DBPF I/O ops in progress: 15 [D 20:01:02.600794] PINT_process_request [D 20:01:02.600805] tiling 16 copies [D 20:01:02.600811] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.600819] lvl 0 el 0 blk 0 by 497025024 [D 20:01:02.600825] to 497025024 ta 469762048 fi 503316480 [D 20:01:02.600832] basic type or contiguous data [D 20:01:02.600838] Request disp [D 20:01:02.600843] contig_offset = 497025024 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.600850] calling distribute [D 20:01:02.600855] PINT_distribute [D 20:01:02.600861] of 497025024 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 497025024 exfl 1 [D 20:01:02.600869] begin iteration loff: 497025024 [D 20:01:02.600875] segment exceeds byte limit [D 20:01:02.600881] file being extended [D 20:01:02.600886] process a segment [D 20:01:02.600892] of 497025024 sz 2097152 [D 20:01:02.600898] add a segment 0 [D 20:01:02.600904] end iteration [D 20:01:02.600910] done with segments or bytes [D 20:01:02.600916] of 499122176 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.600922] next loff: 499122176 next poff: 499122176 [D 20:01:02.600929] return value 2097152 [D 20:01:02.600935] finished [D 20:01:02.600940] retval = 2097152 [D 20:01:02.600946] exiting distribute returned less than expected [D 20:01:02.600952] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 499122176 fo 503316480 eof 0 [D 20:01:02.600959] ========================================================= [D 20:01:02.600966] BMI_post_recv: addr: 6843072, offset: 0x2aaabb252010, size: 2097152, tag: 22 [D 20:01:02.600975] openib_check_cq: completed recv wr_id 684230. [D 20:01:02.600983] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea9b0 to 10.1.4.56:39990 remote addr a0e4f000 rkey 108d27 nic_wr_credit 4. [D 20:01:02.600991] openib_post_rr: 10.1.4.56:39990 bh 36 wr_id 684230. [D 20:01:02.601014] post_sr: 10.1.4.56:39990 bh 32 len 48 wr_id 682ea0. [D 20:01:02.601615] openib_check_cq: completed rdma wr_id 7239a0, nic_wr_credit 3. [D 20:01:02.601624] post_sr: 10.1.4.56:39990 bh 33 len 16 wr_id 682ed0. [D 20:01:02.603836] openib_check_cq: completed rdma wr_id 6eaa40, nic_wr_credit 2. [D 20:01:02.603846] post_sr: 10.1.4.56:39990 bh 34 len 16 wr_id 682f00. [D 20:01:02.600669] dbpf_open_cache_put: cached entry. [D 20:01:02.606057] openib_check_cq: completed rdma wr_id 6eadf0, nic_wr_credit 1. [D 20:01:02.560548] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302820) [D 20:01:02.606067] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.606080] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.606087] dbpf_open_cache_put: cached entry. [D 20:01:02.606071] post_sr: 10.1.4.56:39990 bh 35 len 16 wr_id 682f30. [D 20:01:02.600790] to 113246208 ta 100663296 fi 134217728 [D 20:01:02.606127] basic type or contiguous data [D 20:01:02.606133] Request disp [D 20:01:02.606139] contig_offset = 113246208 contig_size = 54525952 lvl_flag = 1 [D 20:01:02.606146] calling distribute [D 20:01:02.606152] PINT_distribute [D 20:01:02.606158] of 113246208 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.606165] begin iteration loff: 113246208 [D 20:01:02.606172] segment exceeds byte limit [D 20:01:02.606178] process a segment [D 20:01:02.606184] of 113246208 sz 2097152 [D 20:01:02.606190] add a segment 0 [D 20:01:02.606196] end iteration [D 20:01:02.606202] done with segments or bytes [D 20:01:02.606208] of 115343360 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.606214] next loff: 115343360 next poff: 115343360 [D 20:01:02.606221] return value 2097152 [D 20:01:02.606227] finished [D 20:01:02.606233] retval = 2097152 [D 20:01:02.606239] exiting distribute returned less than expected [D 20:01:02.606245] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 115343360 fo 134217728 eof 0 [D 20:01:02.606252] ========================================================= [D 20:01:02.606262] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.606268] dbpf_bstream_rw_list: mem_offset: 0x2aaac0619010, mem_size: 2097152 [D 20:01:02.606275] dbpf_bstream_rw_list: stream_offset: 113246208, stream_size: 2097152 [D 20:01:02.606282] dbpf_open_cache_get: called [D 20:01:02.606101] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.600674] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.600774] op_queue add: 0x2aaabd301990 [D 20:01:02.600797] starting delayed I/O operation 0x2aaabd302ac0 (15 in progress) [D 20:01:02.606336] lio_listio called with 1 following aiocbs: [D 20:01:02.606342] aiocb_ptr_array[0]: fd: 13, off: 364904448, bytes: 2097152, buf: 0x2aaab883c010, type: 1 [D 20:01:02.606357] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302ac0 (handle 2863311531, ret 0)) [D 20:01:02.606105] dbpf_open_cache_put: cached entry. [D 20:01:02.606392] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.606399] op_queue add: 0x2aaabd3026d0 [D 20:01:02.606406] DBPF I/O ops in progress: 15 [D 20:01:02.606413] starting delayed I/O operation 0x2aaabd301d60 (15 in progress) [D 20:01:02.606422] lio_listio called with 1 following aiocbs: [D 20:01:02.606428] aiocb_ptr_array[0]: fd: 13, off: 398458880, bytes: 2097152, buf: 0x2aaab8a3d010, type: 1 [D 20:01:02.606437] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301d60 (handle 2863311531, ret 0)) [D 20:01:02.608284] openib_check_cq: completed rdma wr_id 6ea9b0, nic_wr_credit 0. [D 20:01:02.608328] post_sr: 10.1.4.56:39990 bh 36 len 16 wr_id 682f60. [D 20:01:02.608340] openib_check_cq: completed send wr_id 682ea0. [D 20:01:02.608349] openib_check_cq: completed send wr_id 682ed0. [D 20:01:02.608357] openib_check_cq: completed send wr_id 682f00. [D 20:01:02.608364] openib_check_cq: completed send wr_id 682f30. [D 20:01:02.608371] openib_check_cq: completed send wr_id 682f60. [D 20:01:02.608438] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.608459] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.608469] ========================================================= [D 20:01:02.608476] PINT_process_request [D 20:01:02.608484] tiling 15 copies [D 20:01:02.608491] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.608499] lvl 0 el 0 blk 0 by 467664896 [D 20:01:02.608505] to 467664896 ta 436207616 fi 469762048 [D 20:01:02.608513] basic type or contiguous data [D 20:01:02.608519] Request disp [D 20:01:02.608526] contig_offset = 467664896 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.608544] calling distribute [D 20:01:02.608551] PINT_distribute [D 20:01:02.608558] of 467664896 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:02.608567] begin iteration loff: 467664896 [D 20:01:02.608576] process a segment [D 20:01:02.608582] of 467664896 sz 2097152 [D 20:01:02.608591] add a segment 0 [D 20:01:02.608598] end iteration [D 20:01:02.608604] done with segments or bytes [D 20:01:02.608610] of 469762048 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.608616] next loff: 469762048 next poff: 469762048 [D 20:01:02.608623] (return value) 2097152 [D 20:01:02.608630] finished [D 20:01:02.608636] retval = 2097152 [D 20:01:02.608642] exiting distribute returned less than expected [D 20:01:02.608648] done sg 1 sm 64 by 2097152 bm 2097152 ta 436207616 to 469762048 fo 469762048 eof 0 [D 20:01:02.608656] ========================================================= [D 20:01:02.608674] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.608682] dbpf_bstream_rw_list: mem_offset: 0x2aaabc45b010, mem_size: 2097152 [D 20:01:02.608689] dbpf_bstream_rw_list: stream_offset: 467664896, stream_size: 2097152 [D 20:01:02.608698] dbpf_open_cache_get: called [D 20:01:02.608706] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.608714] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.608761] op_queue add: 0x2aaabd32f3e0 [D 20:01:02.608774] delayed I/O operation 0x2aaabd32f3e0 (16 already in progress) [D 20:01:02.608781] DBPF I/O ops in progress: 16 [D 20:01:02.608789] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.608797] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.608804] ========================================================= [D 20:01:02.608811] PINT_process_request [D 20:01:02.608817] tiling 2 copies [D 20:01:02.608823] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.608830] lvl 0 el 0 blk 0 by 29360128 [D 20:01:02.608836] to 29360128 ta 0 fi 33554432 [D 20:01:02.608843] basic type or contiguous data [D 20:01:02.608849] Request disp [D 20:01:02.608855] contig_offset = 29360128 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.608862] calling distribute [D 20:01:02.608868] PINT_distribute [D 20:01:02.608873] of 29360128 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.608881] begin iteration loff: 29360128 [D 20:01:02.608888] segment exceeds byte limit [D 20:01:02.608894] process a segment [D 20:01:02.608900] of 29360128 sz 2097152 [D 20:01:02.608906] add a segment 0 [D 20:01:02.608912] end iteration [D 20:01:02.608918] done with segments or bytes [D 20:01:02.608924] of 31457280 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.608931] next loff: 31457280 next poff: 31457280 [D 20:01:02.608937] return value 2097152 [D 20:01:02.608943] finished [D 20:01:02.608949] retval = 2097152 [D 20:01:02.608955] exiting distribute returned less than expected [D 20:01:02.608961] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 31457280 fo 33554432 eof 0 [D 20:01:02.608969] ========================================================= [D 20:01:02.608976] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.608983] dbpf_bstream_rw_list: mem_offset: 0x2aaabd601010, mem_size: 2097152 [D 20:01:02.608989] dbpf_bstream_rw_list: stream_offset: 29360128, stream_size: 2097152 [D 20:01:02.608996] dbpf_open_cache_get: called [D 20:01:02.609002] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.609008] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.609040] op_queue add: 0x2aaabd32f530 [D 20:01:02.609050] delayed I/O operation 0x2aaabd32f530 (16 already in progress) [D 20:01:02.609057] DBPF I/O ops in progress: 16 [D 20:01:02.609064] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.609072] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.609079] ========================================================= [D 20:01:02.609085] PINT_process_request [D 20:01:02.609092] tiling 3 copies [D 20:01:02.609098] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.609105] lvl 0 el 0 blk 0 by 60817408 [D 20:01:02.609112] to 60817408 ta 33554432 fi 67108864 [D 20:01:02.609118] basic type or contiguous data [D 20:01:02.609124] Request disp [D 20:01:02.609131] contig_offset = 60817408 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.609138] calling distribute [D 20:01:02.609144] PINT_distribute [D 20:01:02.609150] of 60817408 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.609158] begin iteration loff: 60817408 [D 20:01:02.609164] segment exceeds byte limit [D 20:01:02.609170] process a segment [D 20:01:02.609176] of 60817408 sz 2097152 [D 20:01:02.609183] add a segment 0 [D 20:01:02.609189] end iteration [D 20:01:02.609195] done with segments or bytes [D 20:01:02.609201] of 62914560 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.609208] next loff: 62914560 next poff: 62914560 [D 20:01:02.609215] return value 2097152 [D 20:01:02.609221] finished [D 20:01:02.609227] retval = 2097152 [D 20:01:02.609233] exiting distribute returned less than expected [D 20:01:02.609239] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 62914560 fo 67108864 eof 0 [D 20:01:02.609246] ========================================================= [D 20:01:02.609254] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.609261] dbpf_bstream_rw_list: mem_offset: 0x2aaabe408010, mem_size: 2097152 [D 20:01:02.609267] dbpf_bstream_rw_list: stream_offset: 60817408, stream_size: 2097152 [D 20:01:02.609274] dbpf_open_cache_get: called [D 20:01:02.609279] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.609286] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.609316] op_queue add: 0x2aaabd32f680 [D 20:01:02.609325] delayed I/O operation 0x2aaabd32f680 (16 already in progress) [D 20:01:02.609331] DBPF I/O ops in progress: 16 [D 20:01:02.609338] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.609346] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.609353] ========================================================= [D 20:01:02.609359] PINT_process_request [D 20:01:02.609366] tiling 4 copies [D 20:01:02.609373] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.609380] lvl 0 el 0 blk 0 by 90177536 [D 20:01:02.609386] to 90177536 ta 67108864 fi 100663296 [D 20:01:02.609393] basic type or contiguous data [D 20:01:02.609399] Request disp [D 20:01:02.609405] contig_offset = 90177536 contig_size = 44040192 lvl_flag = 1 [D 20:01:02.609412] calling distribute [D 20:01:02.609417] PINT_distribute [D 20:01:02.609423] of 90177536 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.609431] begin iteration loff: 90177536 [D 20:01:02.609437] segment exceeds byte limit [D 20:01:02.609443] process a segment [D 20:01:02.609449] of 90177536 sz 2097152 [D 20:01:02.609455] add a segment 0 [D 20:01:02.609461] end iteration [D 20:01:02.609467] done with segments or bytes [D 20:01:02.609473] of 92274688 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.609480] next loff: 92274688 next poff: 92274688 [D 20:01:02.609486] return value 2097152 [D 20:01:02.609492] finished [D 20:01:02.609498] retval = 2097152 [D 20:01:02.609503] exiting distribute returned less than expected [D 20:01:02.609510] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 92274688 fo 100663296 eof 0 [D 20:01:02.609517] ========================================================= [D 20:01:02.609525] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.609531] dbpf_bstream_rw_list: mem_offset: 0x2aaabf00e010, mem_size: 2097152 [D 20:01:02.609538] dbpf_bstream_rw_list: stream_offset: 90177536, stream_size: 2097152 [D 20:01:02.609545] dbpf_open_cache_get: called [D 20:01:02.609551] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.609557] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.609582] op_queue add: 0x2aaabd38e2e0 [D 20:01:02.609591] delayed I/O operation 0x2aaabd38e2e0 (16 already in progress) [D 20:01:02.609597] DBPF I/O ops in progress: 16 [D 20:01:02.609939] dbpf_open_cache_put: cached entry. [D 20:01:02.609958] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.609966] op_queue add: 0x2aaabd321b00 [D 20:01:02.609974] DBPF I/O ops in progress: 15 [D 20:01:02.609980] starting delayed I/O operation 0x2aaabd301eb0 (15 in progress) [D 20:01:02.609989] lio_listio called with 1 following aiocbs: [D 20:01:02.609995] aiocb_ptr_array[0]: fd: 13, off: 490733568, bytes: 2097152, buf: 0x2aaab8e3f010, type: 1 [D 20:01:02.610010] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302ac0) [D 20:01:02.610020] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.610029] dbpf_open_cache_put: cached entry. [D 20:01:02.610035] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.610041] op_queue add: 0x2aaabd302ac0 [D 20:01:02.610056] dbpf_open_cache_put: cached entry. [D 20:01:02.610067] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.610074] op_queue add: 0x2aaabd302970 [D 20:01:02.610083] dbpf_open_cache_put: cached entry. [D 20:01:02.610095] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.610103] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.610121] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.610129] op_queue add: 0x2aaabd302820 [D 20:01:02.610138] dbpf_open_cache_put: cached entry. [D 20:01:02.610147] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.610154] op_queue add: 0x2aaabd321860 [D 20:01:02.610800] openib_check_cq: completed recv wr_id 684260. [D 20:01:02.610813] openib_post_sr_rdmaw: ibv_post_send wr_id 66f9e0 to 10.1.4.56:39990 remote addr 89437000 rkey 108a27 nic_wr_credit 1. [D 20:01:02.610824] openib_post_rr: 10.1.4.56:39990 bh 37 wr_id 684260. [D 20:01:02.610892] openib_check_cq: completed recv wr_id 684290. [D 20:01:02.610903] openib_post_rr: 10.1.4.56:39990 bh 38 wr_id 684290. [D 20:01:02.610915] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:02.610925] get_data_sync_mode returning 0 [D 20:01:02.610936] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.610942] dbpf_bstream_rw_list: mem_offset: 0x2aaabb252010, mem_size: 2097152 [D 20:01:02.610950] dbpf_bstream_rw_list: stream_offset: 497025024, stream_size: 2097152 [D 20:01:02.610957] dbpf_open_cache_get: called [D 20:01:02.610963] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.610970] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.600785] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.611041] dbpf_open_cache_put: cached entry. [D 20:01:02.611052] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.611060] op_queue add: 0x2aaabd308ad0 [D 20:01:02.611075] op_queue add: 0x2aaab152bf80 [D 20:01:02.611094] op_queue add: 0x2aaabd321320 [D 20:01:02.611112] op_queue add: 0x2aaabd3215c0 [D 20:01:02.611144] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:02.611160] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:01:02.611168] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:02.611176] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:02.611183] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:02.611197] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:01:02.611205] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:02.611213] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:02.611224] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:02.611231] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:01:02.611245] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:02.611253] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.611261] BMI_post_send: addr: 6843072, offset: 0x2aaabf812010, size: 2097152, tag: 25 [D 20:01:02.611277] post_sr: 10.1.4.56:39990 bh 37 len 32 wr_id 682f90. [D 20:01:02.611316] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.611325] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:02.611332] BMI_post_send: addr: 6843072, offset: 0x2aaabbc57010, size: 2097152, tag: 36 [D 20:01:02.611343] post_sr: 10.1.4.56:39990 bh 38 len 32 wr_id 682fc0. [D 20:01:02.611353] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.611362] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.611370] BMI_post_send: addr: 6843072, offset: 0x2aaabdc04010, size: 2097152, tag: 24 [D 20:01:02.611379] post_sr: 10.1.4.56:39990 bh 39 len 32 wr_id 682ff0. [D 20:01:02.611389] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.612557] op_queue add: 0x2aaabd300b50 [D 20:01:02.612609] op_queue add: 0x2aaabd303140 [D 20:01:02.612633] op_queue add: 0x2aaabd321470 [D 20:01:02.614550] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301eb0 (handle 2863311531, ret 0)) [D 20:01:02.614593] op_queue add: 0x2aaabd3011a0 [D 20:01:02.614607] DBPF I/O ops in progress: 15 [D 20:01:02.614615] starting delayed I/O operation 0x2aaabd302000 (15 in progress) [D 20:01:02.614616] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:02.614633] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301d60) [D 20:01:02.614639] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:02.614659] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:02.614669] BMI_post_send: addr: 6843072, offset: 0x2aaabcc5f010, size: 2097152, tag: 23 [D 20:01:02.614685] post_sr: 10.1.4.56:39990 bh 40 len 32 wr_id 683020. [D 20:01:02.614756] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.614767] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:01:02.614627] lio_listio called with 1 following aiocbs: [D 20:01:02.614799] aiocb_ptr_array[0]: fd: 13, off: 432013312, bytes: 2097152, buf: 0x2aaab8c3e010, type: 1 [D 20:01:02.614809] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302000 (handle 2863311531, ret 0)) [D 20:01:02.614650] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.616621] dbpf_open_cache_put: cached entry. [D 20:01:02.616649] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.616658] op_queue add: 0x2aaabd301d60 [D 20:01:02.616689] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:02.616710] DBPF I/O ops in progress: 15 [D 20:01:02.616720] starting delayed I/O operation 0x2aaabd302c10 (15 in progress) [D 20:01:02.616730] lio_listio called with 1 following aiocbs: [D 20:01:02.616736] aiocb_ptr_array[0]: fd: 13, off: 461373440, bytes: 2097152, buf: 0x2aaabbe58010, type: 0 [D 20:01:02.616749] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302c10 (handle 2863311531, ret 0)) [D 20:01:02.616790] DBPF I/O ops in progress: 15 [D 20:01:02.616800] starting delayed I/O operation 0x2aaabd302150 (15 in progress) [D 20:01:02.616811] lio_listio called with 1 following aiocbs: [D 20:01:02.616817] aiocb_ptr_array[0]: fd: 13, off: 23068672, bytes: 2097152, buf: 0x2aaabce60010, type: 0 [D 20:01:02.616829] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302150 (handle 2863311531, ret 0)) [D 20:01:02.616857] DBPF I/O ops in progress: 15 [D 20:01:02.616866] starting delayed I/O operation 0x2aaabd3022a0 (15 in progress) [D 20:01:02.616878] lio_listio called with 1 following aiocbs: [D 20:01:02.616885] aiocb_ptr_array[0]: fd: 13, off: 54525952, bytes: 2097152, buf: 0x2aaabde05010, type: 0 [D 20:01:02.616896] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3022a0 (handle 2863311531, ret 0)) [D 20:01:02.616966] DBPF I/O ops in progress: 15 [D 20:01:02.616981] starting delayed I/O operation 0x2aaabd3023f0 (15 in progress) [D 20:01:02.616992] lio_listio called with 1 following aiocbs: [D 20:01:02.617009] aiocb_ptr_array[0]: fd: 13, off: 83886080, bytes: 2097152, buf: 0x2aaabea0b010, type: 0 [D 20:01:02.617021] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3023f0 (handle 2863311531, ret 0)) [D 20:01:02.617045] op_queue add: 0x2aaab152bb90 [D 20:01:02.617058] delayed I/O operation 0x2aaab152bb90 (16 already in progress) [D 20:01:02.617065] DBPF I/O ops in progress: 16 [D 20:01:02.617076] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.617083] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1522750. [D 20:01:02.617092] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.617785] DBPF I/O ops in progress: 15 [D 20:01:02.617812] starting delayed I/O operation 0x2aaabd302540 (15 in progress) [D 20:01:02.617823] lio_listio called with 1 following aiocbs: [D 20:01:02.617829] aiocb_ptr_array[0]: fd: 13, off: 465567744, bytes: 2097152, buf: 0x2aaab9040010, type: 1 [D 20:01:02.617841] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302540 (handle 2863311531, ret 0)) [D 20:01:02.618454] DBPF I/O ops in progress: 15 [D 20:01:02.618475] starting delayed I/O operation 0x2aaabd302d60 (15 in progress) [D 20:01:02.618487] lio_listio called with 1 following aiocbs: [D 20:01:02.618502] aiocb_ptr_array[0]: fd: 13, off: 31457280, bytes: 2097152, buf: 0x2aaab9241010, type: 1 [D 20:01:02.618513] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302d60 (handle 2863311531, ret 0)) [D 20:01:02.618531] op_queue add: 0x2aaabd390e40 [D 20:01:02.618541] delayed I/O operation 0x2aaabd390e40 (16 already in progress) [D 20:01:02.618550] DBPF I/O ops in progress: 16 [D 20:01:02.618563] ========================================================= [D 20:01:02.618574] PINT_process_request [D 20:01:02.618584] tiling 16 copies [D 20:01:02.618564] ========================================================= [D 20:01:02.618616] PINT_process_request [D 20:01:02.618594] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.618627] tiling 5 copies [D 20:01:02.618603] DBPF I/O ops in progress: 15 [D 20:01:02.618639] lvl 0 el 0 blk 0 by 499122176 [D 20:01:02.618637] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.618666] to 499122176 ta 469762048 fi 503316480 [D 20:01:02.618679] lvl 0 el 0 blk 0 by 115343360 [D 20:01:02.618690] basic type or contiguous data [D 20:01:02.618709] Request disp [D 20:01:02.618716] contig_offset = 499122176 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.618724] calling distribute [D 20:01:02.618731] PINT_distribute [D 20:01:02.618738] of 499122176 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 499122176 exfl 1 [D 20:01:02.618747] begin iteration loff: 499122176 [D 20:01:02.618754] segment exceeds byte limit [D 20:01:02.618761] file being extended [D 20:01:02.618767] process a segment [D 20:01:02.618773] of 499122176 sz 2097152 [D 20:01:02.618780] add a segment 0 [D 20:01:02.618786] end iteration [D 20:01:02.618792] done with segments or bytes [D 20:01:02.618798] of 501219328 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.618805] next loff: 501219328 next poff: 501219328 [D 20:01:02.618812] return value 2097152 [D 20:01:02.618819] finished [D 20:01:02.618825] retval = 2097152 [D 20:01:02.618831] exiting distribute returned less than expected [D 20:01:02.618838] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 501219328 fo 503316480 eof 0 [D 20:01:02.618846] ========================================================= [D 20:01:02.618854] BMI_post_recv: addr: 6843072, offset: 0x2aaabb453010, size: 2097152, tag: 22 [D 20:01:02.618872] openib_check_cq: completed recv wr_id 6842c0. [D 20:01:02.618885] openib_post_sr_rdmaw: ibv_post_send wr_id 6eda80 to 10.1.4.56:39990 remote addr a3251000 rkey 108c27 nic_wr_credit 2. [D 20:01:02.618896] openib_post_rr: 10.1.4.56:39990 bh 39 wr_id 6842c0. [D 20:01:02.618905] openib_check_cq: completed recv wr_id 6842f0. [D 20:01:02.618915] openib_post_sr_rdmaw: ibv_post_send wr_id 6ff030 to 10.1.4.56:39990 remote addr a104f000 rkey 108d27 nic_wr_credit 3. [D 20:01:02.618923] openib_post_rr: 10.1.4.56:39990 bh 40 wr_id 6842f0. [D 20:01:02.618931] openib_check_cq: completed rdma wr_id 66f9e0, nic_wr_credit 2. [D 20:01:02.618941] post_sr: 10.1.4.56:39990 bh 41 len 16 wr_id 683050. [D 20:01:02.618949] openib_check_cq: completed send wr_id 682f90. [D 20:01:02.618956] openib_check_cq: completed send wr_id 682fc0. [D 20:01:02.618964] openib_check_cq: completed send wr_id 682ff0. [D 20:01:02.618971] openib_check_cq: completed recv wr_id 684320. [D 20:01:02.618980] openib_post_sr_rdmaw: ibv_post_send wr_id 66aea0 to 10.1.4.56:39990 remote addr 9ec4d000 rkey 108e27 nic_wr_credit 3. [D 20:01:02.618989] openib_post_rr: 10.1.4.56:39990 bh 41 wr_id 684320. [D 20:01:02.618996] openib_check_cq: completed recv wr_id 684350. [D 20:01:02.619011] openib_post_rr: 10.1.4.56:39990 bh 42 wr_id 684350. [D 20:01:02.619019] openib_check_cq: completed send wr_id 683020. [D 20:01:02.619053] post_sr: 10.1.4.56:39990 bh 42 len 48 wr_id 683080. [D 20:01:02.621133] openib_check_cq: completed rdma wr_id 6eda80, nic_wr_credit 2. [D 20:01:02.621183] post_sr: 10.1.4.56:39990 bh 43 len 16 wr_id 6830b0. [D 20:01:02.623355] openib_check_cq: completed rdma wr_id 6ff030, nic_wr_credit 1. [D 20:01:02.623385] post_sr: 10.1.4.56:39990 bh 44 len 16 wr_id 6830e0. [D 20:01:02.623395] openib_check_cq: completed send wr_id 683050. [D 20:01:02.623407] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.623418] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.624564] openib_check_cq: completed recv wr_id 684380. [D 20:01:02.624621] openib_post_sr_rdmaw: ibv_post_send wr_id 6eaf30 to 10.1.4.56:39990 remote addr 89637000 rkey 108a27 nic_wr_credit 2. [D 20:01:02.624634] openib_post_rr: 10.1.4.56:39990 bh 43 wr_id 684380. [D 20:01:02.625582] openib_check_cq: completed rdma wr_id 66aea0, nic_wr_credit 1. [D 20:01:02.625627] post_sr: 10.1.4.56:39990 bh 45 len 16 wr_id 683110. [D 20:01:02.625638] openib_check_cq: completed send wr_id 683080. [D 20:01:02.625647] openib_check_cq: completed send wr_id 6830b0. [D 20:01:02.625656] openib_check_cq: completed send wr_id 6830e0. [D 20:01:02.625782] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.625807] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.625819] ========================================================= [D 20:01:02.625827] PINT_process_request [D 20:01:02.625836] tiling 2 copies [D 20:01:02.625847] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.625856] lvl 0 el 0 blk 0 by 31457280 [D 20:01:02.625864] to 31457280 ta 0 fi 33554432 [D 20:01:02.625872] basic type or contiguous data [D 20:01:02.625880] Request disp [D 20:01:02.625887] contig_offset = 31457280 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.625896] calling distribute [D 20:01:02.625903] PINT_distribute [D 20:01:02.625911] of 31457280 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.625921] begin iteration loff: 31457280 [D 20:01:02.625931] process a segment [D 20:01:02.625938] of 31457280 sz 2097152 [D 20:01:02.625946] add a segment 0 [D 20:01:02.625957] end iteration [D 20:01:02.625963] done with segments or bytes [D 20:01:02.625970] of 33554432 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.625977] next loff: 33554432 next poff: 33554432 [D 20:01:02.625984] (return value) 2097152 [D 20:01:02.625992] finished [D 20:01:02.625998] retval = 2097152 [D 20:01:02.626005] exiting distribute returned less than expected [D 20:01:02.626012] done sg 1 sm 64 by 2097152 bm 2097152 ta 0 to 33554432 fo 33554432 eof 0 [D 20:01:02.626019] ========================================================= [D 20:01:02.626051] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.626064] dbpf_bstream_rw_list: mem_offset: 0x2aaabd802010, mem_size: 2097152 [D 20:01:02.626071] dbpf_bstream_rw_list: stream_offset: 31457280, stream_size: 2097152 [D 20:01:02.626082] dbpf_open_cache_get: called [D 20:01:02.626092] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.626102] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.618702] to 115343360 ta 100663296 fi 134217728 [D 20:01:02.626267] basic type or contiguous data [D 20:01:02.626275] Request disp [D 20:01:02.626281] contig_offset = 115343360 contig_size = 52428800 lvl_flag = 1 [D 20:01:02.626289] calling distribute [D 20:01:02.626295] PINT_distribute [D 20:01:02.626302] of 115343360 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.626310] begin iteration loff: 115343360 [D 20:01:02.626317] segment exceeds byte limit [D 20:01:02.626323] process a segment [D 20:01:02.626330] of 115343360 sz 2097152 [D 20:01:02.626336] add a segment 0 [D 20:01:02.626342] end iteration [D 20:01:02.626349] done with segments or bytes [D 20:01:02.626355] of 117440512 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.626362] next loff: 117440512 next poff: 117440512 [D 20:01:02.626369] return value 2097152 [D 20:01:02.626375] finished [D 20:01:02.626381] retval = 2097152 [D 20:01:02.626388] exiting distribute returned less than expected [D 20:01:02.626394] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 117440512 fo 134217728 eof 0 [D 20:01:02.626402] ========================================================= [D 20:01:02.626418] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.626426] dbpf_bstream_rw_list: mem_offset: 0x2aaac081a010, mem_size: 2097152 [D 20:01:02.626433] dbpf_bstream_rw_list: stream_offset: 115343360, stream_size: 2097152 [D 20:01:02.626440] dbpf_open_cache_get: called [D 20:01:02.626447] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.626454] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.618654] starting delayed I/O operation 0x2aaabd32c490 (15 in progress) [D 20:01:02.626506] lio_listio called with 1 following aiocbs: [D 20:01:02.626517] aiocb_ptr_array[0]: fd: 13, off: 65011712, bytes: 2097152, buf: 0x2aaab9442010, type: 1 [D 20:01:02.626541] start_delayed_ops_if_any: lio_listio posted 0x2aaabd32c490 (handle 2863311531, ret 0)) [D 20:01:02.626644] DBPF I/O ops in progress: 15 [D 20:01:02.626683] starting delayed I/O operation 0x2aaab1522b60 (15 in progress) [D 20:01:02.626698] lio_listio called with 1 following aiocbs: [D 20:01:02.626704] aiocb_ptr_array[0]: fd: 13, off: 100663296, bytes: 2097152, buf: 0x2aaabfa13010, type: 0 [D 20:01:02.626716] start_delayed_ops_if_any: lio_listio posted 0x2aaab1522b60 (handle 2863311531, ret 0)) [D 20:01:02.626739] DBPF I/O ops in progress: 15 [D 20:01:02.626754] starting delayed I/O operation 0x2aaabd32c5e0 (15 in progress) [D 20:01:02.626769] lio_listio called with 1 following aiocbs: [D 20:01:02.626776] aiocb_ptr_array[0]: fd: 13, off: 98566144, bytes: 2097152, buf: 0x2aaab9643010, type: 1 [D 20:01:02.626790] start_delayed_ops_if_any: lio_listio posted 0x2aaabd32c5e0 (handle 2863311531, ret 0)) [D 20:01:02.626820] DBPF I/O ops in progress: 15 [D 20:01:02.626834] starting delayed I/O operation 0x2aaabd305f70 (15 in progress) [D 20:01:02.626848] lio_listio called with 1 following aiocbs: [D 20:01:02.626857] aiocb_ptr_array[0]: fd: 13, off: 132120576, bytes: 2097152, buf: 0x2aaab9a45010, type: 1 [D 20:01:02.626869] start_delayed_ops_if_any: lio_listio posted 0x2aaabd305f70 (handle 2863311531, ret 0)) [D 20:01:02.626976] DBPF I/O ops in progress: 15 [D 20:01:02.626992] starting delayed I/O operation 0x2aaabd300d80 (15 in progress) [D 20:01:02.627006] lio_listio called with 1 following aiocbs: [D 20:01:02.627014] aiocb_ptr_array[0]: fd: 13, off: 165675008, bytes: 2097152, buf: 0x2aaab9c46010, type: 1 [D 20:01:02.627045] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300d80 (handle 2863311531, ret 0)) [D 20:01:02.627067] DBPF I/O ops in progress: 15 [D 20:01:02.627081] starting delayed I/O operation 0x2aaabd310e70 (15 in progress) [D 20:01:02.627096] lio_listio called with 1 following aiocbs: [D 20:01:02.627104] aiocb_ptr_array[0]: fd: 13, off: 199229440, bytes: 2097152, buf: 0x2aaab9e47010, type: 1 [D 20:01:02.627134] start_delayed_ops_if_any: lio_listio posted 0x2aaabd310e70 (handle 2863311531, ret 0)) [D 20:01:02.627219] DBPF I/O ops in progress: 15 [D 20:01:02.627234] starting delayed I/O operation 0x2aaabd320e90 (15 in progress) [D 20:01:02.627250] lio_listio called with 1 following aiocbs: [D 20:01:02.627257] aiocb_ptr_array[0]: fd: 13, off: 232783872, bytes: 2097152, buf: 0x2aaaba048010, type: 1 [D 20:01:02.627267] start_delayed_ops_if_any: lio_listio posted 0x2aaabd320e90 (handle 2863311531, ret 0)) [D 20:01:02.627287] DBPF I/O ops in progress: 15 [D 20:01:02.627303] starting delayed I/O operation 0x2aaabd301ae0 (15 in progress) [D 20:01:02.627319] lio_listio called with 1 following aiocbs: [D 20:01:02.627325] aiocb_ptr_array[0]: fd: 13, off: 266338304, bytes: 2097152, buf: 0x2aaaba249010, type: 1 [D 20:01:02.627335] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301ae0 (handle 2863311531, ret 0)) [D 20:01:02.627349] op_queue add: 0x2aaabd301c30 [D 20:01:02.627359] delayed I/O operation 0x2aaabd301c30 (16 already in progress) [D 20:01:02.627366] DBPF I/O ops in progress: 16 [D 20:01:02.627383] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.627394] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.627406] ========================================================= [D 20:01:02.627413] PINT_process_request [D 20:01:02.627421] tiling 3 copies [D 20:01:02.627429] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.627437] lvl 0 el 0 blk 0 by 62914560 [D 20:01:02.627444] to 62914560 ta 33554432 fi 67108864 [D 20:01:02.627451] basic type or contiguous data [D 20:01:02.627457] Request disp [D 20:01:02.627464] contig_offset = 62914560 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.627471] calling distribute [D 20:01:02.627478] PINT_distribute [D 20:01:02.627518] of 62914560 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.627528] begin iteration loff: 62914560 [D 20:01:02.627841] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301eb0) [D 20:01:02.627860] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.627872] dbpf_open_cache_put: cached entry. [D 20:01:02.627879] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.627886] op_queue add: 0x2aaabd301eb0 [D 20:01:02.627896] DBPF I/O ops in progress: 15 [D 20:01:02.627904] starting delayed I/O operation 0x2aaabd302eb0 (15 in progress) [D 20:01:02.627914] lio_listio called with 1 following aiocbs: [D 20:01:02.627920] aiocb_ptr_array[0]: fd: 13, off: 299892736, bytes: 2097152, buf: 0x2aaaba44a010, type: 1 [D 20:01:02.627934] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302000) [D 20:01:02.627944] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.627954] dbpf_open_cache_put: cached entry. [D 20:01:02.627961] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.627968] op_queue add: 0x2aaabd302000 [D 20:01:02.627978] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302c10) [D 20:01:02.627987] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.627997] dbpf_open_cache_put: cached entry. [D 20:01:02.628003] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.628010] op_queue add: 0x2aaabd302c10 [D 20:01:02.628199] segment exceeds byte limit [D 20:01:02.628213] process a segment [D 20:01:02.628220] of 62914560 sz 2097152 [D 20:01:02.628227] add a segment 0 [D 20:01:02.628235] end iteration [D 20:01:02.628240] done with segments or bytes [D 20:01:02.628247] of 65011712 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.628255] next loff: 65011712 next poff: 65011712 [D 20:01:02.628263] return value 2097152 [D 20:01:02.628271] finished [D 20:01:02.628277] retval = 2097152 [D 20:01:02.628284] exiting distribute returned less than expected [D 20:01:02.628291] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 65011712 fo 67108864 eof 0 [D 20:01:02.628299] ========================================================= [D 20:01:02.628319] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.628328] dbpf_bstream_rw_list: mem_offset: 0x2aaabe609010, mem_size: 2097152 [D 20:01:02.628334] dbpf_bstream_rw_list: stream_offset: 62914560, stream_size: 2097152 [D 20:01:02.628342] dbpf_open_cache_get: called [D 20:01:02.628350] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.628357] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.628408] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:02.628423] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:02.628432] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:02.628443] BMI_post_send: addr: 6843072, offset: 0x2aaabbe58010, size: 2097152, tag: 36 [D 20:01:02.628465] post_sr: 10.1.4.56:39990 bh 46 len 32 wr_id 683140. [D 20:01:02.628520] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.632576] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3022a0) [D 20:01:02.632658] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.632673] dbpf_open_cache_put: cached entry. [D 20:01:02.632681] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.632689] op_queue add: 0x2aaabd3022a0 [D 20:01:02.632741] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302150) [D 20:01:02.632753] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.632764] dbpf_open_cache_put: cached entry. [D 20:01:02.632771] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.632777] op_queue add: 0x2aaabd302150 [D 20:01:02.636554] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3023f0) [D 20:01:02.636610] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.636625] dbpf_open_cache_put: cached entry. [D 20:01:02.636632] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.636639] op_queue add: 0x2aaabd3023f0 [D 20:01:02.636672] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302eb0 (handle 2863311531, ret 0)) [D 20:01:02.636721] DBPF I/O ops in progress: 15 [D 20:01:02.636736] starting delayed I/O operation 0x2aaab1530a60 (15 in progress) [D 20:01:02.636746] lio_listio called with 1 following aiocbs: [D 20:01:02.636753] aiocb_ptr_array[0]: fd: 13, off: 102760448, bytes: 2097152, buf: 0x2aaabfc14010, type: 0 [D 20:01:02.636763] start_delayed_ops_if_any: lio_listio posted 0x2aaab1530a60 (handle 2863311531, ret 0)) [D 20:01:02.636786] DBPF I/O ops in progress: 15 [D 20:01:02.636795] starting delayed I/O operation 0x2aaab152b470 (15 in progress) [D 20:01:02.636809] lio_listio called with 1 following aiocbs: [D 20:01:02.636816] aiocb_ptr_array[0]: fd: 13, off: 104857600, bytes: 2097152, buf: 0x2aaabfe15010, type: 0 [D 20:01:02.636826] start_delayed_ops_if_any: lio_listio posted 0x2aaab152b470 (handle 2863311531, ret 0)) [D 20:01:02.636840] op_queue add: 0x2aaabd303000 [D 20:01:02.636852] delayed I/O operation 0x2aaabd303000 (16 already in progress) [D 20:01:02.636859] DBPF I/O ops in progress: 16 [D 20:01:02.636885] openib_check_cq: completed recv wr_id 6843b0. [D 20:01:02.636899] openib_post_sr_rdmaw: ibv_post_send wr_id 668920 to 10.1.4.56:39990 remote addr a3451000 rkey 108c27 nic_wr_credit 2. [D 20:01:02.636910] openib_post_rr: 10.1.4.56:39990 bh 44 wr_id 6843b0. [D 20:01:02.636919] openib_check_cq: completed recv wr_id 6843e0. [D 20:01:02.636929] openib_post_sr_rdmaw: ibv_post_send wr_id 6ed990 to 10.1.4.56:39990 remote addr a124f000 rkey 108d27 nic_wr_credit 3. [D 20:01:02.636937] openib_post_rr: 10.1.4.56:39990 bh 45 wr_id 6843e0. [D 20:01:02.636944] openib_check_cq: completed recv wr_id 684410. [D 20:01:02.636955] openib_post_rr: 10.1.4.56:39990 bh 46 wr_id 684410. [D 20:01:02.636962] openib_check_cq: completed rdma wr_id 6eaf30, nic_wr_credit 2. [D 20:01:02.636974] post_sr: 10.1.4.56:39990 bh 47 len 16 wr_id 683170. [D 20:01:02.636983] openib_check_cq: completed send wr_id 683110. [D 20:01:02.636991] openib_check_cq: completed send wr_id 683140. [D 20:01:02.636999] openib_check_cq: completed recv wr_id 684440. [D 20:01:02.637010] openib_post_rr: 10.1.4.56:39990 bh 47 wr_id 684440. [D 20:01:02.637019] openib_check_cq: completed recv wr_id 684470. [D 20:01:02.637029] openib_post_sr_rdmaw: ibv_post_send wr_id 6ee270 to 10.1.4.56:39990 remote addr 9ee4d000 rkey 108e27 nic_wr_credit 3. [D 20:01:02.637037] openib_post_rr: 10.1.4.56:39990 bh 48 wr_id 684470. [D 20:01:02.637059] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.637072] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.637081] ========================================================= [D 20:01:02.637088] PINT_process_request [D 20:01:02.637095] tiling 4 copies [D 20:01:02.637102] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.637110] lvl 0 el 0 blk 0 by 92274688 [D 20:01:02.637112] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302540) [D 20:01:02.637126] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.637134] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.637139] BMI_post_send: addr: 6843072, offset: 0x2aaabde05010, size: 2097152, tag: 24 [D 20:01:02.637150] dbpf_open_cache_put: cached entry. [D 20:01:02.637159] post_sr: 10.1.4.56:39990 bh 48 len 32 wr_id 6831a0. [D 20:01:02.637204] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.637223] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:02.637232] BMI_post_send: addr: 6843072, offset: 0x2aaabce60010, size: 2097152, tag: 23 [D 20:01:02.637241] post_sr: 10.1.4.56:39990 bh 49 len 32 wr_id 6831d0. [D 20:01:02.637251] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.637118] to 92274688 ta 67108864 fi 100663296 [D 20:01:02.637268] basic type or contiguous data [D 20:01:02.637275] Request disp [D 20:01:02.637281] contig_offset = 92274688 contig_size = 41943040 lvl_flag = 1 [D 20:01:02.637289] calling distribute [D 20:01:02.637295] PINT_distribute [D 20:01:02.637302] of 92274688 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.637310] begin iteration loff: 92274688 [D 20:01:02.637318] segment exceeds byte limit [D 20:01:02.637324] process a segment [D 20:01:02.637330] of 92274688 sz 2097152 [D 20:01:02.637337] add a segment 0 [D 20:01:02.637344] end iteration [D 20:01:02.637122] op_queue add: 0x2aaab151fd00 [D 20:01:02.645523] delayed I/O operation 0x2aaab151fd00 (16 already in progress) [D 20:01:02.645556] DBPF I/O ops in progress: 16 [D 20:01:02.645567] flowproto posted 0x2aaab1522750 [D 20:01:02.645576] Job flows in progress (post time): 20 [D 20:01:02.645597] [SM Exiting]: (0x662c90) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:02.645686] [SM Entering]: (0x663f50) io_sm:start_flow (status: 0) [D 20:01:02.645694] [SM frame get]: (0x663f50) op-id: 3 index: 0 base-frm: 0 [D 20:01:02.645708] io_start_flow() issuing flow to read data. [D 20:01:02.645716] flow: fsize: 471859200, server_nr: 0, server_ct: 1 [D 20:01:02.645723] file_req_offset: 134217728, aggregate_size: 33554432, handle: 2863311531 [D 20:01:02.645734] PINT_new_request_states n=1 [D 20:01:02.645742] flowproto posting 0x2aaab151fe50 [D 20:01:02.645755] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.645762] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab151fe50. [D 20:01:02.645850] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:02.647884] done with segments or bytes [D 20:01:02.647980] of 94371840 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.647992] next loff: 94371840 next poff: 94371840 [D 20:01:02.648001] return value 2097152 [D 20:01:02.648008] finished [D 20:01:02.648015] retval = 2097152 [D 20:01:02.648023] exiting distribute returned less than expected [D 20:01:02.648029] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 94371840 fo 100663296 eof 0 [D 20:01:02.648037] ========================================================= [D 20:01:02.648060] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.648070] dbpf_bstream_rw_list: mem_offset: 0x2aaabf20f010, mem_size: 2097152 [D 20:01:02.648078] dbpf_bstream_rw_list: stream_offset: 92274688, stream_size: 2097152 [D 20:01:02.648087] dbpf_open_cache_get: called [D 20:01:02.648095] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.648102] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.648131] op_queue add: 0x2aaabd300a20 [D 20:01:02.648138] delayed I/O operation 0x2aaabd300a20 (16 already in progress) [D 20:01:02.648146] DBPF I/O ops in progress: 16 [D 20:01:02.648163] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:02.648175] get_data_sync_mode returning 0 [D 20:01:02.648184] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.648191] dbpf_bstream_rw_list: mem_offset: 0x2aaabb453010, mem_size: 2097152 [D 20:01:02.648198] dbpf_bstream_rw_list: stream_offset: 499122176, stream_size: 2097152 [D 20:01:02.648204] dbpf_open_cache_get: called [D 20:01:02.648211] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.648217] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.648239] op_queue add: 0x2aaabd300b70 [D 20:01:02.648247] delayed I/O operation 0x2aaabd300b70 (16 already in progress) [D 20:01:02.648254] DBPF I/O ops in progress: 16 [D 20:01:02.648263] ========================================================= [D 20:01:02.648270] PINT_process_request [D 20:01:02.648277] tiling 16 copies [D 20:01:02.648286] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.648294] lvl 0 el 0 blk 0 by 501219328 [D 20:01:02.648300] to 501219328 ta 469762048 fi 503316480 [D 20:01:02.648308] basic type or contiguous data [D 20:01:02.648315] Request disp [D 20:01:02.648322] contig_offset = 501219328 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.648330] calling distribute [D 20:01:02.648336] PINT_distribute [D 20:01:02.648342] of 501219328 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 501219328 exfl 1 [D 20:01:02.648352] begin iteration loff: 501219328 [D 20:01:02.648359] file being extended [D 20:01:02.648366] process a segment [D 20:01:02.648372] of 501219328 sz 2097152 [D 20:01:02.648379] add a segment 0 [D 20:01:02.648385] end iteration [D 20:01:02.648391] done with segments or bytes [D 20:01:02.648397] of 503316480 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.648404] next loff: 503316480 next poff: 503316480 [D 20:01:02.648410] (return value) 2097152 [D 20:01:02.648417] finished [D 20:01:02.648423] retval = 2097152 [D 20:01:02.648429] exiting distribute returned less than expected [D 20:01:02.648435] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 503316480 fo 503316480 eof 0 [D 20:01:02.648442] ========================================================= [D 20:01:02.648451] BMI_post_recv: addr: 6843072, offset: 0x2aaabc85d010, size: 2097152, tag: 22 [D 20:01:02.648470] openib_check_cq: completed rdma wr_id 668920, nic_wr_credit 2. [D 20:01:02.648489] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.648503] BMI_post_send: addr: 6843072, offset: 0x2aaabea0b010, size: 2097152, tag: 25 [D 20:01:02.648528] DBPF I/O ops in progress: 15 [D 20:01:02.648590] post_sr: 10.1.4.56:39990 bh 50 len 16 wr_id 683200. [D 20:01:02.648613] openib_check_cq: completed rdma wr_id 6ed990, nic_wr_credit 1. [D 20:01:02.648622] post_sr: 10.1.4.56:39990 bh 51 len 16 wr_id 683230. [D 20:01:02.648630] openib_check_cq: completed send wr_id 683170. [D 20:01:02.648638] openib_check_cq: completed recv wr_id 6844a0. [D 20:01:02.648650] openib_post_sr_rdmaw: ibv_post_send wr_id 6ee130 to 10.1.4.56:39990 remote addr 89837000 rkey 108a27 nic_wr_credit 2. [D 20:01:02.648659] openib_post_rr: 10.1.4.56:39990 bh 49 wr_id 6844a0. [D 20:01:02.648668] openib_check_cq: completed rdma wr_id 6ee270, nic_wr_credit 1. [D 20:01:02.648678] post_sr: 10.1.4.56:39990 bh 52 len 16 wr_id 683260. [D 20:01:02.648685] openib_check_cq: completed send wr_id 6831a0. [D 20:01:02.648693] openib_check_cq: completed send wr_id 6831d0. [D 20:01:02.648700] openib_check_cq: completed send wr_id 683200. [D 20:01:02.648708] openib_check_cq: completed send wr_id 683230. [D 20:01:02.648766] post_sr: 10.1.4.56:39990 bh 53 len 48 wr_id 683290. [D 20:01:02.648801] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.648813] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.648823] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.648832] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.648839] ========================================================= [D 20:01:02.648845] PINT_process_request [D 20:01:02.648852] tiling 3 copies [D 20:01:02.648858] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.648866] lvl 0 el 0 blk 0 by 65011712 [D 20:01:02.648872] to 65011712 ta 33554432 fi 67108864 [D 20:01:02.648880] basic type or contiguous data [D 20:01:02.648886] Request disp [D 20:01:02.648892] contig_offset = 65011712 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.648899] calling distribute [D 20:01:02.648905] PINT_distribute [D 20:01:02.648911] of 65011712 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.648919] begin iteration loff: 65011712 [D 20:01:02.648926] process a segment [D 20:01:02.648932] of 65011712 sz 2097152 [D 20:01:02.648938] add a segment 0 [D 20:01:02.648944] end iteration [D 20:01:02.648950] done with segments or bytes [D 20:01:02.648956] of 67108864 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.648963] next loff: 67108864 next poff: 67108864 [D 20:01:02.648969] (return value) 2097152 [D 20:01:02.648975] finished [D 20:01:02.648982] retval = 2097152 [D 20:01:02.648988] exiting distribute returned less than expected [D 20:01:02.648994] done sg 1 sm 64 by 2097152 bm 2097152 ta 33554432 to 67108864 fo 67108864 eof 0 [D 20:01:02.649001] ========================================================= [D 20:01:02.649015] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.649023] dbpf_bstream_rw_list: mem_offset: 0x2aaabe80a010, mem_size: 2097152 [D 20:01:02.649035] dbpf_bstream_rw_list: stream_offset: 65011712, stream_size: 2097152 [D 20:01:02.649044] dbpf_open_cache_get: called [D 20:01:02.649051] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.649059] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.649095] post_sr: 10.1.4.56:39990 bh 54 len 32 wr_id 6832c0. [D 20:01:02.649118] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.648596] ========================================================= [D 20:01:02.649151] PINT_process_request [D 20:01:02.649158] tiling 6 copies [D 20:01:02.649164] skipping ahead to target_offset [D 20:01:02.649170] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.649177] lvl 0 el 0 blk 0 by 0 [D 20:01:02.649184] to 0 ta 134217728 fi 167772160 [D 20:01:02.649190] basic type or contiguous data [D 20:01:02.649197] Request disp [D 20:01:02.649202] contig_offset = 0 contig_size = 201326592 lvl_flag = 1 [D 20:01:02.649209] process logical skip [D 20:01:02.649215] retval = 134217728 [D 20:01:02.649221] exiting logical skip because distribute indicates done [D 20:01:02.649227] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.649235] lvl 0 el 0 blk 0 by 134217728 [D 20:01:02.649241] to 134217728 ta 134217728 fi 167772160 [D 20:01:02.649247] basic type or contiguous data [D 20:01:02.649253] Request disp [D 20:01:02.649259] contig_offset = 134217728 contig_size = 67108864 lvl_flag = 1 [D 20:01:02.649266] calling distribute [D 20:01:02.649272] PINT_distribute [D 20:01:02.649279] of 134217728 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.649287] begin iteration loff: 134217728 [D 20:01:02.649293] segment exceeds byte limit [D 20:01:02.649299] process a segment [D 20:01:02.649305] of 134217728 sz 2097152 [D 20:01:02.649311] add a segment 0 [D 20:01:02.649317] end iteration [D 20:01:02.649323] done with segments or bytes [D 20:01:02.649329] of 136314880 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.649336] next loff: 136314880 next poff: 136314880 [D 20:01:02.649343] return value 2097152 [D 20:01:02.649349] finished [D 20:01:02.649355] retval = 2097152 [D 20:01:02.649361] exiting distribute returned less than expected [D 20:01:02.649367] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 136314880 fo 167772160 eof 0 [D 20:01:02.649374] ========================================================= [D 20:01:02.649386] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.649394] dbpf_bstream_rw_list: mem_offset: 0x2aaac0a1b010, mem_size: 2097152 [D 20:01:02.649400] dbpf_bstream_rw_list: stream_offset: 134217728, stream_size: 2097152 [D 20:01:02.649407] dbpf_open_cache_get: called [D 20:01:02.649414] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.649420] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.648563] starting delayed I/O operation 0x2aaab1525840 (15 in progress) [D 20:01:02.649447] lio_listio called with 1 following aiocbs: [D 20:01:02.649455] aiocb_ptr_array[0]: fd: 13, off: 106954752, bytes: 2097152, buf: 0x2aaac0016010, type: 0 [D 20:01:02.649469] start_delayed_ops_if_any: lio_listio posted 0x2aaab1525840 (handle 2863311531, ret 0)) [D 20:01:02.649518] DBPF I/O ops in progress: 15 [D 20:01:02.649531] starting delayed I/O operation 0x2aaabd310fc0 (15 in progress) [D 20:01:02.649542] lio_listio called with 1 following aiocbs: [D 20:01:02.649548] aiocb_ptr_array[0]: fd: 13, off: 333447168, bytes: 2097152, buf: 0x2aaaba64b010, type: 1 [D 20:01:02.649558] start_delayed_ops_if_any: lio_listio posted 0x2aaabd310fc0 (handle 2863311531, ret 0)) [D 20:01:02.649578] DBPF I/O ops in progress: 15 [D 20:01:02.649590] starting delayed I/O operation 0x2aaabd321710 (15 in progress) [D 20:01:02.649599] lio_listio called with 1 following aiocbs: [D 20:01:02.649606] aiocb_ptr_array[0]: fd: 13, off: 367001600, bytes: 2097152, buf: 0x2aaaba84c010, type: 1 [D 20:01:02.649616] start_delayed_ops_if_any: lio_listio posted 0x2aaabd321710 (handle 2863311531, ret 0)) [D 20:01:02.649627] op_queue add: 0x2aaabd301170 [D 20:01:02.649634] delayed I/O operation 0x2aaabd301170 (16 already in progress) [D 20:01:02.649641] DBPF I/O ops in progress: 16 [D 20:01:02.649652] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.649661] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.649891] openib_check_cq: completed recv wr_id 6844d0. [D 20:01:02.649901] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea480 to 10.1.4.56:39990 remote addr a3651000 rkey 108c27 nic_wr_credit 2. [D 20:01:02.649911] openib_post_rr: 10.1.4.56:39990 bh 50 wr_id 6844d0. [D 20:01:02.650893] openib_check_cq: completed rdma wr_id 6ee130, nic_wr_credit 1. [D 20:01:02.650908] post_sr: 10.1.4.56:39990 bh 55 len 16 wr_id 6832f0. [D 20:01:02.650920] openib_check_cq: completed send wr_id 683260. [D 20:01:02.650928] openib_check_cq: completed send wr_id 683290. [D 20:01:02.650935] openib_check_cq: completed send wr_id 6832c0. [D 20:01:02.650945] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.650954] (send callback time) ini posts: 0, pending: 3, last: 0, src_list emtpy: no [D 20:01:02.650964] ========================================================= [D 20:01:02.650972] PINT_process_request [D 20:01:02.650979] tiling 4 copies [D 20:01:02.650987] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.650995] lvl 0 el 0 blk 0 by 94371840 [D 20:01:02.651001] to 94371840 ta 67108864 fi 100663296 [D 20:01:02.651010] basic type or contiguous data [D 20:01:02.651016] Request disp [D 20:01:02.651024] contig_offset = 94371840 contig_size = 39845888 lvl_flag = 1 [D 20:01:02.651031] calling distribute [D 20:01:02.651037] PINT_distribute [D 20:01:02.651044] of 94371840 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.651053] begin iteration loff: 94371840 [D 20:01:02.651060] segment exceeds byte limit [D 20:01:02.651066] process a segment [D 20:01:02.651072] of 94371840 sz 2097152 [D 20:01:02.651079] add a segment 0 [D 20:01:02.651085] end iteration [D 20:01:02.651091] done with segments or bytes [D 20:01:02.651097] of 96468992 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.651104] next loff: 96468992 next poff: 96468992 [D 20:01:02.651111] return value 2097152 [D 20:01:02.651117] finished [D 20:01:02.651123] retval = 2097152 [D 20:01:02.651129] exiting distribute returned less than expected [D 20:01:02.651135] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 96468992 fo 100663296 eof 0 [D 20:01:02.651144] ========================================================= [D 20:01:02.651159] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.651168] dbpf_bstream_rw_list: mem_offset: 0x2aaabf410010, mem_size: 2097152 [D 20:01:02.651175] dbpf_bstream_rw_list: stream_offset: 94371840, stream_size: 2097152 [D 20:01:02.651183] dbpf_open_cache_get: called [D 20:01:02.651191] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.651198] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.651225] op_queue add: 0x2aaabd3012c0 [D 20:01:02.651238] delayed I/O operation 0x2aaabd3012c0 (16 already in progress) [D 20:01:02.651245] DBPF I/O ops in progress: 16 [D 20:01:02.653533] openib_check_cq: completed rdma wr_id 6ea480, nic_wr_credit 0. [D 20:01:02.653567] post_sr: 10.1.4.56:39990 bh 56 len 16 wr_id 683320. [D 20:01:02.653578] openib_check_cq: completed send wr_id 6832f0. [D 20:01:02.653587] openib_check_cq: completed send wr_id 683320. [D 20:01:02.653598] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.653608] (send callback time) ini posts: 0, pending: 2, last: 0, src_list emtpy: no [D 20:01:02.653621] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.653629] (send callback time) ini posts: 0, pending: 2, last: 0, src_list emtpy: no [D 20:01:02.653811] openib_check_cq: completed recv wr_id 684500. [D 20:01:02.653820] openib_post_sr_rdmaw: ibv_post_send wr_id 6ecf80 to 10.1.4.56:39990 remote addr a144f000 rkey 108d27 nic_wr_credit 1. [D 20:01:02.653829] openib_post_rr: 10.1.4.56:39990 bh 51 wr_id 684500. [D 20:01:02.653837] openib_check_cq: completed recv wr_id 684530. [D 20:01:02.653844] openib_post_sr_rdmaw: ibv_post_send wr_id 6ee3b0 to 10.1.4.56:39990 remote addr 9f04d000 rkey 108e27 nic_wr_credit 2. [D 20:01:02.653852] openib_post_rr: 10.1.4.56:39990 bh 52 wr_id 684530. [D 20:01:02.637160] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.654140] op_queue add: 0x2aaabd302540 [D 20:01:02.654153] DBPF I/O ops in progress: 15 [D 20:01:02.654161] starting delayed I/O operation 0x2aaabd3008d0 (15 in progress) [D 20:01:02.654171] lio_listio called with 1 following aiocbs: [D 20:01:02.654178] aiocb_ptr_array[0]: fd: 13, off: 400556032, bytes: 2097152, buf: 0x2aaabaa4d010, type: 1 [D 20:01:02.654195] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302d60) [D 20:01:02.654207] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.654218] dbpf_open_cache_put: cached entry. [D 20:01:02.654225] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.654231] op_queue add: 0x2aaabd302d60 [D 20:01:02.654244] --- aio_progress_notification called with handle 2863311531 (0x2aaabd32c490) [D 20:01:02.654253] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.654264] dbpf_open_cache_put: cached entry. [D 20:01:02.654270] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.654276] op_queue add: 0x2aaabd32c490 [D 20:01:02.654287] --- aio_progress_notification called with handle 2863311531 (0x2aaab1522b60) [D 20:01:02.654296] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.654306] dbpf_open_cache_put: cached entry. [D 20:01:02.654313] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.654319] op_queue add: 0x2aaab1522b60 [D 20:01:02.654330] --- aio_progress_notification called with handle 2863311531 (0x2aaabd32c5e0) [D 20:01:02.654339] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.654349] dbpf_open_cache_put: cached entry. [D 20:01:02.654356] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.654362] op_queue add: 0x2aaabd32c5e0 [D 20:01:02.654806] openib_check_cq: completed recv wr_id 684560. [D 20:01:02.654822] openib_post_rr: 10.1.4.56:39990 bh 53 wr_id 684560. [D 20:01:02.654836] flowproto-multiqueue bmi_recv_callback_fn, error code: 0, flow: 0x2aaab1505370. [D 20:01:02.654847] get_data_sync_mode returning 0 [D 20:01:02.654861] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.654869] dbpf_bstream_rw_list: mem_offset: 0x2aaabc85d010, mem_size: 2097152 [D 20:01:02.654877] dbpf_bstream_rw_list: stream_offset: 501219328, stream_size: 2097152 [D 20:01:02.654885] dbpf_open_cache_get: called [D 20:01:02.654892] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.654901] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.654958] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:02.654973] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aadd0. [D 20:01:02.654980] flowproto completing 0x6aadd0 [D 20:01:02.655028] Job flows in progress (callback time): 19 [D 20:01:02.655036] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6aecb0. [D 20:01:02.655043] flowproto completing 0x6aecb0 [D 20:01:02.655059] Job flows in progress (callback time): 18 [D 20:01:02.655068] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:02.655078] BMI_post_send: addr: 6843072, offset: 0x2aaabfa13010, size: 2097152, tag: 26 [D 20:01:02.655108] post_sr: 10.1.4.56:39990 bh 57 len 32 wr_id 683350. [D 20:01:02.655149] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.655160] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b2df0. [D 20:01:02.655168] flowproto completing 0x6b2df0 [D 20:01:02.655185] Job flows in progress (callback time): 17 [D 20:01:02.656569] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3008d0 (handle 2863311531, ret 0)) [D 20:01:02.660576] --- aio_progress_notification called with handle 2863311531 (0x2aaabd305f70) [D 20:01:02.660637] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.660650] dbpf_open_cache_put: cached entry. [D 20:01:02.660658] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.660665] op_queue add: 0x2aaabd305f70 [D 20:01:02.660692] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6b7070. [D 20:01:02.660704] flowproto completing 0x6b7070 [D 20:01:02.660734] Job flows in progress (callback time): 16 [D 20:01:02.660749] DBPF I/O ops in progress: 15 [D 20:01:02.660757] starting delayed I/O operation 0x2aaabd303290 (15 in progress) [D 20:01:02.660768] lio_listio called with 1 following aiocbs: [D 20:01:02.660775] aiocb_ptr_array[0]: fd: 13, off: 492830720, bytes: 2097152, buf: 0x2aaab9844010, type: 1 [D 20:01:02.660789] start_delayed_ops_if_any: lio_listio posted 0x2aaabd303290 (handle 2863311531, ret 0)) [D 20:01:02.660827] DBPF I/O ops in progress: 15 [D 20:01:02.660837] starting delayed I/O operation 0x2aaab152b210 (15 in progress) [D 20:01:02.660848] lio_listio called with 1 following aiocbs: [D 20:01:02.660855] aiocb_ptr_array[0]: fd: 13, off: 109051904, bytes: 2097152, buf: 0x2aaac0217010, type: 0 [D 20:01:02.660864] start_delayed_ops_if_any: lio_listio posted 0x2aaab152b210 (handle 2863311531, ret 0)) [D 20:01:02.660887] DBPF I/O ops in progress: 15 [D 20:01:02.660897] starting delayed I/O operation 0x2aaabd3033e0 (15 in progress) [D 20:01:02.660907] lio_listio called with 1 following aiocbs: [D 20:01:02.660913] aiocb_ptr_array[0]: fd: 13, off: 434110464, bytes: 2097152, buf: 0x2aaabac4f010, type: 1 [D 20:01:02.660923] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3033e0 (handle 2863311531, ret 0)) [D 20:01:02.660977] DBPF I/O ops in progress: 15 [D 20:01:02.660990] starting delayed I/O operation 0x2aaabd300ed0 (15 in progress) [D 20:01:02.661001] lio_listio called with 1 following aiocbs: [D 20:01:02.661007] aiocb_ptr_array[0]: fd: 13, off: 463470592, bytes: 2097152, buf: 0x2aaabc059010, type: 0 [D 20:01:02.661018] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300ed0 (handle 2863311531, ret 0)) [D 20:01:02.661055] DBPF I/O ops in progress: 15 [D 20:01:02.661066] starting delayed I/O operation 0x2aaabd301020 (15 in progress) [D 20:01:02.661076] lio_listio called with 1 following aiocbs: [D 20:01:02.661082] aiocb_ptr_array[0]: fd: 13, off: 25165824, bytes: 2097152, buf: 0x2aaabd061010, type: 0 [D 20:01:02.661092] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301020 (handle 2863311531, ret 0)) [D 20:01:02.661106] op_queue add: 0x2aaabd301920 [D 20:01:02.661117] delayed I/O operation 0x2aaabd301920 (16 already in progress) [D 20:01:02.661123] DBPF I/O ops in progress: 16 [D 20:01:02.661147] openib_check_cq: completed recv wr_id 684590. [D 20:01:02.661162] openib_post_sr_rdmaw: ibv_post_send wr_id 6edc10 to 10.1.4.56:39990 remote addr 89a37000 rkey 108a27 nic_wr_credit 3. [D 20:01:02.661174] openib_post_rr: 10.1.4.56:39990 bh 54 wr_id 684590. [D 20:01:02.661183] openib_check_cq: completed rdma wr_id 6ecf80, nic_wr_credit 2. [D 20:01:02.661193] post_sr: 10.1.4.56:39990 bh 58 len 16 wr_id 683380. [D 20:01:02.661200] openib_check_cq: completed recv wr_id 6845c0. [D 20:01:02.661208] openib_post_sr_rdmaw: ibv_post_send wr_id 6d2c90 to 10.1.4.56:39990 remote addr a3851000 rkey 108c27 nic_wr_credit 3. [D 20:01:02.661219] op_queue add: 0x2aaab1520020 [D 20:01:02.735583] delayed I/O operation 0x2aaab1520020 (16 already in progress) [D 20:01:02.735603] DBPF I/O ops in progress: 16 [D 20:01:02.735614] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.735621] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab151fe50. [D 20:01:02.735630] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.735641] ========================================================= [D 20:01:02.735648] PINT_process_request [D 20:01:02.735660] tiling 6 copies [D 20:01:02.735668] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.735675] lvl 0 el 0 blk 0 by 136314880 [D 20:01:02.735682] to 136314880 ta 134217728 fi 167772160 [D 20:01:02.735689] basic type or contiguous data [D 20:01:02.735695] Request disp [D 20:01:02.735701] contig_offset = 136314880 contig_size = 65011712 lvl_flag = 1 [D 20:01:02.735708] calling distribute [D 20:01:02.735717] PINT_distribute [D 20:01:02.735724] of 136314880 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.735732] begin iteration loff: 136314880 [D 20:01:02.735738] segment exceeds byte limit [D 20:01:02.735745] process a segment [D 20:01:02.735750] of 136314880 sz 2097152 [D 20:01:02.735757] add a segment 0 [D 20:01:02.735763] end iteration [D 20:01:02.735768] done with segments or bytes [D 20:01:02.735774] of 138412032 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.735781] next loff: 138412032 next poff: 138412032 [D 20:01:02.735787] return value 2097152 [D 20:01:02.735794] finished [D 20:01:02.735799] retval = 2097152 [D 20:01:02.735805] exiting distribute returned less than expected [D 20:01:02.735811] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 138412032 fo 167772160 eof 0 [D 20:01:02.735818] ========================================================= [D 20:01:02.735829] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.735836] dbpf_bstream_rw_list: mem_offset: 0x2aaab9a45010, mem_size: 2097152 [D 20:01:02.735842] dbpf_bstream_rw_list: stream_offset: 136314880, stream_size: 2097152 [D 20:01:02.735849] dbpf_open_cache_get: called [D 20:01:02.735856] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.735862] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.735877] op_queue add: 0x2aaab1522b60 [D 20:01:02.735884] delayed I/O operation 0x2aaab1522b60 (16 already in progress) [D 20:01:02.735890] DBPF I/O ops in progress: 16 [D 20:01:02.735896] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.735902] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab151fe50. [D 20:01:02.735910] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.735917] ========================================================= [D 20:01:02.735923] PINT_process_request [D 20:01:02.735929] tiling 6 copies [D 20:01:02.735935] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.735942] lvl 0 el 0 blk 0 by 138412032 [D 20:01:02.735948] to 138412032 ta 134217728 fi 167772160 [D 20:01:02.735954] basic type or contiguous data [D 20:01:02.735960] Request disp [D 20:01:02.735966] contig_offset = 138412032 contig_size = 62914560 lvl_flag = 1 [D 20:01:02.735973] calling distribute [D 20:01:02.735978] PINT_distribute [D 20:01:02.735984] of 138412032 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.735991] begin iteration loff: 138412032 [D 20:01:02.735997] segment exceeds byte limit [D 20:01:02.736003] process a segment [D 20:01:02.736008] of 138412032 sz 2097152 [D 20:01:02.736014] add a segment 0 [D 20:01:02.736020] end iteration [D 20:01:02.736026] done with segments or bytes [D 20:01:02.736032] of 140509184 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.736039] next loff: 140509184 next poff: 140509184 [D 20:01:02.736045] return value 2097152 [D 20:01:02.736051] finished [D 20:01:02.736057] retval = 2097152 [D 20:01:02.736063] exiting distribute returned less than expected [D 20:01:02.736069] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 140509184 fo 167772160 eof 0 [D 20:01:02.736076] ========================================================= [D 20:01:02.736084] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.736090] dbpf_bstream_rw_list: mem_offset: 0x2aaab7a35010, mem_size: 2097152 [D 20:01:02.736097] dbpf_bstream_rw_list: stream_offset: 138412032, stream_size: 2097152 [D 20:01:02.736103] dbpf_open_cache_get: called [D 20:01:02.736109] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.736116] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.736129] op_queue add: 0x2aaab152bf70 [D 20:01:02.736135] delayed I/O operation 0x2aaab152bf70 (16 already in progress) [D 20:01:02.736142] DBPF I/O ops in progress: 16 [D 20:01:02.736148] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.736154] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab151fe50. [D 20:01:02.736161] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.736169] ========================================================= [D 20:01:02.736175] PINT_process_request [D 20:01:02.736180] tiling 6 copies [D 20:01:02.736186] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.736193] lvl 0 el 0 blk 0 by 140509184 [D 20:01:02.736200] to 140509184 ta 134217728 fi 167772160 [D 20:01:02.736206] basic type or contiguous data [D 20:01:02.736212] Request disp [D 20:01:02.736218] contig_offset = 140509184 contig_size = 60817408 lvl_flag = 1 [D 20:01:02.736224] calling distribute [D 20:01:02.736230] PINT_distribute [D 20:01:02.736236] of 140509184 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.736243] begin iteration loff: 140509184 [D 20:01:02.736249] segment exceeds byte limit [D 20:01:02.736255] process a segment [D 20:01:02.736261] of 140509184 sz 2097152 [D 20:01:02.736267] add a segment 0 [D 20:01:02.736272] end iteration [D 20:01:02.736278] done with segments or bytes [D 20:01:02.736284] of 142606336 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.736291] next loff: 142606336 next poff: 142606336 [D 20:01:02.736297] return value 2097152 [D 20:01:02.736303] finished [D 20:01:02.736308] retval = 2097152 [D 20:01:02.736314] exiting distribute returned less than expected [D 20:01:02.736320] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 142606336 fo 167772160 eof 0 [D 20:01:02.736328] ========================================================= [D 20:01:02.736334] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.736341] dbpf_bstream_rw_list: mem_offset: 0x2aaab5e26010, mem_size: 2097152 [D 20:01:02.736348] dbpf_bstream_rw_list: stream_offset: 140509184, stream_size: 2097152 [D 20:01:02.736354] dbpf_open_cache_get: called [D 20:01:02.736360] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.736366] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.736387] op_queue add: 0x2aaab152c0c0 [D 20:01:02.736396] delayed I/O operation 0x2aaab152c0c0 (16 already in progress) [D 20:01:02.736402] DBPF I/O ops in progress: 16 [D 20:01:02.736408] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.736415] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab151fe50. [D 20:01:02.736422] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.736429] ========================================================= [D 20:01:02.736436] PINT_process_request [D 20:01:02.736442] tiling 6 copies [D 20:01:02.736448] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.736455] lvl 0 el 0 blk 0 by 142606336 [D 20:01:02.736462] to 142606336 ta 134217728 fi 167772160 [D 20:01:02.736468] basic type or contiguous data [D 20:01:02.736474] Request disp [D 20:01:02.736480] contig_offset = 142606336 contig_size = 58720256 lvl_flag = 1 [D 20:01:02.736487] calling distribute [D 20:01:02.736493] PINT_distribute [D 20:01:02.736498] of 142606336 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.736506] begin iteration loff: 142606336 [D 20:01:02.736512] segment exceeds byte limit [D 20:01:02.736518] process a segment [D 20:01:02.736523] of 142606336 sz 2097152 [D 20:01:02.736529] add a segment 0 [D 20:01:02.736538] end iteration [D 20:01:02.664568] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300d80) [D 20:01:02.668573] --- aio_progress_notification called with handle 2863311531 (0x2aaabd310e70) [D 20:01:02.736574] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.736580] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.736587] dbpf_open_cache_put: cached entry. [D 20:01:02.736598] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.669797] --- aio_progress_notification called with handle 2863311531 (0x2aaabd320e90) [D 20:01:02.736600] dbpf_open_cache_put: cached entry. [D 20:01:02.676577] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301ae0) [D 20:01:02.736616] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.680567] --- aio_progress_notification called with handle 2863311531 (0x2aaab1530a60) [D 20:01:02.736627] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.736632] dbpf_open_cache_put: cached entry. [D 20:01:02.680623] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302eb0) [D 20:01:02.736618] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.683756] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b470) [D 20:01:02.736669] dbpf_open_cache_put: cached entry. [D 20:01:02.736607] op_queue add: 0x2aaabd300d80 [D 20:01:02.683804] --- aio_progress_notification called with handle 2863311531 (0x2aaab1525840) [D 20:01:02.736679] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.688578] --- aio_progress_notification called with handle 2863311531 (0x2aaabd310fc0) [D 20:01:02.736698] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.696573] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3008d0) [D 20:01:02.736695] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bb330. [D 20:01:02.736687] DBPF I/O ops in progress: 15 [D 20:01:02.736735] flowproto completing 0x6bb330 [D 20:01:02.736656] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.698249] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303290) [D 20:01:02.736760] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.707760] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300ed0) [D 20:01:02.736764] Job flows in progress (callback time): 15 [D 20:01:02.736545] openib_post_rr: 10.1.4.56:39990 bh 55 wr_id 6845c0. [D 20:01:02.736545] done with segments or bytes [D 20:01:02.736715] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.692629] --- aio_progress_notification called with handle 2863311531 (0x2aaabd321710) [D 20:01:02.736788] openib_check_cq: completed rdma wr_id 6ee3b0, nic_wr_credit 2. [D 20:01:02.736804] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.736813] post_sr: 10.1.4.56:39990 bh 59 len 16 wr_id 6833b0. [D 20:01:02.736718] dbpf_open_cache_put: cached entry. [D 20:01:02.736782] of 144703488 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.707806] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301020) [D 20:01:02.736702] op_queue add: 0x2aaabd301ae0 [D 20:01:02.736828] openib_check_cq: completed send wr_id 683350. [D 20:01:02.736855] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.736863] op_queue add: 0x2aaabd310e70 [D 20:01:02.701601] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b210) [D 20:01:02.704572] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3033e0) [D 20:01:02.736879] dbpf_open_cache_put: cached entry. [D 20:01:02.736887] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.736894] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.736852] next loff: 144703488 next poff: 144703488 [D 20:01:02.736905] dbpf_open_cache_put: cached entry. [D 20:01:02.736868] openib_check_cq: completed recv wr_id 6845f0. [D 20:01:02.736924] return value 2097152 [D 20:01:02.736943] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec5d0 to 10.1.4.56:39990 remote addr 9c44b000 rkey 108f27 nic_wr_credit 3. [D 20:01:02.736919] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.736948] finished [D 20:01:02.736959] dbpf_open_cache_put: cached entry. [D 20:01:02.736975] op_queue add: 0x2aaabd320e90 [D 20:01:02.736984] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.736846] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.737013] dbpf_open_cache_put: cached entry. [D 20:01:02.736740] starting delayed I/O operation 0x2aaabd320fe0 (15 in progress) [D 20:01:02.736995] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c7f30. [D 20:01:02.736979] retval = 2097152 [D 20:01:02.736929] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.736890] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.736881] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.737072] dbpf_open_cache_put: cached entry. [D 20:01:02.737083] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.736911] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.737025] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.737106] dbpf_open_cache_put: cached entry. [D 20:01:02.737115] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.736958] openib_post_rr: 10.1.4.56:39990 bh 56 wr_id 6845f0. [D 20:01:02.737044] flowproto completing 0x6c7f30 [D 20:01:02.736672] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.737148] Job flows in progress (callback time): 14 [D 20:01:02.737055] exiting distribute returned less than expected [D 20:01:02.737160] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6bf6b0. [D 20:01:02.737169] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 144703488 fo 167772160 eof 0 [D 20:01:02.737188] ========================================================= [D 20:01:02.737196] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.737203] dbpf_bstream_rw_list: mem_offset: 0x2aaab3e15010, mem_size: 2097152 [D 20:01:02.737209] dbpf_bstream_rw_list: stream_offset: 142606336, stream_size: 2097152 [D 20:01:02.737216] dbpf_open_cache_get: called [D 20:01:02.737134] openib_check_cq: completed rdma wr_id 6edc10, nic_wr_credit 2. [D 20:01:02.737002] op_queue add: 0x2aaabd310fc0 [D 20:01:02.737235] post_sr: 10.1.4.56:39990 bh 60 len 16 wr_id 6833e0. [D 20:01:02.737179] flowproto completing 0x6bf6b0 [D 20:01:02.737152] dbpf_open_cache_put: cached entry. [D 20:01:02.737262] Job flows in progress (callback time): 13 [D 20:01:02.736727] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.737286] dbpf_open_cache_put: cached entry. [D 20:01:02.737294] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.737300] op_queue add: 0x2aaabd3008d0 [D 20:01:02.737309] op_queue add: 0x2aaab1525840 [D 20:01:02.737248] openib_check_cq: completed send wr_id 683380. [D 20:01:02.737273] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6c0290. [D 20:01:02.737331] openib_check_cq: completed recv wr_id 684620. [D 20:01:02.737340] flowproto completing 0x6c0290 [D 20:01:02.737353] openib_post_sr_rdmaw: ibv_post_send wr_id 6676a0 to 10.1.4.56:39990 remote addr a164f000 rkey 108d27 nic_wr_credit 3. [D 20:01:02.737364] Job flows in progress (callback time): 12 [D 20:01:02.737036] lio_listio called with 1 following aiocbs: [D 20:01:02.737383] aiocb_ptr_array[0]: fd: 13, off: 56623104, bytes: 2097152, buf: 0x2aaabe006010, type: 0 [D 20:01:02.737396] start_delayed_ops_if_any: lio_listio posted 0x2aaabd320fe0 (handle 2863311531, ret 0)) [D 20:01:02.737408] DBPF I/O ops in progress: 15 [D 20:01:02.737419] starting delayed I/O operation 0x2aaab1517400 (15 in progress) [D 20:01:02.737365] openib_post_rr: 10.1.4.56:39990 bh 57 wr_id 684620. [D 20:01:02.737319] op_queue add: 0x2aaabd3033e0 [D 20:01:02.737368] dbpf_open_cache_put: cached entry. [D 20:01:02.737430] lio_listio called with 1 following aiocbs: [D 20:01:02.737446] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.737475] dbpf_open_cache_put: cached entry. [D 20:01:02.737478] (send callback time) ini posts: 0, pending: 2, last: 0, src_list emtpy: no [D 20:01:02.737490] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.737508] op_queue add: 0x2aaab152b210 [D 20:01:02.737469] aiocb_ptr_array[0]: fd: 13, off: 111149056, bytes: 2097152, buf: 0x2aaac0418010, type: 0 [D 20:01:02.737526] start_delayed_ops_if_any: lio_listio posted 0x2aaab1517400 (handle 2863311531, ret 0)) [D 20:01:02.737542] DBPF I/O ops in progress: 15 [D 20:01:02.737550] starting delayed I/O operation 0x2aaabd321130 (15 in progress) [D 20:01:02.737559] lio_listio called with 1 following aiocbs: [D 20:01:02.737565] aiocb_ptr_array[0]: fd: 13, off: 85983232, bytes: 2097152, buf: 0x2aaabec0c010, type: 0 [D 20:01:02.737574] start_delayed_ops_if_any: lio_listio posted 0x2aaabd321130 (handle 2863311531, ret 0)) [D 20:01:02.738775] openib_check_cq: completed rdma wr_id 6d2c90, nic_wr_credit 2. [D 20:01:02.738785] post_sr: 10.1.4.56:39990 bh 61 len 16 wr_id 683410. [D 20:01:02.738793] openib_check_cq: completed send wr_id 6833b0. [D 20:01:02.738802] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.738810] (send callback time) ini posts: 0, pending: 2, last: 0, src_list emtpy: no [D 20:01:02.738817] ========================================================= [D 20:01:02.738823] PINT_process_request [D 20:01:02.738830] tiling 4 copies [D 20:01:02.738836] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.738843] lvl 0 el 0 blk 0 by 96468992 [D 20:01:02.738850] to 96468992 ta 67108864 fi 100663296 [D 20:01:02.738856] basic type or contiguous data [D 20:01:02.738862] Request disp [D 20:01:02.738868] contig_offset = 96468992 contig_size = 37748736 lvl_flag = 1 [D 20:01:02.738875] calling distribute [D 20:01:02.738880] PINT_distribute [D 20:01:02.738886] of 96468992 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.738894] begin iteration loff: 96468992 [D 20:01:02.738900] segment exceeds byte limit [D 20:01:02.738906] process a segment [D 20:01:02.738911] of 96468992 sz 2097152 [D 20:01:02.738917] add a segment 0 [D 20:01:02.738923] end iteration [D 20:01:02.738929] done with segments or bytes [D 20:01:02.738935] of 98566144 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.738942] next loff: 98566144 next poff: 98566144 [D 20:01:02.738949] return value 2097152 [D 20:01:02.738955] finished [D 20:01:02.738960] retval = 2097152 [D 20:01:02.738966] exiting distribute returned less than expected [D 20:01:02.738972] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 98566144 fo 100663296 eof 0 [D 20:01:02.738979] ========================================================= [D 20:01:02.738988] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.738995] dbpf_bstream_rw_list: mem_offset: 0x2aaabf611010, mem_size: 2097152 [D 20:01:02.739001] dbpf_bstream_rw_list: stream_offset: 96468992, stream_size: 2097152 [D 20:01:02.739008] dbpf_open_cache_get: called [D 20:01:02.739014] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.739020] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.739034] op_queue add: 0x2aaabd310e50 [D 20:01:02.739041] delayed I/O operation 0x2aaabd310e50 (16 already in progress) [D 20:01:02.739047] DBPF I/O ops in progress: 16 [D 20:01:02.740002] openib_check_cq: completed recv wr_id 684650. [D 20:01:02.740010] openib_post_sr_rdmaw: ibv_post_send wr_id 6ed220 to 10.1.4.56:39990 remote addr 9f24d000 rkey 108e27 nic_wr_credit 3. [D 20:01:02.740018] openib_post_rr: 10.1.4.56:39990 bh 58 wr_id 684650. [D 20:01:02.741001] openib_check_cq: completed rdma wr_id 6ec5d0, nic_wr_credit 2. [D 20:01:02.741031] post_sr: 10.1.4.56:39990 bh 62 len 16 wr_id 683440. [D 20:01:02.741039] openib_check_cq: completed send wr_id 6833e0. [D 20:01:02.741048] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.741056] (send callback time) ini posts: 0, pending: 1, last: 0, src_list emtpy: no [D 20:01:02.741436] DBPF I/O ops in progress: 15 [D 20:01:02.741449] starting delayed I/O operation 0x2aaabd301530 (15 in progress) [D 20:01:02.741459] lio_listio called with 1 following aiocbs: [D 20:01:02.741465] aiocb_ptr_array[0]: fd: 13, off: 467664896, bytes: 2097152, buf: 0x2aaabae50010, type: 1 [D 20:01:02.741473] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301530 (handle 2863311531, ret 0)) [D 20:01:02.741489] DBPF I/O ops in progress: 15 [D 20:01:02.741497] starting delayed I/O operation 0x2aaabd301680 (15 in progress) [D 20:01:02.741506] lio_listio called with 1 following aiocbs: [D 20:01:02.741513] aiocb_ptr_array[0]: fd: 13, off: 465567744, bytes: 2097152, buf: 0x2aaabc25a010, type: 0 [D 20:01:02.741521] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301680 (handle 2863311531, ret 0)) [D 20:01:02.741552] DBPF I/O ops in progress: 15 [D 20:01:02.741561] starting delayed I/O operation 0x2aaabd310cf0 (15 in progress) [D 20:01:02.741571] lio_listio called with 1 following aiocbs: [D 20:01:02.741577] aiocb_ptr_array[0]: fd: 13, off: 27262976, bytes: 2097152, buf: 0x2aaabd400010, type: 0 [D 20:01:02.741586] start_delayed_ops_if_any: lio_listio posted 0x2aaabd310cf0 (handle 2863311531, ret 0)) [D 20:01:02.741611] DBPF I/O ops in progress: 15 [D 20:01:02.741619] starting delayed I/O operation 0x2aaabd3017d0 (15 in progress) [D 20:01:02.741628] lio_listio called with 1 following aiocbs: [D 20:01:02.741635] aiocb_ptr_array[0]: fd: 13, off: 58720256, bytes: 2097152, buf: 0x2aaabe207010, type: 0 [D 20:01:02.741643] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3017d0 (handle 2863311531, ret 0)) [D 20:01:02.741666] DBPF I/O ops in progress: 15 [D 20:01:02.741674] starting delayed I/O operation 0x2aaabd32f140 (15 in progress) [D 20:01:02.741683] lio_listio called with 1 following aiocbs: [D 20:01:02.741690] aiocb_ptr_array[0]: fd: 13, off: 88080384, bytes: 2097152, buf: 0x2aaabee0d010, type: 0 [D 20:01:02.741698] start_delayed_ops_if_any: lio_listio posted 0x2aaabd32f140 (handle 2863311531, ret 0)) [D 20:01:02.741720] DBPF I/O ops in progress: 15 [D 20:01:02.741729] starting delayed I/O operation 0x2aaabd32f290 (15 in progress) [D 20:01:02.741738] lio_listio called with 1 following aiocbs: [D 20:01:02.741744] aiocb_ptr_array[0]: fd: 13, off: 494927872, bytes: 2097152, buf: 0x2aaabb051010, type: 1 [D 20:01:02.741753] start_delayed_ops_if_any: lio_listio posted 0x2aaabd32f290 (handle 2863311531, ret 0)) [D 20:01:02.737463] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.741784] op_queue add: 0x2aaabd303290 [D 20:01:02.741792] DBPF I/O ops in progress: 15 [D 20:01:02.741799] starting delayed I/O operation 0x2aaabd32f3e0 (15 in progress) [D 20:01:02.741807] lio_listio called with 1 following aiocbs: [D 20:01:02.741814] aiocb_ptr_array[0]: fd: 13, off: 467664896, bytes: 2097152, buf: 0x2aaabc45b010, type: 0 [D 20:01:02.741822] start_delayed_ops_if_any: lio_listio posted 0x2aaabd32f3e0 (handle 2863311531, ret 0)) [D 20:01:02.741847] dbpf_open_cache_put: cached entry. [D 20:01:02.741858] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.741866] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.741887] op_queue add: 0x2aaab152c210 [D 20:01:02.741895] delayed I/O operation 0x2aaab152c210 (16 already in progress) [D 20:01:02.741902] DBPF I/O ops in progress: 16 [D 20:01:02.741908] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.741915] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab151fe50. [D 20:01:02.741922] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.741930] ========================================================= [D 20:01:02.741936] PINT_process_request [D 20:01:02.741942] tiling 6 copies [D 20:01:02.741948] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.741955] lvl 0 el 0 blk 0 by 144703488 [D 20:01:02.741961] to 144703488 ta 134217728 fi 167772160 [D 20:01:02.741968] basic type or contiguous data [D 20:01:02.741974] Request disp [D 20:01:02.741980] contig_offset = 144703488 contig_size = 56623104 lvl_flag = 1 [D 20:01:02.741986] calling distribute [D 20:01:02.741992] PINT_distribute [D 20:01:02.741998] of 144703488 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.742005] begin iteration loff: 144703488 [D 20:01:02.742011] segment exceeds byte limit [D 20:01:02.742017] process a segment [D 20:01:02.742023] of 144703488 sz 2097152 [D 20:01:02.742029] add a segment 0 [D 20:01:02.742035] end iteration [D 20:01:02.742041] done with segments or bytes [D 20:01:02.742046] of 146800640 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.742053] next loff: 146800640 next poff: 146800640 [D 20:01:02.742059] return value 2097152 [D 20:01:02.742065] finished [D 20:01:02.742071] retval = 2097152 [D 20:01:02.742077] exiting distribute returned less than expected [D 20:01:02.742083] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 146800640 fo 167772160 eof 0 [D 20:01:02.742090] ========================================================= [D 20:01:02.742097] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.742104] dbpf_bstream_rw_list: mem_offset: 0x2aaaba048010, mem_size: 2097152 [D 20:01:02.742110] dbpf_bstream_rw_list: stream_offset: 144703488, stream_size: 2097152 [D 20:01:02.742117] dbpf_open_cache_get: called [D 20:01:02.742123] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.742129] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.742148] op_queue add: 0x2aaab152c360 [D 20:01:02.742156] delayed I/O operation 0x2aaab152c360 (16 already in progress) [D 20:01:02.742163] DBPF I/O ops in progress: 16 [D 20:01:02.742169] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.742175] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab151fe50. [D 20:01:02.742182] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.742190] ========================================================= [D 20:01:02.742196] PINT_process_request [D 20:01:02.742202] tiling 6 copies [D 20:01:02.742208] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.742215] lvl 0 el 0 blk 0 by 146800640 [D 20:01:02.742221] openib_check_cq: completed recv wr_id 684680. [D 20:01:02.742227] op_queue add: 0x2aaabd301020 [D 20:01:02.742246] DBPF I/O ops in progress: 15 [D 20:01:02.742253] starting delayed I/O operation 0x2aaabd32f530 (15 in progress) [D 20:01:02.742265] lio_listio called with 1 following aiocbs: [D 20:01:02.742271] aiocb_ptr_array[0]: fd: 13, off: 29360128, bytes: 2097152, buf: 0x2aaabd601010, type: 0 [D 20:01:02.742280] start_delayed_ops_if_any: lio_listio posted 0x2aaabd32f530 (handle 2863311531, ret 0)) [D 20:01:02.742222] to 146800640 ta 134217728 fi 167772160 [D 20:01:02.742321] basic type or contiguous data [D 20:01:02.742327] Request disp [D 20:01:02.742333] contig_offset = 146800640 contig_size = 54525952 lvl_flag = 1 [D 20:01:02.742340] calling distribute [D 20:01:02.742346] PINT_distribute [D 20:01:02.742352] of 146800640 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.742359] begin iteration loff: 146800640 [D 20:01:02.742365] segment exceeds byte limit [D 20:01:02.742371] process a segment [D 20:01:02.742377] of 146800640 sz 2097152 [D 20:01:02.742383] add a segment 0 [D 20:01:02.742389] end iteration [D 20:01:02.742395] done with segments or bytes [D 20:01:02.742401] of 148897792 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.742408] next loff: 148897792 next poff: 148897792 [D 20:01:02.742414] return value 2097152 [D 20:01:02.742420] finished [D 20:01:02.742426] retval = 2097152 [D 20:01:02.742432] exiting distribute returned less than expected [D 20:01:02.742438] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 148897792 fo 167772160 eof 0 [D 20:01:02.742445] ========================================================= [D 20:01:02.742453] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.742460] dbpf_bstream_rw_list: mem_offset: 0x2aaab8038010, mem_size: 2097152 [D 20:01:02.742466] dbpf_bstream_rw_list: stream_offset: 146800640, stream_size: 2097152 [D 20:01:02.742473] dbpf_open_cache_get: called [D 20:01:02.742479] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.742486] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.742510] op_queue add: 0x2aaab152c4b0 [D 20:01:02.742519] delayed I/O operation 0x2aaab152c4b0 (16 already in progress) [D 20:01:02.742526] DBPF I/O ops in progress: 16 [D 20:01:02.742532] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.742539] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab151fe50. [D 20:01:02.742546] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.742555] ========================================================= [D 20:01:02.742561] PINT_process_request [D 20:01:02.742567] tiling 6 copies [D 20:01:02.742573] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.742580] lvl 0 el 0 blk 0 by 148897792 [D 20:01:02.742587] to 148897792 ta 134217728 fi 167772160 [D 20:01:02.742593] basic type or contiguous data [D 20:01:02.742599] Request disp [D 20:01:02.742605] contig_offset = 148897792 contig_size = 52428800 lvl_flag = 1 [D 20:01:02.742611] calling distribute [D 20:01:02.742617] PINT_distribute [D 20:01:02.742623] of 148897792 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.742630] begin iteration loff: 148897792 [D 20:01:02.742637] segment exceeds byte limit [D 20:01:02.742643] process a segment [D 20:01:02.742648] of 148897792 sz 2097152 [D 20:01:02.742655] add a segment 0 [D 20:01:02.742660] end iteration [D 20:01:02.742666] done with segments or bytes [D 20:01:02.742672] of 150994944 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.742679] next loff: 150994944 next poff: 150994944 [D 20:01:02.742685] return value 2097152 [D 20:01:02.742691] finished [D 20:01:02.742698] retval = 2097152 [D 20:01:02.742703] exiting distribute returned less than expected [D 20:01:02.742710] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 150994944 fo 167772160 eof 0 [D 20:01:02.742718] ========================================================= [D 20:01:02.742725] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.742732] dbpf_bstream_rw_list: mem_offset: 0x2aaab642a010, mem_size: 2097152 [D 20:01:02.742739] dbpf_bstream_rw_list: stream_offset: 148897792, stream_size: 2097152 [D 20:01:02.742746] dbpf_open_cache_get: called [D 20:01:02.742752] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.742759] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.742789] op_queue add: 0x2aaab152c600 [D 20:01:02.742799] delayed I/O operation 0x2aaab152c600 (16 already in progress) [D 20:01:02.742806] DBPF I/O ops in progress: 16 [D 20:01:02.742813] flowproto posted 0x2aaab151fe50 [D 20:01:02.742822] Job flows in progress (post time): 13 [D 20:01:02.742836] [SM Exiting]: (0x663f50) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:02.742849] [SM Entering]: (0x665210) io_sm:start_flow (status: 0) [D 20:01:02.742856] [SM frame get]: (0x665210) op-id: 3 index: 0 base-frm: 0 [D 20:01:02.742867] io_start_flow() issuing flow to read data. [D 20:01:02.742874] flow: fsize: 471859200, server_nr: 0, server_ct: 1 [D 20:01:02.742882] file_req_offset: 167772160, aggregate_size: 33554432, handle: 2863311531 [D 20:01:02.742895] PINT_new_request_states n=1 [D 20:01:02.742905] flowproto posting 0x2aaab152c750 [D 20:01:02.742929] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.742938] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c750. [D 20:01:02.742946] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:02.742954] ========================================================= [D 20:01:02.742961] PINT_process_request [D 20:01:02.742967] tiling 7 copies [D 20:01:02.742973] skipping ahead to target_offset [D 20:01:02.742979] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.742987] lvl 0 el 0 blk 0 by 0 [D 20:01:02.742993] to 0 ta 167772160 fi 201326592 [D 20:01:02.743000] basic type or contiguous data [D 20:01:02.743006] Request disp [D 20:01:02.743012] contig_offset = 0 contig_size = 234881024 lvl_flag = 1 [D 20:01:02.743019] process logical skip [D 20:01:02.743025] retval = 167772160 [D 20:01:02.743031] exiting logical skip because distribute indicates done [D 20:01:02.743037] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.743044] lvl 0 el 0 blk 0 by 167772160 [D 20:01:02.743051] to 167772160 ta 167772160 fi 201326592 [D 20:01:02.743058] basic type or contiguous data [D 20:01:02.743064] Request disp [D 20:01:02.743070] contig_offset = 167772160 contig_size = 67108864 lvl_flag = 1 [D 20:01:02.743076] calling distribute [D 20:01:02.743082] PINT_distribute [D 20:01:02.743088] of 167772160 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.743096] begin iteration loff: 167772160 [D 20:01:02.743102] segment exceeds byte limit [D 20:01:02.743109] process a segment [D 20:01:02.743115] of 167772160 sz 2097152 [D 20:01:02.743121] add a segment 0 [D 20:01:02.743127] end iteration [D 20:01:02.743134] done with segments or bytes [D 20:01:02.743140] of 169869312 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.743147] next loff: 169869312 next poff: 169869312 [D 20:01:02.743153] return value 2097152 [D 20:01:02.743159] finished [D 20:01:02.743165] retval = 2097152 [D 20:01:02.743171] exiting distribute returned less than expected [D 20:01:02.743177] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 169869312 fo 201326592 eof 0 [D 20:01:02.743185] ========================================================= [D 20:01:02.743193] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.743200] dbpf_bstream_rw_list: mem_offset: 0x2aaab4418010, mem_size: 2097152 [D 20:01:02.743208] dbpf_bstream_rw_list: stream_offset: 167772160, stream_size: 2097152 [D 20:01:02.743215] dbpf_open_cache_get: called [D 20:01:02.743221] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.743227] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.743259] op_queue add: 0x2aaab152cbb0 [D 20:01:02.743269] delayed I/O operation 0x2aaab152cbb0 (16 already in progress) [D 20:01:02.743276] DBPF I/O ops in progress: 16 [D 20:01:02.743283] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.743289] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c750. [D 20:01:02.743297] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.743305] ========================================================= [D 20:01:02.743311] PINT_process_request [D 20:01:02.743317] tiling 7 copies [D 20:01:02.743323] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.743331] lvl 0 el 0 blk 0 by 169869312 [D 20:01:02.743337] to 169869312 ta 167772160 fi 201326592 [D 20:01:02.743344] basic type or contiguous data [D 20:01:02.743350] Request disp [D 20:01:02.743356] contig_offset = 169869312 contig_size = 65011712 lvl_flag = 1 [D 20:01:02.743363] calling distribute [D 20:01:02.743369] PINT_distribute [D 20:01:02.743375] of 169869312 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.743383] begin iteration loff: 169869312 [D 20:01:02.743389] segment exceeds byte limit [D 20:01:02.743395] process a segment [D 20:01:02.743401] of 169869312 sz 2097152 [D 20:01:02.743407] add a segment 0 [D 20:01:02.743413] end iteration [D 20:01:02.743419] done with segments or bytes [D 20:01:02.743425] of 171966464 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.743433] next loff: 171966464 next poff: 171966464 [D 20:01:02.743439] return value 2097152 [D 20:01:02.743446] finished [D 20:01:02.743452] retval = 2097152 [D 20:01:02.743458] exiting distribute returned less than expected [D 20:01:02.743464] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 171966464 fo 201326592 eof 0 [D 20:01:02.743471] ========================================================= [D 20:01:02.743480] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.743487] dbpf_bstream_rw_list: mem_offset: 0x2aaab2809010, mem_size: 2097152 [D 20:01:02.743493] dbpf_bstream_rw_list: stream_offset: 169869312, stream_size: 2097152 [D 20:01:02.743500] dbpf_open_cache_get: called [D 20:01:02.743507] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.743514] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.743545] op_queue add: 0x2aaab152cd00 [D 20:01:02.743554] delayed I/O operation 0x2aaab152cd00 (16 already in progress) [D 20:01:02.743561] DBPF I/O ops in progress: 16 [D 20:01:02.743567] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.743574] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c750. [D 20:01:02.743581] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.743589] ========================================================= [D 20:01:02.743596] PINT_process_request [D 20:01:02.743602] tiling 7 copies [D 20:01:02.743608] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.743616] lvl 0 el 0 blk 0 by 171966464 [D 20:01:02.743622] to 171966464 ta 167772160 fi 201326592 [D 20:01:02.743629] basic type or contiguous data [D 20:01:02.743635] Request disp [D 20:01:02.743640] contig_offset = 171966464 contig_size = 62914560 lvl_flag = 1 [D 20:01:02.743647] calling distribute [D 20:01:02.743653] PINT_distribute [D 20:01:02.743659] of 171966464 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.743667] begin iteration loff: 171966464 [D 20:01:02.743674] segment exceeds byte limit [D 20:01:02.743680] process a segment [D 20:01:02.743688] of 171966464 sz 2097152 [D 20:01:02.743694] add a segment 0 [D 20:01:02.743700] end iteration [D 20:01:02.743706] done with segments or bytes [D 20:01:02.743725] of 174063616 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.743733] next loff: 174063616 next poff: 174063616 [D 20:01:02.743740] return value 2097152 [D 20:01:02.743746] finished [D 20:01:02.743752] retval = 2097152 [D 20:01:02.743758] exiting distribute returned less than expected [D 20:01:02.743764] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 174063616 fo 201326592 eof 0 [D 20:01:02.743772] ========================================================= [D 20:01:02.743780] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.743788] dbpf_bstream_rw_list: mem_offset: 0x2aaab0add010, mem_size: 2097152 [D 20:01:02.743794] dbpf_bstream_rw_list: stream_offset: 171966464, stream_size: 2097152 [D 20:01:02.743801] dbpf_open_cache_get: called [D 20:01:02.743808] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.743814] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.743843] op_queue add: 0x2aaab152ce50 [D 20:01:02.743855] delayed I/O operation 0x2aaab152ce50 (16 already in progress) [D 20:01:02.743862] DBPF I/O ops in progress: 16 [D 20:01:02.743869] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.743876] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c750. [D 20:01:02.743884] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.743892] ========================================================= [D 20:01:02.743898] PINT_process_request [D 20:01:02.743904] tiling 7 copies [D 20:01:02.743910] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.743918] lvl 0 el 0 blk 0 by 174063616 [D 20:01:02.743925] to 174063616 ta 167772160 fi 201326592 [D 20:01:02.743931] basic type or contiguous data [D 20:01:02.743937] Request disp [D 20:01:02.743943] contig_offset = 174063616 contig_size = 60817408 lvl_flag = 1 [D 20:01:02.743950] calling distribute [D 20:01:02.743956] PINT_distribute [D 20:01:02.743962] of 174063616 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.743970] begin iteration loff: 174063616 [D 20:01:02.743977] segment exceeds byte limit [D 20:01:02.743983] process a segment [D 20:01:02.743989] of 174063616 sz 2097152 [D 20:01:02.743995] add a segment 0 [D 20:01:02.744001] end iteration [D 20:01:02.744007] done with segments or bytes [D 20:01:02.744013] of 176160768 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.744021] next loff: 176160768 next poff: 176160768 [D 20:01:02.744027] return value 2097152 [D 20:01:02.744033] finished [D 20:01:02.744039] retval = 2097152 [D 20:01:02.744045] exiting distribute returned less than expected [D 20:01:02.744051] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 176160768 fo 201326592 eof 0 [D 20:01:02.744059] ========================================================= [D 20:01:02.744067] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.744073] dbpf_bstream_rw_list: mem_offset: 0x2aaaaeecf010, mem_size: 2097152 [D 20:01:02.744080] dbpf_bstream_rw_list: stream_offset: 174063616, stream_size: 2097152 [D 20:01:02.744087] dbpf_open_cache_get: called [D 20:01:02.744093] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.744100] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.744127] op_queue add: 0x2aaab152cfa0 [D 20:01:02.744136] delayed I/O operation 0x2aaab152cfa0 (16 already in progress) [D 20:01:02.744143] DBPF I/O ops in progress: 16 [D 20:01:02.744150] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.744156] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c750. [D 20:01:02.744164] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.744171] ========================================================= [D 20:01:02.744179] PINT_process_request [D 20:01:02.744185] tiling 7 copies [D 20:01:02.744191] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.744198] lvl 0 el 0 blk 0 by 176160768 [D 20:01:02.744205] to 176160768 ta 167772160 fi 201326592 [D 20:01:02.744212] basic type or contiguous data [D 20:01:02.744218] Request disp [D 20:01:02.744224] contig_offset = 176160768 contig_size = 58720256 lvl_flag = 1 [D 20:01:02.744231] calling distribute [D 20:01:02.744237] PINT_distribute [D 20:01:02.744243] of 176160768 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.744250] begin iteration loff: 176160768 [D 20:01:02.744257] segment exceeds byte limit [D 20:01:02.744263] process a segment [D 20:01:02.744269] of 176160768 sz 2097152 [D 20:01:02.744275] add a segment 0 [D 20:01:02.744281] end iteration [D 20:01:02.744287] done with segments or bytes [D 20:01:02.744293] of 178257920 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.744300] next loff: 178257920 next poff: 178257920 [D 20:01:02.744306] return value 2097152 [D 20:01:02.744312] finished [D 20:01:02.744318] retval = 2097152 [D 20:01:02.744323] exiting distribute returned less than expected [D 20:01:02.744330] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 178257920 fo 201326592 eof 0 [D 20:01:02.744337] ========================================================= [D 20:01:02.744344] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.744351] dbpf_bstream_rw_list: mem_offset: 0x2aaaad2c1010, mem_size: 2097152 [D 20:01:02.744359] dbpf_bstream_rw_list: stream_offset: 176160768, stream_size: 2097152 [D 20:01:02.744365] dbpf_open_cache_get: called [D 20:01:02.744372] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.744378] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.744404] op_queue add: 0x2aaab152d0f0 [D 20:01:02.744415] delayed I/O operation 0x2aaab152d0f0 (16 already in progress) [D 20:01:02.744422] DBPF I/O ops in progress: 16 [D 20:01:02.744428] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.744435] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c750. [D 20:01:02.744442] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.744450] ========================================================= [D 20:01:02.744456] PINT_process_request [D 20:01:02.744462] tiling 7 copies [D 20:01:02.744468] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.744475] lvl 0 el 0 blk 0 by 178257920 [D 20:01:02.744482] to 178257920 ta 167772160 fi 201326592 [D 20:01:02.744488] basic type or contiguous data [D 20:01:02.744494] Request disp [D 20:01:02.744500] contig_offset = 178257920 contig_size = 56623104 lvl_flag = 1 [D 20:01:02.737265] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.742234] openib_post_sr_rdmaw: ibv_post_send wr_id 66af30 to 10.1.4.56:39990 remote addr 89c37000 rkey 108a27 nic_wr_credit 3. [D 20:01:02.744563] openib_post_rr: 10.1.4.56:39990 bh 59 wr_id 684680. [D 20:01:02.744582] openib_check_cq: completed rdma wr_id 6676a0, nic_wr_credit 2. [D 20:01:02.744595] post_sr: 10.1.4.56:39990 bh 63 len 16 wr_id 683470. [D 20:01:02.744603] openib_check_cq: completed send wr_id 683410. [D 20:01:02.744612] openib_check_cq: completed recv wr_id 6846b0. [D 20:01:02.744623] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec210 to 10.1.4.56:39990 remote addr a3a51000 rkey 108c27 nic_wr_credit 3. [D 20:01:02.744632] openib_post_rr: 10.1.4.56:39990 bh 60 wr_id 6846b0. [D 20:01:02.744654] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.744664] (send callback time) ini posts: 0, pending: 1, last: 0, src_list emtpy: no [D 20:01:02.745454] openib_check_cq: completed rdma wr_id 6ed220, nic_wr_credit 2. [D 20:01:02.745478] post_sr: 10.1.4.56:39990 bh 64 len 16 wr_id 6834a0. [D 20:01:02.745489] openib_check_cq: completed send wr_id 683440. [D 20:01:02.745504] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:02.745514] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.745522] ========================================================= [D 20:01:02.745529] PINT_process_request [D 20:01:02.745535] tiling 5 copies [D 20:01:02.745543] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.745551] lvl 0 el 0 blk 0 by 117440512 [D 20:01:02.745558] to 117440512 ta 100663296 fi 134217728 [D 20:01:02.745565] basic type or contiguous data [D 20:01:02.745572] Request disp [D 20:01:02.745578] contig_offset = 117440512 contig_size = 50331648 lvl_flag = 1 [D 20:01:02.745585] calling distribute [D 20:01:02.745592] PINT_distribute [D 20:01:02.745598] of 117440512 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.745608] begin iteration loff: 117440512 [D 20:01:02.745615] segment exceeds byte limit [D 20:01:02.745621] process a segment [D 20:01:02.745627] of 117440512 sz 2097152 [D 20:01:02.745633] add a segment 0 [D 20:01:02.745639] end iteration [D 20:01:02.745646] done with segments or bytes [D 20:01:02.745652] of 119537664 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.745659] next loff: 119537664 next poff: 119537664 [D 20:01:02.745666] return value 2097152 [D 20:01:02.745672] finished [D 20:01:02.745678] retval = 2097152 [D 20:01:02.745685] exiting distribute returned less than expected [D 20:01:02.745692] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 119537664 fo 134217728 eof 0 [D 20:01:02.745699] ========================================================= [D 20:01:02.745716] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.745725] dbpf_bstream_rw_list: mem_offset: 0x2aaabfa13010, mem_size: 2097152 [D 20:01:02.745733] dbpf_bstream_rw_list: stream_offset: 117440512, stream_size: 2097152 [D 20:01:02.745740] dbpf_open_cache_get: called [D 20:01:02.745749] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.745756] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.745785] op_queue add: 0x2aaabd301a70 [D 20:01:02.745799] delayed I/O operation 0x2aaabd301a70 (16 already in progress) [D 20:01:02.745806] DBPF I/O ops in progress: 16 [D 20:01:02.747683] openib_check_cq: completed rdma wr_id 66af30, nic_wr_credit 1. [D 20:01:02.747764] post_sr: 10.1.4.56:39990 bh 65 len 16 wr_id 6834d0. [D 20:01:02.747784] openib_check_cq: completed send wr_id 683470. [D 20:01:02.747810] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.747826] (send callback time) ini posts: 0, pending: 1, last: 0, src_list emtpy: no [D 20:01:02.748882] openib_check_cq: completed recv wr_id 6846e0. [D 20:01:02.748936] openib_post_sr_rdmaw: ibv_post_send wr_id 66aea0 to 10.1.4.56:39990 remote addr a184f000 rkey 108d27 nic_wr_credit 2. [D 20:01:02.748948] openib_post_rr: 10.1.4.56:39990 bh 61 wr_id 6846e0. [D 20:01:02.749901] openib_check_cq: completed rdma wr_id 6ec210, nic_wr_credit 1. [D 20:01:02.749919] post_sr: 10.1.4.56:39990 bh 66 len 16 wr_id 683500. [D 20:01:02.749928] openib_check_cq: completed send wr_id 6834a0. [D 20:01:02.749937] openib_check_cq: completed send wr_id 6834d0. [D 20:01:02.749951] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.749963] (send callback time) ini posts: 0, pending: 1, last: 0, src_list emtpy: no [D 20:01:02.749972] ========================================================= [D 20:01:02.749979] PINT_process_request [D 20:01:02.749988] tiling 4 copies [D 20:01:02.749997] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.750004] lvl 0 el 0 blk 0 by 98566144 [D 20:01:02.750011] to 98566144 ta 67108864 fi 100663296 [D 20:01:02.750018] basic type or contiguous data [D 20:01:02.750026] Request disp [D 20:01:02.750033] contig_offset = 98566144 contig_size = 35651584 lvl_flag = 1 [D 20:01:02.750040] calling distribute [D 20:01:02.750047] PINT_distribute [D 20:01:02.750052] of 98566144 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 440401920 exfl 0 [D 20:01:02.750062] begin iteration loff: 98566144 [D 20:01:02.750070] process a segment [D 20:01:02.750076] of 98566144 sz 2097152 [D 20:01:02.750082] add a segment 0 [D 20:01:02.750088] end iteration [D 20:01:02.750095] done with segments or bytes [D 20:01:02.750101] of 100663296 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.750109] next loff: 100663296 next poff: 100663296 [D 20:01:02.750116] (return value) 2097152 [D 20:01:02.750122] finished [D 20:01:02.750128] retval = 2097152 [D 20:01:02.750135] exiting distribute returned less than expected [D 20:01:02.750141] done sg 1 sm 64 by 2097152 bm 2097152 ta 67108864 to 100663296 fo 100663296 eof 0 [D 20:01:02.750149] ========================================================= [D 20:01:02.750166] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.750175] dbpf_bstream_rw_list: mem_offset: 0x2aaabf812010, mem_size: 2097152 [D 20:01:02.750182] dbpf_bstream_rw_list: stream_offset: 98566144, stream_size: 2097152 [D 20:01:02.750190] dbpf_open_cache_get: called [D 20:01:02.750198] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.750205] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.750233] op_queue add: 0x2aaabd300cc0 [D 20:01:02.750244] delayed I/O operation 0x2aaabd300cc0 (16 already in progress) [D 20:01:02.750251] DBPF I/O ops in progress: 16 [D 20:01:02.750262] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.750269] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.751159] openib_check_cq: completed recv wr_id 684710. [D 20:01:02.751174] openib_post_sr_rdmaw: ibv_post_send wr_id 668920 to 10.1.4.56:39990 remote addr 9f44d000 rkey 108e27 nic_wr_credit 2. [D 20:01:02.751186] openib_post_rr: 10.1.4.56:39990 bh 62 wr_id 684710. [D 20:01:02.752129] openib_check_cq: completed rdma wr_id 66aea0, nic_wr_credit 1. [D 20:01:02.752152] post_sr: 10.1.4.56:39990 bh 67 len 16 wr_id 683530. [D 20:01:02.752162] openib_check_cq: completed send wr_id 683500. [D 20:01:02.752175] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.752185] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.754363] openib_check_cq: completed rdma wr_id 668920, nic_wr_credit 0. [D 20:01:02.754429] post_sr: 10.1.4.56:39990 bh 68 len 16 wr_id 683560. [D 20:01:02.754443] openib_check_cq: completed send wr_id 683530. [D 20:01:02.754451] openib_check_cq: completed send wr_id 683560. [D 20:01:02.754466] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.754478] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.754487] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.754496] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.744531] op_queue add: 0x2aaab152b470 [D 20:01:02.756612] DBPF I/O ops in progress: 15 [D 20:01:02.756622] starting delayed I/O operation 0x2aaabd32f680 (15 in progress) [D 20:01:02.756631] lio_listio called with 1 following aiocbs: [D 20:01:02.756638] aiocb_ptr_array[0]: fd: 13, off: 60817408, bytes: 2097152, buf: 0x2aaabe408010, type: 0 [D 20:01:02.756650] start_delayed_ops_if_any: lio_listio posted 0x2aaabd32f680 (handle 2863311531, ret 0)) [D 20:01:02.756684] --- aio_progress_notification called with handle 2863311531 (0x2aaab1517400) [D 20:01:02.756697] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.756708] dbpf_open_cache_put: cached entry. [D 20:01:02.756715] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.756722] op_queue add: 0x2aaab1517400 [D 20:01:02.756730] DBPF I/O ops in progress: 15 [D 20:01:02.756736] starting delayed I/O operation 0x2aaabd38e2e0 (15 in progress) [D 20:01:02.756745] lio_listio called with 1 following aiocbs: [D 20:01:02.756752] aiocb_ptr_array[0]: fd: 13, off: 90177536, bytes: 2097152, buf: 0x2aaabf00e010, type: 0 [D 20:01:02.756761] start_delayed_ops_if_any: lio_listio posted 0x2aaabd38e2e0 (handle 2863311531, ret 0)) [D 20:01:02.756778] --- aio_progress_notification called with handle 2863311531 (0x2aaabd321130) [D 20:01:02.756788] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.756798] dbpf_open_cache_put: cached entry. [D 20:01:02.756804] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.756810] op_queue add: 0x2aaabd321130 [D 20:01:02.756818] DBPF I/O ops in progress: 15 [D 20:01:02.756825] starting delayed I/O operation 0x2aaab152bb90 (15 in progress) [D 20:01:02.756833] lio_listio called with 1 following aiocbs: [D 20:01:02.756840] aiocb_ptr_array[0]: fd: 13, off: 113246208, bytes: 2097152, buf: 0x2aaac0619010, type: 0 [D 20:01:02.756848] start_delayed_ops_if_any: lio_listio posted 0x2aaab152bb90 (handle 2863311531, ret 0)) [D 20:01:02.756864] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301530) [D 20:01:02.756874] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.756884] dbpf_open_cache_put: cached entry. [D 20:01:02.756890] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.756896] op_queue add: 0x2aaabd301530 [D 20:01:02.756904] DBPF I/O ops in progress: 15 [D 20:01:02.756910] starting delayed I/O operation 0x2aaabd390e40 (15 in progress) [D 20:01:02.756919] lio_listio called with 1 following aiocbs: [D 20:01:02.756926] aiocb_ptr_array[0]: fd: 13, off: 497025024, bytes: 2097152, buf: 0x2aaabb252010, type: 1 [D 20:01:02.756934] start_delayed_ops_if_any: lio_listio posted 0x2aaabd390e40 (handle 2863311531, ret 0)) [D 20:01:02.756952] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301680) [D 20:01:02.756968] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.756981] dbpf_open_cache_put: cached entry. [D 20:01:02.756988] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.756994] op_queue add: 0x2aaabd301680 [D 20:01:02.757000] DBPF I/O ops in progress: 15 [D 20:01:02.757007] starting delayed I/O operation 0x2aaabd301c30 (15 in progress) [D 20:01:02.757016] lio_listio called with 1 following aiocbs: [D 20:01:02.757022] aiocb_ptr_array[0]: fd: 13, off: 31457280, bytes: 2097152, buf: 0x2aaabd802010, type: 0 [D 20:01:02.757031] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301c30 (handle 2863311531, ret 0)) [D 20:01:02.757061] op_queue add: 0x2aaabd321710 [D 20:01:02.757074] DBPF I/O ops in progress: 15 [D 20:01:02.757081] starting delayed I/O operation 0x2aaabd303000 (15 in progress) [D 20:01:02.757089] lio_listio called with 1 following aiocbs: [D 20:01:02.757095] aiocb_ptr_array[0]: fd: 13, off: 62914560, bytes: 2097152, buf: 0x2aaabe609010, type: 0 [D 20:01:02.757104] start_delayed_ops_if_any: lio_listio posted 0x2aaabd303000 (handle 2863311531, ret 0)) [D 20:01:02.757127] op_queue add: 0x2aaabd302eb0 [D 20:01:02.757140] DBPF I/O ops in progress: 15 [D 20:01:02.757147] starting delayed I/O operation 0x2aaab151fd00 (15 in progress) [D 20:01:02.757156] lio_listio called with 1 following aiocbs: [D 20:01:02.757162] aiocb_ptr_array[0]: fd: 13, off: 115343360, bytes: 2097152, buf: 0x2aaac081a010, type: 0 [D 20:01:02.757170] start_delayed_ops_if_any: lio_listio posted 0x2aaab151fd00 (handle 2863311531, ret 0)) [D 20:01:02.759282] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d01a0. [D 20:01:02.759350] flowproto completing 0x6d01a0 [D 20:01:02.759381] Job flows in progress (callback time): 12 [D 20:01:02.759392] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d86a0. [D 20:01:02.759400] flowproto completing 0x6d86a0 [D 20:01:02.759414] Job flows in progress (callback time): 11 [D 20:01:02.759423] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:02.759434] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6dc960. [D 20:01:02.759441] flowproto completing 0x6dc960 [D 20:01:02.759456] Job flows in progress (callback time): 10 [D 20:01:02.759463] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:02.759481] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:02.759491] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:02.759499] BMI_post_send: addr: 6843072, offset: 0x2aaabd061010, size: 2097152, tag: 23 [D 20:01:02.759516] post_sr: 10.1.4.56:39990 bh 69 len 32 wr_id 683590. [D 20:01:02.759558] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.759567] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:02.759575] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:02.759583] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.759590] BMI_post_send: addr: 6843072, offset: 0x2aaabec0c010, size: 2097152, tag: 25 [D 20:01:02.759599] post_sr: 10.1.4.56:39990 bh 70 len 32 wr_id 6835c0. [D 20:01:02.759609] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.759627] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6e0ce0. [D 20:01:02.759636] flowproto completing 0x6e0ce0 [D 20:01:02.759648] Job flows in progress (callback time): 9 [D 20:01:02.759656] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:02.759664] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6d4400. [D 20:01:02.759671] flowproto completing 0x6d4400 [D 20:01:02.759682] Job flows in progress (callback time): 8 [D 20:01:02.759690] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x6cc2b0. [D 20:01:02.759697] flowproto completing 0x6cc2b0 [D 20:01:02.759710] Job flows in progress (callback time): 7 [D 20:01:02.744509] calling distribute [D 20:01:02.759748] PINT_distribute [D 20:01:02.759756] of 178257920 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.759766] begin iteration loff: 178257920 [D 20:01:02.759774] segment exceeds byte limit [D 20:01:02.759780] process a segment [D 20:01:02.759787] of 178257920 sz 2097152 [D 20:01:02.759794] add a segment 0 [D 20:01:02.759800] end iteration [D 20:01:02.759807] done with segments or bytes [D 20:01:02.759812] of 180355072 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.759820] next loff: 180355072 next poff: 180355072 [D 20:01:02.759827] return value 2097152 [D 20:01:02.759833] finished [D 20:01:02.759839] retval = 2097152 [D 20:01:02.759846] exiting distribute returned less than expected [D 20:01:02.759853] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 180355072 fo 201326592 eof 0 [D 20:01:02.759860] ========================================================= [D 20:01:02.759876] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.759884] dbpf_bstream_rw_list: mem_offset: 0x2aaab9e47010, mem_size: 2097152 [D 20:01:02.759891] dbpf_bstream_rw_list: stream_offset: 178257920, stream_size: 2097152 [D 20:01:02.759899] dbpf_open_cache_get: called [D 20:01:02.759906] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.759914] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.759937] op_queue add: 0x2aaab1525840 [D 20:01:02.759945] delayed I/O operation 0x2aaab1525840 (16 already in progress) [D 20:01:02.759952] DBPF I/O ops in progress: 16 [D 20:01:02.759960] flowproto-multiqueue forcing bmi_send_callback_fn. -[D 20:01:02.840248] openib_check_cq: completed send wr_id 683590. [D 20:01:02.840268] openib_check_cq: completed send wr_id 6835c0. [D 20:01:02.840276] openib_check_cq: completed recv wr_id 684740. [D 20:01:02.840285] openib_post_sr_rdmaw: ibv_post_send wr_id 668920 to 10.1.4.56:39990 remote addr a3c51000 rkey 108c27 nic_wr_credit 1. [D 20:01:02.840296] openib_post_rr: 10.1.4.56:39990 bh 63 wr_id 684740. [D 20:01:02.840305] openib_check_cq: completed recv wr_id 684770. [D 20:01:02.840312] openib_post_sr_rdmaw: ibv_post_send wr_id 6ecdf0 to 10.1.4.56:39990 remote addr 9f64d000 rkey 108e27 nic_wr_credit 2. [D 20:01:02.840320] openib_post_rr: 10.1.4.56:39990 bh 64 wr_id 684770. [D 20:01:02.842528] openib_check_cq: completed rdma wr_id 668920, nic_wr_credit 1. [D 20:01:02.842539] post_sr: 10.1.4.56:39990 bh 71 len 16 wr_id 6835f0. [D 20:01:02.844639] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c750. [D 20:01:02.844653] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.844664] ========================================================= [D 20:01:02.844671] PINT_process_request [D 20:01:02.844678] tiling 7 copies [D 20:01:02.844685] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.844692] lvl 0 el 0 blk 0 by 180355072 [D 20:01:02.844698] to 180355072 ta 167772160 fi 201326592 [D 20:01:02.844705] basic type or contiguous data [D 20:01:02.844711] Request disp [D 20:01:02.844717] contig_offset = 180355072 contig_size = 54525952 lvl_flag = 1 [D 20:01:02.844724] calling distribute [D 20:01:02.844730] PINT_distribute [D 20:01:02.844736] of 180355072 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.844745] begin iteration loff: 180355072 [D 20:01:02.844749] openib_check_cq: completed rdma wr_id 6ecdf0, nic_wr_credit 0. [D 20:01:02.844763] post_sr: 10.1.4.56:39990 bh 72 len 16 wr_id 683620. [D 20:01:02.844771] openib_check_cq: completed send wr_id 6835f0. [D 20:01:02.844778] openib_check_cq: completed send wr_id 683620. [D 20:01:02.844788] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:02.844796] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.844804] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:02.844811] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.844752] segment exceeds byte limit [D 20:01:02.844938] process a segment [D 20:01:02.844944] of 180355072 sz 2097152 [D 20:01:02.844950] add a segment 0 [D 20:01:02.844956] end iteration [D 20:01:02.844962] done with segments or bytes [D 20:01:02.844968] of 182452224 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.844975] next loff: 182452224 next poff: 182452224 [D 20:01:02.844982] return value 2097152 [D 20:01:02.844988] finished [D 20:01:02.844994] retval = 2097152 [D 20:01:02.845000] exiting distribute returned less than expected [D 20:01:02.845006] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 182452224 fo 201326592 eof 0 [D 20:01:02.845013] ========================================================= [D 20:01:02.845025] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.845032] dbpf_bstream_rw_list: mem_offset: 0x2aaaba44a010, mem_size: 2097152 [D 20:01:02.845038] dbpf_bstream_rw_list: stream_offset: 180355072, stream_size: 2097152 [D 20:01:02.845046] dbpf_open_cache_get: called [D 20:01:02.845052] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.845059] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.845074] op_queue add: 0x2aaab152b470 [D 20:01:02.845081] delayed I/O operation 0x2aaab152b470 (16 already in progress) [D 20:01:02.845088] DBPF I/O ops in progress: 16 [D 20:01:02.845094] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.845101] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c750. [D 20:01:02.845108] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.845116] ========================================================= [D 20:01:02.845122] PINT_process_request [D 20:01:02.845128] tiling 7 copies [D 20:01:02.845134] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.845141] lvl 0 el 0 blk 0 by 182452224 [D 20:01:02.845147] to 182452224 ta 167772160 fi 201326592 [D 20:01:02.845154] basic type or contiguous data [D 20:01:02.845160] Request disp [D 20:01:02.845166] contig_offset = 182452224 contig_size = 52428800 lvl_flag = 1 [D 20:01:02.845172] calling distribute [D 20:01:02.845178] PINT_distribute [D 20:01:02.845183] of 182452224 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.845191] begin iteration loff: 182452224 [D 20:01:02.845197] segment exceeds byte limit [D 20:01:02.845203] process a segment [D 20:01:02.845209] of 182452224 sz 2097152 [D 20:01:02.845215] add a segment 0 [D 20:01:02.845220] end iteration [D 20:01:02.845226] done with segments or bytes [D 20:01:02.845232] of 184549376 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.845239] next loff: 184549376 next poff: 184549376 [D 20:01:02.845245] return value 2097152 [D 20:01:02.845251] finished [D 20:01:02.845257] retval = 2097152 [D 20:01:02.845263] exiting distribute returned less than expected [D 20:01:02.845269] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 184549376 fo 201326592 eof 0 [D 20:01:02.845276] ========================================================= [D 20:01:02.845283] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.845290] dbpf_bstream_rw_list: mem_offset: 0x2aaab843a010, mem_size: 2097152 [D 20:01:02.845296] dbpf_bstream_rw_list: stream_offset: 182452224, stream_size: 2097152 [D 20:01:02.845303] dbpf_open_cache_get: called [D 20:01:02.845309] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.845315] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.845327] op_queue add: 0x2aaab152b210 [D 20:01:02.845334] delayed I/O operation 0x2aaab152b210 (16 already in progress) [D 20:01:02.845341] DBPF I/O ops in progress: 16 [D 20:01:02.845347] flowproto posted 0x2aaab152c750 [D 20:01:02.845354] Job flows in progress (post time): 8 [D 20:01:02.845364] [SM Exiting]: (0x665210) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:02.845374] [SM Entering]: (0x6664d0) io_sm:start_flow (status: 0) [D 20:01:02.845381] [SM frame get]: (0x6664d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:02.845389] io_start_flow() issuing flow to read data. [D 20:01:02.845396] flow: fsize: 471859200, server_nr: 0, server_ct: 1 [D 20:01:02.845402] file_req_offset: 201326592, aggregate_size: 33554432, handle: 2863311531 [D 20:01:02.845412] PINT_new_request_states n=1 [D 20:01:02.845420] flowproto posting 0x2aaab1517400 [D 20:01:02.845431] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.845438] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517400. [D 20:01:02.845445] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:02.845452] ========================================================= [D 20:01:02.845458] PINT_process_request [D 20:01:02.845464] tiling 8 copies [D 20:01:02.845470] skipping ahead to target_offset [D 20:01:02.845476] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.845483] lvl 0 el 0 blk 0 by 0 [D 20:01:02.845489] to 0 ta 201326592 fi 234881024 [D 20:01:02.845496] basic type or contiguous data [D 20:01:02.845502] Request disp [D 20:01:02.845507] contig_offset = 0 contig_size = 268435456 lvl_flag = 1 [D 20:01:02.845514] process logical skip [D 20:01:02.845519] retval = 201326592 [D 20:01:02.845525] exiting logical skip because distribute indicates done [D 20:01:02.845531] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.845539] lvl 0 el 0 blk 0 by 201326592 [D 20:01:02.845546] to 201326592 ta 201326592 fi 234881024 [D 20:01:02.845552] basic type or contiguous data [D 20:01:02.845558] Request disp [D 20:01:02.845563] contig_offset = 201326592 contig_size = 67108864 lvl_flag = 1 [D 20:01:02.845570] calling distribute [D 20:01:02.845575] PINT_distribute [D 20:01:02.845581] of 201326592 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.845588] begin iteration loff: 201326592 [D 20:01:02.845594] segment exceeds byte limit [D 20:01:02.845600] process a segment [D 20:01:02.845606] of 201326592 sz 2097152 [D 20:01:02.845612] add a segment 0 [D 20:01:02.845618] end iteration [D 20:01:02.845623] done with segments or bytes [D 20:01:02.845629] of 203423744 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.845636] next loff: 203423744 next poff: 203423744 [D 20:01:02.845642] return value 2097152 [D 20:01:02.845648] finished [D 20:01:02.845653] retval = 2097152 [D 20:01:02.845659] exiting distribute returned less than expected [D 20:01:02.845665] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 203423744 fo 234881024 eof 0 [D 20:01:02.845672] ========================================================= [D 20:01:02.845679] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.845686] dbpf_bstream_rw_list: mem_offset: 0x2aaab682c010, mem_size: 2097152 [D 20:01:02.845692] dbpf_bstream_rw_list: stream_offset: 201326592, stream_size: 2097152 [D 20:01:02.845699] dbpf_open_cache_get: called [D 20:01:02.845704] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.845711] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.845736] op_queue add: 0x2aaab153ad80 [D 20:01:02.845744] delayed I/O operation 0x2aaab153ad80 (16 already in progress) [D 20:01:02.845751] DBPF I/O ops in progress: 16 [D 20:01:02.845757] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.845763] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517400. [D 20:01:02.845770] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.845778] ========================================================= [D 20:01:02.845784] PINT_process_request [D 20:01:02.845790] tiling 8 copies [D 20:01:02.845796] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.845803] lvl 0 el 0 blk 0 by 203423744 [D 20:01:02.845809] to 203423744 ta 201326592 fi 234881024 [D 20:01:02.845815] basic type or contiguous data [D 20:01:02.845821] Request disp [D 20:01:02.845827] contig_offset = 203423744 contig_size = 65011712 lvl_flag = 1 [D 20:01:02.845833] calling distribute [D 20:01:02.845839] PINT_distribute [D 20:01:02.845844] of 203423744 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.845852] begin iteration loff: 203423744 [D 20:01:02.845858] segment exceeds byte limit [D 20:01:02.845864] process a segment [D 20:01:02.845869] of 203423744 sz 2097152 [D 20:01:02.845875] add a segment 0 [D 20:01:02.845881] end iteration [D 20:01:02.845887] done with segments or bytes [D 20:01:02.845893] of 205520896 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.845899] next loff: 205520896 next poff: 205520896 [D 20:01:02.845906] return value 2097152 [D 20:01:02.845911] finished [D 20:01:02.845917] retval = 2097152 [D 20:01:02.845923] exiting distribute returned less than expected [D 20:01:02.845929] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 205520896 fo 234881024 eof 0 [D 20:01:02.845936] ========================================================= [D 20:01:02.845943] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.845950] dbpf_bstream_rw_list: mem_offset: 0x2aaab481a010, mem_size: 2097152 [D 20:01:02.845956] dbpf_bstream_rw_list: stream_offset: 203423744, stream_size: 2097152 [D 20:01:02.845962] dbpf_open_cache_get: called [D 20:01:02.845969] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.845975] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.845996] op_queue add: 0x2aaab153aed0 [D 20:01:02.846004] delayed I/O operation 0x2aaab153aed0 (16 already in progress) [D 20:01:02.846011] DBPF I/O ops in progress: 16 [D 20:01:02.846017] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.846023] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517400. [D 20:01:02.846030] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.846037] ========================================================= [D 20:01:02.846044] PINT_process_request [D 20:01:02.846049] tiling 8 copies [D 20:01:02.846055] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.846063] lvl 0 el 0 blk 0 by 205520896 [D 20:01:02.846069] to 205520896 ta 201326592 fi 234881024 [D 20:01:02.846075] basic type or contiguous data [D 20:01:02.846081] Request disp [D 20:01:02.846087] contig_offset = 205520896 contig_size = 62914560 lvl_flag = 1 [D 20:01:02.846093] calling distribute [D 20:01:02.846099] PINT_distribute [D 20:01:02.846105] of 205520896 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.846112] begin iteration loff: 205520896 [D 20:01:02.846118] segment exceeds byte limit [D 20:01:02.846124] process a segment [D 20:01:02.846130] of 205520896 sz 2097152 [D 20:01:02.846136] add a segment 0 [D 20:01:02.846142] end iteration [D 20:01:02.846147] done with segments or bytes [D 20:01:02.846153] of 207618048 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.846160] next loff: 207618048 next poff: 207618048 [D 20:01:02.846166] return value 2097152 [D 20:01:02.846172] finished [D 20:01:02.846178] retval = 2097152 [D 20:01:02.846184] exiting distribute returned less than expected [D 20:01:02.846190] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 207618048 fo 234881024 eof 0 [D 20:01:02.846197] ========================================================= [D 20:01:02.846204] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.846210] dbpf_bstream_rw_list: mem_offset: 0x2aaab2c0b010, mem_size: 2097152 [D 20:01:02.846217] dbpf_bstream_rw_list: stream_offset: 205520896, stream_size: 2097152 [D 20:01:02.846223] dbpf_open_cache_get: called [D 20:01:02.846229] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.846235] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.846254] op_queue add: 0x2aaab153b020 [D 20:01:02.846262] delayed I/O operation 0x2aaab153b020 (16 already in progress) [D 20:01:02.846269] DBPF I/O ops in progress: 16 [D 20:01:02.846275] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.846281] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517400. [D 20:01:02.846288] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.846296] ========================================================= [D 20:01:02.846302] PINT_process_request [D 20:01:02.846307] tiling 8 copies [D 20:01:02.846314] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.846321] lvl 0 el 0 blk 0 by 207618048 [D 20:01:02.846327] to 207618048 ta 201326592 fi 234881024 [D 20:01:02.846333] basic type or contiguous data [D 20:01:02.846339] Request disp [D 20:01:02.846345] contig_offset = 207618048 contig_size = 60817408 lvl_flag = 1 [D 20:01:02.846352] calling distribute [D 20:01:02.846357] PINT_distribute [D 20:01:02.846363] of 207618048 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.846370] begin iteration loff: 207618048 [D 20:01:02.846377] segment exceeds byte limit [D 20:01:02.846382] process a segment [D 20:01:02.846388] of 207618048 sz 2097152 [D 20:01:02.846394] add a segment 0 [D 20:01:02.846400] end iteration [D 20:01:02.846406] done with segments or bytes [D 20:01:02.846829] of 209715200 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.846840] next loff: 209715200 next poff: 209715200 [D 20:01:02.846847] return value 2097152 [D 20:01:02.846853] finished [D 20:01:02.846858] retval = 2097152 [D 20:01:02.846864] exiting distribute returned less than expected [D 20:01:02.846870] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 209715200 fo 234881024 eof 0 [D 20:01:02.846877] ========================================================= [D 20:01:02.846885] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.846891] dbpf_bstream_rw_list: mem_offset: 0x2aaab0edf010, mem_size: 2097152 [D 20:01:02.846898] dbpf_bstream_rw_list: stream_offset: 207618048, stream_size: 2097152 [D 20:01:02.846904] dbpf_open_cache_get: called [D 20:01:02.846910] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.846916] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.846936] op_queue add: 0x2aaab152d4d0 [D 20:01:02.846945] delayed I/O operation 0x2aaab152d4d0 (16 already in progress) [D 20:01:02.846951] DBPF I/O ops in progress: 16 [D 20:01:02.846957] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.846964] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517400. [D 20:01:02.846970] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.846978] ========================================================= [D 20:01:02.846984] PINT_process_request [D 20:01:02.846990] tiling 8 copies [D 20:01:02.846996] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.847003] lvl 0 el 0 blk 0 by 209715200 [D 20:01:02.847009] to 209715200 ta 201326592 fi 234881024 [D 20:01:02.847015] basic type or contiguous data [D 20:01:02.847021] Request disp [D 20:01:02.847027] contig_offset = 209715200 contig_size = 58720256 lvl_flag = 1 [D 20:01:02.847033] calling distribute [D 20:01:02.847039] PINT_distribute [D 20:01:02.847045] of 209715200 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.847052] begin iteration loff: 209715200 [D 20:01:02.847058] segment exceeds byte limit [D 20:01:02.847064] process a segment [D 20:01:02.847069] of 209715200 sz 2097152 [D 20:01:02.847075] add a segment 0 [D 20:01:02.847081] end iteration [D 20:01:02.847087] done with segments or bytes [D 20:01:02.847092] of 211812352 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.847099] next loff: 211812352 next poff: 211812352 [D 20:01:02.847105] return value 2097152 [D 20:01:02.847111] finished [D 20:01:02.847116] retval = 2097152 [D 20:01:02.847122] exiting distribute returned less than expected [D 20:01:02.847128] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 211812352 fo 234881024 eof 0 [D 20:01:02.847135] ========================================================= [D 20:01:02.847142] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.847149] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf2d1010, mem_size: 2097152 [D 20:01:02.847155] dbpf_bstream_rw_list: stream_offset: 209715200, stream_size: 2097152 [D 20:01:02.847161] dbpf_open_cache_get: called [D 20:01:02.847167] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.847173] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.847193] op_queue add: 0x2aaab152d620 [D 20:01:02.847201] delayed I/O operation 0x2aaab152d620 (16 already in progress) [D 20:01:02.847208] DBPF I/O ops in progress: 16 [D 20:01:02.847214] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.847220] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517400. [D 20:01:02.847227] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.847234] ========================================================= [D 20:01:02.847240] PINT_process_request [D 20:01:02.847246] tiling 8 copies [D 20:01:02.847252] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.847259] lvl 0 el 0 blk 0 by 211812352 [D 20:01:02.847266] to 211812352 ta 201326592 fi 234881024 [D 20:01:02.847272] basic type or contiguous data [D 20:01:02.847278] Request disp [D 20:01:02.847284] contig_offset = 211812352 contig_size = 56623104 lvl_flag = 1 [D 20:01:02.847290] calling distribute [D 20:01:02.847296] PINT_distribute [D 20:01:02.847301] of 211812352 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.847309] begin iteration loff: 211812352 [D 20:01:02.847315] segment exceeds byte limit [D 20:01:02.847321] process a segment [D 20:01:02.847327] of 211812352 sz 2097152 [D 20:01:02.847333] add a segment 0 [D 20:01:02.847338] end iteration [D 20:01:02.847344] done with segments or bytes [D 20:01:02.847350] of 213909504 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.847357] next loff: 213909504 next poff: 213909504 [D 20:01:02.847363] return value 2097152 [D 20:01:02.847369] finished [D 20:01:02.847375] retval = 2097152 [D 20:01:02.847380] exiting distribute returned less than expected [D 20:01:02.847386] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 213909504 fo 234881024 eof 0 [D 20:01:02.847394] ========================================================= [D 20:01:02.847401] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.847407] dbpf_bstream_rw_list: mem_offset: 0x2aaaad6c3010, mem_size: 2097152 [D 20:01:02.847414] dbpf_bstream_rw_list: stream_offset: 211812352, stream_size: 2097152 [D 20:01:02.847420] dbpf_open_cache_get: called [D 20:01:02.847426] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.847432] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.847448] op_queue add: 0x2aaab152d770 [D 20:01:02.847457] delayed I/O operation 0x2aaab152d770 (16 already in progress) [D 20:01:02.847463] DBPF I/O ops in progress: 16 [D 20:01:02.847469] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.847475] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517400. [D 20:01:02.847483] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.847490] ========================================================= [D 20:01:02.847496] PINT_process_request [D 20:01:02.847502] tiling 8 copies [D 20:01:02.847508] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.847515] lvl 0 el 0 blk 0 by 213909504 [D 20:01:02.847521] to 213909504 ta 201326592 fi 234881024 [D 20:01:02.847528] basic type or contiguous data [D 20:01:02.847534] Request disp [D 20:01:02.847539] contig_offset = 213909504 contig_size = 54525952 lvl_flag = 1 [D 20:01:02.847546] calling distribute [D 20:01:02.847552] PINT_distribute [D 20:01:02.847557] of 213909504 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.847565] begin iteration loff: 213909504 [D 20:01:02.847571] segment exceeds byte limit [D 20:01:02.847577] process a segment [D 20:01:02.847582] of 213909504 sz 2097152 [D 20:01:02.847588] add a segment 0 [D 20:01:02.847594] end iteration [D 20:01:02.847600] done with segments or bytes [D 20:01:02.847605] of 216006656 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.847612] next loff: 216006656 next poff: 216006656 [D 20:01:02.847618] return value 2097152 [D 20:01:02.847624] finished [D 20:01:02.847630] retval = 2097152 [D 20:01:02.847636] exiting distribute returned less than expected [D 20:01:02.847642] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 216006656 fo 234881024 eof 0 [D 20:01:02.847649] ========================================================= [D 20:01:02.847656] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.847662] dbpf_bstream_rw_list: mem_offset: 0x2aaaba84c010, mem_size: 2097152 [D 20:01:02.847669] dbpf_bstream_rw_list: stream_offset: 213909504, stream_size: 2097152 [D 20:01:02.847675] dbpf_open_cache_get: called [D 20:01:02.848005] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.848014] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.848034] op_queue add: 0x2aaab152d8c0 [D 20:01:02.848043] delayed I/O operation 0x2aaab152d8c0 (16 already in progress) [D 20:01:02.848049] DBPF I/O ops in progress: 16 [D 20:01:02.848055] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.848061] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1517400. [D 20:01:02.848068] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.848076] ========================================================= [D 20:01:02.848082] PINT_process_request [D 20:01:02.848088] tiling 8 copies [D 20:01:02.848094] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.848101] lvl 0 el 0 blk 0 by 216006656 [D 20:01:02.848107] to 216006656 ta 201326592 fi 234881024 [D 20:01:02.848113] basic type or contiguous data [D 20:01:02.848119] Request disp [D 20:01:02.848125] contig_offset = 216006656 contig_size = 52428800 lvl_flag = 1 [D 20:01:02.848131] calling distribute [D 20:01:02.848137] PINT_distribute [D 20:01:02.848143] of 216006656 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.848150] begin iteration loff: 216006656 [D 20:01:02.848156] segment exceeds byte limit [D 20:01:02.848162] process a segment [D 20:01:02.848167] of 216006656 sz 2097152 [D 20:01:02.848173] add a segment 0 [D 20:01:02.848179] end iteration [D 20:01:02.848185] done with segments or bytes [D 20:01:02.848190] of 218103808 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.848197] next loff: 218103808 next poff: 218103808 [D 20:01:02.848203] return value 2097152 [D 20:01:02.848209] finished [D 20:01:02.848215] retval = 2097152 [D 20:01:02.848220] exiting distribute returned less than expected [D 20:01:02.848226] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 218103808 fo 234881024 eof 0 [D 20:01:02.848233] ========================================================= [D 20:01:02.848240] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.848247] dbpf_bstream_rw_list: mem_offset: 0x2aaab883c010, mem_size: 2097152 [D 20:01:02.848253] dbpf_bstream_rw_list: stream_offset: 216006656, stream_size: 2097152 [D 20:01:02.848260] dbpf_open_cache_get: called [D 20:01:02.848265] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.848272] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.848293] op_queue add: 0x2aaab152da10 [D 20:01:02.848301] delayed I/O operation 0x2aaab152da10 (16 already in progress) [D 20:01:02.848308] DBPF I/O ops in progress: 16 [D 20:01:02.848314] flowproto posted 0x2aaab1517400 [D 20:01:02.848320] Job flows in progress (post time): 9 [D 20:01:02.848327] [SM Exiting]: (0x6664d0) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:02.848335] [SM Entering]: (0x667790) io_sm:start_flow (status: 0) [D 20:01:02.848342] [SM frame get]: (0x667790) op-id: 3 index: 0 base-frm: 0 [D 20:01:02.848349] io_start_flow() issuing flow to read data. [D 20:01:02.848356] flow: fsize: 471859200, server_nr: 0, server_ct: 1 [D 20:01:02.848362] file_req_offset: 234881024, aggregate_size: 33554432, handle: 2863311531 [D 20:01:02.848370] PINT_new_request_states n=1 [D 20:01:02.848376] flowproto posting 0x2aaab152db60 [D 20:01:02.848395] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.848403] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152db60. [D 20:01:02.848410] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:02.848418] ========================================================= [D 20:01:02.848424] PINT_process_request [D 20:01:02.848430] tiling 9 copies [D 20:01:02.848435] skipping ahead to target_offset [D 20:01:02.848441] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.848449] lvl 0 el 0 blk 0 by 0 [D 20:01:02.848455] to 0 ta 234881024 fi 268435456 [D 20:01:02.848462] basic type or contiguous data [D 20:01:02.848468] Request disp [D 20:01:02.848473] contig_offset = 0 contig_size = 301989888 lvl_flag = 1 [D 20:01:02.848480] process logical skip [D 20:01:02.848486] retval = 234881024 [D 20:01:02.848492] exiting logical skip because distribute indicates done [D 20:01:02.848498] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.848505] lvl 0 el 0 blk 0 by 234881024 [D 20:01:02.848511] to 234881024 ta 234881024 fi 268435456 [D 20:01:02.848517] basic type or contiguous data [D 20:01:02.848523] Request disp [D 20:01:02.848529] contig_offset = 234881024 contig_size = 67108864 lvl_flag = 1 [D 20:01:02.848535] calling distribute [D 20:01:02.848739] PINT_distribute [D 20:01:02.848747] of 234881024 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.848754] begin iteration loff: 234881024 [D 20:01:02.848760] segment exceeds byte limit [D 20:01:02.848766] process a segment [D 20:01:02.848772] of 234881024 sz 2097152 [D 20:01:02.848778] add a segment 0 [D 20:01:02.848784] end iteration [D 20:01:02.848789] done with segments or bytes [D 20:01:02.848795] of 236978176 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.848802] next loff: 236978176 next poff: 236978176 [D 20:01:02.848808] return value 2097152 [D 20:01:02.848814] finished [D 20:01:02.848819] retval = 2097152 [D 20:01:02.848825] exiting distribute returned less than expected [D 20:01:02.848831] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 236978176 fo 268435456 eof 0 [D 20:01:02.848838] ========================================================= [D 20:01:02.848845] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.848852] dbpf_bstream_rw_list: mem_offset: 0x2aaab6c2e010, mem_size: 2097152 [D 20:01:02.848859] dbpf_bstream_rw_list: stream_offset: 234881024, stream_size: 2097152 [D 20:01:02.848865] dbpf_open_cache_get: called [D 20:01:02.848871] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.848877] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.848898] op_queue add: 0x2aaab152de30 [D 20:01:02.848907] delayed I/O operation 0x2aaab152de30 (16 already in progress) [D 20:01:02.848913] DBPF I/O ops in progress: 16 [D 20:01:02.848919] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.848925] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152db60. [D 20:01:02.759989] --- aio_progress_notification called with handle 2863311531 (0x2aaabd320fe0) [D 20:01:02.848932] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.760552] op_queue add: 0x2aaab1530a60 [D 20:01:02.848945] ========================================================= [D 20:01:02.848952] PINT_process_request [D 20:01:02.848958] tiling 9 copies [D 20:01:02.848960] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:02.849045] BMI_post_send: addr: 6843072, offset: 0x2aaabfc14010, size: 2097152, tag: 26 [D 20:01:02.849090] post_sr: 10.1.4.56:39990 bh 73 len 32 wr_id 683650. [D 20:01:02.849116] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.849124] BMI_post_send: addr: 6843072, offset: 0x2aaabfe15010, size: 2097152, tag: 26 [D 20:01:02.849132] post_sr: 10.1.4.56:39990 bh 74 len 32 wr_id 683680. [D 20:01:02.849140] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.849148] BMI_post_send: addr: 6843072, offset: 0x2aaac0016010, size: 2097152, tag: 26 [D 20:01:02.849155] post_sr: 10.1.4.56:39990 bh 75 len 32 wr_id 6836b0. [D 20:01:02.849164] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:02.849241] BMI_post_send: addr: 6843072, offset: 0x2aaac0217010, size: 2097152, tag: 26 [D 20:01:02.849280] post_sr: 10.1.4.56:39990 bh 76 len 32 wr_id 6836e0. [D 20:01:02.849289] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.849296] BMI_post_send: addr: 6843072, offset: 0x2aaac0418010, size: 2097152, tag: 26 [D 20:01:02.849304] post_sr: 10.1.4.56:39990 bh 77 len 32 wr_id 683710. [D 20:01:02.849313] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 -[D 20:01:02.849328] openib_check_cq: completed send wr_id 683650. [D 20:01:02.849336] openib_check_cq: completed send wr_id 683680. [D 20:01:02.849352] openib_check_cq: completed send wr_id 6836b0. [D 20:01:02.849360] openib_check_cq: completed recv wr_id 6847a0. [D 20:01:02.849437] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec210 to 10.1.4.56:39990 remote addr 9c64b000 rkey 108f27 nic_wr_credit 1. [D 20:01:02.849453] openib_post_rr: 10.1.4.56:39990 bh 65 wr_id 6847a0. [D 20:01:02.849479] openib_check_cq: completed send wr_id 6836e0. [D 20:01:02.849487] openib_check_cq: completed send wr_id 683710. [D 20:01:02.848967] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.849564] lvl 0 el 0 blk 0 by 236978176 [D 20:01:02.849571] to 236978176 ta 234881024 fi 268435456 [D 20:01:02.849578] basic type or contiguous data [D 20:01:02.849584] Request disp [D 20:01:02.849590] contig_offset = 236978176 contig_size = 65011712 lvl_flag = 1 [D 20:01:02.849597] calling distribute [D 20:01:02.849603] PINT_distribute [D 20:01:02.849608] of 236978176 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.849616] begin iteration loff: 236978176 [D 20:01:02.849622] segment exceeds byte limit [D 20:01:02.849628] process a segment [D 20:01:02.849634] of 236978176 sz 2097152 [D 20:01:02.849640] add a segment 0 [D 20:01:02.849646] end iteration [D 20:01:02.849651] done with segments or bytes [D 20:01:02.849657] of 239075328 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.849664] next loff: 239075328 next poff: 239075328 [D 20:01:02.849670] return value 2097152 [D 20:01:02.849677] finished [D 20:01:02.849682] retval = 2097152 [D 20:01:02.849688] exiting distribute returned less than expected [D 20:01:02.849694] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 239075328 fo 268435456 eof 0 [D 20:01:02.849701] ========================================================= [D 20:01:02.849709] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.849716] dbpf_bstream_rw_list: mem_offset: 0x2aaab4c1c010, mem_size: 2097152 [D 20:01:02.849722] dbpf_bstream_rw_list: stream_offset: 236978176, stream_size: 2097152 [D 20:01:02.849729] dbpf_open_cache_get: called [D 20:01:02.849735] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.849741] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.851686] openib_check_cq: completed rdma wr_id 6ec210, nic_wr_credit 0. [D 20:01:02.851770] post_sr: 10.1.4.56:39990 bh 78 len 16 wr_id 683740. [D 20:01:02.851785] openib_check_cq: completed send wr_id 683740. [D 20:01:02.851796] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:02.851803] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:02.851810] ========================================================= [D 20:01:02.851816] PINT_process_request [D 20:01:02.851822] tiling 5 copies [D 20:01:02.851828] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.851835] lvl 0 el 0 blk 0 by 119537664 [D 20:01:02.851849] to 119537664 ta 100663296 fi 134217728 [D 20:01:02.851858] basic type or contiguous data [D 20:01:02.851864] Request disp [D 20:01:02.851956] contig_offset = 119537664 contig_size = 48234496 lvl_flag = 1 [D 20:01:02.851963] calling distribute [D 20:01:02.851979] PINT_distribute [D 20:01:02.851985] of 119537664 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.851993] begin iteration loff: 119537664 [D 20:01:02.851999] segment exceeds byte limit [D 20:01:02.852005] process a segment [D 20:01:02.852011] of 119537664 sz 2097152 [D 20:01:02.852017] add a segment 0 [D 20:01:02.852023] end iteration [D 20:01:02.852029] done with segments or bytes [D 20:01:02.852108] of 121634816 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.852138] next loff: 121634816 next poff: 121634816 [D 20:01:02.852144] return value 2097152 [D 20:01:02.852150] finished [D 20:01:02.852161] retval = 2097152 [D 20:01:02.852167] exiting distribute returned less than expected [D 20:01:02.852176] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 121634816 fo 134217728 eof 0 [D 20:01:02.852185] ========================================================= [D 20:01:02.852195] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.852203] dbpf_bstream_rw_list: mem_offset: 0x2aaabfc14010, mem_size: 2097152 [D 20:01:02.852211] dbpf_bstream_rw_list: stream_offset: 119537664, stream_size: 2097152 [D 20:01:02.852232] dbpf_open_cache_get: called [D 20:01:02.852241] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.852249] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.852265] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852278] dbpf_open_cache_put: cached entry. [D 20:01:02.852286] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.852293] op_queue add: 0x2aaabd320fe0 [D 20:01:02.852312] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.852322] BMI_post_send: addr: 6843072, offset: 0x2aaabe006010, size: 2097152, tag: 24 [D 20:01:02.852331] post_sr: 10.1.4.56:39990 bh 79 len 32 wr_id 683770. [D 20:01:02.852342] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.764584] --- aio_progress_notification called with handle 2863311531 (0x2aaabd32f140) [D 20:01:02.765677] --- aio_progress_notification called with handle 2863311531 (0x2aaabd310cf0) [D 20:01:02.764640] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3017d0) [D 20:01:02.765703] --- aio_progress_notification called with handle 2863311531 (0x2aaabd32f290) [D 20:01:02.772591] --- aio_progress_notification called with handle 2863311531 (0x2aaabd32f680) [D 20:01:02.772691] --- aio_progress_notification called with handle 2863311531 (0x2aaabd32f530) [D 20:01:02.852395] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852409] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.776582] --- aio_progress_notification called with handle 2863311531 (0x2aaab152bb90) [D 20:01:02.780599] --- aio_progress_notification called with handle 2863311531 (0x2aaabd390e40) [D 20:01:02.852433] dbpf_open_cache_put: cached entry. [D 20:01:02.852447] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.782914] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301c30) [D 20:01:02.852471] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.742224] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.848963] DBPF I/O ops in progress: 15 [D 20:01:02.852362] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852504] op_queue add: 0x2aaabd32f290 [D 20:01:02.773418] --- aio_progress_notification called with handle 2863311531 (0x2aaabd38e2e0) [D 20:01:02.786065] --- aio_progress_notification called with handle 2863311531 (0x2aaab151fd00) [D 20:01:02.852546] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:02.852482] dbpf_open_cache_put: cached entry. [D 20:01:02.852380] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.768583] --- aio_progress_notification called with handle 2863311531 (0x2aaabd32f3e0) [D 20:01:02.852587] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852401] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852594] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852513] starting delayed I/O operation 0x2aaabd300a20 (15 in progress) [D 20:01:02.852455] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852553] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852490] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.784568] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303000) [D 20:01:02.852565] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852616] lio_listio called with 1 following aiocbs: [D 20:01:02.852654] aiocb_ptr_array[0]: fd: 13, off: 92274688, bytes: 2097152, buf: 0x2aaabf20f010, type: 0 [D 20:01:02.852667] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300a20 (handle 2863311531, ret 0)) [D 20:01:02.852682] op_queue add: 0x2aaab1530a60 [D 20:01:02.852635] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.852693] delayed I/O operation 0x2aaab1530a60 (16 already in progress) [D 20:01:02.852553] op_queue add: 0x2aaabd300ed0 [D 20:01:02.852574] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.852737] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:02.852746] op_queue add: 0x2aaab152bb90 [D 20:01:02.852724] DBPF I/O ops in progress: 16 [D 20:01:02.852763] BMI_post_send: addr: 6843072, offset: 0x2aaabc059010, size: 2097152, tag: 36 [D 20:01:02.852774] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.852787] op_queue add: 0x2aaabd3008d0 [D 20:01:02.852792] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152db60. [D 20:01:02.852785] post_sr: 10.1.4.56:39990 bh 80 len 32 wr_id 6837a0. [D 20:01:02.852811] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.852800] delayed I/O operation 0x2aaabd3008d0 (16 already in progress) [D 20:01:02.852835] ========================================================= [D 20:01:02.852827] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.852857] BMI_post_send: addr: 6843072, offset: 0x2aaabc25a010, size: 2097152, tag: 36 [D 20:01:02.852846] PINT_process_request [D 20:01:02.852869] post_sr: 10.1.4.56:39990 bh 81 len 32 wr_id 6837d0. [D 20:01:02.852873] tiling 9 copies [D 20:01:02.852838] DBPF I/O ops in progress: 16 [D 20:01:02.852890] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.852883] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.852923] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:02.852931] BMI_post_send: addr: 6843072, offset: 0x2aaac0619010, size: 2097152, tag: 26 [D 20:01:02.852900] openib_check_cq: completed send wr_id 683770. [D 20:01:02.852580] dbpf_open_cache_put: cached entry. [D 20:01:02.852909] lvl 0 el 0 blk 0 by 239075328 [D 20:01:02.852949] openib_check_cq: completed send wr_id 6837a0. [D 20:01:02.852958] to 239075328 ta 234881024 fi 268435456 [D 20:01:02.852975] DBPF I/O ops in progress: 15 [D 20:01:02.852989] starting delayed I/O operation 0x2aaabd300b70 (15 in progress) [D 20:01:02.853000] lio_listio called with 1 following aiocbs: [D 20:01:02.853006] aiocb_ptr_array[0]: fd: 13, off: 499122176, bytes: 2097152, buf: 0x2aaabb453010, type: 1 [D 20:01:02.853015] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300b70 (handle 2863311531, ret 0)) [D 20:01:02.853033] dbpf_open_cache_put: cached entry. [D 20:01:02.853043] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.853052] op_queue add: 0x2aaabd32f140 [D 20:01:02.853060] DBPF I/O ops in progress: 15 [D 20:01:02.853067] starting delayed I/O operation 0x2aaabd301170 (15 in progress) [D 20:01:02.853078] lio_listio called with 1 following aiocbs: [D 20:01:02.853084] aiocb_ptr_array[0]: fd: 13, off: 65011712, bytes: 2097152, buf: 0x2aaabe80a010, type: 0 [D 20:01:02.853093] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301170 (handle 2863311531, ret 0)) [D 20:01:02.852969] openib_check_cq: completed send wr_id 6837d0. [D 20:01:02.852973] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.852981] basic type or contiguous data [D 20:01:02.853125] openib_check_cq: completed recv wr_id 6847d0. [D 20:01:02.853129] op_queue add: 0x2aaabd3017d0 [D 20:01:02.853134] Request disp [D 20:01:02.853159] DBPF I/O ops in progress: 15 [D 20:01:02.853153] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec210 to 10.1.4.56:39990 remote addr a1a4f000 rkey 108d27 nic_wr_credit 1. [D 20:01:02.853163] contig_offset = 239075328 contig_size = 62914560 lvl_flag = 1 [D 20:01:02.853180] starting delayed I/O operation 0x2aaabd3012c0 (15 in progress) [D 20:01:02.853213] lio_listio called with 1 following aiocbs: [D 20:01:02.853201] calling distribute [D 20:01:02.853194] openib_post_rr: 10.1.4.56:39990 bh 66 wr_id 6847d0. [D 20:01:02.853236] PINT_distribute [D 20:01:02.853252] openib_check_cq: completed recv wr_id 684800. [D 20:01:02.853260] of 239075328 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.853274] openib_post_sr_rdmaw: ibv_post_send wr_id 6f0f90 to 10.1.4.56:39990 remote addr 89e37000 rkey 108a27 nic_wr_credit 2. [D 20:01:02.853286] begin iteration loff: 239075328 [D 20:01:02.853293] openib_post_rr: 10.1.4.56:39990 bh 67 wr_id 684800. [D 20:01:02.853299] segment exceeds byte limit [D 20:01:02.853315] openib_check_cq: completed recv wr_id 684830. [D 20:01:02.853336] openib_post_sr_rdmaw: ibv_post_send wr_id 668920 to 10.1.4.56:39990 remote addr 9c84b000 rkey 108f27 nic_wr_credit 3. [D 20:01:02.853227] aiocb_ptr_array[0]: fd: 13, off: 94371840, bytes: 2097152, buf: 0x2aaabf410010, type: 0 [D 20:01:02.853323] process a segment [D 20:01:02.853347] openib_post_rr: 10.1.4.56:39990 bh 68 wr_id 684830. [D 20:01:02.853354] of 239075328 sz 2097152 [D 20:01:02.853371] post_sr: 10.1.4.56:39990 bh 82 len 32 wr_id 683800. [D 20:01:02.853379] add a segment 0 [D 20:01:02.853402] end iteration [D 20:01:02.853408] done with segments or bytes [D 20:01:02.853414] of 241172480 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.853421] next loff: 241172480 next poff: 241172480 [D 20:01:02.853427] return value 2097152 [D 20:01:02.853433] finished [D 20:01:02.853439] retval = 2097152 [D 20:01:02.853445] exiting distribute returned less than expected [D 20:01:02.853451] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 241172480 fo 268435456 eof 0 [D 20:01:02.853458] ========================================================= [D 20:01:02.853466] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.853473] dbpf_bstream_rw_list: mem_offset: 0x2aaab300e010, mem_size: 2097152 [D 20:01:02.853479] dbpf_bstream_rw_list: stream_offset: 239075328, stream_size: 2097152 [D 20:01:02.853486] dbpf_open_cache_get: called [D 20:01:02.855430] openib_check_cq: completed rdma wr_id 6ec210, nic_wr_credit 2. [D 20:01:02.855488] post_sr: 10.1.4.56:39990 bh 83 len 16 wr_id 683830. [D 20:01:02.857653] openib_check_cq: completed rdma wr_id 6f0f90, nic_wr_credit 1. [D 20:01:02.857703] post_sr: 10.1.4.56:39990 bh 84 len 16 wr_id 683860. [D 20:01:02.859876] openib_check_cq: completed rdma wr_id 668920, nic_wr_credit 0. [D 20:01:02.859937] post_sr: 10.1.4.56:39990 bh 85 len 16 wr_id 683890. [D 20:01:02.859952] openib_check_cq: completed send wr_id 683800. [D 20:01:02.859961] openib_check_cq: completed send wr_id 683830. [D 20:01:02.859969] openib_check_cq: completed send wr_id 683860. [D 20:01:02.859976] openib_check_cq: completed send wr_id 683890. [D 20:01:02.853396] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:02.860058] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.860072] BMI_post_send: addr: 6843072, offset: 0x2aaabee0d010, size: 2097152, tag: 25 [D 20:01:02.860090] post_sr: 10.1.4.56:39990 bh 86 len 32 wr_id 6838c0. [D 20:01:02.860118] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.860129] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.860137] BMI_post_send: addr: 6843072, offset: 0x2aaabe207010, size: 2097152, tag: 24 [D 20:01:02.860146] post_sr: 10.1.4.56:39990 bh 87 len 32 wr_id 6838f0. [D 20:01:02.860155] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.860177] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:02.860195] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:02.860204] ========================================================= [D 20:01:02.860211] PINT_process_request [D 20:01:02.860218] tiling 5 copies [D 20:01:02.860226] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.860234] lvl 0 el 0 blk 0 by 121634816 [D 20:01:02.860241] to 121634816 ta 100663296 fi 134217728 [D 20:01:02.860248] basic type or contiguous data [D 20:01:02.860255] Request disp [D 20:01:02.860262] contig_offset = 121634816 contig_size = 46137344 lvl_flag = 1 [D 20:01:02.860269] calling distribute [D 20:01:02.860275] PINT_distribute [D 20:01:02.860282] of 121634816 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.860291] begin iteration loff: 121634816 [D 20:01:02.860299] segment exceeds byte limit [D 20:01:02.860306] process a segment [D 20:01:02.860312] of 121634816 sz 2097152 [D 20:01:02.860319] add a segment 0 [D 20:01:02.860326] end iteration [D 20:01:02.860332] done with segments or bytes [D 20:01:02.860338] of 123731968 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.860345] next loff: 123731968 next poff: 123731968 [D 20:01:02.860352] return value 2097152 [D 20:01:02.860359] finished [D 20:01:02.860365] retval = 2097152 [D 20:01:02.860371] exiting distribute returned less than expected [D 20:01:02.860378] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 123731968 fo 134217728 eof 0 [D 20:01:02.860385] ========================================================= [D 20:01:02.860400] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.860408] dbpf_bstream_rw_list: mem_offset: 0x2aaabfe15010, mem_size: 2097152 [D 20:01:02.860415] dbpf_bstream_rw_list: stream_offset: 121634816, stream_size: 2097152 [D 20:01:02.860423] dbpf_open_cache_get: called [D 20:01:02.860436] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3012c0 (handle 2863311531, ret 0)) [D 20:01:02.853254] dbpf_open_cache_put: cached entry. [D 20:01:02.860508] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.860517] op_queue add: 0x2aaabd32f3e0 [D 20:01:02.860548] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518270. [D 20:01:02.860557] BMI_post_send: addr: 6843072, offset: 0x2aaabc45b010, size: 2097152, tag: 36 [D 20:01:02.860570] post_sr: 10.1.4.56:39990 bh 88 len 32 wr_id 683920. [D 20:01:02.860583] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 1 [D 20:01:02.860596] DBPF I/O ops in progress: 15 [D 20:01:02.860606] starting delayed I/O operation 0x2aaabd301920 (15 in progress) [D 20:01:02.860617] lio_listio called with 1 following aiocbs: [D 20:01:02.860624] aiocb_ptr_array[0]: fd: 13, off: 501219328, bytes: 2097152, buf: 0x2aaabc85d010, type: 1 [D 20:01:02.860639] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301920 (handle 2863311531, ret 0)) [D 20:01:02.860768] DBPF I/O ops in progress: 15 [D 20:01:02.860783] starting delayed I/O operation 0x2aaab1520020 (15 in progress) [D 20:01:02.860794] lio_listio called with 1 following aiocbs: [D 20:01:02.860801] aiocb_ptr_array[0]: fd: 13, off: 134217728, bytes: 2097152, buf: 0x2aaac0a1b010, type: 0 [D 20:01:02.860815] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300a20) [D 20:01:02.860825] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.860836] dbpf_open_cache_put: cached entry. [D 20:01:02.860842] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.860848] op_queue add: 0x2aaabd300a20 [D 20:01:02.860861] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.860861] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300b70) [D 20:01:02.860893] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.860870] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301170) [D 20:01:02.860916] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.864559] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3012c0) [D 20:01:02.864614] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.864646] start_delayed_ops_if_any: lio_listio posted 0x2aaab1520020 (handle 2863311531, ret 0)) [D 20:01:02.864666] DBPF I/O ops in progress: 15 [D 20:01:02.864678] starting delayed I/O operation 0x2aaab1522b60 (15 in progress) [D 20:01:02.864690] lio_listio called with 1 following aiocbs: [D 20:01:02.864697] aiocb_ptr_array[0]: fd: 13, off: 136314880, bytes: 2097152, buf: 0x2aaab9a45010, type: 0 [D 20:01:02.864707] start_delayed_ops_if_any: lio_listio posted 0x2aaab1522b60 (handle 2863311531, ret 0)) [D 20:01:02.864749] DBPF I/O ops in progress: 15 [D 20:01:02.864763] starting delayed I/O operation 0x2aaab152bf70 (15 in progress) [D 20:01:02.864774] lio_listio called with 1 following aiocbs: [D 20:01:02.864780] aiocb_ptr_array[0]: fd: 13, off: 138412032, bytes: 2097152, buf: 0x2aaab7a35010, type: 0 [D 20:01:02.864789] start_delayed_ops_if_any: lio_listio posted 0x2aaab152bf70 (handle 2863311531, ret 0)) [D 20:01:02.864810] DBPF I/O ops in progress: 15 [D 20:01:02.864821] starting delayed I/O operation 0x2aaab152c0c0 (15 in progress) [D 20:01:02.864831] lio_listio called with 1 following aiocbs: [D 20:01:02.864837] aiocb_ptr_array[0]: fd: 13, off: 140509184, bytes: 2097152, buf: 0x2aaab5e26010, type: 0 [D 20:01:02.864847] start_delayed_ops_if_any: lio_listio posted 0x2aaab152c0c0 (handle 2863311531, ret 0)) [D 20:01:02.868640] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301920) [D 20:01:02.868782] aio_progress_notification: BSTREAM_WRITE_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.860867] dbpf_open_cache_put: cached entry. [D 20:01:02.868851] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.868862] op_queue add: 0x2aaabd310cf0 [D 20:01:02.868901] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:02.868917] BMI_post_send: addr: 6843072, offset: 0x2aaabd400010, size: 2097152, tag: 23 [D 20:01:02.868941] post_sr: 10.1.4.56:39990 bh 89 len 32 wr_id 683950. [D 20:01:02.868982] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.869001] DBPF I/O ops in progress: 15 [D 20:01:02.869010] starting delayed I/O operation 0x2aaabd310e50 (15 in progress) [D 20:01:02.869023] lio_listio called with 1 following aiocbs: [D 20:01:02.869030] aiocb_ptr_array[0]: fd: 13, off: 96468992, bytes: 2097152, buf: 0x2aaabf611010, type: 0 [D 20:01:02.869044] start_delayed_ops_if_any: lio_listio posted 0x2aaabd310e50 (handle 2863311531, ret 0)) [D 20:01:02.869131] dbpf_open_cache_put: cached entry. [D 20:01:02.869150] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.869158] op_queue add: 0x2aaabd32f680 [D 20:01:02.869177] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.869189] BMI_post_send: addr: 6843072, offset: 0x2aaabe408010, size: 2097152, tag: 24 [D 20:01:02.869214] post_sr: 10.1.4.56:39990 bh 90 len 32 wr_id 683980. [D 20:01:02.869226] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:02.869239] DBPF I/O ops in progress: 15 [D 20:01:02.869248] starting delayed I/O operation 0x2aaab152c210 (15 in progress) [D 20:01:02.869261] lio_listio called with 1 following aiocbs: [D 20:01:02.869267] aiocb_ptr_array[0]: fd: 13, off: 142606336, bytes: 2097152, buf: 0x2aaab3e15010, type: 0 [D 20:01:02.869278] start_delayed_ops_if_any: lio_listio posted 0x2aaab152c210 (handle 2863311531, ret 0)) [D 20:01:02.869349] dbpf_open_cache_put: cached entry. [D 20:01:02.869366] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.869375] op_queue add: 0x2aaabd32f530 [D 20:01:02.869391] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:02.869400] BMI_post_send: addr: 6843072, offset: 0x2aaabd601010, size: 2097152, tag: 23 [D 20:01:02.869409] post_sr: 10.1.4.56:39990 bh 91 len 32 wr_id 6839b0. [D 20:01:02.869419] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.869446] DBPF I/O ops in progress: 15 [D 20:01:02.869457] starting delayed I/O operation 0x2aaab152c360 (15 in progress) [D 20:01:02.869467] lio_listio called with 1 following aiocbs: [D 20:01:02.869473] aiocb_ptr_array[0]: fd: 13, off: 144703488, bytes: 2097152, buf: 0x2aaaba048010, type: 0 [D 20:01:02.869488] --- aio_progress_notification called with handle 2863311531 (0x2aaab1520020) [D 20:01:02.869500] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.869512] dbpf_open_cache_put: cached entry. [D 20:01:02.869518] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.869525] op_queue add: 0x2aaab1520020 [D 20:01:02.869543] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:02.869553] BMI_post_send: addr: 6843072, offset: 0x2aaac0a1b010, size: 2097152, tag: 27 [D 20:01:02.869562] post_sr: 10.1.4.56:39990 bh 92 len 32 wr_id 6839e0. [D 20:01:02.869592] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:02.869608] start_delayed_ops_if_any: lio_listio posted 0x2aaab152c360 (handle 2863311531, ret 0)) [D 20:01:02.869628] DBPF I/O ops in progress: 15 [D 20:01:02.869636] starting delayed I/O operation 0x2aaab152c4b0 (15 in progress) [D 20:01:02.869649] lio_listio called with 1 following aiocbs: [D 20:01:02.869656] aiocb_ptr_array[0]: fd: 13, off: 146800640, bytes: 2097152, buf: 0x2aaab8038010, type: 0 [D 20:01:02.869665] start_delayed_ops_if_any: lio_listio posted 0x2aaab152c4b0 (handle 2863311531, ret 0)) [D 20:01:02.869711] dbpf_open_cache_put: cached entry. [D 20:01:02.870345] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.870372] op_queue add: 0x2aaabd38e2e0 [D 20:01:02.870400] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.870412] BMI_post_send: addr: 6843072, offset: 0x2aaabf00e010, size: 2097152, tag: 25 [D 20:01:02.870426] post_sr: 10.1.4.56:39990 bh 93 len 32 wr_id 683a10. [D 20:01:02.870447] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.870455] BMI_post_send: addr: 6843072, offset: 0x2aaabf20f010, size: 2097152, tag: 25 [D 20:01:02.870464] post_sr: 10.1.4.56:39990 bh 94 len 32 wr_id 683a40. [D 20:01:02.870473] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:02.870490] DBPF I/O ops in progress: 15 [D 20:01:02.870499] starting delayed I/O operation 0x2aaab152c600 (15 in progress) [D 20:01:02.870510] lio_listio called with 1 following aiocbs: [D 20:01:02.870516] aiocb_ptr_array[0]: fd: 13, off: 148897792, bytes: 2097152, buf: 0x2aaab642a010, type: 0 [D 20:01:02.870527] start_delayed_ops_if_any: lio_listio posted 0x2aaab152c600 (handle 2863311531, ret 0)) [D 20:01:02.870595] dbpf_open_cache_put: cached entry. [D 20:01:02.870612] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.870621] op_queue add: 0x2aaabd301c30 [D 20:01:02.870639] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517ef0. [D 20:01:02.870648] BMI_post_send: addr: 6843072, offset: 0x2aaabd802010, size: 2097152, tag: 23 [D 20:01:02.870657] post_sr: 10.1.4.56:39990 bh 95 len 32 wr_id 683a70. [D 20:01:02.870667] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 1 [D 20:01:02.870679] DBPF I/O ops in progress: 15 [D 20:01:02.870687] starting delayed I/O operation 0x2aaab152cbb0 (15 in progress) [D 20:01:02.870698] lio_listio called with 1 following aiocbs: [D 20:01:02.870704] aiocb_ptr_array[0]: fd: 13, off: 167772160, bytes: 2097152, buf: 0x2aaab4418010, type: 0 [D 20:01:02.870714] start_delayed_ops_if_any: lio_listio posted 0x2aaab152cbb0 (handle 2863311531, ret 0)) [D 20:01:02.870759] dbpf_open_cache_put: cached entry. [D 20:01:02.870773] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.870781] op_queue add: 0x2aaab151fd00 [D 20:01:02.870803] DBPF I/O ops in progress: 15 [D 20:01:02.870812] starting delayed I/O operation 0x2aaab152cd00 (15 in progress) [D 20:01:02.870823] lio_listio called with 1 following aiocbs: [D 20:01:02.870830] aiocb_ptr_array[0]: fd: 13, off: 169869312, bytes: 2097152, buf: 0x2aaab2809010, type: 0 [D 20:01:02.870839] start_delayed_ops_if_any: lio_listio posted 0x2aaab152cd00 (handle 2863311531, ret 0)) [D 20:01:02.870881] dbpf_open_cache_put: cached entry. [D 20:01:02.870895] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.870903] op_queue add: 0x2aaabd303000 [D 20:01:02.870912] DBPF I/O ops in progress: 15 [D 20:01:02.870920] starting delayed I/O operation 0x2aaab152ce50 (15 in progress) [D 20:01:02.870930] lio_listio called with 1 following aiocbs: [D 20:01:02.870936] aiocb_ptr_array[0]: fd: 13, off: 171966464, bytes: 2097152, buf: 0x2aaab0add010, type: 0 [D 20:01:02.870945] start_delayed_ops_if_any: lio_listio posted 0x2aaab152ce50 (handle 2863311531, ret 0)) [D 20:01:02.870982] dbpf_open_cache_put: cached entry. [D 20:01:02.870997] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.871005] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.871004] op_queue add: 0x2aaabd390e40 [D 20:01:02.871026] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.871049] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.871063] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.871032] DBPF I/O ops in progress: 15 [D 20:01:02.871092] starting delayed I/O operation 0x2aaab152cfa0 (15 in progress) [D 20:01:02.871104] lio_listio called with 1 following aiocbs: [D 20:01:02.871111] aiocb_ptr_array[0]: fd: 13, off: 174063616, bytes: 2097152, buf: 0x2aaaaeecf010, type: 0 [D 20:01:02.871121] start_delayed_ops_if_any: lio_listio posted 0x2aaab152cfa0 (handle 2863311531, ret 0)) [D 20:01:02.871133] op_queue add: 0x2aaab152bb90 [D 20:01:02.871145] delayed I/O operation 0x2aaab152bb90 (16 already in progress) [D 20:01:02.871146] dbpf_open_cache_put: cached entry. [D 20:01:02.871154] DBPF I/O ops in progress: 16 [D 20:01:02.871172] dbpf_open_cache_put: cached entry. [D 20:01:02.871180] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.871181] op_queue add: 0x2aaabd310fa0 [D 20:01:02.871214] delayed I/O operation 0x2aaabd310fa0 (16 already in progress) [D 20:01:02.871221] DBPF I/O ops in progress: 16 [D 20:01:02.871234] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:02.871246] (send callback time) ini posts: 0, pending: 2, last: 0, src_list emtpy: no [D 20:01:02.871255] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:02.871263] (send callback time) ini posts: 0, pending: 2, last: 1, src_list emtpy: yes [D 20:01:02.871282] openib_check_cq: completed send wr_id 6838c0. [D 20:01:02.871293] openib_check_cq: completed send wr_id 6838f0. [D 20:01:02.871300] openib_check_cq: completed send wr_id 683920. [D 20:01:02.871308] openib_check_cq: completed recv wr_id 684860. [D 20:01:02.871322] openib_post_sr_rdmaw: ibv_post_send wr_id 668920 to 10.1.4.56:39990 remote addr 9f84d000 rkey 108e27 nic_wr_credit 1. [D 20:01:02.871332] openib_post_rr: 10.1.4.56:39990 bh 69 wr_id 684860. [D 20:01:02.871341] openib_check_cq: completed recv wr_id 684890. [D 20:01:02.871349] openib_post_sr_rdmaw: ibv_post_send wr_id 6663a0 to 10.1.4.56:39990 remote addr 8a037000 rkey 108a27 nic_wr_credit 2. [D 20:01:02.871357] openib_post_rr: 10.1.4.56:39990 bh 70 wr_id 684890. [D 20:01:02.871366] openib_check_cq: completed recv wr_id 6848c0. [D 20:01:02.871375] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec210 to 10.1.4.56:39990 remote addr a1c4f000 rkey 108d27 nic_wr_credit 3. [D 20:01:02.871383] openib_post_rr: 10.1.4.56:39990 bh 71 wr_id 6848c0. [D 20:01:02.871391] openib_check_cq: completed recv wr_id 6848f0. [D 20:01:02.871398] openib_post_sr_rdmaw: ibv_post_send wr_id 6ecdf0 to 10.1.4.56:39990 remote addr 9ca4b000 rkey 108f27 nic_wr_credit 4. [D 20:01:02.871406] openib_post_rr: 10.1.4.56:39990 bh 72 wr_id 6848f0. [D 20:01:02.871414] openib_check_cq: completed send wr_id 683950. [D 20:01:02.871422] openib_check_cq: completed recv wr_id 684920. [D 20:01:02.871430] openib_post_sr_rdmaw: ibv_post_send wr_id 6df4e0 to 10.1.4.56:39990 remote addr a3e51000 rkey 108c27 nic_wr_credit 5. [D 20:01:02.871438] openib_post_rr: 10.1.4.56:39990 bh 73 wr_id 684920. [D 20:01:02.871446] openib_check_cq: completed send wr_id 683980. [D 20:01:02.871453] openib_check_cq: completed send wr_id 6839b0. [D 20:01:02.871460] openib_check_cq: completed send wr_id 6839e0. [D 20:01:02.871467] openib_check_cq: completed recv wr_id 684950. [D 20:01:02.871475] openib_post_sr_rdmaw: ibv_post_send wr_id 6ed990 to 10.1.4.56:39990 remote addr 9a449000 rkey 109126 nic_wr_credit 6. [D 20:01:02.871483] openib_post_rr: 10.1.4.56:39990 bh 74 wr_id 684950. [D 20:01:02.871491] openib_check_cq: completed send wr_id 683a10. [D 20:01:02.871498] openib_check_cq: completed send wr_id 683a40. [D 20:01:02.871505] openib_check_cq: completed send wr_id 683a70. [D 20:01:02.872558] --- aio_progress_notification called with handle 2863311531 (0x2aaab1522b60) [D 20:01:02.872588] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.873570] openib_check_cq: completed rdma wr_id 668920, nic_wr_credit 5. [D 20:01:02.873616] post_sr: 10.1.4.56:39990 bh 96 len 16 wr_id 683aa0. [D 20:01:02.875792] openib_check_cq: completed rdma wr_id 6663a0, nic_wr_credit 4. [D 20:01:02.875864] post_sr: 10.1.4.56:39990 bh 97 len 16 wr_id 683ad0. [D 20:01:02.876564] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:02.876601] BMI_post_send: addr: 6843072, offset: 0x2aaac081a010, size: 2097152, tag: 26 [D 20:01:02.876627] post_sr: 10.1.4.56:39990 bh 98 len 32 wr_id 683b00. [D 20:01:02.876665] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:02.876758] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.876779] BMI_post_send: addr: 6843072, offset: 0x2aaabe609010, size: 2097152, tag: 24 [D 20:01:02.876790] post_sr: 10.1.4.56:39990 bh 99 len 32 wr_id 683b30. [D 20:01:02.876801] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:02.876812] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:02.878013] openib_check_cq: completed rdma wr_id 6ec210, nic_wr_credit 3. [D 20:01:02.878082] post_sr: 10.1.4.56:39990 bh 0 len 16 wr_id 6828a0. [D 20:01:02.880234] openib_check_cq: completed rdma wr_id 6ecdf0, nic_wr_credit 2. [D 20:01:02.880305] post_sr: 10.1.4.56:39990 bh 1 len 16 wr_id 6828d0. [D 20:01:02.881115] --- aio_progress_notification called with handle 2863311531 (0x2aaab152bf70) [D 20:01:02.881146] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.881165] --- aio_progress_notification called with handle 2863311531 (0x2aaab152c0c0) [D 20:01:02.881175] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.881188] --- aio_progress_notification called with handle 2863311531 (0x2aaabd310e50) [D 20:01:02.881197] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.881215] --- aio_progress_notification called with handle 2863311531 (0x2aaab152c210) [D 20:01:02.881230] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.882460] openib_check_cq: completed rdma wr_id 6df4e0, nic_wr_credit 1. [D 20:01:02.882525] post_sr: 10.1.4.56:39990 bh 2 len 16 wr_id 682900. [D 20:01:02.884683] openib_check_cq: completed rdma wr_id 6ed990, nic_wr_credit 0. [D 20:01:02.884753] post_sr: 10.1.4.56:39990 bh 3 len 16 wr_id 682930. [D 20:01:02.884765] openib_check_cq: completed send wr_id 683aa0. [D 20:01:02.884773] openib_check_cq: completed send wr_id 683ad0. [D 20:01:02.884781] openib_check_cq: completed send wr_id 683b00. [D 20:01:02.884788] openib_check_cq: completed send wr_id 683b30. [D 20:01:02.884795] openib_check_cq: completed send wr_id 6828a0. [D 20:01:02.884803] openib_check_cq: completed send wr_id 6828d0. [D 20:01:02.884810] openib_check_cq: completed send wr_id 682900. [D 20:01:02.884817] openib_check_cq: completed send wr_id 682930. [D 20:01:02.884837] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:02.884851] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:02.884862] ========================================================= [D 20:01:02.884869] PINT_process_request [D 20:01:02.884877] tiling 5 copies [D 20:01:02.884885] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.884893] lvl 0 el 0 blk 0 by 123731968 [D 20:01:02.884900] to 123731968 ta 100663296 fi 134217728 [D 20:01:02.884908] basic type or contiguous data [D 20:01:02.884914] Request disp [D 20:01:02.884921] contig_offset = 123731968 contig_size = 44040192 lvl_flag = 1 [D 20:01:02.884929] calling distribute [D 20:01:02.884935] PINT_distribute [D 20:01:02.884942] of 123731968 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:02.884952] begin iteration loff: 123731968 [D 20:01:02.884959] segment exceeds byte limit [D 20:01:02.884966] process a segment [D 20:01:02.884972] of 123731968 sz 2097152 [D 20:01:02.884979] add a segment 0 [D 20:01:02.884985] end iteration [D 20:01:02.884992] done with segments or bytes [D 20:01:02.884998] of 125829120 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.885005] next loff: 125829120 next poff: 125829120 [D 20:01:02.885012] return value 2097152 [D 20:01:02.885019] finished [D 20:01:02.885025] retval = 2097152 [D 20:01:02.885032] exiting distribute returned less than expected [D 20:01:02.885038] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 125829120 fo 134217728 eof 0 [D 20:01:02.885046] ========================================================= [D 20:01:02.885064] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.885073] dbpf_bstream_rw_list: mem_offset: 0x2aaac0016010, mem_size: 2097152 [D 20:01:02.885080] dbpf_bstream_rw_list: stream_offset: 123731968, stream_size: 2097152 [D 20:01:02.885089] dbpf_open_cache_get: called [D 20:01:02.885123] dbpf_open_cache_put: cached entry. [D 20:01:02.885146] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.885156] op_queue add: 0x2aaabd3012c0 [D 20:01:02.885186] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:02.885200] BMI_post_send: addr: 6843072, offset: 0x2aaabf410010, size: 2097152, tag: 25 [D 20:01:02.885215] post_sr: 10.1.4.56:39990 bh 4 len 32 wr_id 682960. [D 20:01:02.885249] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.885266] DBPF I/O ops in progress: 15 [D 20:01:02.885277] starting delayed I/O operation 0x2aaab152d0f0 (15 in progress) [D 20:01:02.885288] lio_listio called with 1 following aiocbs: [D 20:01:02.885296] aiocb_ptr_array[0]: fd: 13, off: 176160768, bytes: 2097152, buf: 0x2aaaad2c1010, type: 0 [D 20:01:02.885312] start_delayed_ops_if_any: lio_listio posted 0x2aaab152d0f0 (handle 2863311531, ret 0)) [D 20:01:02.871193] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.885380] op_queue add: 0x2aaabd301170 [D 20:01:02.871205] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152db60. [D 20:01:02.885401] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1518130. [D 20:01:02.885409] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.885412] BMI_post_send: addr: 6843072, offset: 0x2aaabe80a010, size: 2097152, tag: 24 [D 20:01:02.885414] DBPF I/O ops in progress: 15 [D 20:01:02.885423] ========================================================= [D 20:01:02.885436] post_sr: 10.1.4.56:39990 bh 5 len 32 wr_id 682990. [D 20:01:02.885444] PINT_process_request [D 20:01:02.885457] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 1 [D 20:01:02.885467] tiling 9 copies [D 20:01:02.885485] dbpf_open_cache_put: cached entry. [D 20:01:02.885490] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.871169] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.885511] lvl 0 el 0 blk 0 by 241172480 [D 20:01:02.885502] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.885521] to 241172480 ta 234881024 fi 268435456 [D 20:01:02.885459] starting delayed I/O operation 0x2aaabd301a70 (15 in progress) [D 20:01:02.885541] basic type or contiguous data [D 20:01:02.885555] lio_listio called with 1 following aiocbs: [D 20:01:02.885565] Request disp [D 20:01:02.885515] op_queue add: 0x2aaabd300b70 [D 20:01:02.885576] aiocb_ptr_array[0]: fd: 13, off: 117440512, bytes: 2097152, buf: 0x2aaabfa13010, type: 0 [D 20:01:02.885588] contig_offset = 241172480 contig_size = 60817408 lvl_flag = 1 [D 20:01:02.885620] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301a70 (handle 2863311531, ret 0)) [D 20:01:02.885629] calling distribute [D 20:01:02.885622] op_queue add: 0x2aaabd301920 [D 20:01:02.885650] PINT_distribute [D 20:01:02.885673] of 241172480 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.885666] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:02.885694] flowproto-multiqueue trove_write_callback_fn, error_code: 0, flow: 0x2aaab1505370. [D 20:01:02.885702] flowproto completing 0x2aaab1505370 [D 20:01:02.885717] Job flows in progress (callback time): 8 [D 20:01:02.885669] DBPF I/O ops in progress: 15 [D 20:01:02.885684] begin iteration loff: 241172480 [D 20:01:02.885736] starting delayed I/O operation 0x2aaabd300cc0 (15 in progress) [D 20:01:02.885746] segment exceeds byte limit [D 20:01:02.885758] lio_listio called with 1 following aiocbs: [D 20:01:02.885776] aiocb_ptr_array[0]: fd: 13, off: 98566144, bytes: 2097152, buf: 0x2aaabf812010, type: 0 [D 20:01:02.885765] process a segment [D 20:01:02.885787] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300cc0 (handle 2863311531, ret 0)) [D 20:01:02.885794] of 241172480 sz 2097152 [D 20:01:02.885813] DBPF I/O ops in progress: 15 [D 20:01:02.885827] starting delayed I/O operation 0x2aaab1525840 (15 in progress) [D 20:01:02.885814] add a segment 0 [D 20:01:02.885838] lio_listio called with 1 following aiocbs: [D 20:01:02.885856] aiocb_ptr_array[0]: fd: 13, off: 178257920, bytes: 2097152, buf: 0x2aaab9e47010, type: 0 [D 20:01:02.885844] end iteration [D 20:01:02.885867] start_delayed_ops_if_any: lio_listio posted 0x2aaab1525840 (handle 2863311531, ret 0)) [D 20:01:02.885871] done with segments or bytes [D 20:01:02.885888] of 243269632 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.885895] next loff: 243269632 next poff: 243269632 [D 20:01:02.885902] return value 2097152 [D 20:01:02.885908] finished [D 20:01:02.885914] retval = 2097152 [D 20:01:02.885920] exiting distribute returned less than expected [D 20:01:02.885926] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 243269632 fo 268435456 eof 0 [D 20:01:02.885933] ========================================================= [D 20:01:02.885942] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.885949] dbpf_bstream_rw_list: mem_offset: 0x2aaab12e1010, mem_size: 2097152 [D 20:01:02.885956] dbpf_bstream_rw_list: stream_offset: 241172480, stream_size: 2097152 [D 20:01:02.885963] dbpf_open_cache_get: called [D 20:01:02.885969] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.885976] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.885991] op_queue add: 0x2aaab151fd00 [D 20:01:02.885998] delayed I/O operation 0x2aaab151fd00 (16 already in progress) [D 20:01:02.886004] DBPF I/O ops in progress: 16 [D 20:01:02.886011] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.886018] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152db60. [D 20:01:02.886025] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.886032] ========================================================= [D 20:01:02.886039] PINT_process_request [D 20:01:02.886044] tiling 9 copies [D 20:01:02.886050] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.886058] lvl 0 el 0 blk 0 by 243269632 [D 20:01:02.886064] to 243269632 ta 234881024 fi 268435456 [D 20:01:02.886070] basic type or contiguous data [D 20:01:02.886076] Request disp [D 20:01:02.886082] contig_offset = 243269632 contig_size = 58720256 lvl_flag = 1 [D 20:01:02.886089] calling distribute [D 20:01:02.886095] PINT_distribute [D 20:01:02.886100] of 243269632 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.886108] begin iteration loff: 243269632 [D 20:01:02.886114] segment exceeds byte limit [D 20:01:02.886120] process a segment [D 20:01:02.886125] of 243269632 sz 2097152 [D 20:01:02.886132] add a segment 0 [D 20:01:02.886138] end iteration [D 20:01:02.886143] done with segments or bytes [D 20:01:02.886149] of 245366784 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.886156] next loff: 245366784 next poff: 245366784 [D 20:01:02.886162] return value 2097152 [D 20:01:02.886168] finished [D 20:01:02.886174] retval = 2097152 [D 20:01:02.886180] exiting distribute returned less than expected [D 20:01:02.886186] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 245366784 fo 268435456 eof 0 [D 20:01:02.886193] ========================================================= [D 20:01:02.886200] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.886207] dbpf_bstream_rw_list: mem_offset: 0x2aaabc85d010, mem_size: 2097152 [D 20:01:02.886213] dbpf_bstream_rw_list: stream_offset: 243269632, stream_size: 2097152 [D 20:01:02.886220] dbpf_open_cache_get: called [D 20:01:02.886226] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.886232] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.886244] op_queue add: 0x2aaab1520020 [D 20:01:02.886251] delayed I/O operation 0x2aaab1520020 (16 already in progress) [D 20:01:02.886257] DBPF I/O ops in progress: 16 [D 20:01:02.886263] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.886269] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152db60. [D 20:01:02.886276] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.886284] ========================================================= [D 20:01:02.886291] PINT_process_request [D 20:01:02.886297] tiling 9 copies [D 20:01:02.886303] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.886310] lvl 0 el 0 blk 0 by 245366784 [D 20:01:02.886316] to 245366784 ta 234881024 fi 268435456 [D 20:01:02.886323] basic type or contiguous data [D 20:01:02.886329] Request disp [D 20:01:02.886334] contig_offset = 245366784 contig_size = 56623104 lvl_flag = 1 [D 20:01:02.886341] calling distribute [D 20:01:02.886347] PINT_distribute [D 20:01:02.886352] of 245366784 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.886360] begin iteration loff: 245366784 [D 20:01:02.886366] segment exceeds byte limit [D 20:01:02.886372] process a segment [D 20:01:02.886377] of 245366784 sz 2097152 [D 20:01:02.886384] add a segment 0 [D 20:01:02.886389] end iteration [D 20:01:02.886395] done with segments or bytes [D 20:01:02.886401] of 247463936 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.886408] next loff: 247463936 next poff: 247463936 [D 20:01:02.886414] return value 2097152 [D 20:01:02.886420] finished [D 20:01:02.886425] retval = 2097152 [D 20:01:02.886431] exiting distribute returned less than expected [D 20:01:02.886437] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 247463936 fo 268435456 eof 0 [D 20:01:02.886444] ========================================================= [D 20:01:02.886452] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.886458] dbpf_bstream_rw_list: mem_offset: 0x2aaabb453010, mem_size: 2097152 [D 20:01:02.886465] dbpf_bstream_rw_list: stream_offset: 245366784, stream_size: 2097152 [D 20:01:02.886471] dbpf_open_cache_get: called [D 20:01:02.886477] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.886483] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.886511] op_queue add: 0x2aaab15054a0 [D 20:01:02.886519] delayed I/O operation 0x2aaab15054a0 (16 already in progress) [D 20:01:02.886526] DBPF I/O ops in progress: 16 [D 20:01:02.886532] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.886538] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152db60. [D 20:01:02.886546] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.886553] ========================================================= [D 20:01:02.886559] PINT_process_request [D 20:01:02.886565] tiling 9 copies [D 20:01:02.886571] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.886578] lvl 0 el 0 blk 0 by 247463936 [D 20:01:02.886585] to 247463936 ta 234881024 fi 268435456 [D 20:01:02.886591] basic type or contiguous data [D 20:01:02.886597] Request disp [D 20:01:02.886603] contig_offset = 247463936 contig_size = 54525952 lvl_flag = 1 [D 20:01:02.886610] calling distribute [D 20:01:02.886616] PINT_distribute [D 20:01:02.886621] of 247463936 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.886629] begin iteration loff: 247463936 [D 20:01:02.886635] segment exceeds byte limit [D 20:01:02.886641] process a segment [D 20:01:02.886647] of 247463936 sz 2097152 [D 20:01:02.886653] add a segment 0 [D 20:01:02.886659] end iteration [D 20:01:02.886665] done with segments or bytes [D 20:01:02.886670] of 249561088 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.886677] next loff: 249561088 next poff: 249561088 [D 20:01:02.886684] return value 2097152 [D 20:01:02.886689] finished [D 20:01:02.886695] retval = 2097152 [D 20:01:02.886701] exiting distribute returned less than expected [D 20:01:02.886707] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 249561088 fo 268435456 eof 0 [D 20:01:02.886714] ========================================================= [D 20:01:02.886721] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.886728] dbpf_bstream_rw_list: mem_offset: 0x2aaabb252010, mem_size: 2097152 [D 20:01:02.886735] dbpf_bstream_rw_list: stream_offset: 247463936, stream_size: 2097152 [D 20:01:02.886742] dbpf_open_cache_get: called [D 20:01:02.886748] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.886755] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.886774] op_queue add: 0x2aaab15055f0 [D 20:01:02.886783] delayed I/O operation 0x2aaab15055f0 (16 already in progress) [D 20:01:02.886789] DBPF I/O ops in progress: 16 [D 20:01:02.886795] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.886802] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152db60. [D 20:01:02.886809] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.886816] ========================================================= [D 20:01:02.886823] PINT_process_request [D 20:01:02.886829] tiling 9 copies [D 20:01:02.886835] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.886842] lvl 0 el 0 blk 0 by 249561088 [D 20:01:02.886848] to 249561088 ta 234881024 fi 268435456 [D 20:01:02.886855] basic type or contiguous data [D 20:01:02.886861] Request disp [D 20:01:02.886866] contig_offset = 249561088 contig_size = 52428800 lvl_flag = 1 [D 20:01:02.886873] calling distribute [D 20:01:02.886879] PINT_distribute [D 20:01:02.886885] of 249561088 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:02.886892] begin iteration loff: 249561088 [D 20:01:02.886898] segment exceeds byte limit [D 20:01:02.886904] process a segment [D 20:01:02.886910] of 249561088 sz 2097152 [D 20:01:02.886916] add a segment 0 [D 20:01:02.886922] end iteration [D 20:01:02.886928] done with segments or bytes [D 20:01:02.886934] of 251658240 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.886940] next loff: 251658240 next poff: 251658240 [D 20:01:02.886947] return value 2097152 [D 20:01:02.886953] finished [D 20:01:02.886958] retval = 2097152 [D 20:01:02.886964] exiting distribute returned less than expected [D 20:01:02.886970] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 251658240 fo 268435456 eof 0 [D 20:01:02.886978] ========================================================= [D 20:01:02.886985] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.886992] dbpf_bstream_rw_list: mem_offset: 0x2aaabb051010, mem_size: 2097152 [D 20:01:02.886998] dbpf_bstream_rw_list: stream_offset: 249561088, stream_size: 2097152 [D 20:01:02.887005] dbpf_open_cache_get: called [D 20:01:02.887011] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.887017] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.887036] op_queue add: 0x2aaab1505740 [D 20:01:02.887044] delayed I/O operation 0x2aaab1505740 (16 already in progress) [D 20:01:02.887051] DBPF I/O ops in progress: 16 [D 20:01:02.887058] flowproto posted 0x2aaab152db60 [D 20:01:02.887064] Job flows in progress (post time): 9 [D 20:01:02.887075] [SM Exiting]: (0x667790) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:02.887084] [SM Entering]: (0x668a50) io_sm:start_flow (status: 0) [D 20:01:02.887091] [SM frame get]: (0x668a50) op-id: 3 index: 0 base-frm: 0 [D 20:01:02.887099] io_start_flow() issuing flow to read data. [D 20:01:02.887106] flow: fsize: 476053504, server_nr: 0, server_ct: 1 [D 20:01:02.887113] file_req_offset: 268435456, aggregate_size: 33554432, handle: 2863311531 [D 20:01:02.887122] PINT_new_request_states n=1 [D 20:01:02.887130] flowproto posting 0x2aaab1505890 [D 20:01:02.887148] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.887156] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1505890. [D 20:01:02.887164] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:02.887171] ========================================================= [D 20:01:02.887178] PINT_process_request [D 20:01:02.887184] tiling 10 copies [D 20:01:02.887190] skipping ahead to target_offset [D 20:01:02.887196] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.887204] lvl 0 el 0 blk 0 by 0 [D 20:01:02.887210] to 0 ta 268435456 fi 301989888 [D 20:01:02.887216] basic type or contiguous data [D 20:01:02.887222] Request disp [D 20:01:02.887228] contig_offset = 0 contig_size = 335544320 lvl_flag = 1 [D 20:01:02.887235] process logical skip [D 20:01:02.887241] retval = 268435456 [D 20:01:02.887247] exiting logical skip because distribute indicates done [D 20:01:02.887253] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.887260] lvl 0 el 0 blk 0 by 268435456 [D 20:01:02.887266] to 268435456 ta 268435456 fi 301989888 [D 20:01:02.887273] basic type or contiguous data [D 20:01:02.887279] Request disp [D 20:01:02.887285] contig_offset = 268435456 contig_size = 67108864 lvl_flag = 1 [D 20:01:02.887291] calling distribute [D 20:01:02.887297] PINT_distribute [D 20:01:02.887303] of 268435456 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:02.887310] begin iteration loff: 268435456 [D 20:01:02.887317] segment exceeds byte limit [D 20:01:02.887323] process a segment [D 20:01:02.887328] of 268435456 sz 2097152 [D 20:01:02.887334] add a segment 0 [D 20:01:02.887340] end iteration [D 20:01:02.887346] done with segments or bytes [D 20:01:02.887352] of 270532608 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.887359] next loff: 270532608 next poff: 270532608 [D 20:01:02.887365] return value 2097152 [D 20:01:02.887371] finished [D 20:01:02.887377] retval = 2097152 [D 20:01:02.887383] exiting distribute returned less than expected [D 20:01:02.887389] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 270532608 fo 301989888 eof 0 [D 20:01:02.887396] ========================================================= [D 20:01:02.887403] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.887410] dbpf_bstream_rw_list: mem_offset: 0x2aaab9844010, mem_size: 2097152 [D 20:01:02.887417] dbpf_bstream_rw_list: stream_offset: 268435456, stream_size: 2097152 [D 20:01:02.887423] dbpf_open_cache_get: called [D 20:01:02.887429] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.887435] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.887455] op_queue add: 0x2aaab1505d50 [D 20:01:02.887464] delayed I/O operation 0x2aaab1505d50 (16 already in progress) [D 20:01:02.887470] DBPF I/O ops in progress: 16 [D 20:01:02.887476] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.887483] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1505890. [D 20:01:02.887490] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.887497] ========================================================= [D 20:01:02.887504] PINT_process_request [D 20:01:02.887509] tiling 10 copies [D 20:01:02.887516] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.887523] lvl 0 el 0 blk 0 by 270532608 [D 20:01:02.887529] to 270532608 ta 268435456 fi 301989888 [D 20:01:02.887536] basic type or contiguous data [D 20:01:02.887542] Request disp [D 20:01:02.887547] contig_offset = 270532608 contig_size = 65011712 lvl_flag = 1 [D 20:01:02.887554] calling distribute [D 20:01:02.887559] PINT_distribute [D 20:01:02.887565] of 270532608 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:02.887573] begin iteration loff: 270532608 [D 20:01:02.887579] segment exceeds byte limit [D 20:01:02.887585] process a segment [D 20:01:02.887591] of 270532608 sz 2097152 [D 20:01:02.887597] add a segment 0 [D 20:01:02.887603] end iteration [D 20:01:02.887608] done with segments or bytes [D 20:01:02.887614] of 272629760 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.887621] next loff: 272629760 next poff: 272629760 [D 20:01:02.887627] return value 2097152 [D 20:01:02.887634] finished [D 20:01:02.887640] retval = 2097152 [D 20:01:02.887646] exiting distribute returned less than expected [D 20:01:02.887652] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 272629760 fo 301989888 eof 0 [D 20:01:02.887659] ========================================================= [D 20:01:02.887666] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.887672] dbpf_bstream_rw_list: mem_offset: 0x2aaab8e3f010, mem_size: 2097152 [D 20:01:02.887679] dbpf_bstream_rw_list: stream_offset: 270532608, stream_size: 2097152 [D 20:01:02.887685] dbpf_open_cache_get: called [D 20:01:02.887691] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.887698] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.887715] op_queue add: 0x2aaab1505ea0 [D 20:01:02.887723] delayed I/O operation 0x2aaab1505ea0 (16 already in progress) [D 20:01:02.887730] DBPF I/O ops in progress: 16 [D 20:01:02.887737] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.887743] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1505890. [D 20:01:02.887750] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.887757] ========================================================= [D 20:01:02.887764] PINT_process_request [D 20:01:02.887769] tiling 10 copies [D 20:01:02.887776] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.887783] lvl 0 el 0 blk 0 by 272629760 [D 20:01:02.887789] to 272629760 ta 268435456 fi 301989888 [D 20:01:02.887796] basic type or contiguous data [D 20:01:02.887802] Request disp [D 20:01:02.887807] contig_offset = 272629760 contig_size = 62914560 lvl_flag = 1 [D 20:01:02.887814] calling distribute [D 20:01:02.887820] PINT_distribute [D 20:01:02.887825] of 272629760 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:02.887833] begin iteration loff: 272629760 [D 20:01:02.887839] segment exceeds byte limit [D 20:01:02.887845] process a segment [D 20:01:02.887851] of 272629760 sz 2097152 [D 20:01:02.887857] add a segment 0 [D 20:01:02.887863] end iteration [D 20:01:02.887868] done with segments or bytes [D 20:01:02.887874] of 274726912 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.887881] next loff: 274726912 next poff: 274726912 [D 20:01:02.887887] return value 2097152 [D 20:01:02.887893] finished [D 20:01:02.887899] retval = 2097152 [D 20:01:02.887905] exiting distribute returned less than expected [D 20:01:02.887911] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 274726912 fo 301989888 eof 0 [D 20:01:02.887918] ========================================================= [D 20:01:02.887925] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.887932] dbpf_bstream_rw_list: mem_offset: 0x2aaab5c25010, mem_size: 2097152 [D 20:01:02.887938] dbpf_bstream_rw_list: stream_offset: 272629760, stream_size: 2097152 [D 20:01:02.887945] dbpf_open_cache_get: called [D 20:01:02.887951] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.887957] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.887976] op_queue add: 0x2aaab1505ff0 [D 20:01:02.887985] delayed I/O operation 0x2aaab1505ff0 (16 already in progress) [D 20:01:02.887991] DBPF I/O ops in progress: 16 [D 20:01:02.887997] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.888004] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1505890. [D 20:01:02.888011] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.888018] ========================================================= [D 20:01:02.888024] PINT_process_request [D 20:01:02.888030] tiling 10 copies [D 20:01:02.888036] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.888044] lvl 0 el 0 blk 0 by 274726912 [D 20:01:02.888050] to 274726912 ta 268435456 fi 301989888 [D 20:01:02.888056] basic type or contiguous data [D 20:01:02.888063] Request disp [D 20:01:02.888069] contig_offset = 274726912 contig_size = 60817408 lvl_flag = 1 [D 20:01:02.888075] calling distribute [D 20:01:02.888081] PINT_distribute [D 20:01:02.888087] of 274726912 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:02.888094] begin iteration loff: 274726912 [D 20:01:02.888100] segment exceeds byte limit [D 20:01:02.888106] process a segment [D 20:01:02.888112] of 274726912 sz 2097152 [D 20:01:02.888118] add a segment 0 [D 20:01:02.888124] end iteration [D 20:01:02.888129] done with segments or bytes [D 20:01:02.888135] of 276824064 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.888142] next loff: 276824064 next poff: 276824064 [D 20:01:02.888148] return value 2097152 [D 20:01:02.888154] finished [D 20:01:02.888160] retval = 2097152 [D 20:01:02.888166] exiting distribute returned less than expected [D 20:01:02.888172] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 276824064 fo 301989888 eof 0 [D 20:01:02.888179] ========================================================= [D 20:01:02.888186] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.888193] dbpf_bstream_rw_list: mem_offset: 0x2aaab521f010, mem_size: 2097152 [D 20:01:02.888199] dbpf_bstream_rw_list: stream_offset: 274726912, stream_size: 2097152 [D 20:01:02.888205] dbpf_open_cache_get: called [D 20:01:02.888211] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.888218] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.888233] op_queue add: 0x2aaab1506140 [D 20:01:02.888241] delayed I/O operation 0x2aaab1506140 (16 already in progress) [D 20:01:02.888248] DBPF I/O ops in progress: 16 [D 20:01:02.888254] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.888260] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1505890. [D 20:01:02.888268] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.888275] ========================================================= [D 20:01:02.888281] PINT_process_request [D 20:01:02.888287] tiling 10 copies [D 20:01:02.888293] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.888301] lvl 0 el 0 blk 0 by 276824064 [D 20:01:02.888307] to 276824064 ta 268435456 fi 301989888 [D 20:01:02.888313] basic type or contiguous data [D 20:01:02.888319] Request disp [D 20:01:02.888325] contig_offset = 276824064 contig_size = 58720256 lvl_flag = 1 [D 20:01:02.888332] calling distribute [D 20:01:02.888337] PINT_distribute [D 20:01:02.888343] of 276824064 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:02.888350] begin iteration loff: 276824064 [D 20:01:02.888357] segment exceeds byte limit [D 20:01:02.888363] process a segment [D 20:01:02.888368] of 276824064 sz 2097152 [D 20:01:02.888374] add a segment 0 [D 20:01:02.888380] end iteration [D 20:01:02.888386] done with segments or bytes [D 20:01:02.888392] of 278921216 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.888399] next loff: 278921216 next poff: 278921216 [D 20:01:02.888405] return value 2097152 [D 20:01:02.888411] finished [D 20:01:02.888417] retval = 2097152 [D 20:01:02.888422] exiting distribute returned less than expected [D 20:01:02.888428] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 278921216 fo 301989888 eof 0 [D 20:01:02.888436] ========================================================= [D 20:01:02.888443] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.888449] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf6d3010, mem_size: 2097152 [D 20:01:02.888456] dbpf_bstream_rw_list: stream_offset: 276824064, stream_size: 2097152 [D 20:01:02.888462] dbpf_open_cache_get: called [D 20:01:02.888468] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.888474] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:02.888497] op_queue add: 0x2aaab1506290 [D 20:01:02.888506] delayed I/O operation 0x2aaab1506290 (16 already in progress) [D 20:01:02.888513] DBPF I/O ops in progress: 16 [D 20:01:02.888519] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.888525] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1505890. [D 20:01:02.888533] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:02.888547] ========================================================= [D 20:01:02.888556] PINT_process_request [D 20:01:02.888576] tiling 10 copies [D 20:01:02.888582] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:02.888590] lvl 0 el 0 blk 0 by 278921216 [D 20:01:02.888596] to 278921216 ta 268435456 fi 301989888 [D 20:01:02.888603] basic type or contiguous data [D 20:01:02.888609] Request disp [D 20:01:02.888615] contig_offset = 278921216 contig_size = 56623104 lvl_flag = 1 [D 20:01:02.888622] calling distribute [D 20:01:02.888629] PINT_distribute [D 20:01:02.888635] of 278921216 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:02.888643] begin iteration loff: 278921216 [D 20:01:02.888649] segment exceeds byte limit [D 20:01:02.888655] process a segment [D 20:01:02.888661] of 278921216 sz 2097152 [D 20:01:02.888667] add a segment 0 [D 20:01:02.888673] end iteration [D 20:01:02.888679] done with segments or bytes [D 20:01:02.888685] of 281018368 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:02.888693] next loff: 281018368 next poff: 281018368 [D 20:01:02.888699] return value 2097152 [D 20:01:02.888705] finished [D 20:01:02.888711] retval = 2097152 [D 20:01:02.888717] exiting distribute returned less than expected [D 20:01:02.888723] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 281018368 fo 301989888 eof 0 [D 20:01:02.888731] ========================================================= [D 20:01:02.888739] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:02.888746] dbpf_bstream_rw_list: mem_offset: 0x2aaaadac5010, mem_size: 2097152 [D 20:01:02.888753] dbpf_bstream_rw_list: stream_offset: 278921216, stream_size: 2097152 [D 20:01:02.888759] dbpf_open_cache_get: called [D 20:01:02.888548] dbpf_open_cache_put: cached entry. [D 20:01:02.888774] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.888784] op_queue add: 0x2aaab1522b60 [D 20:01:02.888801] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:02.888810] BMI_post_send: addr: 6843072, offset: 0x2aaab9a45010, size: 2097152, tag: 27 [D 20:01:02.888821] post_sr: 10.1.4.56:39990 bh 6 len 32 wr_id 6829c0. [D 20:01:02.888836] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:02.888850] DBPF I/O ops in progress: 15 [D 20:01:02.888858] starting delayed I/O operation 0x2aaab152b470 (15 in progress) [D 20:01:02.888869] lio_listio called with 1 following aiocbs: [D 20:01:02.888876] aiocb_ptr_array[0]: fd: 13, off: 180355072, bytes: 2097152, buf: 0x2aaaba44a010, type: 0 [D 20:01:02.888886] start_delayed_ops_if_any: lio_listio posted 0x2aaab152b470 (handle 2863311531, ret 0)) [D 20:01:02.888910] dbpf_open_cache_put: cached entry. [D 20:01:02.888924] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.888932] op_queue add: 0x2aaab152bf70 [D 20:01:02.888948] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:02.888956] BMI_post_send: addr: 6843072, offset: 0x2aaab7a35010, size: 2097152, tag: 27 [D 20:01:02.888965] post_sr: 10.1.4.56:39990 bh 7 len 32 wr_id 6829f0. [D 20:01:02.888975] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:02.888988] DBPF I/O ops in progress: 15 [D 20:01:02.888997] starting delayed I/O operation 0x2aaab152b210 (15 in progress) [D 20:01:02.889006] lio_listio called with 1 following aiocbs: [D 20:01:02.889013] aiocb_ptr_array[0]: fd: 13, off: 182452224, bytes: 2097152, buf: 0x2aaab843a010, type: 0 [D 20:01:04.209801] start_delayed_ops_if_any: lio_listio posted 0x2aaab152b210 (handle 2863311531, ret 0)) [D 20:01:02.889025] dbpf_open_cache_put: cached entry. [D 20:01:04.209991] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.892600] --- aio_progress_notification called with handle 2863311531 (0x2aaab152c600) [D 20:01:04.209993] dbpf_open_cache_put: cached entry. [D 20:01:04.210006] op_queue add: 0x2aaab152c0c0 [D 20:01:04.210037] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:02.892639] --- aio_progress_notification called with handle 2863311531 (0x2aaab152c360) [D 20:01:04.210055] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.210072] BMI_post_send: addr: 6843072, offset: 0x2aaab5e26010, size: 2097152, tag: 27 [D 20:01:04.210074] dbpf_open_cache_put: cached entry. [D 20:01:04.210089] post_sr: 10.1.4.56:39990 bh 8 len 32 wr_id 682a20. [D 20:01:04.210094] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.210059] op_queue add: 0x2aaabd310e50 [D 20:01:04.210112] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:02.892648] --- aio_progress_notification called with handle 2863311531 (0x2aaab152c4b0) [D 20:01:04.210115] DBPF I/O ops in progress: 15 [D 20:01:04.210136] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.210137] starting delayed I/O operation 0x2aaab153ad80 (15 in progress) [D 20:01:04.210151] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.210163] lio_listio called with 1 following aiocbs: [D 20:01:04.210179] aiocb_ptr_array[0]: fd: 13, off: 201326592, bytes: 2097152, buf: 0x2aaab682c010, type: 0 [D 20:01:04.210185] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:02.892657] --- aio_progress_notification called with handle 2863311531 (0x2aaab152cbb0) [D 20:01:04.210188] start_delayed_ops_if_any: lio_listio posted 0x2aaab153ad80 (handle 2863311531, ret 0)) [D 20:01:04.210198] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.210117] op_queue add: 0x2aaab152c210 [D 20:01:02.900570] --- aio_progress_notification called with handle 2863311531 (0x2aaab152cfa0) [D 20:01:02.900575] --- aio_progress_notification called with handle 2863311531 (0x2aaab152d0f0) [D 20:01:04.210226] op_queue add: 0x2aaab15063e0 [D 20:01:04.210244] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210276] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:04.210287] dbpf_open_cache_put: cached entry. [D 20:01:04.210294] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210253] delayed I/O operation 0x2aaab15063e0 (16 already in progress) [D 20:01:04.210289] BMI_post_send: addr: 6843072, offset: 0x2aaabf611010, size: 2097152, tag: 25 [D 20:01:04.210316] DBPF I/O ops in progress: 16 [D 20:01:04.210326] post_sr: 10.1.4.56:39990 bh 9 len 32 wr_id 682a50. [D 20:01:04.210299] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.210345] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:02.896574] --- aio_progress_notification called with handle 2863311531 (0x2aaab152ce50) [D 20:01:04.210353] op_queue add: 0x2aaab152d0f0 [D 20:01:04.210358] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.210380] BMI_post_send: addr: 6843072, offset: 0x2aaab3e15010, size: 2097152, tag: 27 [D 20:01:04.210389] post_sr: 10.1.4.56:39990 bh 10 len 32 wr_id 682a80. [D 20:01:04.210398] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.210407] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.210362] op_queue add: 0x2aaabd300a20 [D 20:01:04.210427] delayed I/O operation 0x2aaabd300a20 (16 already in progress) [D 20:01:04.210433] DBPF I/O ops in progress: 16 [D 20:01:02.896655] --- aio_progress_notification called with handle 2863311531 (0x2aaab152cd00) [D 20:01:04.210335] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:02.908252] --- aio_progress_notification called with handle 2863311531 (0x2aaab1525840) [D 20:01:02.908299] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b470) [D 20:01:04.210443] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:04.210455] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210468] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210016] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:02.904598] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300cc0) [D 20:01:04.210456] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1505890. [D 20:01:02.904686] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301a70) [D 20:01:04.210464] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210524] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210490] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210443] DBPF I/O ops in progress: 15 [D 20:01:04.210493] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210474] (send callback time) ini posts: 0, pending: 1, last: 1, src_list emtpy: yes [D 20:01:04.210581] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:04.210589] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:04.210596] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:04.210604] (send callback time) ini posts: 0, pending: 3, last: 1, src_list emtpy: yes [D 20:01:04.210611] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:04.210618] (send callback time) ini posts: 0, pending: 2, last: 1, src_list emtpy: yes [D 20:01:04.210630] openib_check_cq: completed send wr_id 682960. [D 20:01:04.210638] openib_check_cq: completed send wr_id 682990. [D 20:01:04.210646] openib_check_cq: completed recv wr_id 684980. [D 20:01:04.210655] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec2a0 to 10.1.4.56:39990 remote addr a4051000 rkey 108c27 nic_wr_credit 1. [D 20:01:04.210663] openib_post_rr: 10.1.4.56:39990 bh 75 wr_id 684980. [D 20:01:04.210671] openib_check_cq: completed recv wr_id 6849b0. [D 20:01:04.210679] openib_post_sr_rdmaw: ibv_post_send wr_id 6aad30 to 10.1.4.56:39990 remote addr 9fa4d000 rkey 108e27 nic_wr_credit 2. [D 20:01:04.210687] openib_post_rr: 10.1.4.56:39990 bh 76 wr_id 6849b0. [D 20:01:04.210694] openib_check_cq: completed send wr_id 6829c0. [D 20:01:04.210701] openib_check_cq: completed send wr_id 6829f0. [D 20:01:04.210707] openib_check_cq: completed recv wr_id 6849e0. [D 20:01:04.210715] openib_post_sr_rdmaw: ibv_post_send wr_id 668920 to 10.1.4.56:39990 remote addr 9a649000 rkey 109126 nic_wr_credit 3. [D 20:01:04.210723] openib_post_rr: 10.1.4.56:39990 bh 77 wr_id 6849e0. [D 20:01:04.210730] openib_check_cq: completed recv wr_id 684a10. [D 20:01:04.210736] openib_post_sr_rdmaw: ibv_post_send wr_id 6ed4e0 to 10.1.4.56:39990 remote addr 9cc4b000 rkey 108f27 nic_wr_credit 4. [D 20:01:04.210744] openib_post_rr: 10.1.4.56:39990 bh 78 wr_id 684a10. [D 20:01:04.210751] openib_check_cq: completed recv wr_id 684a40. [D 20:01:04.210759] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec680 to 10.1.4.56:39990 remote addr 8a237000 rkey 108a27 nic_wr_credit 5. [D 20:01:04.210766] openib_post_rr: 10.1.4.56:39990 bh 79 wr_id 684a40. [D 20:01:04.210774] openib_check_cq: completed recv wr_id 684a70. [D 20:01:04.210781] openib_post_sr_rdmaw: ibv_post_send wr_id 6eda80 to 10.1.4.56:39990 remote addr a1e4f000 rkey 108d27 nic_wr_credit 6. [D 20:01:04.210788] openib_post_rr: 10.1.4.56:39990 bh 80 wr_id 684a70. [D 20:01:04.210795] openib_check_cq: completed send wr_id 682a20. [D 20:01:04.210802] openib_check_cq: completed send wr_id 682a50. [D 20:01:04.210808] openib_check_cq: completed send wr_id 682a80. [D 20:01:04.210820] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:04.210829] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [D 20:01:04.210836] ========================================================= [D 20:01:04.210843] PINT_process_request [D 20:01:04.210849] tiling 6 copies [D 20:01:04.210856] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.210864] lvl 0 el 0 blk 0 by 150994944 [D 20:01:04.210870] to 150994944 ta 134217728 fi 167772160 [D 20:01:04.210877] basic type or contiguous data [D 20:01:04.210884] Request disp [D 20:01:04.210890] contig_offset = 150994944 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.210897] calling distribute [D 20:01:04.210903] PINT_distribute [D 20:01:04.210909] of 150994944 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.210917] begin iteration loff: 150994944 [D 20:01:04.210924] segment exceeds byte limit [D 20:01:04.210930] process a segment [D 20:01:04.210936] of 150994944 sz 2097152 [D 20:01:04.210943] add a segment 0 [D 20:01:04.210949] end iteration [D 20:01:04.210955] done with segments or bytes [D 20:01:04.210961] of 153092096 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.210968] next loff: 153092096 next poff: 153092096 [D 20:01:04.210974] return value 2097152 [D 20:01:04.210980] finished [D 20:01:04.210986] retval = 2097152 [D 20:01:04.210992] exiting distribute returned less than expected [D 20:01:04.210998] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 153092096 fo 167772160 eof 0 [D 20:01:04.211006] ========================================================= [D 20:01:04.211016] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.211023] dbpf_bstream_rw_list: mem_offset: 0x2aaac0a1b010, mem_size: 2097152 [D 20:01:04.211030] dbpf_bstream_rw_list: stream_offset: 150994944, stream_size: 2097152 [D 20:01:04.211037] dbpf_open_cache_get: called [D 20:01:04.210518] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.210364] dbpf_open_cache_put: cached entry. [D 20:01:04.210559] starting delayed I/O operation 0x2aaab153aed0 (15 in progress) [D 20:01:04.211071] dbpf_open_cache_put: cached entry. [D 20:01:04.211080] lio_listio called with 1 following aiocbs: [D 20:01:04.210237] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.211083] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211115] dbpf_open_cache_put: cached entry. [D 20:01:04.211127] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211147] dbpf_open_cache_put: cached entry. [D 20:01:04.211156] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211160] dbpf_open_cache_put: cached entry. [D 20:01:04.211172] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.210522] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.211176] dbpf_open_cache_put: cached entry. [D 20:01:04.211118] op_queue add: 0x2aaab152cd00 [D 20:01:04.211202] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211211] op_queue add: 0x2aaab1525840 [D 20:01:04.211218] dbpf_open_cache_put: cached entry. [D 20:01:04.211224] op_queue add: 0x2aaab152cfa0 [D 20:01:04.210447] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.211245] dbpf_open_cache_put: cached entry. [D 20:01:04.211231] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211264] dbpf_open_cache_put: cached entry. [D 20:01:04.211292] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211196] ========================================================= [D 20:01:04.211311] PINT_process_request [D 20:01:04.211294] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.211322] tiling 10 copies [D 20:01:04.211341] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.211351] lvl 0 el 0 blk 0 by 281018368 [D 20:01:04.211358] to 281018368 ta 268435456 fi 301989888 [D 20:01:04.211364] basic type or contiguous data [D 20:01:04.211371] Request disp [D 20:01:04.211377] contig_offset = 281018368 contig_size = 54525952 lvl_flag = 1 [D 20:01:04.211384] calling distribute [D 20:01:04.211390] PINT_distribute [D 20:01:04.211396] of 281018368 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.211405] begin iteration loff: 281018368 [D 20:01:04.211411] segment exceeds byte limit [D 20:01:04.211417] process a segment [D 20:01:04.211423] of 281018368 sz 2097152 [D 20:01:04.211430] add a segment 0 [D 20:01:04.211436] end iteration [D 20:01:04.211442] done with segments or bytes [D 20:01:04.211448] of 283115520 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.211455] next loff: 283115520 next poff: 283115520 [D 20:01:04.211462] return value 2097152 [D 20:01:04.211468] finished [D 20:01:04.211474] retval = 2097152 [D 20:01:04.211480] exiting distribute returned less than expected [D 20:01:04.211487] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 283115520 fo 301989888 eof 0 [D 20:01:04.211494] ========================================================= [D 20:01:04.211502] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.211510] dbpf_bstream_rw_list: mem_offset: 0x2aaabae50010, mem_size: 2097152 [D 20:01:04.211517] dbpf_bstream_rw_list: stream_offset: 281018368, stream_size: 2097152 [D 20:01:04.211524] dbpf_open_cache_get: called [D 20:01:04.211262] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211264] op_queue add: 0x2aaab152b470 [D 20:01:04.211332] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.211092] aiocb_ptr_array[0]: fd: 13, off: 203423744, bytes: 2097152, buf: 0x2aaab481a010, type: 0 [D 20:01:04.211068] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211578] start_delayed_ops_if_any: lio_listio posted 0x2aaab153aed0 (handle 2863311531, ret 0)) [D 20:01:04.211582] op_queue add: 0x2aaab152cbb0 [D 20:01:04.211594] DBPF I/O ops in progress: 15 [D 20:01:04.211601] starting delayed I/O operation 0x2aaab153b020 (15 in progress) [D 20:01:04.211610] lio_listio called with 1 following aiocbs: [D 20:01:04.211616] aiocb_ptr_array[0]: fd: 13, off: 205520896, bytes: 2097152, buf: 0x2aaab2c0b010, type: 0 [D 20:01:04.211625] start_delayed_ops_if_any: lio_listio posted 0x2aaab153b020 (handle 2863311531, ret 0)) [D 20:01:04.211642] op_queue add: 0x2aaab152c600 [D 20:01:04.211657] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.211665] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.211672] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.211679] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.211686] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.211693] BMI_post_send: addr: 6843072, offset: 0x2aaab4418010, size: 2097152, tag: 28 [D 20:01:04.211702] post_sr: 10.1.4.56:39990 bh 11 len 32 wr_id 682ab0. [D 20:01:04.211713] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.211721] BMI_post_send: addr: 6843072, offset: 0x2aaab2809010, size: 2097152, tag: 28 [D 20:01:04.211729] post_sr: 10.1.4.56:39990 bh 12 len 32 wr_id 682ae0. [D 20:01:04.211738] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:04.211749] DBPF I/O ops in progress: 15 [D 20:01:04.211757] starting delayed I/O operation 0x2aaab152d4d0 (15 in progress) [D 20:01:04.211770] lio_listio called with 1 following aiocbs: [D 20:01:04.211777] aiocb_ptr_array[0]: fd: 13, off: 207618048, bytes: 2097152, buf: 0x2aaab0edf010, type: 0 [D 20:01:04.211785] start_delayed_ops_if_any: lio_listio posted 0x2aaab152d4d0 (handle 2863311531, ret 0)) [D 20:01:04.211815] dbpf_open_cache_put: cached entry. [D 20:01:04.211825] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.211832] op_queue add: 0x2aaabd301a70 [D 20:01:04.211839] DBPF I/O ops in progress: 15 [D 20:01:04.211846] starting delayed I/O operation 0x2aaab152d620 (15 in progress) [D 20:01:04.211854] lio_listio called with 1 following aiocbs: [D 20:01:04.211861] aiocb_ptr_array[0]: fd: 13, off: 209715200, bytes: 2097152, buf: 0x2aaaaf2d1010, type: 0 [D 20:01:04.211869] start_delayed_ops_if_any: lio_listio posted 0x2aaab152d620 (handle 2863311531, ret 0)) [D 20:01:04.211891] DBPF I/O ops in progress: 15 [D 20:01:04.211900] starting delayed I/O operation 0x2aaab152d770 (15 in progress) [D 20:01:04.211908] lio_listio called with 1 following aiocbs: [D 20:01:04.211914] aiocb_ptr_array[0]: fd: 13, off: 211812352, bytes: 2097152, buf: 0x2aaaad6c3010, type: 0 [D 20:01:04.211923] start_delayed_ops_if_any: lio_listio posted 0x2aaab152d770 (handle 2863311531, ret 0)) [D 20:01:04.211944] op_queue add: 0x2aaabd300cc0 [D 20:01:04.211955] DBPF I/O ops in progress: 15 [D 20:01:04.211957] op_queue add: 0x2aaab152c4b0 [D 20:01:04.211964] starting delayed I/O operation 0x2aaab152d8c0 (15 in progress) [D 20:01:04.211979] op_queue add: 0x2aaab152c360 [D 20:01:04.211967] dbpf_open_cache_put: cached entry. [D 20:01:04.211990] lio_listio called with 1 following aiocbs: [D 20:01:04.212008] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.212034] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.212002] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.212018] aiocb_ptr_array[0]: fd: 13, off: 213909504, bytes: 2097152, buf: 0x2aaaba84c010, type: 0 [D 20:01:04.212060] op_queue add: 0x2aaab152ce50 [D 20:01:04.212072] start_delayed_ops_if_any: lio_listio posted 0x2aaab152d8c0 (handle 2863311531, ret 0)) [D 20:01:04.212118] DBPF I/O ops in progress: 15 [D 20:01:04.212128] starting delayed I/O operation 0x2aaab152da10 (15 in progress) [D 20:01:04.212138] lio_listio called with 1 following aiocbs: [D 20:01:04.212145] aiocb_ptr_array[0]: fd: 13, off: 216006656, bytes: 2097152, buf: 0x2aaab883c010, type: 0 [D 20:01:04.212154] start_delayed_ops_if_any: lio_listio posted 0x2aaab152da10 (handle 2863311531, ret 0)) [D 20:01:04.212186] DBPF I/O ops in progress: 15 [D 20:01:04.212196] starting delayed I/O operation 0x2aaab152de30 (15 in progress) [D 20:01:04.212207] lio_listio called with 1 following aiocbs: [D 20:01:04.212214] aiocb_ptr_array[0]: fd: 13, off: 234881024, bytes: 2097152, buf: 0x2aaab6c2e010, type: 0 [D 20:01:04.212223] start_delayed_ops_if_any: lio_listio posted 0x2aaab152de30 (handle 2863311531, ret 0)) [D 20:01:04.212252] DBPF I/O ops in progress: 15 [D 20:01:04.212262] starting delayed I/O operation 0x2aaab1530a60 (15 in progress) [D 20:01:04.212272] lio_listio called with 1 following aiocbs: [D 20:01:04.212279] aiocb_ptr_array[0]: fd: 13, off: 236978176, bytes: 2097152, buf: 0x2aaab4c1c010, type: 0 [D 20:01:04.212287] start_delayed_ops_if_any: lio_listio posted 0x2aaab1530a60 (handle 2863311531, ret 0)) [D 20:01:04.212302] op_queue add: 0x2aaabd300e10 [D 20:01:04.212313] delayed I/O operation 0x2aaabd300e10 (16 already in progress) [D 20:01:04.212320] DBPF I/O ops in progress: 16 [D 20:01:04.212331] DBPF I/O ops in progress: 15 [D 20:01:04.212342] starting delayed I/O operation 0x2aaabd3008d0 (15 in progress) [D 20:01:04.212351] lio_listio called with 1 following aiocbs: [D 20:01:04.212357] aiocb_ptr_array[0]: fd: 13, off: 119537664, bytes: 2097152, buf: 0x2aaabfc14010, type: 0 [D 20:01:04.212366] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3008d0 (handle 2863311531, ret 0)) [D 20:01:04.212388] DBPF I/O ops in progress: 15 [D 20:01:04.212397] starting delayed I/O operation 0x2aaab152bb90 (15 in progress) [D 20:01:04.212405] lio_listio called with 1 following aiocbs: [D 20:01:04.212412] aiocb_ptr_array[0]: fd: 13, off: 239075328, bytes: 2097152, buf: 0x2aaab300e010, type: 0 [D 20:01:04.212420] start_delayed_ops_if_any: lio_listio posted 0x2aaab152bb90 (handle 2863311531, ret 0)) [D 20:01:04.212431] DBPF I/O ops in progress: 15 [D 20:01:04.212438] starting delayed I/O operation 0x2aaabd310fa0 (15 in progress) [D 20:01:04.212446] lio_listio called with 1 following aiocbs: [D 20:01:04.212453] aiocb_ptr_array[0]: fd: 13, off: 121634816, bytes: 2097152, buf: 0x2aaabfe15010, type: 0 [D 20:01:04.212461] start_delayed_ops_if_any: lio_listio posted 0x2aaabd310fa0 (handle 2863311531, ret 0)) [D 20:01:04.212494] DBPF I/O ops in progress: 15 [D 20:01:04.212503] starting delayed I/O operation 0x2aaab151fd00 (15 in progress) [D 20:01:04.212512] lio_listio called with 1 following aiocbs: [D 20:01:04.212518] aiocb_ptr_array[0]: fd: 13, off: 241172480, bytes: 2097152, buf: 0x2aaab12e1010, type: 0 [D 20:01:04.212527] start_delayed_ops_if_any: lio_listio posted 0x2aaab151fd00 (handle 2863311531, ret 0)) [D 20:01:04.212537] op_queue add: 0x2aaab1522b60 [D 20:01:04.212544] delayed I/O operation 0x2aaab1522b60 (16 already in progress) [D 20:01:04.212550] DBPF I/O ops in progress: 16 [D 20:01:04.212558] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.212565] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1505890. [D 20:01:04.212578] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.212587] ========================================================= [D 20:01:04.212593] PINT_process_request [D 20:01:04.212599] tiling 10 copies [D 20:01:04.212605] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.212613] lvl 0 el 0 blk 0 by 283115520 [D 20:01:04.212619] to 283115520 ta 268435456 fi 301989888 [D 20:01:04.212625] basic type or contiguous data [D 20:01:04.212631] Request disp [D 20:01:04.212637] contig_offset = 283115520 contig_size = 52428800 lvl_flag = 1 [D 20:01:04.212644] calling distribute [D 20:01:04.212649] PINT_distribute [D 20:01:04.212655] of 283115520 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.212663] begin iteration loff: 283115520 [D 20:01:04.212669] segment exceeds byte limit [D 20:01:04.212675] process a segment [D 20:01:04.212680] of 283115520 sz 2097152 [D 20:01:04.212686] add a segment 0 [D 20:01:04.212692] end iteration [D 20:01:04.212698] done with segments or bytes [D 20:01:04.212704] of 285212672 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.212711] next loff: 285212672 next poff: 285212672 [D 20:01:04.212717] return value 2097152 [D 20:01:04.212723] finished [D 20:01:04.212728] retval = 2097152 [D 20:01:04.212734] exiting distribute returned less than expected [D 20:01:04.212740] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 285212672 fo 301989888 eof 0 [D 20:01:04.212748] ========================================================= [D 20:01:04.212755] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.212761] dbpf_bstream_rw_list: mem_offset: 0x2aaab9040010, mem_size: 2097152 [D 20:01:04.212768] dbpf_bstream_rw_list: stream_offset: 283115520, stream_size: 2097152 [D 20:01:04.212775] dbpf_open_cache_get: called [D 20:01:04.212781] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.212787] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.212800] op_queue add: 0x2aaab1525840 [D 20:01:04.212806] delayed I/O operation 0x2aaab1525840 (16 already in progress) [D 20:01:04.212813] DBPF I/O ops in progress: 16 [D 20:01:04.212819] flowproto posted 0x2aaab1505890 [D 20:01:04.212826] Job flows in progress (post time): 10 [D 20:01:04.212837] [SM Exiting]: (0x668a50) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:04.212847] [SM Entering]: (0x669d10) io_sm:start_flow (status: 0) [D 20:01:04.212854] [SM frame get]: (0x669d10) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.212862] io_start_flow() issuing flow to read data. [D 20:01:04.212869] flow: fsize: 476053504, server_nr: 0, server_ct: 1 [D 20:01:04.212876] file_req_offset: 301989888, aggregate_size: 33554432, handle: 2863311531 [D 20:01:04.212886] PINT_new_request_states n=1 [D 20:01:04.212906] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.212895] flowproto posting 0x2aaab152b470 [D 20:01:04.212921] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:04.212929] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.212894] openib_check_cq: completed rdma wr_id 6ec2a0, nic_wr_credit 5. [D 20:01:04.212931] BMI_post_send: addr: 6843072, offset: 0x2aaabfa13010, size: 2097152, tag: 26 [D 20:01:04.212945] post_sr: 10.1.4.56:39990 bh 13 len 16 wr_id 682b10. [D 20:01:04.212953] DBPF I/O ops in progress: 15 [D 20:01:04.212940] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b470. [D 20:01:04.212965] post_sr: 10.1.4.56:39990 bh 14 len 32 wr_id 682b40. [D 20:01:04.212980] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:04.212997] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.213008] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b630. [D 20:01:04.213016] BMI_post_send: addr: 6843072, offset: 0x2aaabf812010, size: 2097152, tag: 25 [D 20:01:04.212974] starting delayed I/O operation 0x2aaab1520020 (15 in progress) [D 20:01:04.213001] ========================================================= [D 20:01:04.213025] post_sr: 10.1.4.56:39990 bh 15 len 32 wr_id 682b70. [D 20:01:04.213052] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 1 [D 20:01:04.213061] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.213068] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.213075] BMI_post_send: addr: 6843072, offset: 0x2aaaba048010, size: 2097152, tag: 27 [D 20:01:04.213040] PINT_process_request [D 20:01:04.213083] post_sr: 10.1.4.56:39990 bh 16 len 32 wr_id 682ba0. [D 20:01:04.213087] tiling 11 copies [D 20:01:04.213096] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.215210] BMI_post_send: addr: 6843072, offset: 0x2aaab8038010, size: 2097152, tag: 27 [D 20:01:04.213105] lio_listio called with 1 following aiocbs: [D 20:01:04.215125] openib_check_cq: completed rdma wr_id 6aad30, nic_wr_credit 4. [D 20:01:04.215192] skipping ahead to target_offset [D 20:01:04.215236] aiocb_ptr_array[0]: fd: 13, off: 243269632, bytes: 2097152, buf: 0x2aaabc85d010, type: 0 [D 20:01:04.215243] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.215250] start_delayed_ops_if_any: lio_listio posted 0x2aaab1520020 (handle 2863311531, ret 0)) [D 20:01:04.215254] lvl 0 el 0 blk 0 by 0 [D 20:01:04.215271] to 0 ta 301989888 fi 335544320 [D 20:01:04.215279] basic type or contiguous data [D 20:01:04.215285] Request disp [D 20:01:04.215292] contig_offset = 0 contig_size = 369098752 lvl_flag = 1 [D 20:01:04.215299] process logical skip [D 20:01:04.215305] retval = 301989888 [D 20:01:04.215311] exiting logical skip because distribute indicates done [D 20:01:04.215317] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.215325] lvl 0 el 0 blk 0 by 301989888 [D 20:01:04.215332] to 301989888 ta 301989888 fi 335544320 [D 20:01:04.215338] basic type or contiguous data [D 20:01:04.215344] Request disp [D 20:01:04.215350] contig_offset = 301989888 contig_size = 67108864 lvl_flag = 1 [D 20:01:04.215357] calling distribute [D 20:01:04.215364] PINT_distribute [D 20:01:04.215371] of 301989888 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.215381] begin iteration loff: 301989888 [D 20:01:04.215388] segment exceeds byte limit [D 20:01:04.215394] process a segment [D 20:01:04.215400] of 301989888 sz 2097152 [D 20:01:04.215407] add a segment 0 [D 20:01:04.215413] end iteration [D 20:01:04.215422] done with segments or bytes [D 20:01:04.215429] of 304087040 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.215436] next loff: 304087040 next poff: 304087040 [D 20:01:04.215442] return value 2097152 [D 20:01:04.215449] finished [D 20:01:04.215455] retval = 2097152 [D 20:01:04.215461] exiting distribute returned less than expected [D 20:01:04.215468] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 304087040 fo 335544320 eof 0 [D 20:01:04.215475] ========================================================= [D 20:01:04.215496] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.215505] dbpf_bstream_rw_list: mem_offset: 0x2aaab7231010, mem_size: 2097152 [D 20:01:04.215511] dbpf_bstream_rw_list: stream_offset: 301989888, stream_size: 2097152 [D 20:01:04.215519] dbpf_open_cache_get: called [D 20:01:04.215527] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.215534] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.215558] op_queue add: 0x2aaab152ce40 [D 20:01:04.215566] delayed I/O operation 0x2aaab152ce40 (16 already in progress) [D 20:01:04.215572] DBPF I/O ops in progress: 16 [D 20:01:04.215580] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.215587] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b470. [D 20:01:04.215596] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.215608] ========================================================= [D 20:01:04.215614] PINT_process_request [D 20:01:04.215622] tiling 11 copies [D 20:01:04.215629] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.215636] lvl 0 el 0 blk 0 by 304087040 [D 20:01:04.215642] to 304087040 ta 301989888 fi 335544320 [D 20:01:04.215649] basic type or contiguous data [D 20:01:04.215655] Request disp [D 20:01:04.215661] contig_offset = 304087040 contig_size = 65011712 lvl_flag = 1 [D 20:01:04.215668] calling distribute [D 20:01:04.215674] PINT_distribute [D 20:01:04.215681] of 304087040 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.215688] begin iteration loff: 304087040 [D 20:01:04.215694] segment exceeds byte limit [D 20:01:04.215701] process a segment [D 20:01:04.215706] of 304087040 sz 2097152 [D 20:01:04.215713] add a segment 0 [D 20:01:04.215719] end iteration [D 20:01:04.215725] done with segments or bytes [D 20:01:04.215731] of 306184192 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.215738] next loff: 306184192 next poff: 306184192 [D 20:01:04.215744] return value 2097152 [D 20:01:04.215751] finished [D 20:01:04.215756] retval = 2097152 [D 20:01:04.215762] exiting distribute returned less than expected [D 20:01:04.215769] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 306184192 fo 335544320 eof 0 [D 20:01:04.215777] ========================================================= [D 20:01:04.215785] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.215791] dbpf_bstream_rw_list: mem_offset: 0x2aaab5420010, mem_size: 2097152 [D 20:01:04.215798] dbpf_bstream_rw_list: stream_offset: 304087040, stream_size: 2097152 [D 20:01:04.215805] dbpf_open_cache_get: called [D 20:01:04.215811] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.215817] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.215836] op_queue add: 0x2aaab152cf90 [D 20:01:04.215844] delayed I/O operation 0x2aaab152cf90 (16 already in progress) [D 20:01:04.215851] DBPF I/O ops in progress: 16 [D 20:01:04.215858] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.215864] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b470. [D 20:01:04.215872] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.215880] ========================================================= [D 20:01:04.215886] PINT_process_request [D 20:01:04.215892] tiling 11 copies [D 20:01:04.215899] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.215907] lvl 0 el 0 blk 0 by 306184192 [D 20:01:04.215913] to 306184192 ta 301989888 fi 335544320 [D 20:01:04.215920] basic type or contiguous data [D 20:01:04.215926] Request disp [D 20:01:04.215932] contig_offset = 306184192 contig_size = 62914560 lvl_flag = 1 [D 20:01:04.215939] calling distribute [D 20:01:04.215945] PINT_distribute [D 20:01:04.215951] of 306184192 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.215959] begin iteration loff: 306184192 [D 20:01:04.215965] segment exceeds byte limit [D 20:01:04.215971] process a segment [D 20:01:04.215976] of 306184192 sz 2097152 [D 20:01:04.215983] add a segment 0 [D 20:01:04.215989] end iteration [D 20:01:04.215995] done with segments or bytes [D 20:01:04.216000] of 308281344 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.216008] next loff: 308281344 next poff: 308281344 [D 20:01:04.216014] return value 2097152 [D 20:01:04.216020] finished [D 20:01:04.216026] retval = 2097152 [D 20:01:04.216031] exiting distribute returned less than expected [D 20:01:04.216038] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 308281344 fo 335544320 eof 0 [D 20:01:04.216045] ========================================================= [D 20:01:04.216052] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.216060] dbpf_bstream_rw_list: mem_offset: 0x2aaab3611010, mem_size: 2097152 [D 20:01:04.216066] dbpf_bstream_rw_list: stream_offset: 306184192, stream_size: 2097152 [D 20:01:04.216073] dbpf_open_cache_get: called [D 20:01:04.216079] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.216085] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.216102] op_queue add: 0x2aaab152d0e0 [D 20:01:04.216111] delayed I/O operation 0x2aaab152d0e0 (16 already in progress) [D 20:01:04.216118] DBPF I/O ops in progress: 16 [D 20:01:04.216124] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.216131] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b470. [D 20:01:04.216138] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.216147] ========================================================= [D 20:01:04.216153] PINT_process_request [D 20:01:04.216159] tiling 11 copies [D 20:01:04.216165] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.216172] lvl 0 el 0 blk 0 by 308281344 [D 20:01:04.216178] to 308281344 ta 301989888 fi 335544320 [D 20:01:04.216185] basic type or contiguous data [D 20:01:04.216191] Request disp [D 20:01:04.216197] contig_offset = 308281344 contig_size = 60817408 lvl_flag = 1 [D 20:01:04.216204] calling distribute [D 20:01:04.216210] PINT_distribute [D 20:01:04.216215] of 308281344 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.216223] begin iteration loff: 308281344 [D 20:01:04.216229] segment exceeds byte limit [D 20:01:04.216235] process a segment [D 20:01:04.216241] of 308281344 sz 2097152 [D 20:01:04.216247] add a segment 0 [D 20:01:04.216253] end iteration [D 20:01:04.216259] done with segments or bytes [D 20:01:04.216265] of 310378496 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.216272] next loff: 310378496 next poff: 310378496 [D 20:01:04.216278] return value 2097152 [D 20:01:04.216284] finished [D 20:01:04.216290] retval = 2097152 [D 20:01:04.216295] exiting distribute returned less than expected [D 20:01:04.216302] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 310378496 fo 335544320 eof 0 [D 20:01:04.216309] ========================================================= [D 20:01:04.216317] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.216324] dbpf_bstream_rw_list: mem_offset: 0x2aaab1a02010, mem_size: 2097152 [D 20:01:04.216331] dbpf_bstream_rw_list: stream_offset: 308281344, stream_size: 2097152 [D 20:01:04.216338] dbpf_open_cache_get: called [D 20:01:04.216345] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.216351] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.216372] op_queue add: 0x2aaab152bf70 [D 20:01:04.216379] delayed I/O operation 0x2aaab152bf70 (16 already in progress) [D 20:01:04.216386] DBPF I/O ops in progress: 16 [D 20:01:04.216392] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.216399] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b470. [D 20:01:04.216406] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.216414] ========================================================= [D 20:01:04.216420] PINT_process_request [D 20:01:04.216427] tiling 11 copies [D 20:01:04.216433] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.216440] lvl 0 el 0 blk 0 by 310378496 [D 20:01:04.216447] to 310378496 ta 301989888 fi 335544320 [D 20:01:04.216453] basic type or contiguous data [D 20:01:04.216459] Request disp [D 20:01:04.216465] contig_offset = 310378496 contig_size = 58720256 lvl_flag = 1 [D 20:01:04.216472] calling distribute [D 20:01:04.216478] PINT_distribute [D 20:01:04.216484] of 310378496 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.216491] begin iteration loff: 310378496 [D 20:01:04.216497] segment exceeds byte limit [D 20:01:04.216503] process a segment [D 20:01:04.216509] of 310378496 sz 2097152 [D 20:01:04.216515] add a segment 0 [D 20:01:04.216521] end iteration [D 20:01:04.216527] done with segments or bytes [D 20:01:04.216533] of 312475648 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.216539] next loff: 312475648 next poff: 312475648 [D 20:01:04.216546] return value 2097152 [D 20:01:04.216552] finished [D 20:01:04.216557] retval = 2097152 [D 20:01:04.216563] exiting distribute returned less than expected [D 20:01:04.216569] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 312475648 fo 335544320 eof 0 [D 20:01:04.216589] ========================================================= [D 20:01:04.216597] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.216604] dbpf_bstream_rw_list: mem_offset: 0x2aaaafcd6010, mem_size: 2097152 [D 20:01:04.216610] dbpf_bstream_rw_list: stream_offset: 310378496, stream_size: 2097152 [D 20:01:04.216617] dbpf_open_cache_get: called [D 20:01:04.216623] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.216629] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.216650] op_queue add: 0x2aaab1561b70 [D 20:01:04.216659] delayed I/O operation 0x2aaab1561b70 (16 already in progress) [D 20:01:04.216666] DBPF I/O ops in progress: 16 [D 20:01:04.216672] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.216679] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b470. [D 20:01:04.216686] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.216695] ========================================================= [D 20:01:04.216701] PINT_process_request [D 20:01:04.216707] tiling 11 copies [D 20:01:04.216713] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.216721] lvl 0 el 0 blk 0 by 312475648 [D 20:01:04.216727] to 312475648 ta 301989888 fi 335544320 [D 20:01:04.216734] basic type or contiguous data [D 20:01:04.216740] Request disp [D 20:01:04.216746] contig_offset = 312475648 contig_size = 56623104 lvl_flag = 1 [D 20:01:04.216753] calling distribute [D 20:01:04.216758] PINT_distribute [D 20:01:04.216764] of 312475648 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.216771] begin iteration loff: 312475648 [D 20:01:04.216778] segment exceeds byte limit [D 20:01:04.216784] process a segment [D 20:01:04.216790] of 312475648 sz 2097152 [D 20:01:04.216796] add a segment 0 [D 20:01:04.216802] end iteration [D 20:01:04.216808] done with segments or bytes [D 20:01:04.216815] of 314572800 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.216823] next loff: 314572800 next poff: 314572800 [D 20:01:04.216829] return value 2097152 [D 20:01:04.216835] finished [D 20:01:04.216841] retval = 2097152 [D 20:01:04.216847] exiting distribute returned less than expected [D 20:01:04.216853] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 314572800 fo 335544320 eof 0 [D 20:01:04.216860] ========================================================= [D 20:01:04.216868] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.216875] dbpf_bstream_rw_list: mem_offset: 0x2aaaae0c8010, mem_size: 2097152 [D 20:01:04.216882] dbpf_bstream_rw_list: stream_offset: 312475648, stream_size: 2097152 [D 20:01:04.216888] dbpf_open_cache_get: called [D 20:01:04.216894] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.216901] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.216921] op_queue add: 0x2aaab1561cc0 [D 20:01:04.216929] delayed I/O operation 0x2aaab1561cc0 (16 already in progress) [D 20:01:04.216935] DBPF I/O ops in progress: 16 [D 20:01:04.216942] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.216948] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b470. [D 20:01:04.216956] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.216963] ========================================================= [D 20:01:04.216970] PINT_process_request [D 20:01:04.216976] tiling 11 copies [D 20:01:04.216982] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.216989] lvl 0 el 0 blk 0 by 314572800 [D 20:01:04.216996] to 314572800 ta 301989888 fi 335544320 [D 20:01:04.217002] basic type or contiguous data [D 20:01:04.217008] Request disp [D 20:01:04.217015] contig_offset = 314572800 contig_size = 54525952 lvl_flag = 1 [D 20:01:04.217022] calling distribute [D 20:01:04.217027] PINT_distribute [D 20:01:04.217033] of 314572800 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.217041] begin iteration loff: 314572800 [D 20:01:04.217047] segment exceeds byte limit [D 20:01:04.217053] process a segment [D 20:01:04.217058] of 314572800 sz 2097152 [D 20:01:04.217064] add a segment 0 [D 20:01:04.217071] end iteration [D 20:01:04.217077] done with segments or bytes [D 20:01:04.217083] of 316669952 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.217090] next loff: 316669952 next poff: 316669952 [D 20:01:04.217096] return value 2097152 [D 20:01:04.217103] finished [D 20:01:04.217108] retval = 2097152 [D 20:01:04.217114] exiting distribute returned less than expected [D 20:01:04.217121] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 316669952 fo 335544320 eof 0 [D 20:01:04.217128] ========================================================= [D 20:01:04.217135] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.217142] dbpf_bstream_rw_list: mem_offset: 0x2aaabac4f010, mem_size: 2097152 [D 20:01:04.217149] dbpf_bstream_rw_list: stream_offset: 314572800, stream_size: 2097152 [D 20:01:04.217156] dbpf_open_cache_get: called [D 20:01:04.217162] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.217168] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.217186] op_queue add: 0x2aaab1561e10 [D 20:01:04.217193] delayed I/O operation 0x2aaab1561e10 (16 already in progress) [D 20:01:04.217200] DBPF I/O ops in progress: 16 [D 20:01:04.217207] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.217214] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152b470. [D 20:01:04.217221] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.217229] ========================================================= [D 20:01:04.217236] PINT_process_request [D 20:01:04.217242] tiling 11 copies [D 20:01:04.217248] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.217256] lvl 0 el 0 blk 0 by 316669952 [D 20:01:04.217263] to 316669952 ta 301989888 fi 335544320 [D 20:01:04.217269] basic type or contiguous data [D 20:01:04.217275] Request disp [D 20:01:04.217281] contig_offset = 316669952 contig_size = 52428800 lvl_flag = 1 [D 20:01:04.217288] calling distribute [D 20:01:04.217294] PINT_distribute [D 20:01:04.217299] of 316669952 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.217307] begin iteration loff: 316669952 [D 20:01:04.217313] segment exceeds byte limit [D 20:01:04.217319] process a segment [D 20:01:04.217324] of 316669952 sz 2097152 [D 20:01:04.217331] add a segment 0 [D 20:01:04.217337] end iteration [D 20:01:04.217342] done with segments or bytes [D 20:01:04.217348] of 318767104 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.217355] next loff: 318767104 next poff: 318767104 [D 20:01:04.217362] return value 2097152 [D 20:01:04.217368] finished [D 20:01:04.217374] retval = 2097152 [D 20:01:04.217379] exiting distribute returned less than expected [D 20:01:04.217385] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 318767104 fo 335544320 eof 0 [D 20:01:04.217393] ========================================================= [D 20:01:04.217401] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.217408] dbpf_bstream_rw_list: mem_offset: 0x2aaab8c3e010, mem_size: 2097152 [D 20:01:04.217415] dbpf_bstream_rw_list: stream_offset: 316669952, stream_size: 2097152 [D 20:01:04.217422] dbpf_open_cache_get: called [D 20:01:04.217428] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.217434] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.217456] op_queue add: 0x2aaab152c0c0 [D 20:01:04.217464] delayed I/O operation 0x2aaab152c0c0 (16 already in progress) [D 20:01:04.217470] DBPF I/O ops in progress: 16 [D 20:01:04.217477] flowproto posted 0x2aaab152b470 [D 20:01:04.217486] Job flows in progress (post time): 11 [D 20:01:04.217503] [SM Exiting]: (0x669d10) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:04.217518] [SM Entering]: (0x66afd0) io_sm:start_flow (status: 0) [D 20:01:04.217525] [SM frame get]: (0x66afd0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.217534] io_start_flow() issuing flow to read data. [D 20:01:04.217542] flow: fsize: 476053504, server_nr: 0, server_ct: 1 [D 20:01:04.217549] file_req_offset: 335544320, aggregate_size: 33554432, handle: 2863311531 [D 20:01:04.217564] PINT_new_request_states n=1 [D 20:01:04.217574] flowproto posting 0x2aaab152c210 [D 20:01:04.217590] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.217599] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c210. [D 20:01:04.217607] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:04.217615] ========================================================= [D 20:01:04.217621] PINT_process_request [D 20:01:04.217627] tiling 12 copies [D 20:01:04.217634] skipping ahead to target_offset [D 20:01:04.217640] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.217647] lvl 0 el 0 blk 0 by 0 [D 20:01:04.217654] to 0 ta 335544320 fi 369098752 [D 20:01:04.217660] basic type or contiguous data [D 20:01:04.217666] Request disp [D 20:01:04.217672] contig_offset = 0 contig_size = 402653184 lvl_flag = 1 [D 20:01:04.217679] process logical skip [D 20:01:04.217685] retval = 335544320 [D 20:01:04.217691] exiting logical skip because distribute indicates done [D 20:01:04.217697] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.217705] lvl 0 el 0 blk 0 by 335544320 [D 20:01:04.217711] to 335544320 ta 335544320 fi 369098752 [D 20:01:04.217717] basic type or contiguous data [D 20:01:04.217723] Request disp [D 20:01:04.217729] contig_offset = 335544320 contig_size = 67108864 lvl_flag = 1 [D 20:01:04.217736] calling distribute [D 20:01:04.217743] PINT_distribute [D 20:01:04.217749] of 335544320 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.217757] begin iteration loff: 335544320 [D 20:01:04.217763] segment exceeds byte limit [D 20:01:04.217769] process a segment [D 20:01:04.217775] of 335544320 sz 2097152 [D 20:01:04.217781] add a segment 0 [D 20:01:04.217787] end iteration [D 20:01:04.217793] done with segments or bytes [D 20:01:04.217798] of 337641472 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.217806] next loff: 337641472 next poff: 337641472 [D 20:01:04.217812] return value 2097152 [D 20:01:04.217818] finished [D 20:01:04.217825] retval = 2097152 [D 20:01:04.217830] exiting distribute returned less than expected [D 20:01:04.217837] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 337641472 fo 369098752 eof 0 [D 20:01:04.217844] ========================================================= [D 20:01:04.217852] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.217859] dbpf_bstream_rw_list: mem_offset: 0x2aaab7030010, mem_size: 2097152 [D 20:01:04.217866] dbpf_bstream_rw_list: stream_offset: 335544320, stream_size: 2097152 [D 20:01:04.217872] dbpf_open_cache_get: called [D 20:01:04.217879] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.217886] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.217908] op_queue add: 0x2aaab15067c0 [D 20:01:04.217916] delayed I/O operation 0x2aaab15067c0 (16 already in progress) [D 20:01:04.217923] DBPF I/O ops in progress: 16 [D 20:01:04.217929] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.217936] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c210. [D 20:01:04.217943] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.217951] ========================================================= [D 20:01:04.217958] PINT_process_request [D 20:01:04.217963] tiling 12 copies [D 20:01:04.217969] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.217977] lvl 0 el 0 blk 0 by 337641472 [D 20:01:04.217983] to 337641472 ta 335544320 fi 369098752 [D 20:01:04.217990] basic type or contiguous data [D 20:01:04.217996] Request disp [D 20:01:04.218002] contig_offset = 337641472 contig_size = 65011712 lvl_flag = 1 [D 20:01:04.218009] calling distribute [D 20:01:04.218015] PINT_distribute [D 20:01:04.218021] of 337641472 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.218028] begin iteration loff: 337641472 [D 20:01:04.218035] segment exceeds byte limit [D 20:01:04.218041] process a segment [D 20:01:04.218047] of 337641472 sz 2097152 [D 20:01:04.218053] add a segment 0 [D 20:01:04.218059] end iteration [D 20:01:04.218065] done with segments or bytes [D 20:01:04.218071] of 339738624 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.218078] next loff: 339738624 next poff: 339738624 [D 20:01:04.218084] return value 2097152 [D 20:01:04.218090] finished [D 20:01:04.218096] retval = 2097152 [D 20:01:04.218102] exiting distribute returned less than expected [D 20:01:04.218108] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 339738624 fo 369098752 eof 0 [D 20:01:04.218116] ========================================================= [D 20:01:04.218123] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.218130] dbpf_bstream_rw_list: mem_offset: 0x2aaab501e010, mem_size: 2097152 [D 20:01:04.218136] dbpf_bstream_rw_list: stream_offset: 337641472, stream_size: 2097152 [D 20:01:04.218143] dbpf_open_cache_get: called [D 20:01:04.218149] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.218155] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.218221] op_queue add: 0x2aaab1552390 [D 20:01:04.218244] delayed I/O operation 0x2aaab1552390 (16 already in progress) [D 20:01:04.218251] DBPF I/O ops in progress: 16 [D 20:01:04.218258] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.218265] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c210. [D 20:01:04.218272] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.218281] ========================================================= [D 20:01:04.218287] PINT_process_request [D 20:01:04.218294] tiling 12 copies [D 20:01:04.218300] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.218307] lvl 0 el 0 blk 0 by 339738624 [D 20:01:04.218313] to 339738624 ta 335544320 fi 369098752 [D 20:01:04.218320] basic type or contiguous data [D 20:01:04.218326] Request disp [D 20:01:04.218332] contig_offset = 339738624 contig_size = 62914560 lvl_flag = 1 [D 20:01:04.218339] calling distribute [D 20:01:04.218345] PINT_distribute [D 20:01:04.218351] of 339738624 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.218359] begin iteration loff: 339738624 [D 20:01:04.218365] segment exceeds byte limit [D 20:01:04.218371] process a segment [D 20:01:04.218377] of 339738624 sz 2097152 [D 20:01:04.218383] add a segment 0 [D 20:01:04.218389] end iteration [D 20:01:04.218395] done with segments or bytes [D 20:01:04.218401] of 341835776 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.218408] next loff: 341835776 next poff: 341835776 [D 20:01:04.218414] return value 2097152 [D 20:01:04.218420] finished [D 20:01:04.218426] retval = 2097152 [D 20:01:04.218432] exiting distribute returned less than expected [D 20:01:04.218438] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 341835776 fo 369098752 eof 0 [D 20:01:04.218445] ========================================================= [D 20:01:04.218453] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.218460] dbpf_bstream_rw_list: mem_offset: 0x2aaab3410010, mem_size: 2097152 [D 20:01:04.218466] dbpf_bstream_rw_list: stream_offset: 339738624, stream_size: 2097152 [D 20:01:04.218473] dbpf_open_cache_get: called [D 20:01:04.218479] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.218485] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.218509] op_queue add: 0x2aaab15524e0 [D 20:01:04.218521] delayed I/O operation 0x2aaab15524e0 (16 already in progress) [D 20:01:04.218527] DBPF I/O ops in progress: 16 [D 20:01:04.218534] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.218541] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c210. [D 20:01:04.218548] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.218556] ========================================================= [D 20:01:04.218563] PINT_process_request [D 20:01:04.218569] tiling 12 copies [D 20:01:04.218575] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.218583] lvl 0 el 0 blk 0 by 341835776 [D 20:01:04.218589] to 341835776 ta 335544320 fi 369098752 [D 20:01:04.218596] basic type or contiguous data [D 20:01:04.218602] Request disp [D 20:01:04.218608] contig_offset = 341835776 contig_size = 60817408 lvl_flag = 1 [D 20:01:04.218614] calling distribute [D 20:01:04.218620] PINT_distribute [D 20:01:04.218626] of 341835776 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.218633] begin iteration loff: 341835776 [D 20:01:04.218639] segment exceeds byte limit [D 20:01:04.218646] process a segment [D 20:01:04.218651] of 341835776 sz 2097152 [D 20:01:04.218658] add a segment 0 [D 20:01:04.218663] end iteration [D 20:01:04.218669] done with segments or bytes [D 20:01:04.218675] of 343932928 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.218682] next loff: 343932928 next poff: 343932928 [D 20:01:04.218688] return value 2097152 [D 20:01:04.218695] finished [D 20:01:04.218700] retval = 2097152 [D 20:01:04.218707] exiting distribute returned less than expected [D 20:01:04.218713] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 343932928 fo 369098752 eof 0 [D 20:01:04.218721] ========================================================= [D 20:01:04.218729] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.218736] dbpf_bstream_rw_list: mem_offset: 0x2aaab1801010, mem_size: 2097152 [D 20:01:04.218743] dbpf_bstream_rw_list: stream_offset: 341835776, stream_size: 2097152 [D 20:01:04.218749] dbpf_open_cache_get: called [D 20:01:04.218755] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.218762] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.218793] op_queue add: 0x2aaab1552630 [D 20:01:04.218803] delayed I/O operation 0x2aaab1552630 (16 already in progress) [D 20:01:04.218810] DBPF I/O ops in progress: 16 [D 20:01:04.218817] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.218824] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c210. [D 20:01:04.218831] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.218839] ========================================================= [D 20:01:04.218845] PINT_process_request [D 20:01:04.218851] tiling 12 copies [D 20:01:04.218858] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.218865] lvl 0 el 0 blk 0 by 343932928 [D 20:01:04.218871] to 343932928 ta 335544320 fi 369098752 [D 20:01:04.218877] basic type or contiguous data [D 20:01:04.218883] Request disp [D 20:01:04.218889] contig_offset = 343932928 contig_size = 58720256 lvl_flag = 1 [D 20:01:04.218896] calling distribute [D 20:01:04.218901] PINT_distribute [D 20:01:04.218907] of 343932928 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.218914] begin iteration loff: 343932928 [D 20:01:04.218921] segment exceeds byte limit [D 20:01:04.218927] process a segment [D 20:01:04.218933] of 343932928 sz 2097152 [D 20:01:04.218939] add a segment 0 [D 20:01:04.218945] end iteration [D 20:01:04.218951] done with segments or bytes [D 20:01:04.218957] of 346030080 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.218964] next loff: 346030080 next poff: 346030080 [D 20:01:04.218970] return value 2097152 [D 20:01:04.218976] finished [D 20:01:04.218982] retval = 2097152 [D 20:01:04.218988] exiting distribute returned less than expected [D 20:01:04.218994] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 346030080 fo 369098752 eof 0 [D 20:01:04.219002] ========================================================= [D 20:01:04.219009] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.219016] dbpf_bstream_rw_list: mem_offset: 0x2aaaafad5010, mem_size: 2097152 [D 20:01:04.219023] dbpf_bstream_rw_list: stream_offset: 343932928, stream_size: 2097152 [D 20:01:04.219029] dbpf_open_cache_get: called [D 20:01:04.219035] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.219042] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.219074] op_queue add: 0x2aaab1506910 [D 20:01:04.219084] delayed I/O operation 0x2aaab1506910 (16 already in progress) [D 20:01:04.219091] DBPF I/O ops in progress: 16 [D 20:01:04.219097] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.219103] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c210. [D 20:01:04.219111] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.219118] ========================================================= [D 20:01:04.219125] PINT_process_request [D 20:01:04.219130] tiling 12 copies [D 20:01:04.219137] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.219144] lvl 0 el 0 blk 0 by 346030080 [D 20:01:04.219150] to 346030080 ta 335544320 fi 369098752 [D 20:01:04.219157] basic type or contiguous data [D 20:01:04.219163] Request disp [D 20:01:04.219169] contig_offset = 346030080 contig_size = 56623104 lvl_flag = 1 [D 20:01:04.219176] calling distribute [D 20:01:04.219182] PINT_distribute [D 20:01:04.219189] of 346030080 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.219196] begin iteration loff: 346030080 [D 20:01:04.219203] segment exceeds byte limit [D 20:01:04.219208] process a segment [D 20:01:04.219214] of 346030080 sz 2097152 [D 20:01:04.219220] add a segment 0 [D 20:01:04.219227] end iteration [D 20:01:04.219232] done with segments or bytes [D 20:01:04.219238] of 348127232 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.219245] next loff: 348127232 next poff: 348127232 [D 20:01:04.219252] return value 2097152 [D 20:01:04.219258] finished [D 20:01:04.219264] retval = 2097152 [D 20:01:04.219270] exiting distribute returned less than expected [D 20:01:04.219276] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 348127232 fo 369098752 eof 0 [D 20:01:04.219283] ========================================================= [D 20:01:04.219291] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.219298] dbpf_bstream_rw_list: mem_offset: 0x2aaaadec7010, mem_size: 2097152 [D 20:01:04.219305] dbpf_bstream_rw_list: stream_offset: 346030080, stream_size: 2097152 [D 20:01:04.219311] dbpf_open_cache_get: called [D 20:01:04.219317] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.219324] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.219352] op_queue add: 0x2aaab1506a60 [D 20:01:04.219362] delayed I/O operation 0x2aaab1506a60 (16 already in progress) [D 20:01:04.219368] DBPF I/O ops in progress: 16 [D 20:01:04.219375] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.219381] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c210. [D 20:01:04.219388] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.219396] ========================================================= [D 20:01:04.219403] PINT_process_request [D 20:01:04.219409] tiling 12 copies [D 20:01:04.219415] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.219422] lvl 0 el 0 blk 0 by 348127232 [D 20:01:04.219429] to 348127232 ta 335544320 fi 369098752 [D 20:01:04.219436] basic type or contiguous data [D 20:01:04.219442] Request disp [D 20:01:04.219448] contig_offset = 348127232 contig_size = 54525952 lvl_flag = 1 [D 20:01:04.219455] calling distribute [D 20:01:04.219460] PINT_distribute [D 20:01:04.219466] of 348127232 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.219473] begin iteration loff: 348127232 [D 20:01:04.219480] segment exceeds byte limit [D 20:01:04.219486] process a segment [D 20:01:04.219493] of 348127232 sz 2097152 [D 20:01:04.219499] add a segment 0 [D 20:01:04.219505] end iteration [D 20:01:04.219510] done with segments or bytes [D 20:01:04.219516] of 350224384 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.219523] next loff: 350224384 next poff: 350224384 [D 20:01:04.219529] return value 2097152 [D 20:01:04.219536] finished [D 20:01:04.219541] retval = 2097152 [D 20:01:04.219547] exiting distribute returned less than expected [D 20:01:04.219553] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 350224384 fo 369098752 eof 0 [D 20:01:04.219561] ========================================================= [D 20:01:04.219568] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.219575] dbpf_bstream_rw_list: mem_offset: 0x2aaabaa4d010, mem_size: 2097152 [D 20:01:04.219582] --- aio_progress_notification called with handle 2863311531 (0x2aaab153ad80) [D 20:01:04.219591] post_sr: 10.1.4.56:39990 bh 17 len 16 wr_id 682bd0. [D 20:01:04.219602] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.219611] openib_check_cq: completed rdma wr_id 668920, nic_wr_credit 3. [D 20:01:04.219621] --- aio_progress_notification called with handle 2863311531 (0x2aaab153aed0) [D 20:01:04.219628] post_sr: 10.1.4.56:39990 bh 18 len 16 wr_id 682c00. [D 20:01:04.342659] openib_check_cq: completed rdma wr_id 6ed4e0, nic_wr_credit 2. [D 20:01:04.342676] post_sr: 10.1.4.56:39990 bh 19 len 16 wr_id 682c30. [D 20:01:04.342684] openib_check_cq: completed rdma wr_id 6ec680, nic_wr_credit 1. [D 20:01:04.342693] post_sr: 10.1.4.56:39990 bh 20 len 16 wr_id 682c60. [D 20:01:04.342700] openib_check_cq: completed rdma wr_id 6eda80, nic_wr_credit 0. [D 20:01:04.342787] post_sr: 10.1.4.56:39990 bh 21 len 16 wr_id 682c90. [D 20:01:04.342796] openib_check_cq: completed send wr_id 682ab0. [D 20:01:04.342804] openib_check_cq: completed send wr_id 682ae0. [D 20:01:04.342811] openib_check_cq: completed send wr_id 682b10. [D 20:01:04.342817] openib_check_cq: completed send wr_id 682b40. [D 20:01:04.342824] openib_check_cq: completed send wr_id 682b70. [D 20:01:04.342831] openib_check_cq: completed send wr_id 682ba0. [D 20:01:04.342838] openib_check_cq: completed send wr_id 682bd0. [D 20:01:04.342845] openib_check_cq: completed recv wr_id 684aa0. [D 20:01:04.342856] openib_post_sr_rdmaw: ibv_post_send wr_id 6ed990 to 10.1.4.56:39990 remote addr 98447000 rkey 109226 nic_wr_credit 1. [D 20:01:04.219616] dbpf_open_cache_put: cached entry. [D 20:01:04.219586] dbpf_bstream_rw_list: stream_offset: 348127232, stream_size: 2097152 [D 20:01:04.342939] dbpf_open_cache_get: called [D 20:01:04.219571] --- aio_progress_notification called with handle 2863311531 (0x2aaab152b210) [D 20:01:04.343068] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.219632] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.224600] --- aio_progress_notification called with handle 2863311531 (0x2aaab152d4d0) [D 20:01:04.343165] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.224663] --- aio_progress_notification called with handle 2863311531 (0x2aaab153b020) [D 20:01:04.343241] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.228602] --- aio_progress_notification called with handle 2863311531 (0x2aaab152d770) [D 20:01:04.343338] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.228612] --- aio_progress_notification called with handle 2863311531 (0x2aaab152d620) [D 20:01:04.343417] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.232688] --- aio_progress_notification called with handle 2863311531 (0x2aaab152da10) [D 20:01:04.343566] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.232724] --- aio_progress_notification called with handle 2863311531 (0x2aaab152d8c0) [D 20:01:04.343585] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.344031] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.344041] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.236609] --- aio_progress_notification called with handle 2863311531 (0x2aaab152de30) [D 20:01:04.344048] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344086] op_queue add: 0x2aaab1506bb0 [D 20:01:04.344088] op_queue add: 0x2aaab153ad80 [D 20:01:04.344099] delayed I/O operation 0x2aaab1506bb0 (16 already in progress) [D 20:01:04.344091] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.344121] DBPF I/O ops in progress: 16 [D 20:01:04.344137] dbpf_open_cache_put: cached entry. [D 20:01:04.344144] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.344146] DBPF I/O ops in progress: 15 [D 20:01:04.344163] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab152c210. [D 20:01:04.344171] starting delayed I/O operation 0x2aaab15054a0 (15 in progress) [D 20:01:04.344183] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.344191] lio_listio called with 1 following aiocbs: [D 20:01:04.344198] ========================================================= [D 20:01:04.344200] aiocb_ptr_array[0]: fd: 13, off: 245366784, bytes: 2097152, buf: 0x2aaabb453010, type: 0 [D 20:01:04.344209] PINT_process_request [D 20:01:04.344225] start_delayed_ops_if_any: lio_listio posted 0x2aaab15054a0 (handle 2863311531, ret 0)) [D 20:01:04.344229] tiling 12 copies [D 20:01:04.344241] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.344249] lvl 0 el 0 blk 0 by 350224384 [D 20:01:04.236602] --- aio_progress_notification called with handle 2863311531 (0x2aaab1530a60) [D 20:01:04.344153] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344255] to 350224384 ta 335544320 fi 369098752 [D 20:01:04.240613] --- aio_progress_notification called with handle 2863311531 (0x2aaab152bb90) [D 20:01:04.240710] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3008d0) [D 20:01:04.344267] op_queue add: 0x2aaab152de30 [D 20:01:04.344278] basic type or contiguous data [D 20:01:04.344310] Request disp [D 20:01:04.344317] contig_offset = 350224384 contig_size = 52428800 lvl_flag = 1 [D 20:01:04.344326] calling distribute [D 20:01:04.344332] PINT_distribute [D 20:01:04.344340] of 350224384 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.344349] begin iteration loff: 350224384 [D 20:01:04.344357] segment exceeds byte limit [D 20:01:04.344364] process a segment [D 20:01:04.344370] of 350224384 sz 2097152 [D 20:01:04.344377] add a segment 0 [D 20:01:04.344383] end iteration [D 20:01:04.344389] done with segments or bytes [D 20:01:04.344395] of 352321536 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.344402] next loff: 352321536 next poff: 352321536 [D 20:01:04.344409] return value 2097152 [D 20:01:04.344416] finished [D 20:01:04.344422] retval = 2097152 [D 20:01:04.344429] exiting distribute returned less than expected [D 20:01:04.344436] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 352321536 fo 369098752 eof 0 [D 20:01:04.344443] ========================================================= [D 20:01:04.344460] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.344468] dbpf_bstream_rw_list: mem_offset: 0x2aaab8a3d010, mem_size: 2097152 [D 20:01:04.344474] dbpf_bstream_rw_list: stream_offset: 350224384, stream_size: 2097152 [D 20:01:04.344481] dbpf_open_cache_get: called [D 20:01:04.344258] dbpf_open_cache_put: cached entry. [D 20:01:04.344498] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344506] op_queue add: 0x2aaab152b210 [D 20:01:04.344517] dbpf_open_cache_put: cached entry. [D 20:01:04.344529] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344537] op_queue add: 0x2aaab153aed0 [D 20:01:04.344548] dbpf_open_cache_put: cached entry. [D 20:01:04.344558] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344565] op_queue add: 0x2aaab152d4d0 [D 20:01:04.344583] dbpf_open_cache_put: cached entry. [D 20:01:04.344592] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344600] op_queue add: 0x2aaab153b020 [D 20:01:04.344615] dbpf_open_cache_put: cached entry. [D 20:01:04.344625] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344633] op_queue add: 0x2aaab152d770 [D 20:01:04.344649] dbpf_open_cache_put: cached entry. [D 20:01:04.344660] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344667] op_queue add: 0x2aaab152d620 [D 20:01:04.344284] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.344301] DBPF I/O ops in progress: 15 [D 20:01:04.344696] starting delayed I/O operation 0x2aaab15055f0 (15 in progress) [D 20:01:04.344702] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.344708] lio_listio called with 1 following aiocbs: [D 20:01:04.244607] --- aio_progress_notification called with handle 2863311531 (0x2aaab151fd00) [D 20:01:04.344719] aiocb_ptr_array[0]: fd: 13, off: 247463936, bytes: 2097152, buf: 0x2aaabb252010, type: 0 [D 20:01:04.344291] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.344735] start_delayed_ops_if_any: lio_listio posted 0x2aaab15055f0 (handle 2863311531, ret 0)) [D 20:01:04.344744] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.344697] dbpf_open_cache_put: cached entry. [D 20:01:04.344753] DBPF I/O ops in progress: 15 [D 20:01:04.344778] starting delayed I/O operation 0x2aaab1505740 (15 in progress) [D 20:01:04.344769] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.344792] lio_listio called with 1 following aiocbs: [D 20:01:04.344796] op_queue add: 0x2aaab1530a60 [D 20:01:04.344801] aiocb_ptr_array[0]: fd: 13, off: 249561088, bytes: 2097152, buf: 0x2aaabb051010, type: 0 [D 20:01:04.344825] start_delayed_ops_if_any: lio_listio posted 0x2aaab1505740 (handle 2863311531, ret 0)) [D 20:01:04.344844] DBPF I/O ops in progress: 15 [D 20:01:04.344853] starting delayed I/O operation 0x2aaab1505d50 (15 in progress) [D 20:01:04.344865] lio_listio called with 1 following aiocbs: [D 20:01:04.344871] aiocb_ptr_array[0]: fd: 13, off: 268435456, bytes: 2097152, buf: 0x2aaab9844010, type: 0 [D 20:01:04.344881] start_delayed_ops_if_any: lio_listio posted 0x2aaab1505d50 (handle 2863311531, ret 0)) [D 20:01:04.344899] DBPF I/O ops in progress: 15 [D 20:01:04.344909] starting delayed I/O operation 0x2aaab1505ea0 (15 in progress) [D 20:01:04.344920] lio_listio called with 1 following aiocbs: [D 20:01:04.344927] aiocb_ptr_array[0]: fd: 13, off: 270532608, bytes: 2097152, buf: 0x2aaab8e3f010, type: 0 [D 20:01:04.344936] start_delayed_ops_if_any: lio_listio posted 0x2aaab1505ea0 (handle 2863311531, ret 0)) [D 20:01:04.344987] DBPF I/O ops in progress: 15 [D 20:01:04.344999] starting delayed I/O operation 0x2aaab1505ff0 (15 in progress) [D 20:01:04.345011] lio_listio called with 1 following aiocbs: [D 20:01:04.345017] aiocb_ptr_array[0]: fd: 13, off: 272629760, bytes: 2097152, buf: 0x2aaab5c25010, type: 0 [D 20:01:04.345027] start_delayed_ops_if_any: lio_listio posted 0x2aaab1505ff0 (handle 2863311531, ret 0)) [D 20:01:04.345062] DBPF I/O ops in progress: 15 [D 20:01:04.345072] starting delayed I/O operation 0x2aaab1506140 (15 in progress) [D 20:01:04.345083] lio_listio called with 1 following aiocbs: [D 20:01:04.345089] aiocb_ptr_array[0]: fd: 13, off: 274726912, bytes: 2097152, buf: 0x2aaab521f010, type: 0 [D 20:01:04.345098] start_delayed_ops_if_any: lio_listio posted 0x2aaab1506140 (handle 2863311531, ret 0)) [D 20:01:04.345129] DBPF I/O ops in progress: 15 [D 20:01:04.345139] starting delayed I/O operation 0x2aaab1506290 (15 in progress) [D 20:01:04.345149] lio_listio called with 1 following aiocbs: [D 20:01:04.345155] aiocb_ptr_array[0]: fd: 13, off: 276824064, bytes: 2097152, buf: 0x2aaaaf6d3010, type: 0 [D 20:01:04.345165] start_delayed_ops_if_any: lio_listio posted 0x2aaab1506290 (handle 2863311531, ret 0)) [D 20:01:04.345180] DBPF I/O ops in progress: 15 [D 20:01:04.345190] dbpf_open_cache_put: cached entry. [D 20:01:04.345191] starting delayed I/O operation 0x2aaab15063e0 (15 in progress) [D 20:01:04.345209] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.345210] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.345220] lio_listio called with 1 following aiocbs: [D 20:01:04.345230] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.345246] op_queue add: 0x2aaab152d8c0 [D 20:01:04.345255] aiocb_ptr_array[0]: fd: 13, off: 278921216, bytes: 2097152, buf: 0x2aaaadac5010, type: 0 [D 20:01:04.345308] dbpf_open_cache_put: cached entry. [D 20:01:04.345311] start_delayed_ops_if_any: lio_listio posted 0x2aaab15063e0 (handle 2863311531, ret 0)) [D 20:01:04.345340] op_queue add: 0x2aaab1506d00 [D 20:01:04.345362] delayed I/O operation 0x2aaab1506d00 (16 already in progress) [D 20:01:04.345369] DBPF I/O ops in progress: 16 [D 20:01:04.345378] flowproto posted 0x2aaab152c210 [D 20:01:04.345388] Job flows in progress (post time): 12 [D 20:01:04.345403] [SM Exiting]: (0x66afd0) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:04.345416] [SM Entering]: (0x66c290) io_sm:start_flow (status: 0) [D 20:01:04.345423] [SM frame get]: (0x66c290) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.345435] io_start_flow() issuing flow to read data. [D 20:01:04.345443] flow: fsize: 478150656, server_nr: 0, server_ct: 1 [D 20:01:04.345449] file_req_offset: 369098752, aggregate_size: 33554432, handle: 2863311531 [D 20:01:04.345463] PINT_new_request_states n=1 [D 20:01:04.345473] flowproto posting 0x2aaab1506e50 [D 20:01:04.345506] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.345523] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1506e50. [D 20:01:04.345532] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:04.345544] ========================================================= [D 20:01:04.345550] PINT_process_request [D 20:01:04.345558] tiling 13 copies [D 20:01:04.345564] skipping ahead to target_offset [D 20:01:04.345571] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.345579] lvl 0 el 0 blk 0 by 0 [D 20:01:04.345585] to 0 ta 369098752 fi 402653184 [D 20:01:04.345593] basic type or contiguous data [D 20:01:04.345599] Request disp [D 20:01:04.345605] contig_offset = 0 contig_size = 436207616 lvl_flag = 1 [D 20:01:04.345612] process logical skip [D 20:01:04.345618] retval = 369098752 [D 20:01:04.345624] exiting logical skip because distribute indicates done [D 20:01:04.345630] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.345637] lvl 0 el 0 blk 0 by 369098752 [D 20:01:04.345644] to 369098752 ta 369098752 fi 402653184 [D 20:01:04.345650] basic type or contiguous data [D 20:01:04.345656] Request disp [D 20:01:04.345663] contig_offset = 369098752 contig_size = 67108864 lvl_flag = 1 [D 20:01:04.345670] calling distribute [D 20:01:04.345675] PINT_distribute [D 20:01:04.345681] of 369098752 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.345689] begin iteration loff: 369098752 [D 20:01:04.345696] segment exceeds byte limit [D 20:01:04.345702] process a segment [D 20:01:04.345708] of 369098752 sz 2097152 [D 20:01:04.345714] add a segment 0 [D 20:01:04.345720] end iteration [D 20:01:04.345726] done with segments or bytes [D 20:01:04.345732] of 371195904 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.345739] next loff: 371195904 next poff: 371195904 [D 20:01:04.345746] return value 2097152 [D 20:01:04.345752] finished [D 20:01:04.345758] retval = 2097152 [D 20:01:04.345764] exiting distribute returned less than expected [D 20:01:04.345770] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 371195904 fo 402653184 eof 0 [D 20:01:04.345778] ========================================================= [D 20:01:04.345788] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.345796] dbpf_bstream_rw_list: mem_offset: 0x2aaab6e2f010, mem_size: 2097152 [D 20:01:04.345802] dbpf_bstream_rw_list: stream_offset: 369098752, stream_size: 2097152 [D 20:01:04.345810] dbpf_open_cache_get: called [D 20:01:04.345324] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.345821] op_queue add: 0x2aaab152da10 [D 20:01:04.345486] DBPF I/O ops in progress: 15 [D 20:01:04.345840] starting delayed I/O operation 0x2aaabd300a20 (15 in progress) [D 20:01:04.345851] lio_listio called with 1 following aiocbs: [D 20:01:04.345858] aiocb_ptr_array[0]: fd: 13, off: 123731968, bytes: 2097152, buf: 0x2aaac0016010, type: 0 [D 20:01:04.345868] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300a20 (handle 2863311531, ret 0)) [D 20:01:04.345912] DBPF I/O ops in progress: 15 [D 20:01:04.345922] starting delayed I/O operation 0x2aaabd300e10 (15 in progress) [D 20:01:04.345933] lio_listio called with 1 following aiocbs: [D 20:01:04.345939] aiocb_ptr_array[0]: fd: 13, off: 150994944, bytes: 2097152, buf: 0x2aaac0a1b010, type: 0 [D 20:01:04.345951] start_delayed_ops_if_any: lio_listio posted 0x2aaabd300e10 (handle 2863311531, ret 0)) [D 20:01:04.345330] dbpf_open_cache_put: cached entry. [D 20:01:04.345987] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.345996] op_queue add: 0x2aaabd3008d0 [D 20:01:04.346004] DBPF I/O ops in progress: 15 [D 20:01:04.346011] starting delayed I/O operation 0x2aaab1522b60 (15 in progress) [D 20:01:04.346022] lio_listio called with 1 following aiocbs: [D 20:01:04.346029] aiocb_ptr_array[0]: fd: 13, off: 281018368, bytes: 2097152, buf: 0x2aaabae50010, type: 0 [D 20:01:04.346038] start_delayed_ops_if_any: lio_listio posted 0x2aaab1522b60 (handle 2863311531, ret 0)) [D 20:01:04.346071] dbpf_open_cache_put: cached entry. [D 20:01:04.346084] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.346091] op_queue add: 0x2aaab152bb90 [D 20:01:04.346099] DBPF I/O ops in progress: 15 [D 20:01:04.346107] starting delayed I/O operation 0x2aaab1525840 (15 in progress) [D 20:01:04.346118] lio_listio called with 1 following aiocbs: [D 20:01:04.346124] aiocb_ptr_array[0]: fd: 13, off: 283115520, bytes: 2097152, buf: 0x2aaab9040010, type: 0 [D 20:01:04.346133] start_delayed_ops_if_any: lio_listio posted 0x2aaab1525840 (handle 2863311531, ret 0)) [D 20:01:04.346169] dbpf_open_cache_put: cached entry. [D 20:01:04.346183] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.346191] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.346219] op_queue add: 0x2aaab15073b0 [D 20:01:04.346228] delayed I/O operation 0x2aaab15073b0 (16 already in progress) [D 20:01:04.346235] DBPF I/O ops in progress: 16 [D 20:01:04.346243] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.346250] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1506e50. [D 20:01:04.346259] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.346270] ========================================================= [D 20:01:04.346277] PINT_process_request [D 20:01:04.346284] tiling 13 copies [D 20:01:04.346291] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.346298] lvl 0 el 0 blk 0 by 371195904 [D 20:01:04.346305] to 371195904 ta 369098752 fi 402653184 [D 20:01:04.346312] basic type or contiguous data [D 20:01:04.346319] Request disp [D 20:01:04.346325] contig_offset = 371195904 contig_size = 65011712 lvl_flag = 1 [D 20:01:04.346333] calling distribute [D 20:01:04.346339] PINT_distribute [D 20:01:04.346345] of 371195904 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.346352] begin iteration loff: 371195904 [D 20:01:04.346359] segment exceeds byte limit [D 20:01:04.346365] process a segment [D 20:01:04.346371] of 371195904 sz 2097152 [D 20:01:04.346378] add a segment 0 [D 20:01:04.346384] end iteration [D 20:01:04.346390] done with segments or bytes [D 20:01:04.346396] of 373293056 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.346403] next loff: 373293056 next poff: 373293056 [D 20:01:04.346409] return value 2097152 [D 20:01:04.346415] finished [D 20:01:04.346421] retval = 2097152 [D 20:01:04.346428] exiting distribute returned less than expected [D 20:01:04.346434] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 373293056 fo 402653184 eof 0 [D 20:01:04.346442] ========================================================= [D 20:01:04.346451] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.346458] dbpf_bstream_rw_list: mem_offset: 0x2aaab4e1d010, mem_size: 2097152 [D 20:01:04.346465] dbpf_bstream_rw_list: stream_offset: 371195904, stream_size: 2097152 [D 20:01:04.346471] dbpf_open_cache_get: called [D 20:01:04.346477] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.346484] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.346510] op_queue add: 0x2aaab1507500 [D 20:01:04.346520] delayed I/O operation 0x2aaab1507500 (16 already in progress) [D 20:01:04.346527] DBPF I/O ops in progress: 16 [D 20:01:04.346534] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.346540] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1506e50. [D 20:01:04.346547] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.346556] ========================================================= [D 20:01:04.346562] PINT_process_request [D 20:01:04.346568] tiling 13 copies [D 20:01:04.346574] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.346582] lvl 0 el 0 blk 0 by 373293056 [D 20:01:04.346588] to 373293056 ta 369098752 fi 402653184 [D 20:01:04.346595] basic type or contiguous data [D 20:01:04.346601] Request disp [D 20:01:04.346607] contig_offset = 373293056 contig_size = 62914560 lvl_flag = 1 [D 20:01:04.346614] calling distribute [D 20:01:04.346619] PINT_distribute [D 20:01:04.346625] of 373293056 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.346632] begin iteration loff: 373293056 [D 20:01:04.346639] segment exceeds byte limit [D 20:01:04.346645] process a segment [D 20:01:04.346650] of 373293056 sz 2097152 [D 20:01:04.346657] add a segment 0 [D 20:01:04.346663] end iteration [D 20:01:04.346669] done with segments or bytes [D 20:01:04.346675] of 375390208 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.346682] next loff: 375390208 next poff: 375390208 [D 20:01:04.346688] return value 2097152 [D 20:01:04.346694] finished [D 20:01:04.346700] retval = 2097152 [D 20:01:04.346706] exiting distribute returned less than expected [D 20:01:04.346713] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 375390208 fo 402653184 eof 0 [D 20:01:04.346720] ========================================================= [D 20:01:04.346728] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.346734] dbpf_bstream_rw_list: mem_offset: 0x2aaab320f010, mem_size: 2097152 [D 20:01:04.346741] dbpf_bstream_rw_list: stream_offset: 373293056, stream_size: 2097152 [D 20:01:04.346748] dbpf_open_cache_get: called [D 20:01:04.346754] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.346761] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.346790] op_queue add: 0x2aaab1507650 [D 20:01:04.346799] delayed I/O operation 0x2aaab1507650 (16 already in progress) [D 20:01:04.346806] DBPF I/O ops in progress: 16 [D 20:01:04.346812] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.346819] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1506e50. [D 20:01:04.346826] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.346835] ========================================================= [D 20:01:04.346841] PINT_process_request [D 20:01:04.346847] tiling 13 copies [D 20:01:04.346853] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.346861] lvl 0 el 0 blk 0 by 375390208 [D 20:01:04.346867] to 375390208 ta 369098752 fi 402653184 [D 20:01:04.346874] basic type or contiguous data [D 20:01:04.346880] Request disp [D 20:01:04.346885] contig_offset = 375390208 contig_size = 60817408 lvl_flag = 1 [D 20:01:04.346893] calling distribute [D 20:01:04.346898] PINT_distribute [D 20:01:04.346904] of 375390208 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.346912] begin iteration loff: 375390208 [D 20:01:04.346918] segment exceeds byte limit [D 20:01:04.346924] process a segment [D 20:01:04.346929] of 375390208 sz 2097152 [D 20:01:04.346936] add a segment 0 [D 20:01:04.346942] end iteration [D 20:01:04.346948] done with segments or bytes [D 20:01:04.346954] of 377487360 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.346961] next loff: 377487360 next poff: 377487360 [D 20:01:04.346968] return value 2097152 [D 20:01:04.346974] finished [D 20:01:04.346980] retval = 2097152 [D 20:01:04.346986] exiting distribute returned less than expected [D 20:01:04.346993] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 377487360 fo 402653184 eof 0 [D 20:01:04.347000] ========================================================= [D 20:01:04.347007] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.347014] dbpf_bstream_rw_list: mem_offset: 0x2aaab1600010, mem_size: 2097152 [D 20:01:04.347020] dbpf_bstream_rw_list: stream_offset: 375390208, stream_size: 2097152 [D 20:01:04.347027] dbpf_open_cache_get: called [D 20:01:04.347033] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.347040] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.347064] op_queue add: 0x2aaab15077a0 [D 20:01:04.347075] delayed I/O operation 0x2aaab15077a0 (16 already in progress) [D 20:01:04.347081] DBPF I/O ops in progress: 16 [D 20:01:04.347088] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.347095] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1506e50. [D 20:01:04.347102] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.347111] ========================================================= [D 20:01:04.347117] PINT_process_request [D 20:01:04.347123] tiling 13 copies [D 20:01:04.347129] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.347137] lvl 0 el 0 blk 0 by 377487360 [D 20:01:04.347143] to 377487360 ta 369098752 fi 402653184 [D 20:01:04.347150] basic type or contiguous data [D 20:01:04.347156] Request disp [D 20:01:04.347162] contig_offset = 377487360 contig_size = 58720256 lvl_flag = 1 [D 20:01:04.347168] calling distribute [D 20:01:04.347174] PINT_distribute [D 20:01:04.347180] of 377487360 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.347187] begin iteration loff: 377487360 [D 20:01:04.347194] segment exceeds byte limit [D 20:01:04.347200] process a segment [D 20:01:04.347206] of 377487360 sz 2097152 [D 20:01:04.347212] add a segment 0 [D 20:01:04.347218] end iteration [D 20:01:04.347224] done with segments or bytes [D 20:01:04.347230] of 379584512 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.347237] next loff: 379584512 next poff: 379584512 [D 20:01:04.347243] return value 2097152 [D 20:01:04.347249] finished [D 20:01:04.347255] retval = 2097152 [D 20:01:04.347261] exiting distribute returned less than expected [D 20:01:04.347268] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 379584512 fo 402653184 eof 0 [D 20:01:04.347275] ========================================================= [D 20:01:04.347283] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.347289] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf8d4010, mem_size: 2097152 [D 20:01:04.347296] dbpf_bstream_rw_list: stream_offset: 377487360, stream_size: 2097152 [D 20:01:04.347303] dbpf_open_cache_get: called [D 20:01:04.347309] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.347316] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.347346] op_queue add: 0x2aaab15078f0 [D 20:01:04.347356] delayed I/O operation 0x2aaab15078f0 (16 already in progress) [D 20:01:04.347363] DBPF I/O ops in progress: 16 [D 20:01:04.347369] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.347376] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1506e50. [D 20:01:04.347383] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.347392] ========================================================= [D 20:01:04.347399] PINT_process_request [D 20:01:04.347405] tiling 13 copies [D 20:01:04.347411] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.347418] lvl 0 el 0 blk 0 by 379584512 [D 20:01:04.347424] to 379584512 ta 369098752 fi 402653184 [D 20:01:04.347431] basic type or contiguous data [D 20:01:04.347438] Request disp [D 20:01:04.347444] contig_offset = 379584512 contig_size = 56623104 lvl_flag = 1 [D 20:01:04.347451] calling distribute [D 20:01:04.347457] PINT_distribute [D 20:01:04.347462] of 379584512 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.347470] begin iteration loff: 379584512 [D 20:01:04.347476] segment exceeds byte limit [D 20:01:04.347482] process a segment [D 20:01:04.347488] of 379584512 sz 2097152 [D 20:01:04.347494] add a segment 0 [D 20:01:04.347500] end iteration [D 20:01:04.347506] done with segments or bytes [D 20:01:04.347512] of 381681664 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.347519] next loff: 381681664 next poff: 381681664 [D 20:01:04.347526] return value 2097152 [D 20:01:04.347533] finished [D 20:01:04.347538] retval = 2097152 [D 20:01:04.347544] exiting distribute returned less than expected [D 20:01:04.347551] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 381681664 fo 402653184 eof 0 [D 20:01:04.347558] ========================================================= [D 20:01:04.347566] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.347574] dbpf_bstream_rw_list: mem_offset: 0x2aaaadcc6010, mem_size: 2097152 [D 20:01:04.347580] dbpf_bstream_rw_list: stream_offset: 379584512, stream_size: 2097152 [D 20:01:04.347587] dbpf_open_cache_get: called [D 20:01:04.347593] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.347600] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.347629] op_queue add: 0x2aaab1507a40 [D 20:01:04.347638] delayed I/O operation 0x2aaab1507a40 (16 already in progress) [D 20:01:04.347645] DBPF I/O ops in progress: 16 [D 20:01:04.347652] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.347658] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1506e50. [D 20:01:04.347665] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.347673] ========================================================= [D 20:01:04.347680] PINT_process_request [D 20:01:04.347686] tiling 13 copies [D 20:01:04.347692] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.347699] lvl 0 el 0 blk 0 by 381681664 [D 20:01:04.347706] to 381681664 ta 369098752 fi 402653184 [D 20:01:04.347712] basic type or contiguous data [D 20:01:04.347718] Request disp [D 20:01:04.347724] contig_offset = 381681664 contig_size = 54525952 lvl_flag = 1 [D 20:01:04.347731] calling distribute [D 20:01:04.347736] PINT_distribute [D 20:01:04.347742] of 381681664 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.347750] begin iteration loff: 381681664 [D 20:01:04.347756] segment exceeds byte limit [D 20:01:04.347762] process a segment [D 20:01:04.347768] of 381681664 sz 2097152 [D 20:01:04.347775] add a segment 0 [D 20:01:04.347781] end iteration [D 20:01:04.347787] done with segments or bytes [D 20:01:04.347794] of 383778816 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.347801] next loff: 383778816 next poff: 383778816 [D 20:01:04.347807] return value 2097152 [D 20:01:04.347813] finished [D 20:01:04.347819] retval = 2097152 [D 20:01:04.347825] exiting distribute returned less than expected [D 20:01:04.347831] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 383778816 fo 402653184 eof 0 [D 20:01:04.347838] ========================================================= [D 20:01:04.347847] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.347854] dbpf_bstream_rw_list: mem_offset: 0x2aaaba64b010, mem_size: 2097152 [D 20:01:04.347861] dbpf_bstream_rw_list: stream_offset: 381681664, stream_size: 2097152 [D 20:01:04.347867] dbpf_open_cache_get: called [D 20:01:04.347873] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.347880] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.347903] op_queue add: 0x2aaab15b8110 [D 20:01:04.347913] delayed I/O operation 0x2aaab15b8110 (16 already in progress) [D 20:01:04.347920] DBPF I/O ops in progress: 16 [D 20:01:04.347926] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.347933] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1506e50. [D 20:01:04.347940] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.347949] ========================================================= [D 20:01:04.347955] PINT_process_request [D 20:01:04.347961] tiling 13 copies [D 20:01:04.347967] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.347975] lvl 0 el 0 blk 0 by 383778816 [D 20:01:04.347981] to 383778816 ta 369098752 fi 402653184 [D 20:01:04.347988] basic type or contiguous data [D 20:01:04.347994] Request disp [D 20:01:04.348000] contig_offset = 383778816 contig_size = 52428800 lvl_flag = 1 [D 20:01:04.348007] calling distribute [D 20:01:04.348013] PINT_distribute [D 20:01:04.348019] of 383778816 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.348026] begin iteration loff: 383778816 [D 20:01:04.348032] segment exceeds byte limit [D 20:01:04.348038] process a segment [D 20:01:04.348044] of 383778816 sz 2097152 [D 20:01:04.348050] add a segment 0 [D 20:01:04.348056] end iteration [D 20:01:04.348062] done with segments or bytes [D 20:01:04.348068] of 385875968 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.348075] next loff: 385875968 next poff: 385875968 [D 20:01:04.348081] return value 2097152 [D 20:01:04.348088] finished [D 20:01:04.348093] retval = 2097152 [D 20:01:04.348099] exiting distribute returned less than expected [D 20:01:04.348106] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 385875968 fo 402653184 eof 0 [D 20:01:04.348113] ========================================================= [D 20:01:04.348120] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.348128] dbpf_bstream_rw_list: mem_offset: 0x2aaab863b010, mem_size: 2097152 [D 20:01:04.348134] dbpf_bstream_rw_list: stream_offset: 383778816, stream_size: 2097152 [D 20:01:04.348141] dbpf_open_cache_get: called [D 20:01:04.348148] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.348154] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.348183] op_queue add: 0x2aaab15bac70 [D 20:01:04.348194] delayed I/O operation 0x2aaab15bac70 (16 already in progress) [D 20:01:04.348201] DBPF I/O ops in progress: 16 [D 20:01:04.348208] flowproto posted 0x2aaab1506e50 [D 20:01:04.348216] Job flows in progress (post time): 13 [D 20:01:04.348230] [SM Exiting]: (0x66c290) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:04.348242] [SM Entering]: (0x66d550) io_sm:start_flow (status: 0) [D 20:01:04.348250] [SM frame get]: (0x66d550) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.348261] io_start_flow() issuing flow to read data. [D 20:01:04.348268] flow: fsize: 478150656, server_nr: 0, server_ct: 1 [D 20:01:04.348275] file_req_offset: 402653184, aggregate_size: 33554432, handle: 2863311531 [D 20:01:04.348288] PINT_new_request_states n=1 [D 20:01:04.348298] flowproto posting 0x2aaab1507b90 [D 20:01:04.348323] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.348333] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1507b90. [D 20:01:04.348341] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:04.348349] ========================================================= [D 20:01:04.348355] PINT_process_request [D 20:01:04.348361] tiling 14 copies [D 20:01:04.348367] skipping ahead to target_offset [D 20:01:04.348374] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.348381] lvl 0 el 0 blk 0 by 0 [D 20:01:04.348388] to 0 ta 402653184 fi 436207616 [D 20:01:04.348394] basic type or contiguous data [D 20:01:04.348401] Request disp [D 20:01:04.348406] contig_offset = 0 contig_size = 469762048 lvl_flag = 1 [D 20:01:04.348414] process logical skip [D 20:01:04.348420] retval = 402653184 [D 20:01:04.348426] exiting logical skip because distribute indicates done [D 20:01:04.348432] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.348439] lvl 0 el 0 blk 0 by 402653184 [D 20:01:04.348446] to 402653184 ta 402653184 fi 436207616 [D 20:01:04.348452] basic type or contiguous data [D 20:01:04.348458] Request disp [D 20:01:04.348464] contig_offset = 402653184 contig_size = 67108864 lvl_flag = 1 [D 20:01:04.348471] calling distribute [D 20:01:04.348477] PINT_distribute [D 20:01:04.348483] of 402653184 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.348491] begin iteration loff: 402653184 [D 20:01:04.348497] segment exceeds byte limit [D 20:01:04.348503] process a segment [D 20:01:04.348509] of 402653184 sz 2097152 [D 20:01:04.348515] add a segment 0 [D 20:01:04.348521] end iteration [D 20:01:04.348527] done with segments or bytes [D 20:01:04.348533] of 404750336 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.348540] next loff: 404750336 next poff: 404750336 [D 20:01:04.348547] return value 2097152 [D 20:01:04.348553] finished [D 20:01:04.348558] retval = 2097152 [D 20:01:04.348565] exiting distribute returned less than expected [D 20:01:04.348571] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 404750336 fo 436207616 eof 0 [D 20:01:04.348593] ========================================================= [D 20:01:04.348608] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.348617] dbpf_bstream_rw_list: mem_offset: 0x2aaab6a2d010, mem_size: 2097152 [D 20:01:04.348624] dbpf_bstream_rw_list: stream_offset: 402653184, stream_size: 2097152 [D 20:01:04.348630] dbpf_open_cache_get: called [D 20:01:04.348637] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.348643] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.348666] op_queue add: 0x2aaab15bfff0 [D 20:01:04.348676] delayed I/O operation 0x2aaab15bfff0 (16 already in progress) [D 20:01:04.348682] DBPF I/O ops in progress: 16 [D 20:01:04.348689] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.348695] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1507b90. [D 20:01:04.348702] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.348710] ========================================================= [D 20:01:04.348716] PINT_process_request [D 20:01:04.348722] tiling 14 copies [D 20:01:04.348729] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.348736] lvl 0 el 0 blk 0 by 404750336 [D 20:01:04.348743] to 404750336 ta 402653184 fi 436207616 [D 20:01:04.348749] basic type or contiguous data [D 20:01:04.348755] Request disp [D 20:01:04.348761] contig_offset = 404750336 contig_size = 65011712 lvl_flag = 1 [D 20:01:04.348767] calling distribute [D 20:01:04.348773] PINT_distribute [D 20:01:04.348779] of 404750336 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.348786] begin iteration loff: 404750336 [D 20:01:04.348793] segment exceeds byte limit [D 20:01:04.348798] process a segment [D 20:01:04.348804] of 404750336 sz 2097152 [D 20:01:04.348810] add a segment 0 [D 20:01:04.348816] end iteration [D 20:01:04.348822] done with segments or bytes [D 20:01:04.348828] of 406847488 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.348835] next loff: 406847488 next poff: 406847488 [D 20:01:04.348841] return value 2097152 [D 20:01:04.348848] finished [D 20:01:04.348854] retval = 2097152 [D 20:01:04.348860] exiting distribute returned less than expected [D 20:01:04.348866] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 406847488 fo 436207616 eof 0 [D 20:01:04.348874] ========================================================= [D 20:01:04.348881] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.348890] dbpf_bstream_rw_list: mem_offset: 0x2aaab4a1b010, mem_size: 2097152 [D 20:01:04.348896] dbpf_bstream_rw_list: stream_offset: 404750336, stream_size: 2097152 [D 20:01:04.348903] dbpf_open_cache_get: called [D 20:01:04.348909] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.348915] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.348946] op_queue add: 0x2aaab15c2b50 [D 20:01:04.348957] delayed I/O operation 0x2aaab15c2b50 (16 already in progress) [D 20:01:04.348963] DBPF I/O ops in progress: 16 [D 20:01:04.348970] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.348976] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1507b90. [D 20:01:04.348984] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.348991] ========================================================= [D 20:01:04.348998] PINT_process_request [D 20:01:04.349003] tiling 14 copies [D 20:01:04.349009] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.349017] lvl 0 el 0 blk 0 by 406847488 [D 20:01:04.349023] to 406847488 ta 402653184 fi 436207616 [D 20:01:04.349030] basic type or contiguous data [D 20:01:04.349036] Request disp [D 20:01:04.349041] contig_offset = 406847488 contig_size = 62914560 lvl_flag = 1 [D 20:01:04.349048] calling distribute [D 20:01:04.349054] PINT_distribute [D 20:01:04.349060] of 406847488 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.349068] begin iteration loff: 406847488 [D 20:01:04.349074] segment exceeds byte limit [D 20:01:04.349080] process a segment [D 20:01:04.349086] of 406847488 sz 2097152 [D 20:01:04.349092] add a segment 0 [D 20:01:04.349098] end iteration [D 20:01:04.349104] done with segments or bytes [D 20:01:04.349110] of 408944640 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.349117] next loff: 408944640 next poff: 408944640 [D 20:01:04.349123] return value 2097152 [D 20:01:04.349130] finished [D 20:01:04.349135] retval = 2097152 [D 20:01:04.349141] exiting distribute returned less than expected [D 20:01:04.349148] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 408944640 fo 436207616 eof 0 [D 20:01:04.349155] ========================================================= [D 20:01:04.349162] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.349169] dbpf_bstream_rw_list: mem_offset: 0x2aaab2e0c010, mem_size: 2097152 [D 20:01:04.349175] dbpf_bstream_rw_list: stream_offset: 406847488, stream_size: 2097152 [D 20:01:04.349182] dbpf_open_cache_get: called [D 20:01:04.349188] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.349195] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.349220] op_queue add: 0x2aaab15c56b0 [D 20:01:04.349231] delayed I/O operation 0x2aaab15c56b0 (16 already in progress) [D 20:01:04.349237] DBPF I/O ops in progress: 16 [D 20:01:04.349244] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.349250] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1507b90. [D 20:01:04.349258] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.349266] ========================================================= [D 20:01:04.349272] PINT_process_request [D 20:01:04.349278] tiling 14 copies [D 20:01:04.349284] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.349292] lvl 0 el 0 blk 0 by 408944640 [D 20:01:04.349298] to 408944640 ta 402653184 fi 436207616 [D 20:01:04.349304] basic type or contiguous data [D 20:01:04.349310] Request disp [D 20:01:04.349316] contig_offset = 408944640 contig_size = 60817408 lvl_flag = 1 [D 20:01:04.349323] calling distribute [D 20:01:04.349328] PINT_distribute [D 20:01:04.349334] of 408944640 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.349341] begin iteration loff: 408944640 [D 20:01:04.349348] segment exceeds byte limit [D 20:01:04.349355] process a segment [D 20:01:04.349361] of 408944640 sz 2097152 [D 20:01:04.349367] add a segment 0 [D 20:01:04.349373] end iteration [D 20:01:04.349379] done with segments or bytes [D 20:01:04.349385] of 411041792 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.349392] next loff: 411041792 next poff: 411041792 [D 20:01:04.349398] return value 2097152 [D 20:01:04.349404] finished [D 20:01:04.349410] retval = 2097152 [D 20:01:04.349416] exiting distribute returned less than expected [D 20:01:04.349422] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 411041792 fo 436207616 eof 0 [D 20:01:04.349430] ========================================================= [D 20:01:04.349436] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.349443] dbpf_bstream_rw_list: mem_offset: 0x2aaab10e0010, mem_size: 2097152 [D 20:01:04.349450] dbpf_bstream_rw_list: stream_offset: 408944640, stream_size: 2097152 [D 20:01:04.349456] dbpf_open_cache_get: called [D 20:01:04.349462] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.349469] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.349494] op_queue add: 0x2aaab15c8210 [D 20:01:04.349502] delayed I/O operation 0x2aaab15c8210 (16 already in progress) [D 20:01:04.349509] DBPF I/O ops in progress: 16 [D 20:01:04.349516] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.349522] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1507b90. [D 20:01:04.349530] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.349538] ========================================================= [D 20:01:04.349544] PINT_process_request [D 20:01:04.349551] tiling 14 copies [D 20:01:04.349557] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.349565] lvl 0 el 0 blk 0 by 411041792 [D 20:01:04.349571] to 411041792 ta 402653184 fi 436207616 [D 20:01:04.349577] basic type or contiguous data [D 20:01:04.349584] Request disp [D 20:01:04.349589] contig_offset = 411041792 contig_size = 58720256 lvl_flag = 1 [D 20:01:04.349596] calling distribute [D 20:01:04.349602] PINT_distribute [D 20:01:04.349608] of 411041792 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.349616] begin iteration loff: 411041792 [D 20:01:04.349622] segment exceeds byte limit [D 20:01:04.349628] process a segment [D 20:01:04.349633] of 411041792 sz 2097152 [D 20:01:04.349640] add a segment 0 [D 20:01:04.349646] end iteration [D 20:01:04.349652] done with segments or bytes [D 20:01:04.349658] of 413138944 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.349665] next loff: 413138944 next poff: 413138944 [D 20:01:04.349672] return value 2097152 [D 20:01:04.349678] finished [D 20:01:04.349684] retval = 2097152 [D 20:01:04.349690] exiting distribute returned less than expected [D 20:01:04.349697] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 413138944 fo 436207616 eof 0 [D 20:01:04.349704] ========================================================= [D 20:01:04.349711] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.349718] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf4d2010, mem_size: 2097152 [D 20:01:04.349725] dbpf_bstream_rw_list: stream_offset: 411041792, stream_size: 2097152 [D 20:01:04.349731] dbpf_open_cache_get: called [D 20:01:04.349737] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.349744] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.349772] op_queue add: 0x2aaab15cad70 [D 20:01:04.349781] delayed I/O operation 0x2aaab15cad70 (16 already in progress) [D 20:01:04.349788] DBPF I/O ops in progress: 16 [D 20:01:04.349794] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.349801] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1507b90. [D 20:01:04.349808] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.349817] ========================================================= [D 20:01:04.349824] PINT_process_request [D 20:01:04.349829] tiling 14 copies [D 20:01:04.349835] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.349843] lvl 0 el 0 blk 0 by 413138944 [D 20:01:04.349849] to 413138944 ta 402653184 fi 436207616 [D 20:01:04.349856] basic type or contiguous data [D 20:01:04.349862] Request disp [D 20:01:04.349868] contig_offset = 413138944 contig_size = 56623104 lvl_flag = 1 [D 20:01:04.349875] calling distribute [D 20:01:04.349880] PINT_distribute [D 20:01:04.349886] of 413138944 sz 23068672 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.349894] begin iteration loff: 413138944 [D 20:01:04.349900] segment exceeds byte limit [D 20:01:04.349906] process a segment [D 20:01:04.349912] of 413138944 sz 2097152 [D 20:01:04.349918] add a segment 0 [D 20:01:04.349924] end iteration [D 20:01:04.349930] done with segments or bytes [D 20:01:04.349936] of 415236096 sz 20971520 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.349943] next loff: 415236096 next poff: 415236096 [D 20:01:04.349949] return value 2097152 [D 20:01:04.349955] finished [D 20:01:04.349961] retval = 2097152 [D 20:01:04.349967] exiting distribute returned less than expected [D 20:01:04.349973] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 415236096 fo 436207616 eof 0 [D 20:01:04.349980] ========================================================= [D 20:01:04.349987] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.349994] dbpf_bstream_rw_list: mem_offset: 0x2aaaad8c4010, mem_size: 2097152 [D 20:01:04.350000] dbpf_bstream_rw_list: stream_offset: 413138944, stream_size: 2097152 [D 20:01:04.350007] dbpf_open_cache_get: called [D 20:01:04.350013] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.350019] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.350047] op_queue add: 0x2aaab15cd8d0 [D 20:01:04.350057] delayed I/O operation 0x2aaab15cd8d0 (16 already in progress) [D 20:01:04.350063] DBPF I/O ops in progress: 16 [D 20:01:04.350070] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.350077] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1507b90. [D 20:01:04.350085] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.350093] ========================================================= [D 20:01:04.350099] PINT_process_request [D 20:01:04.350105] tiling 14 copies [D 20:01:04.350112] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.350120] lvl 0 el 0 blk 0 by 415236096 [D 20:01:04.350126] to 415236096 ta 402653184 fi 436207616 [D 20:01:04.350133] basic type or contiguous data [D 20:01:04.350139] Request disp [D 20:01:04.350144] contig_offset = 415236096 contig_size = 54525952 lvl_flag = 1 [D 20:01:04.350151] calling distribute [D 20:01:04.350157] PINT_distribute [D 20:01:04.350163] of 415236096 sz 20971520 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.350170] begin iteration loff: 415236096 [D 20:01:04.350177] segment exceeds byte limit [D 20:01:04.350183] process a segment [D 20:01:04.350188] of 415236096 sz 2097152 [D 20:01:04.350194] add a segment 0 [D 20:01:04.350201] end iteration [D 20:01:04.350206] done with segments or bytes [D 20:01:04.350212] of 417333248 sz 18874368 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.350219] next loff: 417333248 next poff: 417333248 [D 20:01:04.350225] return value 2097152 [D 20:01:04.350231] finished [D 20:01:04.350237] retval = 2097152 [D 20:01:04.350243] exiting distribute returned less than expected [D 20:01:04.350249] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 417333248 fo 436207616 eof 0 [D 20:01:04.350256] ========================================================= [D 20:01:04.350265] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.350273] dbpf_bstream_rw_list: mem_offset: 0x2aaab7e37010, mem_size: 2097152 [D 20:01:04.350279] dbpf_bstream_rw_list: stream_offset: 415236096, stream_size: 2097152 [D 20:01:04.350286] dbpf_open_cache_get: called [D 20:01:04.350293] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.350299] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.350324] op_queue add: 0x2aaab15d0430 [D 20:01:04.350334] delayed I/O operation 0x2aaab15d0430 (16 already in progress) [D 20:01:04.350341] DBPF I/O ops in progress: 16 [D 20:01:04.350347] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.350353] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab1507b90. [D 20:01:04.350361] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.350369] ========================================================= [D 20:01:04.350375] PINT_process_request [D 20:01:04.350381] tiling 14 copies [D 20:01:04.350387] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.350395] lvl 0 el 0 blk 0 by 417333248 [D 20:01:04.350401] to 417333248 ta 402653184 fi 436207616 [D 20:01:04.350408] basic type or contiguous data [D 20:01:04.350414] Request disp [D 20:01:04.350420] contig_offset = 417333248 contig_size = 52428800 lvl_flag = 1 [D 20:01:04.350426] calling distribute [D 20:01:04.350432] PINT_distribute [D 20:01:04.350438] of 417333248 sz 18874368 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.350446] begin iteration loff: 417333248 [D 20:01:04.350452] segment exceeds byte limit [D 20:01:04.350458] process a segment [D 20:01:04.350464] of 417333248 sz 2097152 [D 20:01:04.350470] add a segment 0 [D 20:01:04.350476] end iteration [D 20:01:04.350482] done with segments or bytes [D 20:01:04.350488] of 419430400 sz 16777216 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.350495] next loff: 419430400 next poff: 419430400 [D 20:01:04.350502] return value 2097152 [D 20:01:04.350508] finished [D 20:01:04.350514] retval = 2097152 [D 20:01:04.350520] exiting distribute returned less than expected [D 20:01:04.350526] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 419430400 fo 436207616 eof 0 [D 20:01:04.350534] ========================================================= [D 20:01:04.350551] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.350561] dbpf_bstream_rw_list: mem_offset: 0x2aaab6228010, mem_size: 2097152 [D 20:01:04.350568] dbpf_bstream_rw_list: stream_offset: 417333248, stream_size: 2097152 [D 20:01:04.350575] dbpf_open_cache_get: called [D 20:01:04.350581] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.350588] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.350609] op_queue add: 0x2aaab15d2f90 [D 20:01:04.350618] delayed I/O operation 0x2aaab15d2f90 (16 already in progress) [D 20:01:04.350625] DBPF I/O ops in progress: 16 [D 20:01:04.350632] flowproto posted 0x2aaab1507b90 [D 20:01:04.350641] Job flows in progress (post time): 14 [D 20:01:04.350656] [SM Exiting]: (0x66d550) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:04.350668] [SM Entering]: (0x66fad0) io_sm:start_flow (status: 0) [D 20:01:04.350675] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.350685] io_start_flow() issuing flow to read data. [D 20:01:04.350692] flow: fsize: 480247808, server_nr: 0, server_ct: 1 [D 20:01:04.350700] file_req_offset: 469762048, aggregate_size: 33554432, handle: 2863311531 [D 20:01:04.350712] PINT_new_request_states n=1 [D 20:01:04.350724] flowproto posting 0x2aaab15d5af0 [D 20:01:04.350755] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.350764] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab15d5af0. [D 20:01:04.350772] (send callback time) ini posts: 7, pending: 0, last: 0, src_list emtpy: yes [D 20:01:04.350781] ========================================================= [D 20:01:04.350789] PINT_process_request [D 20:01:04.350795] tiling 16 copies [D 20:01:04.350801] skipping ahead to target_offset [D 20:01:04.350807] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.350815] lvl 0 el 0 blk 0 by 0 [D 20:01:04.350821] to 0 ta 469762048 fi 503316480 [D 20:01:04.350828] basic type or contiguous data [D 20:01:04.350834] Request disp [D 20:01:04.350840] contig_offset = 0 contig_size = 536870912 lvl_flag = 1 [D 20:01:04.350847] process logical skip [D 20:01:04.350854] retval = 469762048 [D 20:01:04.350860] exiting logical skip because distribute indicates done [D 20:01:04.350866] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.350873] lvl 0 el 0 blk 0 by 469762048 [D 20:01:04.350879] to 469762048 ta 469762048 fi 503316480 [D 20:01:04.350887] basic type or contiguous data [D 20:01:04.350893] Request disp [D 20:01:04.350899] contig_offset = 469762048 contig_size = 67108864 lvl_flag = 1 [D 20:01:04.350906] calling distribute [D 20:01:04.350912] PINT_distribute [D 20:01:04.350918] of 469762048 sz 33554432 ix 0 sm 64 by 0 bm 2097152 fsz 480247808 exfl 0 [D 20:01:04.350926] begin iteration loff: 469762048 [D 20:01:04.350932] segment exceeds byte limit [D 20:01:04.350938] process a segment [D 20:01:04.350944] of 469762048 sz 2097152 [D 20:01:04.350951] add a segment 0 [D 20:01:04.350957] end iteration [D 20:01:04.350963] done with segments or bytes [D 20:01:04.350969] of 471859200 sz 31457280 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.350976] next loff: 471859200 next poff: 471859200 [D 20:01:04.350982] return value 2097152 [D 20:01:04.350988] finished [D 20:01:04.350994] retval = 2097152 [D 20:01:04.351000] exiting distribute returned less than expected [D 20:01:04.351006] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 471859200 fo 503316480 eof 0 [D 20:01:04.351014] ========================================================= [D 20:01:04.351022] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.351029] dbpf_bstream_rw_list: mem_offset: 0x2aaab4217010, mem_size: 2097152 [D 20:01:04.351035] dbpf_bstream_rw_list: stream_offset: 469762048, stream_size: 2097152 [D 20:01:04.351042] dbpf_open_cache_get: called [D 20:01:04.351049] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.351056] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.351090] op_queue add: 0x2aaab15d85e0 [D 20:01:04.351101] delayed I/O operation 0x2aaab15d85e0 (16 already in progress) [D 20:01:04.351108] DBPF I/O ops in progress: 16 [D 20:01:04.351116] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.351122] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab15d5af0. [D 20:01:04.351130] (send callback time) ini posts: 6, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.351139] ========================================================= [D 20:01:04.351145] PINT_process_request [D 20:01:04.351152] tiling 16 copies [D 20:01:04.351158] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.351165] lvl 0 el 0 blk 0 by 471859200 [D 20:01:04.351172] to 471859200 ta 469762048 fi 503316480 [D 20:01:04.351179] basic type or contiguous data [D 20:01:04.351185] Request disp [D 20:01:04.351190] contig_offset = 471859200 contig_size = 65011712 lvl_flag = 1 [D 20:01:04.351197] calling distribute [D 20:01:04.351203] PINT_distribute [D 20:01:04.351209] of 471859200 sz 31457280 ix 0 sm 64 by 0 bm 2097152 fsz 480247808 exfl 0 [D 20:01:04.351217] begin iteration loff: 471859200 [D 20:01:04.351223] segment exceeds byte limit [D 20:01:04.351229] process a segment [D 20:01:04.351235] of 471859200 sz 2097152 [D 20:01:04.351242] add a segment 0 [D 20:01:04.351248] end iteration [D 20:01:04.351254] done with segments or bytes [D 20:01:04.351260] of 473956352 sz 29360128 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.351267] next loff: 473956352 next poff: 473956352 [D 20:01:04.351275] return value 2097152 [D 20:01:04.351281] finished [D 20:01:04.351287] retval = 2097152 [D 20:01:04.351293] exiting distribute returned less than expected [D 20:01:04.351299] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 473956352 fo 503316480 eof 0 [D 20:01:04.351306] ========================================================= [D 20:01:04.351313] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.351321] dbpf_bstream_rw_list: mem_offset: 0x2aaab2608010, mem_size: 2097152 [D 20:01:04.351327] dbpf_bstream_rw_list: stream_offset: 471859200, stream_size: 2097152 [D 20:01:04.351334] dbpf_open_cache_get: called [D 20:01:04.351340] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.351347] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.351370] op_queue add: 0x2aaab15db140 [D 20:01:04.351380] delayed I/O operation 0x2aaab15db140 (16 already in progress) [D 20:01:04.351386] DBPF I/O ops in progress: 16 [D 20:01:04.351393] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.351399] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab15d5af0. [D 20:01:04.351406] (send callback time) ini posts: 5, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.351414] ========================================================= [D 20:01:04.351421] PINT_process_request [D 20:01:04.351426] tiling 16 copies [D 20:01:04.351433] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.351440] lvl 0 el 0 blk 0 by 473956352 [D 20:01:04.351446] to 473956352 ta 469762048 fi 503316480 [D 20:01:04.351453] basic type or contiguous data [D 20:01:04.351459] Request disp [D 20:01:04.351465] contig_offset = 473956352 contig_size = 62914560 lvl_flag = 1 [D 20:01:04.351472] calling distribute [D 20:01:04.351478] PINT_distribute [D 20:01:04.351484] of 473956352 sz 29360128 ix 0 sm 64 by 0 bm 2097152 fsz 480247808 exfl 0 [D 20:01:04.351492] begin iteration loff: 473956352 [D 20:01:04.351498] segment exceeds byte limit [D 20:01:04.351504] process a segment [D 20:01:04.351510] of 473956352 sz 2097152 [D 20:01:04.351517] add a segment 0 [D 20:01:04.351523] end iteration [D 20:01:04.351529] done with segments or bytes [D 20:01:04.351535] of 476053504 sz 27262976 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.351542] next loff: 476053504 next poff: 476053504 [D 20:01:04.351548] return value 2097152 [D 20:01:04.351554] finished [D 20:01:04.351560] retval = 2097152 [D 20:01:04.351566] exiting distribute returned less than expected [D 20:01:04.351572] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 476053504 fo 503316480 eof 0 [D 20:01:04.351580] ========================================================= [D 20:01:04.351587] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.351594] dbpf_bstream_rw_list: mem_offset: 0x2aaab08dc010, mem_size: 2097152 [D 20:01:04.351601] dbpf_bstream_rw_list: stream_offset: 473956352, stream_size: 2097152 [D 20:01:04.351608] dbpf_open_cache_get: called [D 20:01:04.351614] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.351620] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.351649] op_queue add: 0x2aaab15ddca0 [D 20:01:04.351659] delayed I/O operation 0x2aaab15ddca0 (16 already in progress) [D 20:01:04.351667] DBPF I/O ops in progress: 16 [D 20:01:04.351673] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.351679] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab15d5af0. [D 20:01:04.351687] (send callback time) ini posts: 4, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.351694] ========================================================= [D 20:01:04.351701] PINT_process_request [D 20:01:04.351707] tiling 16 copies [D 20:01:04.351713] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.351720] lvl 0 el 0 blk 0 by 476053504 [D 20:01:04.351726] to 476053504 ta 469762048 fi 503316480 [D 20:01:04.351733] basic type or contiguous data [D 20:01:04.351740] Request disp [D 20:01:04.351746] contig_offset = 476053504 contig_size = 60817408 lvl_flag = 1 [D 20:01:04.351752] calling distribute [D 20:01:04.351758] PINT_distribute [D 20:01:04.351764] of 476053504 sz 27262976 ix 0 sm 64 by 0 bm 2097152 fsz 480247808 exfl 0 [D 20:01:04.351772] begin iteration loff: 476053504 [D 20:01:04.351779] segment exceeds byte limit [D 20:01:04.351785] process a segment [D 20:01:04.351790] of 476053504 sz 2097152 [D 20:01:04.351797] add a segment 0 [D 20:01:04.351802] end iteration [D 20:01:04.351808] done with segments or bytes [D 20:01:04.351814] of 478150656 sz 25165824 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.351821] next loff: 478150656 next poff: 478150656 [D 20:01:04.351828] return value 2097152 [D 20:01:04.351834] finished [D 20:01:04.351840] retval = 2097152 [D 20:01:04.351846] exiting distribute returned less than expected [D 20:01:04.351852] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 478150656 fo 503316480 eof 0 [D 20:01:04.351860] ========================================================= [D 20:01:04.351867] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.351874] dbpf_bstream_rw_list: mem_offset: 0x2aaaaecce010, mem_size: 2097152 [D 20:01:04.351880] dbpf_bstream_rw_list: stream_offset: 476053504, stream_size: 2097152 [D 20:01:04.351887] dbpf_open_cache_get: called [D 20:01:04.351894] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.351900] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.351929] op_queue add: 0x2aaab15e0800 [D 20:01:04.351939] delayed I/O operation 0x2aaab15e0800 (16 already in progress) [D 20:01:04.351946] DBPF I/O ops in progress: 16 [D 20:01:04.351952] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.351959] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab15d5af0. [D 20:01:04.351966] (send callback time) ini posts: 3, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.351974] ========================================================= [D 20:01:04.351980] PINT_process_request [D 20:01:04.351986] tiling 16 copies [D 20:01:04.351993] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.352000] lvl 0 el 0 blk 0 by 478150656 [D 20:01:04.352007] to 478150656 ta 469762048 fi 503316480 [D 20:01:04.352013] basic type or contiguous data [D 20:01:04.352019] Request disp [D 20:01:04.352025] contig_offset = 478150656 contig_size = 58720256 lvl_flag = 1 [D 20:01:04.352032] calling distribute [D 20:01:04.352038] PINT_distribute [D 20:01:04.352044] of 478150656 sz 25165824 ix 0 sm 64 by 0 bm 2097152 fsz 480247808 exfl 0 [D 20:01:04.352052] begin iteration loff: 478150656 [D 20:01:04.352058] segment exceeds byte limit [D 20:01:04.352064] process a segment [D 20:01:04.352070] of 478150656 sz 2097152 [D 20:01:04.352076] add a segment 0 [D 20:01:04.352082] end iteration [D 20:01:04.352088] done with segments or bytes [D 20:01:04.352094] of 480247808 sz 23068672 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.352101] next loff: 480247808 next poff: 480247808 [D 20:01:04.352107] [return value] 25165824 (EOF) [D 20:01:04.352114] return value 2097152 (EOF) [D 20:01:04.352120] finished [D 20:01:04.352126] retval = 2097152 [D 20:01:04.352132] exiting distribute returned less than expected [D 20:01:04.352138] done sg 1 sm 64 by 2097152 bm 2097152 ta 469762048 to 480247808 fo 503316480 eof 1 [D 20:01:04.352146] ========================================================= [D 20:01:04.352154] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.352161] dbpf_bstream_rw_list: mem_offset: 0x2aaaad0c0010, mem_size: 2097152 [D 20:01:04.352168] dbpf_bstream_rw_list: stream_offset: 478150656, stream_size: 2097152 [D 20:01:04.352174] dbpf_open_cache_get: called [D 20:01:04.352180] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.352187] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.436081] op_queue add: 0x2aaab15e3360 [D 20:01:04.436093] delayed I/O operation 0x2aaab15e3360 (16 already in progress) [D 20:01:04.436099] DBPF I/O ops in progress: 16 [D 20:01:04.436108] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.436114] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab15d5af0. [D 20:01:04.436123] (send callback time) ini posts: 2, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.436130] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.436137] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab15d5af0. [D 20:01:04.436144] (send callback time) ini posts: 1, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.342865] openib_post_rr: 10.1.4.56:39990 bh 81 wr_id 684aa0. [D 20:01:04.436166] openib_check_cq: completed recv wr_id 684ad0. [D 20:01:04.436179] openib_post_sr_rdmaw: ibv_post_send wr_id 6ed430 to 10.1.4.56:39990 remote addr a4251000 rkey 108c27 nic_wr_credit 2. [D 20:01:04.436189] openib_post_rr: 10.1.4.56:39990 bh 82 wr_id 684ad0. [D 20:01:04.436197] openib_check_cq: completed recv wr_id 684b00. [D 20:01:04.436204] openib_post_sr_rdmaw: ibv_post_send wr_id 6e1030 to 10.1.4.56:39990 remote addr 9fc4d000 rkey 108e27 nic_wr_credit 3. [D 20:01:04.436212] openib_post_rr: 10.1.4.56:39990 bh 83 wr_id 684b00. [D 20:01:04.436219] openib_check_cq: completed send wr_id 682c00. [D 20:01:04.436227] openib_check_cq: completed send wr_id 682c30. [D 20:01:04.436234] openib_check_cq: completed send wr_id 682c60. [D 20:01:04.436241] openib_check_cq: completed send wr_id 682c90. [D 20:01:04.436247] openib_check_cq: completed recv wr_id 684b30. [D 20:01:04.436254] openib_post_sr_rdmaw: ibv_post_send wr_id 64d9b0 to 10.1.4.56:39990 remote addr 9ce4b000 rkey 108f27 nic_wr_credit 4. [D 20:01:04.436262] openib_post_rr: 10.1.4.56:39990 bh 84 wr_id 684b30. [D 20:01:04.436269] openib_check_cq: completed rdma wr_id 6ed990, nic_wr_credit 3. [D 20:01:04.436278] post_sr: 10.1.4.56:39990 bh 22 len 16 wr_id 682cc0. [D 20:01:04.436286] openib_check_cq: completed recv wr_id 684b60. [D 20:01:04.436294] openib_post_sr_rdmaw: ibv_post_send wr_id 6bfab0 to 10.1.4.56:39990 remote addr a204f000 rkey 108d27 nic_wr_credit 4. [D 20:01:04.436302] openib_post_rr: 10.1.4.56:39990 bh 85 wr_id 684b60. [D 20:01:04.436309] openib_check_cq: completed recv wr_id 684b90. [D 20:01:04.436316] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec210 to 10.1.4.56:39990 remote addr 9a849000 rkey 109126 nic_wr_credit 5. [D 20:01:04.436324] openib_post_rr: 10.1.4.56:39990 bh 86 wr_id 684b90. [D 20:01:04.436339] flowproto-multiqueue forcing bmi_send_callback_fn. [D 20:01:04.352204] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.436343] post_sr: 10.1.4.56:39990 bh 23 len 32 wr_id 682cf0. [D 20:01:04.436348] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 1, flow: 0x2aaab15d5af0. [D 20:01:04.436430] (send callback time) ini posts: 0, pending: 0, last: 0, src_list emtpy: no [D 20:01:04.436437] flowproto posted 0x2aaab15d5af0 [D 20:01:04.436540] Job flows in progress (post time): 15 [D 20:01:04.436551] [SM Exiting]: (0x66fad0) io_sm:start_flow (error code: 0), (action: DEFERRED) [D 20:01:04.436560] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [D 20:01:04.244617] --- aio_progress_notification called with handle 2863311531 (0x2aaabd310fa0) [D 20:01:04.352596] --- aio_progress_notification called with handle 2863311531 (0x2aaab1505740) [D 20:01:04.352617] --- aio_progress_notification called with handle 2863311531 (0x2aaab15055f0) [D 20:01:04.437648] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.437661] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.437671] dbpf_open_cache_put: cached entry. [D 20:01:04.437652] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.437686] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.437694] dbpf_open_cache_put: cached entry. [D 20:01:04.356604] --- aio_progress_notification called with handle 2863311531 (0x2aaab1505ff0) [D 20:01:04.437696] op_queue add: 0x2aaabd310fa0 [D 20:01:04.437713] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.437716] DBPF I/O ops in progress: 15 [D 20:01:04.437731] dbpf_open_cache_put: cached entry. [D 20:01:04.437733] starting delayed I/O operation 0x2aaab152ce40 (15 in progress) [D 20:01:04.437744] dbpf_open_cache_put: cached entry. [D 20:01:04.356682] --- aio_progress_notification called with handle 2863311531 (0x2aaab1505ea0) [D 20:01:04.437754] lio_listio called with 1 following aiocbs: [D 20:01:04.437742] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.356715] --- aio_progress_notification called with handle 2863311531 (0x2aaab1505d50) [D 20:01:04.437767] aiocb_ptr_array[0]: fd: 13, off: 301989888, bytes: 2097152, buf: 0x2aaab7231010, type: 0 [D 20:01:04.437786] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.437792] start_delayed_ops_if_any: lio_listio posted 0x2aaab152ce40 (handle 2863311531, ret 0)) [D 20:01:04.437800] dbpf_open_cache_put: cached entry. [D 20:01:04.437810] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.352598] --- aio_progress_notification called with handle 2863311531 (0x2aaab15054a0) [D 20:01:04.360442] --- aio_progress_notification called with handle 2863311531 (0x2aaab1506140) [D 20:01:04.437705] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.437777] op_queue add: 0x2aaab1505ff0 [D 20:01:04.437848] DBPF I/O ops in progress: 15 [D 20:01:04.437856] starting delayed I/O operation 0x2aaab152cf90 (15 in progress) [D 20:01:04.437868] lio_listio called with 1 following aiocbs: [D 20:01:04.437874] aiocb_ptr_array[0]: fd: 13, off: 304087040, bytes: 2097152, buf: 0x2aaab5420010, type: 0 [D 20:01:04.437884] start_delayed_ops_if_any: lio_listio posted 0x2aaab152cf90 (handle 2863311531, ret 0)) [D 20:01:04.437756] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.437909] op_queue add: 0x2aaab15055f0 [D 20:01:04.437918] DBPF I/O ops in progress: 15 [D 20:01:04.437926] starting delayed I/O operation 0x2aaab152d0e0 (15 in progress) [D 20:01:04.437937] lio_listio called with 1 following aiocbs: [D 20:01:04.437943] aiocb_ptr_array[0]: fd: 13, off: 306184192, bytes: 2097152, buf: 0x2aaab3611010, type: 0 [D 20:01:04.437952] start_delayed_ops_if_any: lio_listio posted 0x2aaab152d0e0 (handle 2863311531, ret 0)) [D 20:01:04.368601] --- aio_progress_notification called with handle 2863311531 (0x2aaab1522b60) [D 20:01:04.437977] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.437995] dbpf_open_cache_put: cached entry. [D 20:01:04.438002] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.438009] op_queue add: 0x2aaab1522b60 [D 20:01:04.438017] DBPF I/O ops in progress: 15 [D 20:01:04.438025] starting delayed I/O operation 0x2aaab152bf70 (15 in progress) [D 20:01:04.438035] lio_listio called with 1 following aiocbs: [D 20:01:04.438041] aiocb_ptr_array[0]: fd: 13, off: 308281344, bytes: 2097152, buf: 0x2aaab1a02010, type: 0 [D 20:01:04.438050] start_delayed_ops_if_any: lio_listio posted 0x2aaab152bf70 (handle 2863311531, ret 0)) [D 20:01:04.438065] op_queue add: 0x2aaab1505d50 [D 20:01:04.438076] DBPF I/O ops in progress: 15 [D 20:01:04.438085] starting delayed I/O operation 0x2aaab1561b70 (15 in progress) [D 20:01:04.438096] lio_listio called with 1 following aiocbs: [D 20:01:04.438103] aiocb_ptr_array[0]: fd: 13, off: 310378496, bytes: 2097152, buf: 0x2aaaafcd6010, type: 0 [D 20:01:04.438111] start_delayed_ops_if_any: lio_listio posted 0x2aaab1561b70 (handle 2863311531, ret 0)) [D 20:01:04.368601] --- aio_progress_notification called with handle 2863311531 (0x2aaab15063e0) [D 20:01:04.438167] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.438189] dbpf_open_cache_put: cached entry. [D 20:01:04.438197] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.438204] op_queue add: 0x2aaab15063e0 [D 20:01:04.438213] DBPF I/O ops in progress: 15 [D 20:01:04.438219] starting delayed I/O operation 0x2aaab1561cc0 (15 in progress) [D 20:01:04.438230] lio_listio called with 1 following aiocbs: [D 20:01:04.438237] aiocb_ptr_array[0]: fd: 13, off: 312475648, bytes: 2097152, buf: 0x2aaaae0c8010, type: 0 [D 20:01:04.438247] start_delayed_ops_if_any: lio_listio posted 0x2aaab1561cc0 (handle 2863311531, ret 0)) [D 20:01:04.438280] op_queue add: 0x2aaab1505740 [D 20:01:04.438293] DBPF I/O ops in progress: 15 [D 20:01:04.438301] starting delayed I/O operation 0x2aaab1561e10 (15 in progress) [D 20:01:04.438313] lio_listio called with 1 following aiocbs: [D 20:01:04.438319] aiocb_ptr_array[0]: fd: 13, off: 314572800, bytes: 2097152, buf: 0x2aaabac4f010, type: 0 [D 20:01:04.438329] start_delayed_ops_if_any: lio_listio posted 0x2aaab1561e10 (handle 2863311531, ret 0)) [D 20:01:04.368622] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300e10) [D 20:01:04.438369] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.438391] dbpf_open_cache_put: cached entry. [D 20:01:04.438398] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.438405] op_queue add: 0x2aaabd300e10 [D 20:01:04.438413] DBPF I/O ops in progress: 15 [D 20:01:04.438421] starting delayed I/O operation 0x2aaab152c0c0 (15 in progress) [D 20:01:04.438433] lio_listio called with 1 following aiocbs: [D 20:01:04.438440] aiocb_ptr_array[0]: fd: 13, off: 316669952, bytes: 2097152, buf: 0x2aaab8c3e010, type: 0 [D 20:01:04.438450] start_delayed_ops_if_any: lio_listio posted 0x2aaab152c0c0 (handle 2863311531, ret 0)) [D 20:01:04.368615] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300a20) [D 20:01:04.440636] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.440694] dbpf_open_cache_put: cached entry. [D 20:01:04.440714] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.440730] op_queue add: 0x2aaabd300a20 [D 20:01:04.440747] DBPF I/O ops in progress: 15 [D 20:01:04.440763] starting delayed I/O operation 0x2aaab15067c0 (15 in progress) [D 20:01:04.440780] lio_listio called with 1 following aiocbs: [D 20:01:04.440794] aiocb_ptr_array[0]: fd: 13, off: 335544320, bytes: 2097152, buf: 0x2aaab7030010, type: 0 [D 20:01:04.440814] start_delayed_ops_if_any: lio_listio posted 0x2aaab15067c0 (handle 2863311531, ret 0)) [D 20:01:04.437829] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.440876] dbpf_open_cache_put: cached entry. [D 20:01:04.440894] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.440909] op_queue add: 0x2aaab1506140 [D 20:01:04.440924] DBPF I/O ops in progress: 15 [D 20:01:04.440939] starting delayed I/O operation 0x2aaab1552390 (15 in progress) [D 20:01:04.440957] lio_listio called with 1 following aiocbs: [D 20:01:04.440971] aiocb_ptr_array[0]: fd: 13, off: 337641472, bytes: 2097152, buf: 0x2aaab501e010, type: 0 [D 20:01:04.440990] start_delayed_ops_if_any: lio_listio posted 0x2aaab1552390 (handle 2863311531, ret 0)) [D 20:01:04.370337] --- aio_progress_notification called with handle 2863311531 (0x2aaab1525840) [D 20:01:04.245861] --- aio_progress_notification called with handle 2863311531 (0x2aaab1520020) [D 20:01:04.436450] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:04.436440] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.441152] (send callback time) ini posts: 0, pending: 4, last: 0, src_list emtpy: no [P 20:01:04.436802] Start times (hr:min:sec): 20:01:00.143 20:00:55.876 20:00:54.832 20:00:53.796 20:00:52.756 20:00:51.720 [D 20:01:04.441161] BMI_post_send: addr: 6843072, offset: 0x2aaab642a010, size: 2097152, tag: 27 [P 20:01:04.441175] Intervals (hr:min:sec) : 00:00:04.293 00:00:04.267 00:00:01.044 00:00:01.036 00:00:01.040 00:00:01.036 [D 20:01:04.441185] post_sr: 10.1.4.56:39990 bh 24 len 32 wr_id 682d20. [P 20:01:04.441187] ------------------------------------------------------------------------------------------------------------- [P 20:01:04.441215] bytes read : 121634816 0 0 0 0 0 [D 20:01:04.441216] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.441243] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.441251] BMI_post_send: addr: 6843072, offset: 0x2aaab0add010, size: 2097152, tag: 28 [D 20:01:04.441166] ========================================================= [D 20:01:04.441259] post_sr: 10.1.4.56:39990 bh 25 len 32 wr_id 682d50. [D 20:01:04.441276] PINT_process_request [D 20:01:04.441287] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:04.441307] BMI_post_send: addr: 6843072, offset: 0x2aaaaeecf010, size: 2097152, tag: 28 [D 20:01:04.441323] post_sr: 10.1.4.56:39990 bh 26 len 32 wr_id 682d80. [D 20:01:04.441333] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.441340] BMI_post_send: addr: 6843072, offset: 0x2aaaad2c1010, size: 2097152, tag: 28 [P 20:01:04.441226] bytes written : 245366784 257949696 0 0 0 0 [P 20:01:04.441362] metadata reads : 42 42 26 25 24 24 [P 20:01:04.441369] metadata writes : 7 7 7 7 7 7 [P 20:01:04.441376] metadata dspace ops : 0 0 0 0 0 0 [P 20:01:04.441382] metadata keyval ops : 0 0 0 0 0 0 [P 20:01:04.441389] request scheduler : 30 30 14 13 12 12 [D 20:01:04.441409] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:01:04.441421] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:01:04.441433] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) [D 20:01:04.441485] [SM Entering]: (0x64db10) io_sm:send_completion_ack (status: 0) [D 20:01:04.441495] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.441505] PINT_encode_release [D 20:01:04.441513] lebf_encode_rel [D 20:01:04.441524] PINT_encode [D 20:01:04.441533] encode_common [D 20:01:04.441541] lebf_encode_resp [D 20:01:04.441552] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 8 [D 20:01:04.441559] element 0: offset: 0x2aaab152b170, size: 24 [D 20:01:04.441297] tiling 5 copies [D 20:01:04.441348] post_sr: 10.1.4.56:39990 bh 27 len 32 wr_id 682db0. [D 20:01:04.441579] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.441593] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.441612] post_sr: 10.1.4.56:39990 bh 28 len 40 wr_id 682de0. [D 20:01:04.441619] BMI_post_send: addr: 6843072, offset: 0x2aaab9e47010, size: 2097152, tag: 28 [D 20:01:04.441628] [SM Exiting]: (0x64db10) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.441632] post_sr: 10.1.4.56:39990 bh 29 len 32 wr_id 682e10. [D 20:01:04.441643] [SM Entering]: (0x64edd0) io_sm:send_completion_ack (status: 0) [D 20:01:04.441655] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.441675] BMI_post_send: addr: 6843072, offset: 0x2aaaba44a010, size: 2097152, tag: 28 [D 20:01:04.441707] post_sr: 10.1.4.56:39990 bh 30 len 32 wr_id 682e40. [D 20:01:04.441717] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.441734] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.441741] BMI_post_send: addr: 6843072, offset: 0x2aaab682c010, size: 2097152, tag: 29 [D 20:01:04.441750] post_sr: 10.1.4.56:39990 bh 31 len 32 wr_id 682e70. [D 20:01:04.441606] lvl 0 el 0 blk 0 by 125829120 [D 20:01:04.441772] to 125829120 ta 100663296 fi 134217728 [D 20:01:04.441780] basic type or contiguous data [D 20:01:04.441788] Request disp [D 20:01:04.441794] contig_offset = 125829120 contig_size = 41943040 lvl_flag = 1 [D 20:01:04.441801] calling distribute [D 20:01:04.441808] PINT_distribute [D 20:01:04.441815] of 125829120 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:04.441824] begin iteration loff: 125829120 [D 20:01:04.441832] segment exceeds byte limit [D 20:01:04.441840] process a segment [D 20:01:04.441845] of 125829120 sz 2097152 [D 20:01:04.441852] add a segment 0 [D 20:01:04.441858] end iteration [D 20:01:04.441865] done with segments or bytes [D 20:01:04.441871] of 127926272 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.441878] next loff: 127926272 next poff: 127926272 [D 20:01:04.441884] return value 2097152 [D 20:01:04.441891] finished [D 20:01:04.441897] retval = 2097152 [D 20:01:04.441904] exiting distribute returned less than expected [D 20:01:04.441910] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 127926272 fo 134217728 eof 0 [D 20:01:04.441918] ========================================================= [D 20:01:04.441934] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.441943] dbpf_bstream_rw_list: mem_offset: 0x2aaac0217010, mem_size: 2097152 [D 20:01:04.441950] dbpf_bstream_rw_list: stream_offset: 125829120, stream_size: 2097152 [D 20:01:04.441958] dbpf_open_cache_get: called [D 20:01:04.441965] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.441973] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.441998] op_queue add: 0x2aaabd301030 [D 20:01:04.442005] delayed I/O operation 0x2aaabd301030 (16 already in progress) [D 20:01:04.442012] DBPF I/O ops in progress: 16 [D 20:01:04.442021] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518270. [D 20:01:04.442029] (send callback time) ini posts: 0, pending: 0, last: 1, src_list emtpy: yes [D 20:01:04.442036] flowproto completing 0x2aaab1518270 [D 20:01:04.442053] Job flows in progress (callback time): 14 [D 20:01:04.442062] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:04.442069] (send callback time) ini posts: 0, pending: 2, last: 1, src_list emtpy: yes [D 20:01:04.442078] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:04.442087] (send callback time) ini posts: 0, pending: 1, last: 1, src_list emtpy: yes [D 20:01:04.442095] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:04.442103] (send callback time) ini posts: 0, pending: 4, last: 1, src_list emtpy: yes [D 20:01:04.442120] openib_check_cq: completed rdma wr_id 6ed430, nic_wr_credit 4. [D 20:01:04.442132] post_sr: 10.1.4.56:39990 bh 32 len 16 wr_id 682ea0. [D 20:01:04.442140] openib_check_cq: completed rdma wr_id 6e1030, nic_wr_credit 3. [D 20:01:04.442149] post_sr: 10.1.4.56:39990 bh 33 len 16 wr_id 682ed0. [D 20:01:04.442172] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:04.442182] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:04.442190] ========================================================= [D 20:01:04.442197] PINT_process_request [D 20:01:04.442203] tiling 6 copies [D 20:01:04.442210] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.442218] lvl 0 el 0 blk 0 by 153092096 [D 20:01:04.442225] to 153092096 ta 134217728 fi 167772160 [D 20:01:04.442232] basic type or contiguous data [D 20:01:04.442238] Request disp [D 20:01:04.442245] contig_offset = 153092096 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.442252] calling distribute [D 20:01:04.442258] PINT_distribute [D 20:01:04.442264] of 153092096 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.442272] begin iteration loff: 153092096 [D 20:01:04.442279] segment exceeds byte limit [D 20:01:04.442285] process a segment [D 20:01:04.442291] of 153092096 sz 2097152 [D 20:01:04.442297] add a segment 0 [D 20:01:04.442303] end iteration [D 20:01:04.442309] done with segments or bytes [D 20:01:04.442315] of 155189248 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.442323] next loff: 155189248 next poff: 155189248 [D 20:01:04.442329] return value 2097152 [D 20:01:04.442335] finished [D 20:01:04.442341] retval = 2097152 [D 20:01:04.442348] exiting distribute returned less than expected [D 20:01:04.442354] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 155189248 fo 167772160 eof 0 [D 20:01:04.442362] ========================================================= [D 20:01:04.442372] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.442379] dbpf_bstream_rw_list: mem_offset: 0x2aaab9a45010, mem_size: 2097152 [D 20:01:04.442386] dbpf_bstream_rw_list: stream_offset: 153092096, stream_size: 2097152 [D 20:01:04.442393] dbpf_open_cache_get: called [D 20:01:04.442400] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.442407] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.442426] op_queue add: 0x2aaabd301180 [D 20:01:04.442433] delayed I/O operation 0x2aaabd301180 (16 already in progress) [D 20:01:04.442439] DBPF I/O ops in progress: 16 [D 20:01:04.442863] openib_check_cq: completed rdma wr_id 64d9b0, nic_wr_credit 2. [D 20:01:04.442875] post_sr: 10.1.4.56:39990 bh 34 len 16 wr_id 682f00. [D 20:01:04.442884] openib_check_cq: completed send wr_id 682cc0. [D 20:01:04.442897] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c750. [D 20:01:04.442906] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:04.442915] ========================================================= [D 20:01:04.442921] PINT_process_request [D 20:01:04.442927] tiling 7 copies [D 20:01:04.442934] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.442941] lvl 0 el 0 blk 0 by 184549376 [D 20:01:04.442948] to 184549376 ta 167772160 fi 201326592 [D 20:01:04.442955] basic type or contiguous data [D 20:01:04.442962] Request disp [D 20:01:04.442968] contig_offset = 184549376 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.442975] calling distribute [D 20:01:04.442981] PINT_distribute [D 20:01:04.442987] of 184549376 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.442995] begin iteration loff: 184549376 [D 20:01:04.443002] segment exceeds byte limit [D 20:01:04.443008] process a segment [D 20:01:04.443014] of 184549376 sz 2097152 [D 20:01:04.443021] add a segment 0 [D 20:01:04.443027] end iteration [D 20:01:04.443033] done with segments or bytes [D 20:01:04.443039] of 186646528 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.443046] next loff: 186646528 next poff: 186646528 [D 20:01:04.443052] return value 2097152 [D 20:01:04.443059] finished [D 20:01:04.443065] retval = 2097152 [D 20:01:04.443071] exiting distribute returned less than expected [D 20:01:04.443080] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 186646528 fo 201326592 eof 0 [D 20:01:04.443088] ========================================================= [D 20:01:04.443098] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.443106] dbpf_bstream_rw_list: mem_offset: 0x2aaab4418010, mem_size: 2097152 [D 20:01:04.443113] dbpf_bstream_rw_list: stream_offset: 184549376, stream_size: 2097152 [D 20:01:04.443120] dbpf_open_cache_get: called [D 20:01:04.443126] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.443134] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.443157] op_queue add: 0x2aaabd3012d0 [D 20:01:04.443166] delayed I/O operation 0x2aaabd3012d0 (16 already in progress) [D 20:01:04.443174] DBPF I/O ops in progress: 16 [D 20:01:04.444089] openib_check_cq: completed recv wr_id 684bc0. [D 20:01:04.444124] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea9b0 to 10.1.4.56:39990 remote addr 98647000 rkey 109226 nic_wr_credit 3. [D 20:01:04.444138] openib_post_rr: 10.1.4.56:39990 bh 87 wr_id 684bc0. [D 20:01:04.441665] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.441759] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.444655] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.444663] BMI_post_send: addr: 6843072, offset: 0x2aaab6c2e010, size: 2097152, tag: 30 [D 20:01:04.444635] PINT_encode_release [D 20:01:04.444693] lebf_encode_rel [D 20:01:04.444679] post_sr: 10.1.4.56:39990 bh 35 len 32 wr_id 682f30. [D 20:01:04.444719] PINT_encode [D 20:01:04.444725] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.444739] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.444752] BMI_post_send: addr: 6843072, offset: 0x2aaab843a010, size: 2097152, tag: 28 [D 20:01:04.444761] post_sr: 10.1.4.56:39990 bh 36 len 32 wr_id 682f60. [D 20:01:04.444770] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.444778] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.444785] BMI_post_send: addr: 6843072, offset: 0x2aaab481a010, size: 2097152, tag: 29 [D 20:01:04.444793] post_sr: 10.1.4.56:39990 bh 37 len 32 wr_id 682f90. [D 20:01:04.444802] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:04.444811] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.444828] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.444835] BMI_post_send: addr: 6843072, offset: 0x2aaab2c0b010, size: 2097152, tag: 29 [D 20:01:04.444859] post_sr: 10.1.4.56:39990 bh 38 len 32 wr_id 682fc0. [D 20:01:04.444729] encode_common [D 20:01:04.444914] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:04.444927] lebf_encode_resp [D 20:01:04.444936] BMI_post_send: addr: 6843072, offset: 0x2aaab0edf010, size: 2097152, tag: 29 [D 20:01:04.444952] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 9 [D 20:01:04.444964] element 0: offset: 0x2aaab150e9f0, size: 24 [D 20:01:04.444987] post_sr: 10.1.4.56:39990 bh 39 len 32 wr_id 682ff0. [D 20:01:04.445000] op_queue add: 0x2aaab151fd00 [D 20:01:04.445025] DBPF I/O ops in progress: 15 [D 20:01:04.445005] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.445013] post_sr: 10.1.4.56:39990 bh 40 len 40 wr_id 683020. [D 20:01:04.445070] [SM Exiting]: (0x64edd0) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445083] [SM Entering]: (0x650090) io_sm:send_completion_ack (status: 0) [D 20:01:04.445091] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445099] PINT_encode_release [D 20:01:04.445105] lebf_encode_rel [D 20:01:04.445114] PINT_encode [D 20:01:04.445121] encode_common [D 20:01:04.445127] lebf_encode_resp [D 20:01:04.445135] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 10 [D 20:01:04.445143] element 0: offset: 0x2aaab1513550, size: 24 [D 20:01:04.445151] post_sr: 10.1.4.56:39990 bh 41 len 40 wr_id 683050. [D 20:01:04.445160] [SM Exiting]: (0x650090) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445168] [SM Entering]: (0x651350) io_sm:send_completion_ack (status: 0) [D 20:01:04.445175] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445183] PINT_encode_release [D 20:01:04.445189] lebf_encode_rel [D 20:01:04.445197] PINT_encode [D 20:01:04.445203] encode_common [D 20:01:04.445210] lebf_encode_resp [D 20:01:04.445217] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 11 [D 20:01:04.445224] element 0: offset: 0x2aaab152b5a0, size: 24 [D 20:01:04.445232] post_sr: 10.1.4.56:39990 bh 42 len 40 wr_id 683080. [D 20:01:04.445240] [SM Exiting]: (0x651350) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445249] [SM Entering]: (0x652610) io_sm:send_completion_ack (status: 0) [D 20:01:04.445256] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445264] PINT_encode_release [D 20:01:04.445270] lebf_encode_rel [D 20:01:04.445278] PINT_encode [D 20:01:04.445284] encode_common [D 20:01:04.445291] lebf_encode_resp [D 20:01:04.445298] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 12 [D 20:01:04.445305] element 0: offset: 0x2aaab1514e20, size: 24 [D 20:01:04.445314] post_sr: 10.1.4.56:39990 bh 43 len 40 wr_id 6830b0. [D 20:01:04.445322] [SM Exiting]: (0x652610) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445330] [SM Entering]: (0x655e50) io_sm:send_completion_ack (status: 0) [D 20:01:04.445337] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445345] PINT_encode_release [D 20:01:04.445351] lebf_encode_rel [D 20:01:04.445359] PINT_encode [D 20:01:04.445365] encode_common [D 20:01:04.445371] lebf_encode_resp [D 20:01:04.445378] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 15 [D 20:01:04.445385] element 0: offset: 0x2aaab1522880, size: 24 [D 20:01:04.445393] post_sr: 10.1.4.56:39990 bh 44 len 40 wr_id 6830e0. [D 20:01:04.445401] [SM Exiting]: (0x655e50) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445410] [SM Entering]: (0x6538d0) io_sm:send_completion_ack (status: 0) [D 20:01:04.445418] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445425] PINT_encode_release [D 20:01:04.445432] lebf_encode_rel [D 20:01:04.445440] PINT_encode [D 20:01:04.445446] encode_common [D 20:01:04.445452] lebf_encode_resp [D 20:01:04.445459] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 13 [D 20:01:04.445466] element 0: offset: 0x2aaab152c5e0, size: 24 [D 20:01:04.445474] post_sr: 10.1.4.56:39990 bh 45 len 40 wr_id 683110. [D 20:01:04.445482] [SM Exiting]: (0x6538d0) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445491] [SM Entering]: (0x654b90) io_sm:send_completion_ack (status: 0) [D 20:01:04.445498] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445506] PINT_encode_release [D 20:01:04.445512] lebf_encode_rel [D 20:01:04.445520] PINT_encode [D 20:01:04.445526] encode_common [D 20:01:04.445532] lebf_encode_resp [D 20:01:04.445539] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 14 [D 20:01:04.445546] element 0: offset: 0x2aaab1517d20, size: 24 [D 20:01:04.445554] post_sr: 10.1.4.56:39990 bh 46 len 40 wr_id 683140. [D 20:01:04.445563] [SM Exiting]: (0x654b90) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445571] [SM Entering]: (0x6583d0) io_sm:send_completion_ack (status: 0) [D 20:01:04.445578] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445586] PINT_encode_release [D 20:01:04.445592] lebf_encode_rel [D 20:01:04.445601] PINT_encode [D 20:01:04.445607] encode_common [D 20:01:04.445613] lebf_encode_resp [D 20:01:04.445620] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 17 [D 20:01:04.445631] element 0: offset: 0x2aaab150d6b0, size: 24 [D 20:01:04.445640] post_sr: 10.1.4.56:39990 bh 47 len 40 wr_id 683170. [D 20:01:04.445648] [SM Exiting]: (0x6583d0) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445657] [SM Entering]: (0x65a950) io_sm:send_completion_ack (status: 0) [D 20:01:04.445663] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445671] PINT_encode_release [D 20:01:04.445678] lebf_encode_rel [D 20:01:04.445687] PINT_encode [D 20:01:04.445693] encode_common [D 20:01:04.445699] lebf_encode_resp [D 20:01:04.445707] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 19 [D 20:01:04.445714] element 0: offset: 0x2aaab1518a30, size: 24 [D 20:01:04.445723] post_sr: 10.1.4.56:39990 bh 48 len 40 wr_id 6831a0. [D 20:01:04.445731] [SM Exiting]: (0x65a950) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445739] [SM Entering]: (0x65bc10) io_sm:send_completion_ack (status: 0) [D 20:01:04.445746] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445753] PINT_encode_release [D 20:01:04.445760] lebf_encode_rel [D 20:01:04.445769] PINT_encode [D 20:01:04.445775] encode_common [D 20:01:04.445781] lebf_encode_resp [D 20:01:04.445789] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 20 [D 20:01:04.445796] element 0: offset: 0x2aaab1518cc0, size: 24 [D 20:01:04.445804] post_sr: 10.1.4.56:39990 bh 49 len 40 wr_id 6831d0. [D 20:01:04.445813] [SM Exiting]: (0x65bc10) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445821] [SM Entering]: (0x65ced0) io_sm:send_completion_ack (status: 0) [D 20:01:04.445828] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445836] PINT_encode_release [D 20:01:04.445842] lebf_encode_rel [D 20:01:04.445850] PINT_encode [D 20:01:04.445856] encode_common [D 20:01:04.445862] lebf_encode_resp [D 20:01:04.445870] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 21 [D 20:01:04.445876] element 0: offset: 0x2aaab1518f50, size: 24 [D 20:01:04.445884] post_sr: 10.1.4.56:39990 bh 50 len 40 wr_id 683200. [D 20:01:04.445893] [SM Exiting]: (0x65ced0) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445901] [SM Entering]: (0x659690) io_sm:send_completion_ack (status: 0) [D 20:01:04.445908] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445915] PINT_encode_release [D 20:01:04.445921] lebf_encode_rel [D 20:01:04.445930] PINT_encode [D 20:01:04.445936] encode_common [D 20:01:04.445943] lebf_encode_resp [D 20:01:04.445950] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 18 [D 20:01:04.445957] element 0: offset: 0x2aaab15191e0, size: 24 [D 20:01:04.445965] post_sr: 10.1.4.56:39990 bh 51 len 40 wr_id 683230. [D 20:01:04.445973] [SM Exiting]: (0x659690) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.445982] [SM Entering]: (0x657110) io_sm:send_completion_ack (status: 0) [D 20:01:04.445989] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.445997] PINT_encode_release [D 20:01:04.446003] lebf_encode_rel [D 20:01:04.446011] PINT_encode [D 20:01:04.446018] encode_common [D 20:01:04.446024] lebf_encode_resp [D 20:01:04.446031] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 16 [D 20:01:04.446038] element 0: offset: 0x2aaab1519470, size: 24 [D 20:01:04.446046] post_sr: 10.1.4.56:39990 bh 52 len 40 wr_id 683260. [D 20:01:04.446055] [SM Exiting]: (0x657110) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.446063] [SM Entering]: (0x65e190) io_sm:send_completion_ack (status: 0) [D 20:01:04.446070] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.446077] PINT_encode_release [D 20:01:04.446083] lebf_encode_rel [D 20:01:04.446094] PINT_encode [D 20:01:04.446101] encode_common [D 20:01:04.446107] lebf_encode_resp [D 20:01:04.446114] BMI_post_send_list: addr: 6843072, count: 1, total_size: 24, tag: 22 [D 20:01:04.446124] element 0: offset: 0x2aaab1519700, size: 24 [D 20:01:04.446132] post_sr: 10.1.4.56:39990 bh 53 len 40 wr_id 683290. [D 20:01:04.446140] [SM Exiting]: (0x65e190) io_sm:send_completion_ack (error code: 0), (action: DEFERRED) [D 20:01:04.446161] [SM Entering]: (0x66e810) io_sm:send_completion_ack (status: 0) [D 20:01:04.446170] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.446178] [SM Exiting]: (0x66e810) io_sm:send_completion_ack (error code: 0), (action: COMPLETE) [D 20:01:04.446185] [SM Entering]: (0x66e810) io_sm:release (status: 0) [D 20:01:04.446192] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.446203] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x2aaab150f6a0 [D 20:01:04.446214] [SM Exiting]: (0x66e810) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.446221] [SM Entering]: (0x66e810) io_sm:cleanup (status: 0) [D 20:01:04.446227] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.360492] --- aio_progress_notification called with handle 2863311531 (0x2aaab1506290) [D 20:01:04.446347] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.446378] dbpf_open_cache_put: cached entry. [D 20:01:04.445051] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.446435] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.446442] BMI_post_send: addr: 6843072, offset: 0x2aaaaf2d1010, size: 2097152, tag: 29 [D 20:01:04.446453] post_sr: 10.1.4.56:39990 bh 54 len 32 wr_id 6832c0. [D 20:01:04.446466] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.446473] BMI_post_send: addr: 6843072, offset: 0x2aaaad6c3010, size: 2097152, tag: 29 [D 20:01:04.446388] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.446483] openib_check_cq: completed rdma wr_id 6bfab0, nic_wr_credit 2. [D 20:01:04.446505] post_sr: 10.1.4.56:39990 bh 55 len 16 wr_id 6832f0. [D 20:01:04.445040] starting delayed I/O operation 0x2aaab15524e0 (15 in progress) [D 20:01:04.446584] lio_listio called with 1 following aiocbs: [D 20:01:04.446591] aiocb_ptr_array[0]: fd: 13, off: 339738624, bytes: 2097152, buf: 0x2aaab3410010, type: 0 [D 20:01:04.446603] start_delayed_ops_if_any: lio_listio posted 0x2aaab15524e0 (handle 2863311531, ret 0)) [D 20:01:04.444992] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.446680] dbpf_open_cache_put: cached entry. [D 20:01:04.446687] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.446698] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.446725] dbpf_open_cache_put: cached entry. [D 20:01:04.446733] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.446750] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.446776] dbpf_open_cache_put: cached entry. [D 20:01:04.446784] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.446486] op_queue add: 0x2aaab1506290 [D 20:01:04.448695] op_queue add: 0x2aaab1520020 [D 20:01:04.448718] DBPF I/O ops in progress: 15 [D 20:01:04.448728] starting delayed I/O operation 0x2aaab1552630 (15 in progress) [D 20:01:04.448739] lio_listio called with 1 following aiocbs: [D 20:01:04.448746] aiocb_ptr_array[0]: fd: 13, off: 341835776, bytes: 2097152, buf: 0x2aaab1801010, type: 0 [D 20:01:04.448762] start_delayed_ops_if_any: lio_listio posted 0x2aaab1552630 (handle 2863311531, ret 0)) [D 20:01:04.448798] DBPF I/O ops in progress: 15 [D 20:01:04.448809] starting delayed I/O operation 0x2aaab1506910 (15 in progress) [D 20:01:04.448819] lio_listio called with 1 following aiocbs: [D 20:01:04.448826] aiocb_ptr_array[0]: fd: 13, off: 343932928, bytes: 2097152, buf: 0x2aaaafad5010, type: 0 [D 20:01:04.448835] start_delayed_ops_if_any: lio_listio posted 0x2aaab1506910 (handle 2863311531, ret 0)) [D 20:01:04.446561] post_sr: 10.1.4.56:39990 bh 56 len 32 wr_id 683320. [D 20:01:04.448916] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.448934] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.448943] BMI_post_send: addr: 6843072, offset: 0x2aaab4c1c010, size: 2097152, tag: 30 [D 20:01:04.448958] post_sr: 10.1.4.56:39990 bh 57 len 32 wr_id 683350. [D 20:01:04.448968] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:04.448977] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.448984] BMI_post_send: addr: 6843072, offset: 0x2aaaba84c010, size: 2097152, tag: 29 [D 20:01:04.448993] post_sr: 10.1.4.56:39990 bh 58 len 32 wr_id 683380. [D 20:01:04.449003] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.449033] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.449042] BMI_post_send: addr: 6843072, offset: 0x2aaab883c010, size: 2097152, tag: 29 [D 20:01:04.449052] post_sr: 10.1.4.56:39990 bh 59 len 32 wr_id 6833b0. [D 20:01:04.449062] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.449071] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:04.449081] BMI_post_send: addr: 6843072, offset: 0x2aaabfc14010, size: 2097152, tag: 26 [D 20:01:04.449091] post_sr: 10.1.4.56:39990 bh 60 len 32 wr_id 6833e0. [D 20:01:04.449100] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.449109] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.449116] BMI_post_send: addr: 6843072, offset: 0x2aaab300e010, size: 2097152, tag: 30 [D 20:01:04.449125] post_sr: 10.1.4.56:39990 bh 61 len 32 wr_id 683410. [D 20:01:04.449134] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:04.449142] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:04.449150] BMI_post_send: addr: 6843072, offset: 0x2aaabfe15010, size: 2097152, tag: 26 [D 20:01:04.449159] post_sr: 10.1.4.56:39990 bh 62 len 32 wr_id 683440. [D 20:01:04.449169] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.449179] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.449194] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.449203] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.449211] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.449218] BMI_post_send: addr: 6843072, offset: 0x2aaab9844010, size: 2097152, tag: 31 [D 20:01:04.449227] post_sr: 10.1.4.56:39990 bh 63 len 32 wr_id 683470. [D 20:01:04.449237] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.449245] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.449253] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.449312] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.449328] BMI_post_send: addr: 6843072, offset: 0x2aaac0a1b010, size: 2097152, tag: 27 [D 20:01:04.449337] post_sr: 10.1.4.56:39990 bh 64 len 32 wr_id 6834a0. [D 20:01:04.449348] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.449356] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:04.449363] BMI_post_send: addr: 6843072, offset: 0x2aaac0016010, size: 2097152, tag: 26 [D 20:01:04.449372] post_sr: 10.1.4.56:39990 bh 65 len 32 wr_id 6834d0. [D 20:01:04.449381] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.449389] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.449398] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.449405] BMI_post_send: addr: 6843072, offset: 0x2aaab12e1010, size: 2097152, tag: 30 [D 20:01:04.449414] post_sr: 10.1.4.56:39990 bh 66 len 32 wr_id 683500. [D 20:01:04.449423] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.449431] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.449441] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.449448] BMI_post_send: addr: 6843072, offset: 0x2aaabc85d010, size: 2097152, tag: 30 [D 20:01:04.449457] post_sr: 10.1.4.56:39990 bh 67 len 32 wr_id 683530. [D 20:01:04.449466] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.449497] openib_check_cq: completed rdma wr_id 6ec210, nic_wr_credit 1. [D 20:01:04.449514] post_sr: 10.1.4.56:39990 bh 68 len 16 wr_id 683560. [D 20:01:04.449523] openib_check_cq: completed send wr_id 682cf0. [D 20:01:04.449531] openib_check_cq: completed send wr_id 682d20. [D 20:01:04.449537] openib_check_cq: completed send wr_id 682d50. [D 20:01:04.449545] openib_check_cq: completed send wr_id 682d80. [D 20:01:04.449552] openib_check_cq: completed send wr_id 682db0. [D 20:01:04.449560] openib_check_cq: completed send wr_id 682de0. [D 20:01:04.449567] openib_check_cq: completed send wr_id 682e10. [D 20:01:04.449574] openib_check_cq: completed send wr_id 682e40. [D 20:01:04.449582] openib_check_cq: completed send wr_id 682e70. [D 20:01:04.449589] openib_check_cq: completed send wr_id 682ea0. [D 20:01:04.449596] openib_check_cq: completed send wr_id 682ed0. [D 20:01:04.449602] openib_check_cq: completed send wr_id 682f00. [D 20:01:04.449610] openib_check_cq: completed recv wr_id 684bf0. [D 20:01:04.449625] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd300d10 to 10.1.4.56:39990 remote addr 96445000 rkey 109426 nic_wr_credit 2. [D 20:01:04.449634] openib_post_rr: 10.1.4.56:39990 bh 88 wr_id 684bf0. [D 20:01:04.449643] openib_check_cq: completed recv wr_id 684c20. [D 20:01:04.449652] openib_post_sr_rdmaw: ibv_post_send wr_id 666430 to 10.1.4.56:39990 remote addr 9d04b000 rkey 108f27 nic_wr_credit 3. [D 20:01:04.449660] openib_post_rr: 10.1.4.56:39990 bh 89 wr_id 684c20. [D 20:01:04.449668] openib_check_cq: completed rdma wr_id 6ea9b0, nic_wr_credit 2. [D 20:01:04.449676] post_sr: 10.1.4.56:39990 bh 69 len 16 wr_id 683590. [D 20:01:04.449684] openib_check_cq: completed send wr_id 682f30. [D 20:01:04.449692] openib_check_cq: completed send wr_id 682f60. [D 20:01:04.449699] openib_check_cq: completed send wr_id 682f90. [D 20:01:04.449706] openib_check_cq: completed send wr_id 682fc0. [D 20:01:04.449714] openib_check_cq: completed send wr_id 682ff0. [D 20:01:04.449722] openib_check_cq: completed send wr_id 683020. [D 20:01:04.449729] openib_check_cq: completed send wr_id 683050. [D 20:01:04.449736] openib_check_cq: completed send wr_id 683080. [D 20:01:04.449743] openib_check_cq: completed send wr_id 6830b0. [D 20:01:04.449750] openib_check_cq: completed send wr_id 6830e0. [D 20:01:04.449757] openib_check_cq: completed send wr_id 683110. [D 20:01:04.449763] openib_check_cq: completed send wr_id 683140. [D 20:01:04.449771] openib_check_cq: completed send wr_id 683170. [D 20:01:04.449779] openib_check_cq: completed send wr_id 6831a0. [D 20:01:04.449786] openib_check_cq: completed send wr_id 6831d0. [D 20:01:04.449793] openib_check_cq: completed send wr_id 683200. [D 20:01:04.449800] openib_check_cq: completed send wr_id 683230. [D 20:01:04.449807] openib_check_cq: completed send wr_id 683260. [D 20:01:04.449814] openib_check_cq: completed send wr_id 683290. [D 20:01:04.449821] openib_check_cq: completed send wr_id 6832c0. [D 20:01:04.449828] openib_check_cq: completed send wr_id 6832f0. [D 20:01:04.449835] openib_check_cq: completed send wr_id 683320. [D 20:01:04.449842] openib_check_cq: completed send wr_id 683350. [D 20:01:04.449849] openib_check_cq: completed send wr_id 683380. [D 20:01:04.449856] openib_check_cq: completed send wr_id 6833b0. [D 20:01:04.449863] openib_check_cq: completed send wr_id 6833e0. [D 20:01:04.449870] openib_check_cq: completed send wr_id 683410. [D 20:01:04.449877] openib_check_cq: completed send wr_id 683440. [D 20:01:04.449884] openib_check_cq: completed send wr_id 683470. [D 20:01:04.449891] openib_check_cq: completed send wr_id 6834a0. [D 20:01:04.449898] openib_check_cq: completed send wr_id 6834d0. [D 20:01:04.449905] openib_check_cq: completed send wr_id 683500. [D 20:01:04.449912] openib_check_cq: completed send wr_id 683530. [D 20:01:04.449919] openib_check_cq: completed send wr_id 683560. [D 20:01:04.449949] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:04.449964] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:04.449972] ========================================================= [D 20:01:04.449979] PINT_process_request [D 20:01:04.449987] tiling 5 copies [D 20:01:04.449996] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.450004] lvl 0 el 0 blk 0 by 127926272 [D 20:01:04.450011] to 127926272 ta 100663296 fi 134217728 [D 20:01:04.450019] basic type or contiguous data [D 20:01:04.450026] Request disp [D 20:01:04.450033] contig_offset = 127926272 contig_size = 39845888 lvl_flag = 1 [D 20:01:04.450041] calling distribute [D 20:01:04.450047] PINT_distribute [D 20:01:04.450053] of 127926272 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:04.450063] begin iteration loff: 127926272 [D 20:01:04.450071] segment exceeds byte limit [D 20:01:04.450078] process a segment [D 20:01:04.450083] of 127926272 sz 2097152 [D 20:01:04.450090] add a segment 0 [D 20:01:04.450096] end iteration [D 20:01:04.450102] done with segments or bytes [D 20:01:04.450109] of 130023424 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.450116] next loff: 130023424 next poff: 130023424 [D 20:01:04.450123] return value 2097152 [D 20:01:04.450130] finished [D 20:01:04.450136] retval = 2097152 [D 20:01:04.450142] exiting distribute returned less than expected [D 20:01:04.450149] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 130023424 fo 134217728 eof 0 [D 20:01:04.450157] ========================================================= [D 20:01:04.450171] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.450179] dbpf_bstream_rw_list: mem_offset: 0x2aaac0418010, mem_size: 2097152 [D 20:01:04.450186] dbpf_bstream_rw_list: stream_offset: 127926272, stream_size: 2097152 [D 20:01:04.450194] dbpf_open_cache_get: called [D 20:01:04.450201] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.450208] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.450235] op_queue add: 0x2aaabd301cf0 [D 20:01:04.450244] delayed I/O operation 0x2aaabd301cf0 (16 already in progress) [D 20:01:04.450251] DBPF I/O ops in progress: 16 [D 20:01:04.450259] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:04.450267] (send callback time) ini posts: 0, pending: 3, last: 1, src_list emtpy: yes [D 20:01:04.450276] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517ef0. [D 20:01:04.450283] (send callback time) ini posts: 0, pending: 0, last: 1, src_list emtpy: yes [D 20:01:04.450290] flowproto completing 0x2aaab1517ef0 [D 20:01:04.450311] Job flows in progress (callback time): 13 [D 20:01:04.450321] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:04.450329] (send callback time) ini posts: 0, pending: 1, last: 1, src_list emtpy: yes [D 20:01:04.450337] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:04.450344] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:04.450353] ========================================================= [D 20:01:04.450360] PINT_process_request [D 20:01:04.450366] tiling 6 copies [D 20:01:04.450373] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.450380] lvl 0 el 0 blk 0 by 155189248 [D 20:01:04.450387] to 155189248 ta 134217728 fi 167772160 [D 20:01:04.450394] basic type or contiguous data [D 20:01:04.450399] Request disp [D 20:01:04.450406] contig_offset = 155189248 contig_size = 46137344 lvl_flag = 1 [D 20:01:04.450413] calling distribute [D 20:01:04.450419] PINT_distribute [D 20:01:04.450425] of 155189248 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.450433] begin iteration loff: 155189248 [D 20:01:04.450439] segment exceeds byte limit [D 20:01:04.450445] process a segment [D 20:01:04.450451] of 155189248 sz 2097152 [D 20:01:04.450457] add a segment 0 [D 20:01:04.450463] end iteration [D 20:01:04.450469] done with segments or bytes [D 20:01:04.450475] of 157286400 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.450482] next loff: 157286400 next poff: 157286400 [D 20:01:04.450488] return value 2097152 [D 20:01:04.450494] finished [D 20:01:04.450500] retval = 2097152 [D 20:01:04.450506] exiting distribute returned less than expected [D 20:01:04.450513] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 157286400 fo 167772160 eof 0 [D 20:01:04.450520] ========================================================= [D 20:01:04.450528] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.450535] dbpf_bstream_rw_list: mem_offset: 0x2aaab7a35010, mem_size: 2097152 [D 20:01:04.450542] dbpf_bstream_rw_list: stream_offset: 155189248, stream_size: 2097152 [D 20:01:04.450549] dbpf_open_cache_get: called [D 20:01:04.450556] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.450562] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.450582] op_queue add: 0x2aaabd301e40 [D 20:01:04.450591] delayed I/O operation 0x2aaabd301e40 (16 already in progress) [D 20:01:04.450597] DBPF I/O ops in progress: 16 [D 20:01:04.450608] openib_check_cq: completed recv wr_id 684c50. [D 20:01:04.450621] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd301420 to 10.1.4.56:39990 remote addr 94443000 rkey 109526 nic_wr_credit 3. [D 20:01:04.450632] openib_post_rr: 10.1.4.56:39990 bh 90 wr_id 684c50. [D 20:01:04.451890] openib_check_cq: completed rdma wr_id 2aaabd300d10, nic_wr_credit 2. [D 20:01:04.451937] post_sr: 10.1.4.56:39990 bh 70 len 16 wr_id 6835c0. [D 20:01:04.451953] openib_check_cq: completed recv wr_id 684c80. [D 20:01:04.451968] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd307a50 to 10.1.4.56:39990 remote addr 92441000 rkey 109626 nic_wr_credit 3. [D 20:01:04.451979] openib_post_rr: 10.1.4.56:39990 bh 91 wr_id 684c80. [D 20:01:04.452273] openib_check_cq: completed recv wr_id 684cb0. [D 20:01:04.452284] openib_post_sr_rdmaw: ibv_post_send wr_id 6ecdf0 to 10.1.4.56:39990 remote addr 9fe4d000 rkey 108e27 nic_wr_credit 4. [D 20:01:04.452292] openib_post_rr: 10.1.4.56:39990 bh 92 wr_id 684cb0. [D 20:01:04.454127] openib_check_cq: completed rdma wr_id 666430, nic_wr_credit 3. [D 20:01:04.454205] post_sr: 10.1.4.56:39990 bh 71 len 16 wr_id 6835f0. [D 20:01:04.454219] openib_check_cq: completed send wr_id 683590. [D 20:01:04.454246] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c750. [D 20:01:04.454260] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:04.454271] ========================================================= [D 20:01:04.454278] PINT_process_request [D 20:01:04.454286] tiling 7 copies [D 20:01:04.454294] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.454302] lvl 0 el 0 blk 0 by 186646528 [D 20:01:04.454308] to 186646528 ta 167772160 fi 201326592 [D 20:01:04.454316] basic type or contiguous data [D 20:01:04.454323] Request disp [D 20:01:04.454334] contig_offset = 186646528 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.454343] calling distribute [D 20:01:04.454350] PINT_distribute [D 20:01:04.454356] of 186646528 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.454365] begin iteration loff: 186646528 [D 20:01:04.454372] segment exceeds byte limit [D 20:01:04.454380] process a segment [D 20:01:04.454385] of 186646528 sz 2097152 [D 20:01:04.454392] add a segment 0 [D 20:01:04.454400] end iteration [D 20:01:04.454406] done with segments or bytes [D 20:01:04.454412] of 188743680 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.454420] next loff: 188743680 next poff: 188743680 [D 20:01:04.454427] return value 2097152 [D 20:01:04.454435] finished [D 20:01:04.454441] retval = 2097152 [D 20:01:04.454447] exiting distribute returned less than expected [D 20:01:04.454453] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 188743680 fo 201326592 eof 0 [D 20:01:04.454461] ========================================================= [D 20:01:04.454482] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.454491] dbpf_bstream_rw_list: mem_offset: 0x2aaab2809010, mem_size: 2097152 [D 20:01:04.454498] dbpf_bstream_rw_list: stream_offset: 186646528, stream_size: 2097152 [D 20:01:04.454507] dbpf_open_cache_get: called [D 20:01:04.454515] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.454523] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.454553] op_queue add: 0x2aaabd301f90 [D 20:01:04.454565] delayed I/O operation 0x2aaabd301f90 (16 already in progress) [D 20:01:04.454572] DBPF I/O ops in progress: 16 [D 20:01:04.456349] openib_check_cq: completed rdma wr_id 2aaabd301420, nic_wr_credit 2. [D 20:01:04.456411] post_sr: 10.1.4.56:39990 bh 72 len 16 wr_id 683620. [D 20:01:04.456428] openib_check_cq: completed send wr_id 6835c0. [D 20:01:04.456459] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517400. [D 20:01:04.456472] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.456482] ========================================================= [D 20:01:04.456488] PINT_process_request [D 20:01:04.456495] tiling 8 copies [D 20:01:04.456504] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.456512] lvl 0 el 0 blk 0 by 218103808 [D 20:01:04.456518] to 218103808 ta 201326592 fi 234881024 [D 20:01:04.456526] basic type or contiguous data [D 20:01:04.456533] Request disp [D 20:01:04.456539] contig_offset = 218103808 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.456547] calling distribute [D 20:01:04.456553] PINT_distribute [D 20:01:04.456560] of 218103808 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.456568] begin iteration loff: 218103808 [D 20:01:04.456575] segment exceeds byte limit [D 20:01:04.446403] --- aio_progress_notification called with handle 2863311531 (0x2aaab152cf90) [D 20:01:04.456636] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.456648] dbpf_open_cache_put: cached entry. [D 20:01:04.456655] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.456662] op_queue add: 0x2aaab152cf90 [D 20:01:04.456675] DBPF I/O ops in progress: 15 [D 20:01:04.456683] starting delayed I/O operation 0x2aaab1506a60 (15 in progress) [D 20:01:04.456693] lio_listio called with 1 following aiocbs: [D 20:01:04.456700] aiocb_ptr_array[0]: fd: 13, off: 346030080, bytes: 2097152, buf: 0x2aaaadec7010, type: 0 [D 20:01:04.456713] start_delayed_ops_if_any: lio_listio posted 0x2aaab1506a60 (handle 2863311531, ret 0)) [D 20:01:04.446413] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.456877] dbpf_open_cache_put: cached entry. [D 20:01:04.456885] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.456893] op_queue add: 0x2aaab15054a0 [D 20:01:04.456902] DBPF I/O ops in progress: 15 [D 20:01:04.456913] starting delayed I/O operation 0x2aaab1506bb0 (15 in progress) [D 20:01:04.456925] lio_listio called with 1 following aiocbs: [D 20:01:04.456932] aiocb_ptr_array[0]: fd: 13, off: 348127232, bytes: 2097152, buf: 0x2aaabaa4d010, type: 0 [D 20:01:04.456942] start_delayed_ops_if_any: lio_listio posted 0x2aaab1506bb0 (handle 2863311531, ret 0)) [D 20:01:04.456964] op_queue add: 0x2aaab1525840 [D 20:01:04.460647] --- aio_progress_notification called with handle 2863311531 (0x2aaab152c0c0) [D 20:01:04.460723] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.460739] dbpf_open_cache_put: cached entry. [D 20:01:04.460748] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.460756] op_queue add: 0x2aaab152c0c0 [D 20:01:04.460766] DBPF I/O ops in progress: 15 [D 20:01:04.460774] starting delayed I/O operation 0x2aaab1506d00 (15 in progress) [D 20:01:04.460787] lio_listio called with 1 following aiocbs: [D 20:01:04.460794] aiocb_ptr_array[0]: fd: 13, off: 350224384, bytes: 2097152, buf: 0x2aaab8a3d010, type: 0 [D 20:01:04.460808] start_delayed_ops_if_any: lio_listio posted 0x2aaab1506d00 (handle 2863311531, ret 0)) [D 20:01:04.446408] --- aio_progress_notification called with handle 2863311531 (0x2aaab152d0e0) [D 20:01:04.460847] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.460863] dbpf_open_cache_put: cached entry. [D 20:01:04.460870] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.460876] op_queue add: 0x2aaab152d0e0 [D 20:01:04.460884] DBPF I/O ops in progress: 15 [D 20:01:04.460892] starting delayed I/O operation 0x2aaab15073b0 (15 in progress) [D 20:01:04.460902] lio_listio called with 1 following aiocbs: [D 20:01:04.460909] aiocb_ptr_array[0]: fd: 13, off: 369098752, bytes: 2097152, buf: 0x2aaab6e2f010, type: 0 [D 20:01:04.460918] start_delayed_ops_if_any: lio_listio posted 0x2aaab15073b0 (handle 2863311531, ret 0)) [D 20:01:04.464621] --- aio_progress_notification called with handle 2863311531 (0x2aaab15067c0) [D 20:01:04.464678] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.464690] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.456779] --- aio_progress_notification called with handle 2863311531 (0x2aaab1561cc0) [D 20:01:04.464715] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.464718] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.456783] --- aio_progress_notification called with handle 2863311531 (0x2aaab1561e10) [D 20:01:04.464730] BMI_post_send: addr: 6843072, offset: 0x2aaabb453010, size: 2097152, tag: 30 [D 20:01:04.464738] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.464750] post_sr: 10.1.4.56:39990 bh 73 len 32 wr_id 683650. [D 20:01:04.464796] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.464804] BMI_post_send: addr: 6843072, offset: 0x2aaabb252010, size: 2097152, tag: 30 [D 20:01:04.464814] post_sr: 10.1.4.56:39990 bh 74 len 32 wr_id 683680. [D 20:01:04.464824] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.464831] BMI_post_send: addr: 6843072, offset: 0x2aaabb051010, size: 2097152, tag: 30 [D 20:01:04.464840] post_sr: 10.1.4.56:39990 bh 75 len 32 wr_id 6836b0. [D 20:01:04.464849] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.464858] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.464867] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.464875] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.456595] process a segment [D 20:01:04.464908] of 218103808 sz 2097152 [D 20:01:04.464915] add a segment 0 [D 20:01:04.464927] end iteration [D 20:01:04.464934] done with segments or bytes [D 20:01:04.464940] of 220200960 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.464947] next loff: 220200960 next poff: 220200960 [D 20:01:04.464954] return value 2097152 [D 20:01:04.464960] finished [D 20:01:04.464967] retval = 2097152 [D 20:01:04.464974] exiting distribute returned less than expected [D 20:01:04.464980] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 220200960 fo 234881024 eof 0 [D 20:01:04.464988] ========================================================= [D 20:01:04.465004] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.465014] dbpf_bstream_rw_list: mem_offset: 0x2aaab682c010, mem_size: 2097152 [D 20:01:04.465021] dbpf_bstream_rw_list: stream_offset: 218103808, stream_size: 2097152 [D 20:01:04.465029] dbpf_open_cache_get: called [D 20:01:04.465183] op_queue add: 0x2aaab1505ea0 [D 20:01:04.465210] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.465219] BMI_post_send: addr: 6843072, offset: 0x2aaab8e3f010, size: 2097152, tag: 31 [D 20:01:04.465229] post_sr: 10.1.4.56:39990 bh 76 len 32 wr_id 6836e0. [D 20:01:04.465241] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:04.465249] BMI_post_send: addr: 6843072, offset: 0x2aaab5c25010, size: 2097152, tag: 31 [D 20:01:04.465258] post_sr: 10.1.4.56:39990 bh 77 len 32 wr_id 683710. [D 20:01:04.465268] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:04.465276] BMI_post_send: addr: 6843072, offset: 0x2aaab521f010, size: 2097152, tag: 31 [D 20:01:04.465284] post_sr: 10.1.4.56:39990 bh 78 len 32 wr_id 683740. [D 20:01:04.465294] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.465302] BMI_post_send: addr: 6843072, offset: 0x2aaaaf6d3010, size: 2097152, tag: 31 [D 20:01:04.465310] post_sr: 10.1.4.56:39990 bh 79 len 32 wr_id 683770. [D 20:01:04.465319] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.465327] BMI_post_send: addr: 6843072, offset: 0x2aaaadac5010, size: 2097152, tag: 31 [D 20:01:04.465336] post_sr: 10.1.4.56:39990 bh 80 len 32 wr_id 6837a0. [D 20:01:04.465345] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.465353] BMI_post_send: addr: 6843072, offset: 0x2aaabae50010, size: 2097152, tag: 31 [D 20:01:04.465362] post_sr: 10.1.4.56:39990 bh 81 len 32 wr_id 6837d0. [D 20:01:04.465371] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.465379] BMI_post_send: addr: 6843072, offset: 0x2aaab9040010, size: 2097152, tag: 31 [D 20:01:04.465388] post_sr: 10.1.4.56:39990 bh 82 len 32 wr_id 683800. [D 20:01:04.465397] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.465411] DBPF I/O ops in progress: 15 [D 20:01:04.465420] starting delayed I/O operation 0x2aaab1507500 (15 in progress) [D 20:01:04.465431] lio_listio called with 1 following aiocbs: [D 20:01:04.465438] aiocb_ptr_array[0]: fd: 13, off: 371195904, bytes: 2097152, buf: 0x2aaab4e1d010, type: 0 [D 20:01:04.465454] start_delayed_ops_if_any: lio_listio posted 0x2aaab1507500 (handle 2863311531, ret 0)) [D 20:01:04.465477] --- aio_progress_notification called with handle 2863311531 (0x2aaab15524e0) [D 20:01:04.465490] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.465512] --- aio_progress_notification called with handle 2863311531 (0x2aaab1552390) [D 20:01:04.465523] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.446398] --- aio_progress_notification called with handle 2863311531 (0x2aaab152ce40) [D 20:01:04.465552] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.465570] DBPF I/O ops in progress: 15 [D 20:01:04.465581] starting delayed I/O operation 0x2aaab1507650 (15 in progress) [D 20:01:04.465594] lio_listio called with 1 following aiocbs: [D 20:01:04.465600] aiocb_ptr_array[0]: fd: 13, off: 373293056, bytes: 2097152, buf: 0x2aaab320f010, type: 0 [D 20:01:04.465610] start_delayed_ops_if_any: lio_listio posted 0x2aaab1507650 (handle 2863311531, ret 0)) [D 20:01:04.456767] --- aio_progress_notification called with handle 2863311531 (0x2aaab152bf70) [D 20:01:04.465656] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [A 20:01:04.467615] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66e910: io: finish (Success) [D 20:01:04.467676] PINT_encode_release [D 20:01:04.467685] lebf_encode_rel [D 20:01:04.467714] [SM frame get]: (0x66e810) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.467723] server_state_machine_complete 0x66e810 [D 20:01:04.467730] PINT_decode_release [D 20:01:04.467736] lebf_decode_rel [D 20:01:04.467744] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.467753] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.467760] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 29 [D 20:01:04.467769] [SM Exiting]: (0x66e810) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.467779] server_state_machine_terminate 0x66e810 [D 20:01:04.467823] [SM Entering]: (0x65f450) io_sm:send_completion_ack (status: 0) [D 20:01:04.467831] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.467839] [SM Exiting]: (0x65f450) io_sm:send_completion_ack (error code: 0), (action: COMPLETE) [D 20:01:04.467846] [SM Entering]: (0x65f450) io_sm:release (status: 0) [D 20:01:04.467853] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.467865] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6e5270 [D 20:01:04.467875] [SM Exiting]: (0x65f450) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.467882] [SM Entering]: (0x65f450) io_sm:cleanup (status: 0) [D 20:01:04.467889] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.468595] --- aio_progress_notification called with handle 2863311531 (0x2aaab1552630) [D 20:01:04.468623] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.456773] --- aio_progress_notification called with handle 2863311531 (0x2aaab1561b70) [D 20:01:04.468657] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.464695] dbpf_open_cache_put: cached entry. [D 20:01:04.468691] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.468699] op_queue add: 0x2aaab15067c0 [D 20:01:04.468729] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.468742] BMI_post_send: addr: 6843072, offset: 0x2aaab7030010, size: 2097152, tag: 33 [D 20:01:04.468762] post_sr: 10.1.4.56:39990 bh 83 len 32 wr_id 683830. [D 20:01:04.468793] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.468809] DBPF I/O ops in progress: 15 [D 20:01:04.468817] starting delayed I/O operation 0x2aaab15077a0 (15 in progress) [D 20:01:04.468827] lio_listio called with 1 following aiocbs: [D 20:01:04.468834] aiocb_ptr_array[0]: fd: 13, off: 375390208, bytes: 2097152, buf: 0x2aaab1600010, type: 0 [D 20:01:04.468849] start_delayed_ops_if_any: lio_listio posted 0x2aaab15077a0 (handle 2863311531, ret 0)) [D 20:01:04.468890] dbpf_open_cache_put: cached entry. [D 20:01:04.468904] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.468913] op_queue add: 0x2aaab1561cc0 [D 20:01:04.468930] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.468943] DBPF I/O ops in progress: 15 [D 20:01:04.468951] starting delayed I/O operation 0x2aaab15078f0 (15 in progress) [D 20:01:04.468962] lio_listio called with 1 following aiocbs: [D 20:01:04.468968] aiocb_ptr_array[0]: fd: 13, off: 377487360, bytes: 2097152, buf: 0x2aaaaf8d4010, type: 0 [D 20:01:04.468979] start_delayed_ops_if_any: lio_listio posted 0x2aaab15078f0 (handle 2863311531, ret 0)) [D 20:01:04.469000] dbpf_open_cache_put: cached entry. [D 20:01:04.469020] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.469031] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.469058] op_queue add: 0x2aaabd3023a0 [D 20:01:04.469066] delayed I/O operation 0x2aaabd3023a0 (16 already in progress) [D 20:01:04.469073] DBPF I/O ops in progress: 16 [D 20:01:04.469094] openib_check_cq: completed rdma wr_id 2aaabd307a50, nic_wr_credit 1. [D 20:01:04.469106] post_sr: 10.1.4.56:39990 bh 84 len 16 wr_id 683860. [D 20:01:04.469115] openib_check_cq: completed rdma wr_id 6ecdf0, nic_wr_credit 0. [D 20:01:04.469124] post_sr: 10.1.4.56:39990 bh 85 len 16 wr_id 683890. [D 20:01:04.469132] openib_check_cq: completed send wr_id 6835f0. [D 20:01:04.469140] openib_check_cq: completed send wr_id 683620. [D 20:01:04.469148] openib_check_cq: completed recv wr_id 684ce0. [D 20:01:04.469163] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3016e0 to 10.1.4.56:39990 remote addr 96645000 rkey 109426 nic_wr_credit 1. [D 20:01:04.469173] openib_post_rr: 10.1.4.56:39990 bh 93 wr_id 684ce0. [D 20:01:04.469182] openib_check_cq: completed send wr_id 683650. [D 20:01:04.469191] openib_check_cq: completed send wr_id 683680. [D 20:01:04.469198] openib_check_cq: completed send wr_id 6836b0. [D 20:01:04.469206] openib_check_cq: completed send wr_id 6836e0. [D 20:01:04.469213] openib_check_cq: completed send wr_id 683710. [D 20:01:04.469220] openib_check_cq: completed send wr_id 683740. [D 20:01:04.469227] openib_check_cq: completed send wr_id 683770. [D 20:01:04.469235] openib_check_cq: completed send wr_id 6837a0. [D 20:01:04.469242] openib_check_cq: completed send wr_id 6837d0. [D 20:01:04.469249] openib_check_cq: completed send wr_id 683800. [D 20:01:04.469256] openib_check_cq: completed send wr_id 683830. [D 20:01:04.469263] openib_check_cq: completed send wr_id 683860. [D 20:01:04.469270] openib_check_cq: completed send wr_id 683890. [D 20:01:04.469297] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:04.469311] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:04.469320] ========================================================= [D 20:01:04.469326] PINT_process_request [D 20:01:04.469333] tiling 5 copies [D 20:01:04.469342] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.469349] lvl 0 el 0 blk 0 by 130023424 [D 20:01:04.469356] to 130023424 ta 100663296 fi 134217728 [D 20:01:04.469364] basic type or contiguous data [D 20:01:04.469371] Request disp [D 20:01:04.469378] contig_offset = 130023424 contig_size = 37748736 lvl_flag = 1 [D 20:01:04.469385] calling distribute [D 20:01:04.469391] PINT_distribute [D 20:01:04.469398] of 130023424 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:04.469407] begin iteration loff: 130023424 [D 20:01:04.469414] segment exceeds byte limit [D 20:01:04.469421] process a segment [D 20:01:04.469427] of 130023424 sz 2097152 [D 20:01:04.469434] add a segment 0 [D 20:01:04.469440] end iteration [D 20:01:04.469446] done with segments or bytes [D 20:01:04.469453] of 132120576 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.469460] next loff: 132120576 next poff: 132120576 [D 20:01:04.469466] return value 2097152 [D 20:01:04.469473] finished [D 20:01:04.469479] retval = 2097152 [D 20:01:04.469485] exiting distribute returned less than expected [D 20:01:04.469492] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 132120576 fo 134217728 eof 0 [D 20:01:04.469499] ========================================================= [D 20:01:04.469515] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.469523] dbpf_bstream_rw_list: mem_offset: 0x2aaac0619010, mem_size: 2097152 [D 20:01:04.469530] dbpf_bstream_rw_list: stream_offset: 130023424, stream_size: 2097152 [D 20:01:04.469537] dbpf_open_cache_get: called [D 20:01:04.469545] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.469552] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.469572] op_queue add: 0x2aaabd302ff0 [D 20:01:04.469580] delayed I/O operation 0x2aaabd302ff0 (16 already in progress) [D 20:01:04.469586] DBPF I/O ops in progress: 16 [D 20:01:04.469594] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:04.469602] (send callback time) ini posts: 0, pending: 2, last: 1, src_list emtpy: yes [D 20:01:04.469610] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152db60. [D 20:01:04.469617] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.469625] ========================================================= [D 20:01:04.469631] PINT_process_request [D 20:01:04.469638] tiling 9 copies [D 20:01:04.469644] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.469651] lvl 0 el 0 blk 0 by 251658240 [D 20:01:04.469658] to 251658240 ta 234881024 fi 268435456 [D 20:01:04.469665] basic type or contiguous data [D 20:01:04.469671] Request disp [D 20:01:04.469677] contig_offset = 251658240 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.469684] calling distribute [D 20:01:04.469690] PINT_distribute [D 20:01:04.469696] of 251658240 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.469703] begin iteration loff: 251658240 [D 20:01:04.469710] segment exceeds byte limit [D 20:01:04.469716] process a segment [D 20:01:04.469722] of 251658240 sz 2097152 [D 20:01:04.469728] add a segment 0 [D 20:01:04.469734] end iteration [D 20:01:04.469740] done with segments or bytes [D 20:01:04.469746] of 253755392 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.469753] next loff: 253755392 next poff: 253755392 [D 20:01:04.469760] return value 2097152 [D 20:01:04.469766] finished [D 20:01:04.469772] retval = 2097152 [D 20:01:04.469778] exiting distribute returned less than expected [D 20:01:04.469784] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 253755392 fo 268435456 eof 0 [D 20:01:04.469792] ========================================================= [D 20:01:04.469800] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.469808] dbpf_bstream_rw_list: mem_offset: 0x2aaab6c2e010, mem_size: 2097152 [D 20:01:04.469815] dbpf_bstream_rw_list: stream_offset: 251658240, stream_size: 2097152 [D 20:01:04.469822] dbpf_open_cache_get: called [D 20:01:04.469828] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.469834] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.469854] op_queue add: 0x2aaabd307a50 [D 20:01:04.469862] delayed I/O operation 0x2aaabd307a50 (16 already in progress) [D 20:01:04.469869] DBPF I/O ops in progress: 16 [D 20:01:04.469878] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1505890. [D 20:01:04.469886] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.469894] ========================================================= [D 20:01:04.469900] PINT_process_request [D 20:01:04.469906] tiling 10 copies [D 20:01:04.469912] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.469919] lvl 0 el 0 blk 0 by 285212672 [D 20:01:04.469925] to 285212672 ta 268435456 fi 301989888 [D 20:01:04.469932] basic type or contiguous data [D 20:01:04.469939] Request disp [D 20:01:04.469945] contig_offset = 285212672 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.469952] calling distribute [D 20:01:04.469958] PINT_distribute [D 20:01:04.469964] of 285212672 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.469971] begin iteration loff: 285212672 [D 20:01:04.469978] segment exceeds byte limit [D 20:01:04.469984] process a segment [D 20:01:04.469990] of 285212672 sz 2097152 [D 20:01:04.469996] add a segment 0 [D 20:01:04.470003] end iteration [D 20:01:04.470010] done with segments or bytes [D 20:01:04.470016] of 287309824 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.470023] next loff: 287309824 next poff: 287309824 [D 20:01:04.470029] return value 2097152 [D 20:01:04.470035] finished [D 20:01:04.470041] retval = 2097152 [D 20:01:04.470046] exiting distribute returned less than expected [D 20:01:04.470053] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 287309824 fo 301989888 eof 0 [D 20:01:04.470060] ========================================================= [D 20:01:04.470068] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.470075] dbpf_bstream_rw_list: mem_offset: 0x2aaab9844010, mem_size: 2097152 [D 20:01:04.470081] dbpf_bstream_rw_list: stream_offset: 285212672, stream_size: 2097152 [D 20:01:04.470088] dbpf_open_cache_get: called [D 20:01:04.470094] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.470101] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.470120] op_queue add: 0x2aaabd303140 [D 20:01:04.470128] delayed I/O operation 0x2aaabd303140 (16 already in progress) [D 20:01:04.470135] DBPF I/O ops in progress: 16 [D 20:01:04.471449] openib_check_cq: completed rdma wr_id 2aaabd3016e0, nic_wr_credit 0. [D 20:01:04.471494] post_sr: 10.1.4.56:39990 bh 86 len 16 wr_id 6838c0. [D 20:01:04.471514] openib_check_cq: completed send wr_id 6838c0. [D 20:01:04.471536] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517400. [D 20:01:04.471548] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:04.471558] ========================================================= [D 20:01:04.471565] PINT_process_request [D 20:01:04.471572] tiling 8 copies [D 20:01:04.471580] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.471588] lvl 0 el 0 blk 0 by 220200960 [D 20:01:04.471594] to 220200960 ta 201326592 fi 234881024 [D 20:01:04.471602] basic type or contiguous data [D 20:01:04.471609] Request disp [D 20:01:04.471615] contig_offset = 220200960 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.471623] calling distribute [D 20:01:04.471629] PINT_distribute [D 20:01:04.471636] of 220200960 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.471644] begin iteration loff: 220200960 [D 20:01:04.471652] segment exceeds byte limit [D 20:01:04.471659] process a segment [D 20:01:04.471665] of 220200960 sz 2097152 [D 20:01:04.471672] add a segment 0 [D 20:01:04.471678] end iteration [D 20:01:04.471685] done with segments or bytes [D 20:01:04.471691] of 222298112 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.471698] next loff: 222298112 next poff: 222298112 [D 20:01:04.471705] return value 2097152 [D 20:01:04.471711] finished [D 20:01:04.471717] retval = 2097152 [D 20:01:04.471724] exiting distribute returned less than expected [D 20:01:04.471730] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 222298112 fo 234881024 eof 0 [D 20:01:04.471738] ========================================================= [D 20:01:04.471752] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.471761] dbpf_bstream_rw_list: mem_offset: 0x2aaab481a010, mem_size: 2097152 [D 20:01:04.471767] dbpf_bstream_rw_list: stream_offset: 220200960, stream_size: 2097152 [D 20:01:04.471776] dbpf_open_cache_get: called [D 20:01:04.471784] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.471790] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.471815] op_queue add: 0x2aaabd303290 [D 20:01:04.471825] delayed I/O operation 0x2aaabd303290 (16 already in progress) [D 20:01:04.471832] DBPF I/O ops in progress: 16 [D 20:01:04.473131] --- aio_progress_notification called with handle 2863311531 (0x2aaab1506910) [D 20:01:04.473171] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.473181] dbpf_open_cache_put: cached entry. [D 20:01:04.473189] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.473196] op_queue add: 0x2aaab1506910 [D 20:01:04.473209] DBPF I/O ops in progress: 15 [D 20:01:04.473216] starting delayed I/O operation 0x2aaab1507a40 (15 in progress) [D 20:01:04.473225] lio_listio called with 1 following aiocbs: [D 20:01:04.473231] aiocb_ptr_array[0]: fd: 13, off: 379584512, bytes: 2097152, buf: 0x2aaaadcc6010, type: 0 [D 20:01:04.473245] --- aio_progress_notification called with handle 2863311531 (0x2aaab1506a60) [D 20:01:04.473254] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.473263] dbpf_open_cache_put: cached entry. [D 20:01:04.473269] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.473275] op_queue add: 0x2aaab1506a60 [D 20:01:04.473285] --- aio_progress_notification called with handle 2863311531 (0x2aaab1506bb0) [D 20:01:04.473294] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.473303] dbpf_open_cache_put: cached entry. [D 20:01:04.473309] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.473315] op_queue add: 0x2aaab1506bb0 [D 20:01:04.476615] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.476690] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.476698] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.480419] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.480507] op_queue add: 0x2aaab1561e10 [D 20:01:04.480562] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.480634] dbpf_open_cache_put: cached entry. [D 20:01:04.480652] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.480660] op_queue add: 0x2aaab15524e0 [D 20:01:04.480677] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.480696] dbpf_open_cache_put: cached entry. [D 20:01:04.480710] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.480717] op_queue add: 0x2aaab1552390 [D 20:01:04.480742] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.480752] BMI_post_send: addr: 6843072, offset: 0x2aaab501e010, size: 2097152, tag: 33 [D 20:01:04.480774] post_sr: 10.1.4.56:39990 bh 87 len 32 wr_id 6838f0. [D 20:01:04.480809] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:04.480817] BMI_post_send: addr: 6843072, offset: 0x2aaab3410010, size: 2097152, tag: 33 [D 20:01:04.480827] post_sr: 10.1.4.56:39990 bh 88 len 32 wr_id 683920. [D 20:01:04.480837] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 -[D 20:01:04.480863] openib_check_cq: completed send wr_id 6838f0. [D 20:01:04.480874] openib_check_cq: completed send wr_id 683920. [D 20:01:04.481210] --- aio_progress_notification called with handle 2863311531 (0x2aaab1506d00) [D 20:01:04.481227] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.481237] dbpf_open_cache_put: cached entry. [D 20:01:04.481243] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.481249] op_queue add: 0x2aaab1506d00 [D 20:01:04.481265] --- aio_progress_notification called with handle 2863311531 (0x2aaab15073b0) [D 20:01:04.481280] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.481293] dbpf_open_cache_put: cached entry. [D 20:01:04.481300] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.481306] op_queue add: 0x2aaab15073b0 [D 20:01:04.481319] --- aio_progress_notification called with handle 2863311531 (0x2aaab1507500) [D 20:01:04.481333] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.481346] dbpf_open_cache_put: cached entry. [D 20:01:04.516803] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.481357] --- aio_progress_notification called with handle 2863311531 (0x2aaab1507650) [D 20:01:04.516806] dbpf_open_cache_put: cached entry. [D 20:01:04.516840] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.516847] dbpf_open_cache_put: cached entry. [D 20:01:04.516831] op_queue add: 0x2aaab1507500 [D 20:01:04.485365] start_delayed_ops_if_any: lio_listio posted 0x2aaab1507a40 (handle 2863311531, ret 0)) [D 20:01:04.516865] dbpf_open_cache_put: cached entry. [D 20:01:04.485414] --- aio_progress_notification called with handle 2863311531 (0x2aaab15077a0) [D 20:01:04.488602] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.516883] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.516888] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.516843] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.516900] BMI_post_send: addr: 6843072, offset: 0x2aaab6e2f010, size: 2097152, tag: 34 [D 20:01:04.516891] op_queue add: 0x2aaab1561b70 [D 20:01:04.516929] op_queue add: 0x2aaab152bf70 [D 20:01:04.516860] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.516924] post_sr: 10.1.4.56:39990 bh 89 len 32 wr_id 683950. [D 20:01:04.516950] op_queue add: 0x2aaab1552630 -[D 20:01:04.516976] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.516981] openib_check_cq: completed send wr_id 683950. [D 20:01:04.516997] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.517007] BMI_post_send: addr: 6843072, offset: 0x2aaab4e1d010, size: 2097152, tag: 34 [D 20:01:04.516903] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.517029] dbpf_open_cache_put: cached entry. [D 20:01:04.517036] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.517043] op_queue add: 0x2aaab15077a0 [D 20:01:04.485432] --- aio_progress_notification called with handle 2863311531 (0x2aaab15078f0) [D 20:01:04.517055] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.517071] dbpf_open_cache_put: cached entry. [D 20:01:04.517079] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.517085] op_queue add: 0x2aaab15078f0 [D 20:01:04.487384] --- aio_progress_notification called with handle 2863311531 (0x2aaab1507a40) [D 20:01:04.517095] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.517108] dbpf_open_cache_put: cached entry. [D 20:01:04.517115] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.517121] op_queue add: 0x2aaab1507a40 [D 20:01:04.517130] dbpf_open_cache_put: cached entry. [D 20:01:04.517139] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.517146] op_queue add: 0x2aaab152ce40 [D 20:01:04.517176] dbpf_open_cache_put: cached entry. [D 20:01:04.517185] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.517192] op_queue add: 0x2aaab1507650 [D 20:01:04.517199] DBPF I/O ops in progress: 15 [D 20:01:04.517205] starting delayed I/O operation 0x2aaab15b8110 (15 in progress) [D 20:01:04.517214] lio_listio called with 1 following aiocbs: [D 20:01:04.517220] aiocb_ptr_array[0]: fd: 13, off: 381681664, bytes: 2097152, buf: 0x2aaaba64b010, type: 0 [D 20:01:04.517232] start_delayed_ops_if_any: lio_listio posted 0x2aaab15b8110 (handle 2863311531, ret 0)) [D 20:01:04.517247] DBPF I/O ops in progress: 15 [D 20:01:04.517256] starting delayed I/O operation 0x2aaab15bac70 (15 in progress) [D 20:01:04.517265] lio_listio called with 1 following aiocbs: [D 20:01:04.517271] aiocb_ptr_array[0]: fd: 13, off: 383778816, bytes: 2097152, buf: 0x2aaab863b010, type: 0 [D 20:01:04.517280] start_delayed_ops_if_any: lio_listio posted 0x2aaab15bac70 (handle 2863311531, ret 0)) [D 20:01:04.520601] DBPF I/O ops in progress: 15 [D 20:01:04.520643] starting delayed I/O operation 0x2aaab15bfff0 (15 in progress) [D 20:01:04.520653] lio_listio called with 1 following aiocbs: [D 20:01:04.520659] aiocb_ptr_array[0]: fd: 13, off: 402653184, bytes: 2097152, buf: 0x2aaab6a2d010, type: 0 [D 20:01:04.520671] start_delayed_ops_if_any: lio_listio posted 0x2aaab15bfff0 (handle 2863311531, ret 0)) [D 20:01:04.520708] --- aio_progress_notification called with handle 2863311531 (0x2aaab15b8110) [D 20:01:04.520721] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.520731] dbpf_open_cache_put: cached entry. [D 20:01:04.520737] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.520744] op_queue add: 0x2aaab15b8110 [D 20:01:04.520752] DBPF I/O ops in progress: 15 [D 20:01:04.520758] starting delayed I/O operation 0x2aaab15c2b50 (15 in progress) [D 20:01:04.520767] lio_listio called with 1 following aiocbs: [D 20:01:04.520773] aiocb_ptr_array[0]: fd: 13, off: 404750336, bytes: 2097152, buf: 0x2aaab4a1b010, type: 0 [D 20:01:04.520782] start_delayed_ops_if_any: lio_listio posted 0x2aaab15c2b50 (handle 2863311531, ret 0)) [D 20:01:04.525070] --- aio_progress_notification called with handle 2863311531 (0x2aaab15bac70) [D 20:01:04.525115] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.525126] dbpf_open_cache_put: cached entry. [D 20:01:04.525133] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.525140] op_queue add: 0x2aaab15bac70 [D 20:01:04.525148] DBPF I/O ops in progress: 15 [D 20:01:04.525155] starting delayed I/O operation 0x2aaab15c56b0 (15 in progress) [D 20:01:04.525164] lio_listio called with 1 following aiocbs: [D 20:01:04.525170] aiocb_ptr_array[0]: fd: 13, off: 406847488, bytes: 2097152, buf: 0x2aaab2e0c010, type: 0 [D 20:01:04.525183] start_delayed_ops_if_any: lio_listio posted 0x2aaab15c56b0 (handle 2863311531, ret 0)) [D 20:01:04.525211] --- aio_progress_notification called with handle 2863311531 (0x2aaab15bfff0) [D 20:01:04.525220] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.525229] dbpf_open_cache_put: cached entry. [D 20:01:04.525236] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.525242] op_queue add: 0x2aaab15bfff0 [D 20:01:04.525249] DBPF I/O ops in progress: 15 [D 20:01:04.525256] starting delayed I/O operation 0x2aaab15c8210 (15 in progress) [D 20:01:04.525264] lio_listio called with 1 following aiocbs: [D 20:01:04.525271] aiocb_ptr_array[0]: fd: 13, off: 408944640, bytes: 2097152, buf: 0x2aaab10e0010, type: 0 [D 20:01:04.525278] start_delayed_ops_if_any: lio_listio posted 0x2aaab15c8210 (handle 2863311531, ret 0)) [D 20:01:04.525293] --- aio_progress_notification called with handle 2863311531 (0x2aaab15c2b50) [D 20:01:04.525302] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.525311] dbpf_open_cache_put: cached entry. [D 20:01:04.525317] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.525323] op_queue add: 0x2aaab15c2b50 [D 20:01:04.525330] DBPF I/O ops in progress: 15 [D 20:01:04.525337] starting delayed I/O operation 0x2aaab15cad70 (15 in progress) [D 20:01:04.525345] lio_listio called with 1 following aiocbs: [D 20:01:04.525351] aiocb_ptr_array[0]: fd: 13, off: 411041792, bytes: 2097152, buf: 0x2aaaaf4d2010, type: 0 [D 20:01:04.525359] start_delayed_ops_if_any: lio_listio posted 0x2aaab15cad70 (handle 2863311531, ret 0)) [D 20:01:04.525705] post_sr: 10.1.4.56:39990 bh 90 len 32 wr_id 683980. [D 20:01:04.525747] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:04.525762] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.525771] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.525782] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.525793] BMI_post_send: addr: 6843072, offset: 0x2aaab1801010, size: 2097152, tag: 33 [D 20:01:04.525805] post_sr: 10.1.4.56:39990 bh 91 len 32 wr_id 6839b0. [D 20:01:04.525815] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.525823] BMI_post_send: addr: 6843072, offset: 0x2aaaafad5010, size: 2097152, tag: 33 [D 20:01:04.525831] post_sr: 10.1.4.56:39990 bh 92 len 32 wr_id 6839e0. [D 20:01:04.525840] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.525848] BMI_post_send: addr: 6843072, offset: 0x2aaaadec7010, size: 2097152, tag: 33 [D 20:01:04.525857] post_sr: 10.1.4.56:39990 bh 93 len 32 wr_id 683a10. [D 20:01:04.525866] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.525874] BMI_post_send: addr: 6843072, offset: 0x2aaabaa4d010, size: 2097152, tag: 33 [D 20:01:04.525883] post_sr: 10.1.4.56:39990 bh 94 len 32 wr_id 683a40. [D 20:01:04.525892] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.525900] BMI_post_send: addr: 6843072, offset: 0x2aaab8a3d010, size: 2097152, tag: 33 [D 20:01:04.525908] post_sr: 10.1.4.56:39990 bh 95 len 32 wr_id 683a70. [D 20:01:04.525918] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.525941] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.525950] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.525957] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.525965] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.525973] BMI_post_send: addr: 6843072, offset: 0x2aaab7231010, size: 2097152, tag: 32 [D 20:01:04.525981] post_sr: 10.1.4.56:39990 bh 96 len 32 wr_id 683aa0. [D 20:01:04.525991] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.525999] BMI_post_send: addr: 6843072, offset: 0x2aaab5420010, size: 2097152, tag: 32 [D 20:01:04.526007] post_sr: 10.1.4.56:39990 bh 97 len 32 wr_id 683ad0. [D 20:01:04.526018] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:04.526026] BMI_post_send: addr: 6843072, offset: 0x2aaab3611010, size: 2097152, tag: 32 [D 20:01:04.526034] post_sr: 10.1.4.56:39990 bh 98 len 32 wr_id 683b00. [D 20:01:04.526044] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:04.526051] BMI_post_send: addr: 6843072, offset: 0x2aaab1a02010, size: 2097152, tag: 32 [D 20:01:04.526060] post_sr: 10.1.4.56:39990 bh 99 len 32 wr_id 683b30. [D 20:01:04.526069] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.526077] BMI_post_send: addr: 6843072, offset: 0x2aaaafcd6010, size: 2097152, tag: 32 [D 20:01:04.526086] post_sr: 10.1.4.56:39990 bh 0 len 32 wr_id 6828a0. [D 20:01:04.526095] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.526102] BMI_post_send: addr: 6843072, offset: 0x2aaaae0c8010, size: 2097152, tag: 32 [D 20:01:04.526111] post_sr: 10.1.4.56:39990 bh 1 len 32 wr_id 6828d0. [D 20:01:04.526122] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.526129] BMI_post_send: addr: 6843072, offset: 0x2aaabac4f010, size: 2097152, tag: 32 [D 20:01:04.526139] post_sr: 10.1.4.56:39990 bh 2 len 32 wr_id 682900. [D 20:01:04.526149] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.526157] BMI_post_send: addr: 6843072, offset: 0x2aaab8c3e010, size: 2097152, tag: 32 [D 20:01:04.526165] post_sr: 10.1.4.56:39990 bh 3 len 32 wr_id 682930. [D 20:01:04.526174] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.526182] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.526191] BMI_post_send: addr: 6843072, offset: 0x2aaab320f010, size: 2097152, tag: 34 [D 20:01:04.526201] post_sr: 10.1.4.56:39990 bh 4 len 32 wr_id 682960. [D 20:01:04.526210] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:04.526218] BMI_post_send: addr: 6843072, offset: 0x2aaab1600010, size: 2097152, tag: 34 [D 20:01:04.526227] post_sr: 10.1.4.56:39990 bh 5 len 32 wr_id 682990. [D 20:01:04.526236] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.526243] BMI_post_send: addr: 6843072, offset: 0x2aaaaf8d4010, size: 2097152, tag: 34 [D 20:01:04.526252] post_sr: 10.1.4.56:39990 bh 6 len 32 wr_id 6829c0. [D 20:01:04.526261] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.526270] BMI_post_send: addr: 6843072, offset: 0x2aaaadcc6010, size: 2097152, tag: 34 [D 20:01:04.526278] post_sr: 10.1.4.56:39990 bh 7 len 32 wr_id 6829f0. [D 20:01:04.526287] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.526299] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.526307] BMI_post_send: addr: 6843072, offset: 0x2aaaba64b010, size: 2097152, tag: 34 [D 20:01:04.526316] post_sr: 10.1.4.56:39990 bh 8 len 32 wr_id 682a20. [D 20:01:04.526326] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.526334] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.526341] BMI_post_send: addr: 6843072, offset: 0x2aaab863b010, size: 2097152, tag: 34 [D 20:01:04.526349] post_sr: 10.1.4.56:39990 bh 9 len 32 wr_id 682a50. [D 20:01:04.526359] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.526367] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.526374] BMI_post_send: addr: 6843072, offset: 0x2aaab6a2d010, size: 2097152, tag: 35 [D 20:01:04.526383] post_sr: 10.1.4.56:39990 bh 10 len 32 wr_id 682a80. [D 20:01:04.526392] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.526401] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.526408] BMI_post_send: addr: 6843072, offset: 0x2aaab4a1b010, size: 2097152, tag: 35 [D 20:01:04.526416] post_sr: 10.1.4.56:39990 bh 11 len 32 wr_id 682ab0. [D 20:01:04.526426] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 -[D 20:01:04.526467] openib_check_cq: completed send wr_id 683980. [D 20:01:04.526482] openib_check_cq: completed send wr_id 6839b0. [D 20:01:04.526489] openib_check_cq: completed send wr_id 6839e0. [D 20:01:04.526496] openib_check_cq: completed send wr_id 683a10. [D 20:01:04.526504] openib_check_cq: completed send wr_id 683a40. [D 20:01:04.526511] openib_check_cq: completed send wr_id 683a70. [D 20:01:04.526517] openib_check_cq: completed send wr_id 683aa0. [D 20:01:04.526524] openib_check_cq: completed send wr_id 683ad0. [D 20:01:04.526531] openib_check_cq: completed send wr_id 683b00. [D 20:01:04.526538] openib_check_cq: completed send wr_id 683b30. [D 20:01:04.526545] openib_check_cq: completed send wr_id 6828a0. [D 20:01:04.526552] openib_check_cq: completed send wr_id 6828d0. [D 20:01:04.526559] openib_check_cq: completed send wr_id 682900. [D 20:01:04.526566] openib_check_cq: completed send wr_id 682930. [D 20:01:04.526573] openib_check_cq: completed send wr_id 682960. [D 20:01:04.526580] openib_check_cq: completed send wr_id 682990. [D 20:01:04.526587] openib_check_cq: completed send wr_id 6829c0. [D 20:01:04.526594] openib_check_cq: completed send wr_id 6829f0. [D 20:01:04.526601] openib_check_cq: completed send wr_id 682a20. [D 20:01:04.526608] openib_check_cq: completed send wr_id 682a50. [D 20:01:04.526614] openib_check_cq: completed send wr_id 682a80. [D 20:01:04.526621] openib_check_cq: completed send wr_id 682ab0. [D 20:01:04.528602] --- aio_progress_notification called with handle 2863311531 (0x2aaab15c56b0) [D 20:01:04.528645] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.528657] dbpf_open_cache_put: cached entry. [D 20:01:04.528664] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.528671] op_queue add: 0x2aaab15c56b0 [D 20:01:04.528685] DBPF I/O ops in progress: 15 [D 20:01:04.528693] starting delayed I/O operation 0x2aaab15cd8d0 (15 in progress) [D 20:01:04.528693] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.528719] BMI_post_send: addr: 6843072, offset: 0x2aaab2e0c010, size: 2097152, tag: 35 [D 20:01:04.528746] post_sr: 10.1.4.56:39990 bh 12 len 32 wr_id 682ae0. [D 20:01:04.528780] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:04.528808] openib_check_cq: completed send wr_id 682ae0. [D 20:01:04.528705] lio_listio called with 1 following aiocbs: [D 20:01:04.531377] aiocb_ptr_array[0]: fd: 13, off: 413138944, bytes: 2097152, buf: 0x2aaaad8c4010, type: 0 [D 20:01:04.531392] start_delayed_ops_if_any: lio_listio posted 0x2aaab15cd8d0 (handle 2863311531, ret 0)) [D 20:01:04.531424] --- aio_progress_notification called with handle 2863311531 (0x2aaab15c8210) [D 20:01:04.531435] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.531444] dbpf_open_cache_put: cached entry. [D 20:01:04.531451] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.531457] op_queue add: 0x2aaab15c8210 [D 20:01:04.531469] DBPF I/O ops in progress: 15 [D 20:01:04.531477] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.531499] BMI_post_send: addr: 6843072, offset: 0x2aaab10e0010, size: 2097152, tag: 35 [D 20:01:04.531477] starting delayed I/O operation 0x2aaab15d0430 (15 in progress) [D 20:01:04.533478] lio_listio called with 1 following aiocbs: [D 20:01:04.533488] aiocb_ptr_array[0]: fd: 13, off: 415236096, bytes: 2097152, buf: 0x2aaab7e37010, type: 0 [D 20:01:04.533500] start_delayed_ops_if_any: lio_listio posted 0x2aaab15d0430 (handle 2863311531, ret 0)) [D 20:01:04.531482] --- aio_progress_notification called with handle 2863311531 (0x2aaab15cad70) [D 20:01:04.533538] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.533548] dbpf_open_cache_put: cached entry. [D 20:01:04.533554] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.533561] op_queue add: 0x2aaab15cad70 [D 20:01:04.533568] DBPF I/O ops in progress: 15 [D 20:01:04.533575] starting delayed I/O operation 0x2aaab15d2f90 (15 in progress) [D 20:01:04.533584] lio_listio called with 1 following aiocbs: [D 20:01:04.533590] aiocb_ptr_array[0]: fd: 13, off: 417333248, bytes: 2097152, buf: 0x2aaab6228010, type: 0 [D 20:01:04.533598] start_delayed_ops_if_any: lio_listio posted 0x2aaab15d2f90 (handle 2863311531, ret 0)) [D 20:01:04.533613] --- aio_progress_notification called with handle 2863311531 (0x2aaab15cd8d0) [D 20:01:04.533622] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.533631] dbpf_open_cache_put: cached entry. [D 20:01:04.533637] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.533643] op_queue add: 0x2aaab15cd8d0 [D 20:01:04.533650] DBPF I/O ops in progress: 15 [D 20:01:04.533657] starting delayed I/O operation 0x2aaab15d85e0 (15 in progress) [D 20:01:04.533665] lio_listio called with 1 following aiocbs: [D 20:01:04.533671] aiocb_ptr_array[0]: fd: 13, off: 469762048, bytes: 2097152, buf: 0x2aaab4217010, type: 0 [D 20:01:04.533679] start_delayed_ops_if_any: lio_listio posted 0x2aaab15d85e0 (handle 2863311531, ret 0)) [D 20:01:04.536604] --- aio_progress_notification called with handle 2863311531 (0x2aaab15d0430) [D 20:01:04.536646] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.536656] dbpf_open_cache_put: cached entry. [D 20:01:04.536665] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.536673] op_queue add: 0x2aaab15d0430 [D 20:01:04.536681] DBPF I/O ops in progress: 15 [D 20:01:04.536688] starting delayed I/O operation 0x2aaab15db140 (15 in progress) [D 20:01:04.536697] lio_listio called with 1 following aiocbs: [D 20:01:04.536703] aiocb_ptr_array[0]: fd: 13, off: 471859200, bytes: 2097152, buf: 0x2aaab2608010, type: 0 [D 20:01:04.536714] start_delayed_ops_if_any: lio_listio posted 0x2aaab15db140 (handle 2863311531, ret 0)) [D 20:01:04.540224] post_sr: 10.1.4.56:39990 bh 13 len 32 wr_id 682b10. [D 20:01:04.540332] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.540358] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.540368] BMI_post_send: addr: 6843072, offset: 0x2aaaaf4d2010, size: 2097152, tag: 35 [D 20:01:04.540378] post_sr: 10.1.4.56:39990 bh 14 len 32 wr_id 682b40. [D 20:01:04.540389] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 0 [D 20:01:04.540397] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.540405] BMI_post_send: addr: 6843072, offset: 0x2aaaad8c4010, size: 2097152, tag: 35 [D 20:01:04.540413] post_sr: 10.1.4.56:39990 bh 15 len 32 wr_id 682b70. [D 20:01:04.540423] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.540431] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.540438] BMI_post_send: addr: 6843072, offset: 0x2aaab7e37010, size: 2097152, tag: 35 [D 20:01:04.540447] post_sr: 10.1.4.56:39990 bh 16 len 32 wr_id 682ba0. [D 20:01:04.540457] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 -[D 20:01:04.540494] openib_check_cq: completed send wr_id 682b10. [D 20:01:04.540505] openib_check_cq: completed send wr_id 682b40. [D 20:01:04.540512] openib_check_cq: completed send wr_id 682b70. [D 20:01:04.540519] openib_check_cq: completed send wr_id 682ba0. [D 20:01:04.541593] --- aio_progress_notification called with handle 2863311531 (0x2aaab15d2f90) [D 20:01:04.541623] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.541633] dbpf_open_cache_put: cached entry. [D 20:01:04.541640] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.541647] op_queue add: 0x2aaab15d2f90 [D 20:01:04.541660] DBPF I/O ops in progress: 15 [D 20:01:04.541667] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.541678] --- aio_progress_notification called with handle 2863311531 (0x2aaab15db140) [D 20:01:04.541685] BMI_post_send: addr: 6843072, offset: 0x2aaab6228010, size: 2097152, tag: 35 [D 20:01:04.541667] starting delayed I/O operation 0x2aaab15ddca0 (15 in progress) [D 20:01:04.541707] lio_listio called with 1 following aiocbs: [D 20:01:04.541709] post_sr: 10.1.4.56:39990 bh 17 len 32 wr_id 682bd0. [D 20:01:04.541716] aiocb_ptr_array[0]: fd: 13, off: 473956352, bytes: 2097152, buf: 0x2aaab08dc010, type: 0 [D 20:01:04.541740] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.541762] openib_check_cq: completed send wr_id 682bd0. [D 20:01:04.541689] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.543680] dbpf_open_cache_put: cached entry. [D 20:01:04.543688] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.543695] op_queue add: 0x2aaab15db140 [D 20:01:04.543713] --- aio_progress_notification called with handle 2863311531 (0x2aaab15ddca0) [D 20:01:04.543717] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab15d5af0. [D 20:01:04.541741] start_delayed_ops_if_any: lio_listio posted 0x2aaab15ddca0 (handle 2863311531, ret 0)) [D 20:01:04.541674] --- aio_progress_notification called with handle 2863311531 (0x2aaab15d85e0) [D 20:01:04.543781] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.543791] dbpf_open_cache_put: cached entry. [D 20:01:04.543797] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.543804] op_queue add: 0x2aaab15d85e0 [D 20:01:04.543812] DBPF I/O ops in progress: 15 [D 20:01:04.543815] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab15d5af0. [D 20:01:04.543821] starting delayed I/O operation 0x2aaab15e0800 (15 in progress) [D 20:01:04.543832] BMI_post_send: addr: 6843072, offset: 0x2aaab4217010, size: 2097152, tag: 37 [D 20:01:04.543843] lio_listio called with 1 following aiocbs: [D 20:01:04.543858] aiocb_ptr_array[0]: fd: 13, off: 476053504, bytes: 2097152, buf: 0x2aaaaecce010, type: 0 [D 20:01:04.543861] post_sr: 10.1.4.56:39990 bh 18 len 32 wr_id 682c00. [D 20:01:04.543870] start_delayed_ops_if_any: lio_listio posted 0x2aaab15e0800 (handle 2863311531, ret 0)) [D 20:01:04.543725] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.543887] trove_read_callback_fn: (post send time) ini posts: 0, pending: 1, last: 0 [D 20:01:04.543903] dbpf_open_cache_put: cached entry. [D 20:01:04.543905] BMI_post_send: addr: 6843072, offset: 0x2aaab2608010, size: 2097152, tag: 37 [D 20:01:04.543893] openib_check_cq: completed send wr_id 682c00. [D 20:01:04.543943] post_sr: 10.1.4.56:39990 bh 19 len 32 wr_id 682c30. [D 20:01:04.543954] trove_read_callback_fn: (post send time) ini posts: 0, pending: 2, last: 0 [D 20:01:04.543968] openib_check_cq: completed send wr_id 682c30. [D 20:01:04.543914] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.544621] op_queue add: 0x2aaab15ddca0 [D 20:01:04.544640] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab15d5af0. [D 20:01:04.544649] BMI_post_send: addr: 6843072, offset: 0x2aaab08dc010, size: 2097152, tag: 37 [D 20:01:04.544679] post_sr: 10.1.4.56:39990 bh 20 len 32 wr_id 682c60. [D 20:01:04.544679] DBPF I/O ops in progress: 15 [D 20:01:04.544700] starting delayed I/O operation 0x2aaab15e3360 (15 in progress) [D 20:01:04.544703] trove_read_callback_fn: (post send time) ini posts: 0, pending: 3, last: 0 [D 20:01:04.544711] openib_check_cq: completed send wr_id 682c60. [D 20:01:04.545934] --- aio_progress_notification called with handle 2863311531 (0x2aaab15e0800) [D 20:01:04.545962] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.545971] dbpf_open_cache_put: cached entry. [D 20:01:04.545978] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.545984] op_queue add: 0x2aaab15e0800 [D 20:01:04.546006] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab15d5af0. [D 20:01:04.546014] BMI_post_send: addr: 6843072, offset: 0x2aaaaecce010, size: 2097152, tag: 37 [D 20:01:04.544723] lio_listio called with 1 following aiocbs: [D 20:01:04.548618] aiocb_ptr_array[0]: fd: 13, off: 478150656, bytes: 2097152, buf: 0x2aaaad0c0010, type: 0 [D 20:01:04.548645] start_delayed_ops_if_any: lio_listio posted 0x2aaab15e3360 (handle 2863311531, ret 0)) [D 20:01:04.548670] post_sr: 10.1.4.56:39990 bh 21 len 32 wr_id 682c90. [D 20:01:04.548690] trove_read_callback_fn: (post send time) ini posts: 0, pending: 4, last: 0 [D 20:01:04.548693] openib_check_cq: completed send wr_id 682c90. [D 20:01:04.550647] DBPF I/O ops in progress: 15 [D 20:01:04.550684] starting delayed I/O operation 0x2aaabd301030 (15 in progress) [D 20:01:04.550694] lio_listio called with 1 following aiocbs: [D 20:01:04.550700] aiocb_ptr_array[0]: fd: 13, off: 125829120, bytes: 2097152, buf: 0x2aaac0217010, type: 0 [D 20:01:04.550714] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301030 (handle 2863311531, ret 0)) [D 20:01:04.550745] --- aio_progress_notification called with handle 2863311531 (0x2aaab15e3360) [D 20:01:04.550756] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.550765] dbpf_open_cache_put: cached entry. [D 20:01:04.550772] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.550779] op_queue add: 0x2aaab15e3360 [D 20:01:04.550799] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab15d5af0. [D 20:01:04.550808] BMI_post_send: addr: 6843072, offset: 0x2aaaad0c0010, size: 2097152, tag: 37 [D 20:01:04.550821] post_sr: 10.1.4.56:39990 bh 22 len 32 wr_id 682cc0. [D 20:01:04.550845] trove_read_callback_fn: (post send time) ini posts: 0, pending: 5, last: 1 [D 20:01:04.550847] openib_check_cq: completed send wr_id 682cc0. [D 20:01:04.550860] DBPF I/O ops in progress: 15 [D 20:01:04.550870] starting delayed I/O operation 0x2aaabd301180 (15 in progress) [D 20:01:04.550879] lio_listio called with 1 following aiocbs: [D 20:01:04.550885] aiocb_ptr_array[0]: fd: 13, off: 153092096, bytes: 2097152, buf: 0x2aaab9a45010, type: 0 [D 20:01:04.550893] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301180 (handle 2863311531, ret 0)) [D 20:01:04.554698] DBPF I/O ops in progress: 15 [D 20:01:04.554740] starting delayed I/O operation 0x2aaabd3012d0 (15 in progress) [D 20:01:04.554749] lio_listio called with 1 following aiocbs: [D 20:01:04.554756] aiocb_ptr_array[0]: fd: 13, off: 184549376, bytes: 2097152, buf: 0x2aaab4418010, type: 0 [D 20:01:04.554769] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3012d0 (handle 2863311531, ret 0)) [D 20:01:04.554804] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301030) [D 20:01:04.554816] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.554826] dbpf_open_cache_put: cached entry. [D 20:01:04.554833] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.554839] op_queue add: 0x2aaabd301030 [D 20:01:04.554860] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:04.554871] BMI_post_send: addr: 6843072, offset: 0x2aaac0217010, size: 2097152, tag: 26 [D 20:01:04.554894] post_sr: 10.1.4.56:39990 bh 23 len 32 wr_id 682cf0. [D 20:01:04.554924] trove_read_callback_fn: (post send time) ini posts: 0, pending: 6, last: 0 [D 20:01:04.554926] openib_check_cq: completed send wr_id 682cf0. [D 20:01:04.554944] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301180) [D 20:01:04.554957] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.554966] dbpf_open_cache_put: cached entry. [D 20:01:04.554973] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.554979] op_queue add: 0x2aaabd301180 [D 20:01:04.555028] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.555039] BMI_post_send: addr: 6843072, offset: 0x2aaab9a45010, size: 2097152, tag: 27 [D 20:01:04.554882] DBPF I/O ops in progress: 15 [D 20:01:04.555069] post_sr: 10.1.4.56:39990 bh 24 len 32 wr_id 682d20. [D 20:01:04.555085] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.555092] openib_check_cq: completed send wr_id 682d20. [D 20:01:04.557022] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3012d0) [D 20:01:04.557060] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.557070] dbpf_open_cache_put: cached entry. [D 20:01:04.557077] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.557083] op_queue add: 0x2aaabd3012d0 [D 20:01:04.557104] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.557115] BMI_post_send: addr: 6843072, offset: 0x2aaab4418010, size: 2097152, tag: 28 [D 20:01:04.555072] starting delayed I/O operation 0x2aaabd301cf0 (15 in progress) [D 20:01:04.557130] post_sr: 10.1.4.56:39990 bh 25 len 32 wr_id 682d50. [D 20:01:04.557135] lio_listio called with 1 following aiocbs: [D 20:01:04.557146] aiocb_ptr_array[0]: fd: 13, off: 127926272, bytes: 2097152, buf: 0x2aaac0418010, type: 0 [D 20:01:04.557157] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.557163] openib_check_cq: completed send wr_id 682d50. [D 20:01:04.559041] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301cf0) [D 20:01:04.559079] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.559089] dbpf_open_cache_put: cached entry. [D 20:01:04.559096] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.559102] op_queue add: 0x2aaabd301cf0 [D 20:01:04.559125] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:04.559135] BMI_post_send: addr: 6843072, offset: 0x2aaac0418010, size: 2097152, tag: 26 [D 20:01:04.557165] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301cf0 (handle 2863311531, ret 0)) [D 20:01:04.559150] post_sr: 10.1.4.56:39990 bh 26 len 32 wr_id 682d80. [D 20:01:04.559191] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.559195] openib_check_cq: completed send wr_id 682d80. [D 20:01:04.559208] DBPF I/O ops in progress: 15 [D 20:01:04.559220] starting delayed I/O operation 0x2aaabd301e40 (15 in progress) [D 20:01:04.559229] lio_listio called with 1 following aiocbs: [D 20:01:04.559235] aiocb_ptr_array[0]: fd: 13, off: 155189248, bytes: 2097152, buf: 0x2aaab7a35010, type: 0 [D 20:01:04.559247] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301e40 (handle 2863311531, ret 0)) [D 20:01:04.561198] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301e40) [D 20:01:04.561238] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.561248] dbpf_open_cache_put: cached entry. [D 20:01:04.561255] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.561262] op_queue add: 0x2aaabd301e40 [D 20:01:04.561283] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.561294] BMI_post_send: addr: 6843072, offset: 0x2aaab7a35010, size: 2097152, tag: 27 [D 20:01:04.561317] post_sr: 10.1.4.56:39990 bh 27 len 32 wr_id 682db0. [D 20:01:04.561346] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.561305] DBPF I/O ops in progress: 15 [D 20:01:04.561349] openib_check_cq: completed send wr_id 682db0. [D 20:01:04.561363] starting delayed I/O operation 0x2aaabd301f90 (15 in progress) [D 20:01:04.561376] lio_listio called with 1 following aiocbs: [D 20:01:04.561382] aiocb_ptr_array[0]: fd: 13, off: 186646528, bytes: 2097152, buf: 0x2aaab2809010, type: 0 [D 20:01:04.561394] start_delayed_ops_if_any: lio_listio posted 0x2aaabd301f90 (handle 2863311531, ret 0)) [D 20:01:04.563335] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301f90) [D 20:01:04.563372] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.563382] dbpf_open_cache_put: cached entry. [D 20:01:04.563389] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.563396] op_queue add: 0x2aaabd301f90 [D 20:01:04.563419] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.563429] BMI_post_send: addr: 6843072, offset: 0x2aaab2809010, size: 2097152, tag: 28 [D 20:01:04.563445] DBPF I/O ops in progress: 15 [D 20:01:04.563445] post_sr: 10.1.4.56:39990 bh 28 len 32 wr_id 682de0. [D 20:01:04.563459] starting delayed I/O operation 0x2aaabd3023a0 (15 in progress) [D 20:01:04.563478] lio_listio called with 1 following aiocbs: [D 20:01:04.563485] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.563490] openib_check_cq: completed send wr_id 682de0. [D 20:01:04.563485] aiocb_ptr_array[0]: fd: 13, off: 218103808, bytes: 2097152, buf: 0x2aaab682c010, type: 0 [D 20:01:04.564607] start_delayed_ops_if_any: lio_listio posted 0x2aaabd3023a0 (handle 2863311531, ret 0)) [D 20:01:04.566521] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3023a0) [D 20:01:04.566559] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.566571] dbpf_open_cache_put: cached entry. [D 20:01:04.566578] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.566585] op_queue add: 0x2aaabd3023a0 [D 20:01:04.566610] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.566620] BMI_post_send: addr: 6843072, offset: 0x2aaab682c010, size: 2097152, tag: 29 [D 20:01:04.566635] DBPF I/O ops in progress: 15 [D 20:01:04.566635] post_sr: 10.1.4.56:39990 bh 29 len 32 wr_id 682e10. [D 20:01:04.566651] starting delayed I/O operation 0x2aaabd302ff0 (15 in progress) [D 20:01:04.566670] lio_listio called with 1 following aiocbs: [D 20:01:04.566677] aiocb_ptr_array[0]: fd: 13, off: 130023424, bytes: 2097152, buf: 0x2aaac0619010, type: 0 [D 20:01:04.566679] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:04.566686] openib_check_cq: completed send wr_id 682e10. [D 20:01:04.568596] start_delayed_ops_if_any: lio_listio posted 0x2aaabd302ff0 (handle 2863311531, ret 0)) [D 20:01:04.570478] DBPF I/O ops in progress: 15 [D 20:01:04.570513] starting delayed I/O operation 0x2aaabd307a50 (15 in progress) [D 20:01:04.570522] lio_listio called with 1 following aiocbs: [D 20:01:04.570528] aiocb_ptr_array[0]: fd: 13, off: 251658240, bytes: 2097152, buf: 0x2aaab6c2e010, type: 0 [D 20:01:04.570542] start_delayed_ops_if_any: lio_listio posted 0x2aaabd307a50 (handle 2863311531, ret 0)) [D 20:01:04.570572] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302ff0) [D 20:01:04.570582] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.570592] dbpf_open_cache_put: cached entry. [D 20:01:04.570598] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.570605] op_queue add: 0x2aaabd302ff0 [D 20:01:04.570624] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:04.570635] BMI_post_send: addr: 6843072, offset: 0x2aaac0619010, size: 2097152, tag: 26 [D 20:01:04.570645] DBPF I/O ops in progress: 15 [D 20:01:04.570661] starting delayed I/O operation 0x2aaabd303140 (15 in progress) [D 20:01:04.570691] post_sr: 10.1.4.56:39990 bh 30 len 32 wr_id 682e40. [D 20:01:04.570719] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.570678] lio_listio called with 1 following aiocbs: [D 20:01:04.570722] openib_check_cq: completed send wr_id 682e40. [D 20:01:04.570736] aiocb_ptr_array[0]: fd: 13, off: 285212672, bytes: 2097152, buf: 0x2aaab9844010, type: 0 [D 20:01:04.570748] start_delayed_ops_if_any: lio_listio posted 0x2aaabd303140 (handle 2863311531, ret 0)) [D 20:01:04.574628] DBPF I/O ops in progress: 15 [D 20:01:04.574670] starting delayed I/O operation 0x2aaabd303290 (15 in progress) [D 20:01:04.574679] lio_listio called with 1 following aiocbs: [D 20:01:04.574686] aiocb_ptr_array[0]: fd: 13, off: 220200960, bytes: 2097152, buf: 0x2aaab481a010, type: 0 [D 20:01:04.574700] start_delayed_ops_if_any: lio_listio posted 0x2aaabd303290 (handle 2863311531, ret 0)) [D 20:01:04.574733] --- aio_progress_notification called with handle 2863311531 (0x2aaabd307a50) [D 20:01:04.574745] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.574755] dbpf_open_cache_put: cached entry. [D 20:01:04.574761] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.574768] op_queue add: 0x2aaabd307a50 [D 20:01:04.574789] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.574800] BMI_post_send: addr: 6843072, offset: 0x2aaab6c2e010, size: 2097152, tag: 30 [D 20:01:04.574811] DBPF I/O ops in progress: 15 [D 20:01:04.574820] post_sr: 10.1.4.56:39990 bh 31 len 32 wr_id 682e70. [D 20:01:04.574827] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303140) [D 20:01:04.574841] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.574851] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.574887] openib_check_cq: completed send wr_id 682e70. [D 20:01:04.576804] DBPF I/O ops in progress: 14 [D 20:01:04.576847] DBPF I/O ops in progress: 13 [D 20:01:04.574853] dbpf_open_cache_put: cached entry. [D 20:01:04.576891] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.576899] op_queue add: 0x2aaabd303140 [D 20:01:04.576910] DBPF I/O ops in progress: 12 [D 20:01:04.576921] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.576940] BMI_post_send: addr: 6843072, offset: 0x2aaab9844010, size: 2097152, tag: 31 [D 20:01:04.576948] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303290) [D 20:01:04.576962] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.576966] post_sr: 10.1.4.56:39990 bh 32 len 32 wr_id 682ea0. [D 20:01:04.576972] dbpf_open_cache_put: cached entry. [D 20:01:04.576983] dbpf_open_cache_put: move to unused list. [D 20:01:04.576989] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.576993] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.576999] DBPF I/O ops in progress: 11 [D 20:01:04.577013] openib_check_cq: completed send wr_id 682ea0. [D 20:01:04.576995] op_queue add: 0x2aaabd303290 [D 20:01:04.577041] DBPF I/O ops in progress: 10 [D 20:01:04.577064] DBPF I/O ops in progress: 9 [D 20:01:04.580594] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.580606] BMI_post_send: addr: 6843072, offset: 0x2aaab481a010, size: 2097152, tag: 29 [D 20:01:04.583778] post_sr: 10.1.4.56:39990 bh 33 len 32 wr_id 682ed0. [D 20:01:04.583780] DBPF I/O ops in progress: 8 [D 20:01:04.583794] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.583796] openib_check_cq: completed send wr_id 682ed0. [D 20:01:04.583811] DBPF I/O ops in progress: 7 [D 20:01:04.583826] DBPF I/O ops in progress: 6 [D 20:01:04.583865] DBPF I/O ops in progress: 5 [D 20:01:04.583892] DBPF I/O ops in progress: 4 [D 20:01:04.583919] DBPF I/O ops in progress: 3 [D 20:01:04.583933] DBPF I/O ops in progress: 2 [D 20:01:04.583970] DBPF I/O ops in progress: 1 [D 20:01:04.583996] DBPF I/O ops in progress: 0 [A 20:01:04.674036] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65f550: io: finish (Success) [D 20:01:04.674051] PINT_encode_release [D 20:01:04.674061] lebf_encode_rel [D 20:01:04.674072] [SM frame get]: (0x65f450) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.674081] server_state_machine_complete 0x65f450 [D 20:01:04.674089] PINT_decode_release [D 20:01:04.674097] lebf_decode_rel [D 20:01:04.674106] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.674114] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.674122] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 28 [D 20:01:04.674132] [SM Exiting]: (0x65f450) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.674143] server_state_machine_terminate 0x65f450 [D 20:01:04.674152] [SM Entering]: (0x64db10) io_sm:release (status: 0) [D 20:01:04.674160] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.674173] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6943a0 [D 20:01:04.674183] [SM Exiting]: (0x64db10) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.674192] [SM Entering]: (0x64db10) io_sm:cleanup (status: 0) [D 20:01:04.674200] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.680269] troy.scl@10.1.4.56:39990 H=2863311531 S=0x64dc10: io: finish (Success) [D 20:01:04.680282] PINT_encode_release [D 20:01:04.680290] lebf_encode_rel [D 20:01:04.680300] [SM frame get]: (0x64db10) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.680308] server_state_machine_complete 0x64db10 [D 20:01:04.680316] PINT_decode_release [D 20:01:04.680324] lebf_decode_rel [D 20:01:04.680332] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.680340] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.680349] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 27 [D 20:01:04.680358] [SM Exiting]: (0x64db10) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.680367] server_state_machine_terminate 0x64db10 [D 20:01:04.680375] [SM Entering]: (0x64edd0) io_sm:release (status: 0) [D 20:01:04.680384] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.680394] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x682620 [D 20:01:04.680404] [SM Exiting]: (0x64edd0) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.680413] [SM Entering]: (0x64edd0) io_sm:cleanup (status: 0) [D 20:01:04.680421] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.686140] troy.scl@10.1.4.56:39990 H=2863311531 S=0x64eed0: io: finish (Success) [D 20:01:04.686153] PINT_encode_release [D 20:01:04.686161] lebf_encode_rel [D 20:01:04.686171] [SM frame get]: (0x64edd0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.686179] server_state_machine_complete 0x64edd0 [D 20:01:04.686188] PINT_decode_release [D 20:01:04.686195] lebf_decode_rel [D 20:01:04.686203] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.686211] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.686219] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 26 [D 20:01:04.686229] [SM Exiting]: (0x64edd0) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.686237] server_state_machine_terminate 0x64edd0 [D 20:01:04.686246] [SM Entering]: (0x650090) io_sm:release (status: 0) [D 20:01:04.686255] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.686265] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6b2c40 [D 20:01:04.686275] [SM Exiting]: (0x650090) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.686283] [SM Entering]: (0x650090) io_sm:cleanup (status: 0) [D 20:01:04.686291] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.692137] troy.scl@10.1.4.56:39990 H=2863311531 S=0x650190: io: finish (Success) [D 20:01:04.692150] PINT_encode_release [D 20:01:04.692158] lebf_encode_rel [D 20:01:04.692168] [SM frame get]: (0x650090) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.692176] server_state_machine_complete 0x650090 [D 20:01:04.692184] PINT_decode_release [D 20:01:04.692192] lebf_decode_rel [D 20:01:04.692200] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.692208] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.692217] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 25 [D 20:01:04.692226] [SM Exiting]: (0x650090) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.692235] server_state_machine_terminate 0x650090 [D 20:01:04.692243] [SM Entering]: (0x651350) io_sm:release (status: 0) [D 20:01:04.692252] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.692262] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6b6ec0 [D 20:01:04.692272] [SM Exiting]: (0x651350) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.692280] [SM Entering]: (0x651350) io_sm:cleanup (status: 0) [D 20:01:04.692289] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.698134] troy.scl@10.1.4.56:39990 H=2863311531 S=0x651450: io: finish (Success) [D 20:01:04.698147] PINT_encode_release [D 20:01:04.698156] lebf_encode_rel [D 20:01:04.698165] [SM frame get]: (0x651350) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.698174] server_state_machine_complete 0x651350 [D 20:01:04.698182] PINT_decode_release [D 20:01:04.698189] lebf_decode_rel [D 20:01:04.698198] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.698206] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.698214] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 24 [D 20:01:04.698223] [SM Exiting]: (0x651350) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.698232] server_state_machine_terminate 0x651350 [D 20:01:04.698241] [SM Entering]: (0x652610) io_sm:release (status: 0) [D 20:01:04.698249] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.698259] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6bb180 [D 20:01:04.698269] [SM Exiting]: (0x652610) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.698278] [SM Entering]: (0x652610) io_sm:cleanup (status: 0) [D 20:01:04.698286] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.704006] troy.scl@10.1.4.56:39990 H=2863311531 S=0x652710: io: finish (Success) [D 20:01:04.704019] PINT_encode_release [D 20:01:04.704027] lebf_encode_rel [D 20:01:04.704036] [SM frame get]: (0x652610) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.704045] server_state_machine_complete 0x652610 [D 20:01:04.704053] PINT_decode_release [D 20:01:04.704061] lebf_decode_rel [D 20:01:04.704069] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.704077] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.704085] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 23 [D 20:01:04.704094] [SM Exiting]: (0x652610) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.704103] server_state_machine_terminate 0x652610 [D 20:01:04.704112] [SM Entering]: (0x655e50) io_sm:release (status: 0) [D 20:01:04.704120] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.704130] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6c7cd0 [D 20:01:04.704140] [SM Exiting]: (0x655e50) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.704149] [SM Entering]: (0x655e50) io_sm:cleanup (status: 0) [D 20:01:04.704157] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.709753] troy.scl@10.1.4.56:39990 H=2863311531 S=0x655f50: io: finish (Success) [D 20:01:04.709765] PINT_encode_release [D 20:01:04.709774] lebf_encode_rel [D 20:01:04.709783] [SM frame get]: (0x655e50) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.709792] server_state_machine_complete 0x655e50 [D 20:01:04.709799] PINT_decode_release [D 20:01:04.709806] lebf_decode_rel [D 20:01:04.709814] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.709822] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.709830] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 22 [D 20:01:04.709839] [SM Exiting]: (0x655e50) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.709848] server_state_machine_terminate 0x655e50 [D 20:01:04.709857] [SM Entering]: (0x6538d0) io_sm:release (status: 0) [D 20:01:04.709865] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.709875] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6bf5b0 [D 20:01:04.709885] [SM Exiting]: (0x6538d0) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.709894] [SM Entering]: (0x6538d0) io_sm:cleanup (status: 0) [D 20:01:04.709902] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.715749] troy.scl@10.1.4.56:39990 H=2863311531 S=0x6539d0: io: finish (Success) [D 20:01:04.715761] PINT_encode_release [D 20:01:04.715770] lebf_encode_rel [D 20:01:04.715782] [SM frame get]: (0x6538d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.715791] server_state_machine_complete 0x6538d0 [D 20:01:04.715799] PINT_decode_release [D 20:01:04.715807] lebf_decode_rel [D 20:01:04.715815] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.715823] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.715831] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 21 [D 20:01:04.715841] [SM Exiting]: (0x6538d0) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.715849] server_state_machine_terminate 0x6538d0 [D 20:01:04.715858] [SM Entering]: (0x654b90) io_sm:release (status: 0) [D 20:01:04.715867] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.715877] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6c0210 [D 20:01:04.715886] [SM Exiting]: (0x654b90) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.715895] [SM Entering]: (0x654b90) io_sm:cleanup (status: 0) [D 20:01:04.715903] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.721621] troy.scl@10.1.4.56:39990 H=2863311531 S=0x654c90: io: finish (Success) [D 20:01:04.721632] PINT_encode_release [D 20:01:04.721641] lebf_encode_rel [D 20:01:04.721651] [SM frame get]: (0x654b90) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.721659] server_state_machine_complete 0x654b90 [D 20:01:04.721668] PINT_decode_release [D 20:01:04.721675] lebf_decode_rel [D 20:01:04.721683] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.721691] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.721700] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 20 [D 20:01:04.721709] [SM Exiting]: (0x654b90) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.721718] server_state_machine_terminate 0x654b90 [D 20:01:04.721726] [SM Entering]: (0x6583d0) io_sm:release (status: 0) [D 20:01:04.721735] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.721745] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6cfff0 [D 20:01:04.721755] [SM Exiting]: (0x6583d0) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.721764] [SM Entering]: (0x6583d0) io_sm:cleanup (status: 0) [D 20:01:04.721772] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.727492] troy.scl@10.1.4.56:39990 H=2863311531 S=0x6584d0: io: finish (Success) [D 20:01:04.727505] PINT_encode_release [D 20:01:04.727513] lebf_encode_rel [D 20:01:04.727522] [SM frame get]: (0x6583d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.727531] server_state_machine_complete 0x6583d0 [D 20:01:04.727539] PINT_decode_release [D 20:01:04.727546] lebf_decode_rel [D 20:01:04.727554] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.727562] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.727571] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 19 [D 20:01:04.727580] [SM Exiting]: (0x6583d0) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.727588] server_state_machine_terminate 0x6583d0 [D 20:01:04.727597] [SM Entering]: (0x65a950) io_sm:release (status: 0) [D 20:01:04.727606] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.727616] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6d84f0 [D 20:01:04.727626] [SM Exiting]: (0x65a950) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.727634] [SM Entering]: (0x65a950) io_sm:cleanup (status: 0) [D 20:01:04.727643] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.733239] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65aa50: io: finish (Success) [D 20:01:04.733252] PINT_encode_release [D 20:01:04.733260] lebf_encode_rel [D 20:01:04.733270] [SM frame get]: (0x65a950) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.733278] server_state_machine_complete 0x65a950 [D 20:01:04.733286] PINT_decode_release [D 20:01:04.733294] lebf_decode_rel [D 20:01:04.733302] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.733310] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.733319] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 18 [D 20:01:04.733328] [SM Exiting]: (0x65a950) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.733336] server_state_machine_terminate 0x65a950 [D 20:01:04.733345] [SM Entering]: (0x65bc10) io_sm:release (status: 0) [D 20:01:04.733354] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.733363] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6dc7b0 [D 20:01:04.733373] [SM Exiting]: (0x65bc10) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.733382] [SM Entering]: (0x65bc10) io_sm:cleanup (status: 0) [D 20:01:04.733390] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.739111] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65bd10: io: finish (Success) [D 20:01:04.739123] PINT_encode_release [D 20:01:04.739131] lebf_encode_rel [D 20:01:04.739141] [SM frame get]: (0x65bc10) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.739150] server_state_machine_complete 0x65bc10 [D 20:01:04.739158] PINT_decode_release [D 20:01:04.739165] lebf_decode_rel [D 20:01:04.739173] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.739181] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.739190] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 17 [D 20:01:04.739199] [SM Exiting]: (0x65bc10) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.739207] server_state_machine_terminate 0x65bc10 [D 20:01:04.739216] [SM Entering]: (0x65ced0) io_sm:release (status: 0) [D 20:01:04.739225] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.739234] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6e0be0 [D 20:01:04.739244] [SM Exiting]: (0x65ced0) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.739252] [SM Entering]: (0x65ced0) io_sm:cleanup (status: 0) [D 20:01:04.739261] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.744983] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65cfd0: io: finish (Success) [D 20:01:04.744995] PINT_encode_release [D 20:01:04.745003] lebf_encode_rel [D 20:01:04.745013] [SM frame get]: (0x65ced0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.745021] server_state_machine_complete 0x65ced0 [D 20:01:04.745029] PINT_decode_release [D 20:01:04.745037] lebf_decode_rel [D 20:01:04.745045] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.745053] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.745061] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 16 [D 20:01:04.745071] [SM Exiting]: (0x65ced0) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.745079] server_state_machine_terminate 0x65ced0 [D 20:01:04.745088] [SM Entering]: (0x659690) io_sm:release (status: 0) [D 20:01:04.745097] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.745107] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6d4250 [D 20:01:04.745116] [SM Exiting]: (0x659690) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.745125] [SM Entering]: (0x659690) io_sm:cleanup (status: 0) [D 20:01:04.745133] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.750855] troy.scl@10.1.4.56:39990 H=2863311531 S=0x659790: io: finish (Success) [D 20:01:04.750868] PINT_encode_release [D 20:01:04.750876] lebf_encode_rel [D 20:01:04.750885] [SM frame get]: (0x659690) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.750894] server_state_machine_complete 0x659690 [D 20:01:04.750902] PINT_decode_release [D 20:01:04.750910] lebf_decode_rel [D 20:01:04.750918] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.750926] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.750934] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 15 [D 20:01:04.750943] [SM Exiting]: (0x659690) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.750952] server_state_machine_terminate 0x659690 [D 20:01:04.750961] [SM Entering]: (0x657110) io_sm:release (status: 0) [D 20:01:04.750969] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.750979] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6cc030 [D 20:01:04.750989] [SM Exiting]: (0x657110) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.750998] [SM Entering]: (0x657110) io_sm:cleanup (status: 0) [D 20:01:04.751006] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.756728] troy.scl@10.1.4.56:39990 H=2863311531 S=0x657210: io: finish (Success) [D 20:01:04.756740] PINT_encode_release [D 20:01:04.756749] lebf_encode_rel [D 20:01:04.756758] [SM frame get]: (0x657110) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.756767] server_state_machine_complete 0x657110 [D 20:01:04.756775] PINT_decode_release [D 20:01:04.756783] lebf_decode_rel [D 20:01:04.756791] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.756799] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.756807] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 14 [D 20:01:04.756816] [SM Exiting]: (0x657110) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.756825] server_state_machine_terminate 0x657110 [D 20:01:04.756834] [SM Entering]: (0x65e190) io_sm:release (status: 0) [D 20:01:04.756842] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.756852] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x6e19b0 [D 20:01:04.756862] [SM Exiting]: (0x65e190) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.756871] [SM Entering]: (0x65e190) io_sm:cleanup (status: 0) [D 20:01:04.756879] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [A 20:01:04.762724] troy.scl@10.1.4.56:39990 H=2863311531 S=0x65e290: io: finish (Success) [D 20:01:04.762736] PINT_encode_release [D 20:01:04.762745] lebf_encode_rel [D 20:01:04.762754] [SM frame get]: (0x65e190) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.762763] server_state_machine_complete 0x65e190 [D 20:01:04.762771] PINT_decode_release [D 20:01:04.762778] lebf_decode_rel [D 20:01:04.762786] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.762794] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.762802] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 13 [D 20:01:04.762812] [SM Exiting]: (0x65e190) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.762820] server_state_machine_terminate 0x65e190 -[D 20:01:04.826635] openib_check_cq: completed recv wr_id 684d10. [D 20:01:04.826650] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd302e90 to 10.1.4.56:39990 remote addr 8e43d000 rkey 109926 nic_wr_credit 1. [D 20:01:04.826662] openib_post_rr: 10.1.4.56:39990 bh 94 wr_id 684d10. [D 20:01:04.826968] openib_check_cq: completed recv wr_id 684d40. [D 20:01:04.826979] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3033e0 to 10.1.4.56:39990 remote addr 8c43b000 rkey 109a26 nic_wr_credit 2. [D 20:01:04.826988] openib_post_rr: 10.1.4.56:39990 bh 95 wr_id 684d40. [D 20:01:04.827372] openib_check_cq: completed recv wr_id 684d70. [D 20:01:04.827382] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd303d80 to 10.1.4.56:39990 remote addr 9043f000 rkey 109826 nic_wr_credit 3. [D 20:01:04.827392] openib_post_rr: 10.1.4.56:39990 bh 96 wr_id 684d70. [D 20:01:04.828166] openib_check_cq: completed recv wr_id 684da0. [D 20:01:04.828177] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31d730 to 10.1.4.56:39990 remote addr 8a439000 rkey 109b26 nic_wr_credit 4. [D 20:01:04.828186] openib_post_rr: 10.1.4.56:39990 bh 97 wr_id 684da0. [D 20:01:04.828631] openib_check_cq: completed recv wr_id 684dd0. [D 20:01:04.828642] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31e230 to 10.1.4.56:39990 remote addr 86435000 rkey 109d26 nic_wr_credit 5. [D 20:01:04.828651] openib_post_rr: 10.1.4.56:39990 bh 98 wr_id 684dd0. [D 20:01:04.828925] openib_check_cq: completed rdma wr_id 2aaabd302e90, nic_wr_credit 4. [D 20:01:04.828936] post_sr: 10.1.4.56:39990 bh 34 len 16 wr_id 682f00. [D 20:01:04.830761] openib_check_cq: completed recv wr_id 684e00. [D 20:01:04.830772] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd307210 to 10.1.4.56:39990 remote addr 94643000 rkey 109526 nic_wr_credit 5. [D 20:01:04.830780] openib_post_rr: 10.1.4.56:39990 bh 99 wr_id 684e00. [D 20:01:04.831153] openib_check_cq: completed rdma wr_id 2aaabd3033e0, nic_wr_credit 4. [D 20:01:04.831163] post_sr: 10.1.4.56:39990 bh 35 len 16 wr_id 682f30. [D 20:01:04.831928] openib_check_cq: completed recv wr_id 683b70. [D 20:01:04.831938] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3024f0 to 10.1.4.56:39990 remote addr 92641000 rkey 109626 nic_wr_credit 5. [D 20:01:04.831947] openib_post_rr: 10.1.4.56:39990 bh 0 wr_id 683b70. [D 20:01:04.833094] openib_check_cq: completed recv wr_id 683ba0. [D 20:01:04.833104] openib_post_sr_rdmaw: ibv_post_send wr_id 6ecb00 to 10.1.4.56:39990 remote addr 9d24b000 rkey 108f27 nic_wr_credit 6. [D 20:01:04.833113] openib_post_rr: 10.1.4.56:39990 bh 1 wr_id 683ba0. [D 20:01:04.833377] openib_check_cq: completed rdma wr_id 2aaabd303d80, nic_wr_credit 5. [D 20:01:04.833387] post_sr: 10.1.4.56:39990 bh 36 len 16 wr_id 682f60. [D 20:01:04.834248] openib_check_cq: completed recv wr_id 683bd0. [D 20:01:04.834257] openib_post_sr_rdmaw: ibv_post_send wr_id 6663a0 to 10.1.4.56:39990 remote addr a224f000 rkey 108d27 nic_wr_credit 6. [D 20:01:04.834266] openib_post_rr: 10.1.4.56:39990 bh 2 wr_id 683bd0. [D 20:01:04.835411] openib_check_cq: completed recv wr_id 683c00. [D 20:01:04.835421] openib_post_sr_rdmaw: ibv_post_send wr_id 6eacb0 to 10.1.4.56:39990 remote addr a004d000 rkey 108e27 nic_wr_credit 7. [D 20:01:04.835430] openib_post_rr: 10.1.4.56:39990 bh 3 wr_id 683c00. [D 20:01:04.835602] openib_check_cq: completed rdma wr_id 2aaabd31d730, nic_wr_credit 6. [D 20:01:04.835612] post_sr: 10.1.4.56:39990 bh 37 len 16 wr_id 682f90. [D 20:01:04.836598] openib_check_cq: completed recv wr_id 683c30. [D 20:01:04.836608] openib_post_sr_rdmaw: ibv_post_send wr_id 66af30 to 10.1.4.56:39990 remote addr 9aa49000 rkey 109126 nic_wr_credit 7. [D 20:01:04.836617] openib_post_rr: 10.1.4.56:39990 bh 4 wr_id 683c30. [D 20:01:04.837760] openib_check_cq: completed recv wr_id 683c60. [D 20:01:04.837769] openib_post_sr_rdmaw: ibv_post_send wr_id 6ebff0 to 10.1.4.56:39990 remote addr 98847000 rkey 109226 nic_wr_credit 8. [D 20:01:04.837779] openib_post_rr: 10.1.4.56:39990 bh 5 wr_id 683c60. [D 20:01:04.837826] openib_check_cq: completed rdma wr_id 2aaabd31e230, nic_wr_credit 7. [D 20:01:04.837836] post_sr: 10.1.4.56:39990 bh 38 len 16 wr_id 682fc0. [D 20:01:04.837845] openib_check_cq: completed send wr_id 682f00. [D 20:01:04.837857] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c210. [D 20:01:04.837868] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.837877] ========================================================= [D 20:01:04.837885] PINT_process_request [D 20:01:04.837893] tiling 12 copies [D 20:01:04.837902] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.837910] lvl 0 el 0 blk 0 by 352321536 [D 20:01:04.837918] to 352321536 ta 335544320 fi 369098752 [D 20:01:04.837926] basic type or contiguous data [D 20:01:04.837934] Request disp [D 20:01:04.837941] contig_offset = 352321536 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.837950] calling distribute [D 20:01:04.837957] PINT_distribute [D 20:01:04.837965] of 352321536 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.837975] begin iteration loff: 352321536 [D 20:01:04.837983] segment exceeds byte limit [D 20:01:04.837991] process a segment [D 20:01:04.837998] of 352321536 sz 2097152 [D 20:01:04.838006] add a segment 0 [D 20:01:04.838014] end iteration [D 20:01:04.838021] done with segments or bytes [D 20:01:04.838029] of 354418688 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.838038] next loff: 354418688 next poff: 354418688 [D 20:01:04.838046] return value 2097152 [D 20:01:04.838055] finished [D 20:01:04.838062] retval = 2097152 [D 20:01:04.838070] exiting distribute returned less than expected [D 20:01:04.838078] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 354418688 fo 369098752 eof 0 [D 20:01:04.838087] ========================================================= [D 20:01:04.838099] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.838107] dbpf_bstream_rw_list: mem_offset: 0x2aaab7030010, mem_size: 2097152 [D 20:01:04.838114] dbpf_bstream_rw_list: stream_offset: 352321536, stream_size: 2097152 [D 20:01:04.838121] dbpf_open_cache_get: called [D 20:01:04.838127] dbpf_open_cache_get: found bstream entry in unused list. [D 20:01:04.838134] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.838149] DBPF I/O ops in progress: 1 [D 20:01:04.838155] lio_listio called with the following aiocbs: [D 20:01:04.838162] aiocb_ptr_array[0]: fd: 13, off: 352321536, bytes: 2097152, buf: 0x2aaab7030010, type: 0 [D 20:01:04.838174] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304880 (handle 2863311531, ret 0) [D 20:01:04.838998] openib_check_cq: completed recv wr_id 683c90. [D 20:01:04.839006] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd301840 to 10.1.4.56:39990 remote addr 96845000 rkey 109426 nic_wr_credit 8. [D 20:01:04.839014] openib_post_rr: 10.1.4.56:39990 bh 6 wr_id 683c90. [D 20:01:04.840051] openib_check_cq: completed rdma wr_id 2aaabd307210, nic_wr_credit 7. [D 20:01:04.840060] post_sr: 10.1.4.56:39990 bh 39 len 16 wr_id 682ff0. [D 20:01:04.840068] openib_check_cq: completed send wr_id 682f30. [D 20:01:04.840079] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1506e50. [D 20:01:04.840087] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.840094] ========================================================= [D 20:01:04.840100] PINT_process_request [D 20:01:04.840106] tiling 13 copies [D 20:01:04.840112] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.840119] lvl 0 el 0 blk 0 by 385875968 [D 20:01:04.840126] to 385875968 ta 369098752 fi 402653184 [D 20:01:04.840132] basic type or contiguous data [D 20:01:04.840138] Request disp [D 20:01:04.840144] contig_offset = 385875968 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.840151] calling distribute [D 20:01:04.840156] PINT_distribute [D 20:01:04.840162] of 385875968 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.840169] begin iteration loff: 385875968 [D 20:01:04.840175] segment exceeds byte limit [D 20:01:04.840181] process a segment [D 20:01:04.840187] of 385875968 sz 2097152 [D 20:01:04.840193] add a segment 0 [D 20:01:04.840199] end iteration [D 20:01:04.840205] done with segments or bytes [D 20:01:04.840211] of 387973120 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.840217] next loff: 387973120 next poff: 387973120 [D 20:01:04.840224] return value 2097152 [D 20:01:04.840230] finished [D 20:01:04.840235] retval = 2097152 [D 20:01:04.840241] exiting distribute returned less than expected [D 20:01:04.840247] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 387973120 fo 402653184 eof 0 [D 20:01:04.840255] ========================================================= [D 20:01:04.840262] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.840268] dbpf_bstream_rw_list: mem_offset: 0x2aaab6e2f010, mem_size: 2097152 [D 20:01:04.840275] dbpf_bstream_rw_list: stream_offset: 385875968, stream_size: 2097152 [D 20:01:04.840282] dbpf_open_cache_get: called [D 20:01:04.840288] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.840294] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.840307] DBPF I/O ops in progress: 2 [D 20:01:04.840314] lio_listio called with the following aiocbs: [D 20:01:04.840320] aiocb_ptr_array[0]: fd: 13, off: 385875968, bytes: 2097152, buf: 0x2aaab6e2f010, type: 0 [D 20:01:04.840328] issue_or_delay_io_operation: lio_listio posted 0x2aaabd302e90 (handle 2863311531, ret 0) [D 20:01:04.840337] openib_check_cq: completed recv wr_id 683cc0. [D 20:01:04.840345] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd301420 to 10.1.4.56:39990 remote addr 8e63d000 rkey 109926 nic_wr_credit 8. [D 20:01:04.840353] openib_post_rr: 10.1.4.56:39990 bh 7 wr_id 683cc0. [D 20:01:04.841438] openib_check_cq: completed recv wr_id 683cf0. [D 20:01:04.841481] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd303540 to 10.1.4.56:39990 remote addr 8c63b000 rkey 109a26 nic_wr_credit 9. [D 20:01:04.841494] openib_post_rr: 10.1.4.56:39990 bh 8 wr_id 683cf0. [D 20:01:04.842283] openib_check_cq: completed rdma wr_id 2aaabd3024f0, nic_wr_credit 8. [D 20:01:04.842312] post_sr: 10.1.4.56:39990 bh 40 len 16 wr_id 683020. [D 20:01:04.844509] openib_check_cq: completed rdma wr_id 6ecb00, nic_wr_credit 7. [D 20:01:04.844583] post_sr: 10.1.4.56:39990 bh 41 len 16 wr_id 683050. [D 20:01:04.844616] openib_check_cq: completed send wr_id 682f60. [D 20:01:04.844657] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b470. [D 20:01:04.844675] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.844686] ========================================================= [D 20:01:04.844693] PINT_process_request [D 20:01:04.844701] tiling 11 copies [D 20:01:04.844709] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.844718] lvl 0 el 0 blk 0 by 318767104 [D 20:01:04.844724] to 318767104 ta 301989888 fi 335544320 [D 20:01:04.844733] basic type or contiguous data [D 20:01:04.844740] Request disp [D 20:01:04.844747] contig_offset = 318767104 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.844755] calling distribute [D 20:01:04.844762] PINT_distribute [D 20:01:04.844768] of 318767104 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.844777] begin iteration loff: 318767104 [D 20:01:04.844786] segment exceeds byte limit [D 20:01:04.844793] process a segment [D 20:01:04.844799] of 318767104 sz 2097152 [D 20:01:04.844806] add a segment 0 [D 20:01:04.844813] end iteration [D 20:01:04.844819] done with segments or bytes [D 20:01:04.844826] of 320864256 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.844833] next loff: 320864256 next poff: 320864256 [D 20:01:04.844840] return value 2097152 [D 20:01:04.844847] finished [D 20:01:04.844853] retval = 2097152 [D 20:01:04.844860] exiting distribute returned less than expected [D 20:01:04.844867] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 320864256 fo 335544320 eof 0 [D 20:01:04.844875] ========================================================= [D 20:01:04.844894] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.844903] dbpf_bstream_rw_list: mem_offset: 0x2aaab7231010, mem_size: 2097152 [D 20:01:04.844911] dbpf_bstream_rw_list: stream_offset: 318767104, stream_size: 2097152 [D 20:01:04.844918] dbpf_open_cache_get: called [D 20:01:04.844926] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.844933] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.844961] DBPF I/O ops in progress: 3 [D 20:01:04.844970] lio_listio called with the following aiocbs: [D 20:01:04.844977] aiocb_ptr_array[0]: fd: 13, off: 318767104, bytes: 2097152, buf: 0x2aaab7231010, type: 0 [D 20:01:04.844991] issue_or_delay_io_operation: lio_listio posted 0x2aaabd303d80 (handle 2863311531, ret 0) [D 20:01:04.845717] openib_check_cq: completed recv wr_id 683d20. [D 20:01:04.845751] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd303ee0 to 10.1.4.56:39990 remote addr 9063f000 rkey 109826 nic_wr_credit 8. [D 20:01:04.845763] openib_post_rr: 10.1.4.56:39990 bh 9 wr_id 683d20. [D 20:01:04.846735] openib_check_cq: completed rdma wr_id 6663a0, nic_wr_credit 7. [D 20:01:04.846774] post_sr: 10.1.4.56:39990 bh 42 len 16 wr_id 683080. [D 20:01:04.847397] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304880) [D 20:01:04.847424] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.847436] dbpf_open_cache_put: cached entry. [D 20:01:04.847443] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.847450] op_queue add: 0x2aaabd304880 [D 20:01:04.847477] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.847490] BMI_post_send: addr: 6843072, offset: 0x2aaab7030010, size: 2097152, tag: 33 [D 20:01:04.847503] DBPF I/O ops in progress: 2 [D 20:01:04.847533] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302e90) [D 20:01:04.847543] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.847552] dbpf_open_cache_put: cached entry. [D 20:01:04.847559] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.847565] op_queue add: 0x2aaabd302e90 [D 20:01:04.847572] DBPF I/O ops in progress: 1 [D 20:01:04.847586] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303d80) [D 20:01:04.847595] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.847604] dbpf_open_cache_put: cached entry. [D 20:01:04.847611] dbpf_open_cache_put: move to unused list. [D 20:01:04.847617] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.847624] op_queue add: 0x2aaabd303d80 [D 20:01:04.847631] DBPF I/O ops in progress: 0 [D 20:01:04.848951] openib_check_cq: completed rdma wr_id 6eacb0, nic_wr_credit 6. [D 20:01:04.848963] post_sr: 10.1.4.56:39990 bh 43 len 16 wr_id 6830b0. [D 20:01:04.848971] openib_check_cq: completed send wr_id 682f90. [D 20:01:04.848987] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1507b90. [D 20:01:04.848994] post_sr: 10.1.4.56:39990 bh 44 len 32 wr_id 6830e0. [D 20:01:04.849003] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.849022] ========================================================= [D 20:01:04.849028] PINT_process_request [D 20:01:04.849030] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.849037] tiling 14 copies [D 20:01:04.849051] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.849053] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.849064] BMI_post_send: addr: 6843072, offset: 0x2aaab6e2f010, size: 2097152, tag: 34 [D 20:01:04.849072] lvl 0 el 0 blk 0 by 419430400 [D 20:01:04.849081] post_sr: 10.1.4.56:39990 bh 45 len 32 wr_id 683110. [D 20:01:04.849086] to 419430400 ta 402653184 fi 436207616 [D 20:01:04.849096] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.849100] basic type or contiguous data [D 20:01:04.849108] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.849115] Request disp [D 20:01:04.849125] BMI_post_send: addr: 6843072, offset: 0x2aaab7231010, size: 2097152, tag: 32 [D 20:01:04.849132] contig_offset = 419430400 contig_size = 50331648 lvl_flag = 1 [D 20:01:04.849142] post_sr: 10.1.4.56:39990 bh 46 len 32 wr_id 683140. [D 20:01:04.849149] calling distribute [D 20:01:04.849159] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.849163] PINT_distribute [D 20:01:04.849176] of 419430400 sz 16777216 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.849184] begin iteration loff: 419430400 [D 20:01:04.849191] segment exceeds byte limit [D 20:01:04.849197] process a segment [D 20:01:04.849203] of 419430400 sz 2097152 [D 20:01:04.849209] add a segment 0 [D 20:01:04.849215] end iteration [D 20:01:04.849221] done with segments or bytes [D 20:01:04.849227] of 421527552 sz 14680064 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.849234] next loff: 421527552 next poff: 421527552 [D 20:01:04.849240] return value 2097152 [D 20:01:04.849246] finished [D 20:01:04.849252] retval = 2097152 [D 20:01:04.849258] exiting distribute returned less than expected [D 20:01:04.849264] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 421527552 fo 436207616 eof 0 [D 20:01:04.849271] ========================================================= [D 20:01:04.849281] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.849289] dbpf_bstream_rw_list: mem_offset: 0x2aaab6a2d010, mem_size: 2097152 [D 20:01:04.849296] dbpf_bstream_rw_list: stream_offset: 419430400, stream_size: 2097152 [D 20:01:04.849303] dbpf_open_cache_get: called [D 20:01:04.849310] dbpf_open_cache_get: found bstream entry in unused list. [D 20:01:04.849317] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.849330] DBPF I/O ops in progress: 1 [D 20:01:04.849337] lio_listio called with the following aiocbs: [D 20:01:04.849343] aiocb_ptr_array[0]: fd: 13, off: 419430400, bytes: 2097152, buf: 0x2aaab6a2d010, type: 0 [D 20:01:04.849355] issue_or_delay_io_operation: lio_listio posted 0x2aaabd303370 (handle 2863311531, ret 0) [D 20:01:04.850177] openib_check_cq: completed recv wr_id 683d50. [D 20:01:04.850213] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31d890 to 10.1.4.56:39990 remote addr 8a639000 rkey 109b26 nic_wr_credit 7. [D 20:01:04.850225] openib_post_rr: 10.1.4.56:39990 bh 10 wr_id 683d50. [D 20:01:04.851182] openib_check_cq: completed rdma wr_id 66af30, nic_wr_credit 6. [D 20:01:04.851223] post_sr: 10.1.4.56:39990 bh 47 len 16 wr_id 683170. [D 20:01:04.851666] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303370) [D 20:01:04.851688] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.851698] dbpf_open_cache_put: cached entry. [D 20:01:04.851705] dbpf_open_cache_put: move to unused list. [D 20:01:04.851712] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.851719] op_queue add: 0x2aaabd303370 [D 20:01:04.851742] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.851753] BMI_post_send: addr: 6843072, offset: 0x2aaab6a2d010, size: 2097152, tag: 35 [D 20:01:04.851764] DBPF I/O ops in progress: 0 [D 20:01:04.851805] post_sr: 10.1.4.56:39990 bh 48 len 32 wr_id 6831a0. [D 20:01:04.851829] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.853400] openib_check_cq: completed rdma wr_id 6ebff0, nic_wr_credit 5. [D 20:01:04.853411] post_sr: 10.1.4.56:39990 bh 49 len 16 wr_id 6831d0. [D 20:01:04.853418] openib_check_cq: completed send wr_id 682fc0. [D 20:01:04.853431] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab15d5af0. [D 20:01:04.853441] (send callback time) ini posts: 0, pending: 4, last: 1, src_list emtpy: yes [D 20:01:04.854629] openib_check_cq: completed recv wr_id 683d80. [D 20:01:04.854641] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31e390 to 10.1.4.56:39990 remote addr 86635000 rkey 109d26 nic_wr_credit 6. [D 20:01:04.854650] openib_post_rr: 10.1.4.56:39990 bh 11 wr_id 683d80. [D 20:01:04.855625] openib_check_cq: completed rdma wr_id 2aaabd301840, nic_wr_credit 5. [D 20:01:04.855633] post_sr: 10.1.4.56:39990 bh 50 len 16 wr_id 683200. [D 20:01:04.855640] openib_check_cq: completed send wr_id 682ff0. [D 20:01:04.855651] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152db60. [D 20:01:04.855660] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.855667] ========================================================= [D 20:01:04.855673] PINT_process_request [D 20:01:04.855680] tiling 9 copies [D 20:01:04.855687] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.855694] lvl 0 el 0 blk 0 by 253755392 [D 20:01:04.855701] to 253755392 ta 234881024 fi 268435456 [D 20:01:04.855708] basic type or contiguous data [D 20:01:04.855714] Request disp [D 20:01:04.855720] contig_offset = 253755392 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.855727] calling distribute [D 20:01:04.855733] PINT_distribute [D 20:01:04.855739] of 253755392 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.855747] begin iteration loff: 253755392 [D 20:01:04.855753] segment exceeds byte limit [D 20:01:04.855760] process a segment [D 20:01:04.855767] of 253755392 sz 2097152 [D 20:01:04.855773] add a segment 0 [D 20:01:04.855780] end iteration [D 20:01:04.855788] done with segments or bytes [D 20:01:04.855795] of 255852544 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.855801] next loff: 255852544 next poff: 255852544 [D 20:01:04.855808] return value 2097152 [D 20:01:04.855814] finished [D 20:01:04.855820] retval = 2097152 [D 20:01:04.855826] exiting distribute returned less than expected [D 20:01:04.855832] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 255852544 fo 268435456 eof 0 [D 20:01:04.855839] ========================================================= [D 20:01:04.855849] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.855855] dbpf_bstream_rw_list: mem_offset: 0x2aaab4c1c010, mem_size: 2097152 [D 20:01:04.855862] dbpf_bstream_rw_list: stream_offset: 253755392, stream_size: 2097152 [D 20:01:04.855869] dbpf_open_cache_get: called [D 20:01:04.855875] dbpf_open_cache_get: found bstream entry in unused list. [D 20:01:04.855882] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.855894] DBPF I/O ops in progress: 1 [D 20:01:04.855901] lio_listio called with the following aiocbs: [D 20:01:04.855907] aiocb_ptr_array[0]: fd: 13, off: 253755392, bytes: 2097152, buf: 0x2aaab4c1c010, type: 0 [D 20:01:04.855920] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304880 (handle 2863311531, ret 0) [D 20:01:04.856837] openib_check_cq: completed recv wr_id 683db0. [D 20:01:04.856889] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd307790 to 10.1.4.56:39990 remote addr 94843000 rkey 109526 nic_wr_credit 6. [D 20:01:04.856902] openib_post_rr: 10.1.4.56:39990 bh 12 wr_id 683db0. [D 20:01:04.857853] openib_check_cq: completed rdma wr_id 2aaabd301420, nic_wr_credit 5. [D 20:01:04.857895] post_sr: 10.1.4.56:39990 bh 51 len 16 wr_id 683230. [D 20:01:04.858228] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304880) [D 20:01:04.858251] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.858260] dbpf_open_cache_put: cached entry. [D 20:01:04.858267] dbpf_open_cache_put: move to unused list. [D 20:01:04.858273] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.858280] op_queue add: 0x2aaabd304880 [D 20:01:04.858304] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.858315] BMI_post_send: addr: 6843072, offset: 0x2aaab4c1c010, size: 2097152, tag: 30 [D 20:01:04.858326] DBPF I/O ops in progress: 0 [D 20:01:04.860069] openib_check_cq: completed rdma wr_id 2aaabd303540, nic_wr_credit 4. [D 20:01:04.860081] post_sr: 10.1.4.56:39990 bh 52 len 16 wr_id 683260. [D 20:01:04.860088] openib_check_cq: completed send wr_id 683020. [D 20:01:04.860096] openib_check_cq: completed send wr_id 683050. [D 20:01:04.860111] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:04.860116] post_sr: 10.1.4.56:39990 bh 53 len 32 wr_id 683290. [D 20:01:04.860122] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.860140] ========================================================= [D 20:01:04.860146] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.860146] PINT_process_request [D 20:01:04.860165] tiling 5 copies [D 20:01:04.860172] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.860179] lvl 0 el 0 blk 0 by 132120576 [D 20:01:04.860185] to 132120576 ta 100663296 fi 134217728 [D 20:01:04.860192] basic type or contiguous data [D 20:01:04.860198] Request disp [D 20:01:04.860204] contig_offset = 132120576 contig_size = 35651584 lvl_flag = 1 [D 20:01:04.860211] calling distribute [D 20:01:04.860217] PINT_distribute [D 20:01:04.860223] of 132120576 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 442499072 exfl 0 [D 20:01:04.860231] begin iteration loff: 132120576 [D 20:01:04.860238] process a segment [D 20:01:04.860245] of 132120576 sz 2097152 [D 20:01:04.860252] add a segment 0 [D 20:01:04.860258] end iteration [D 20:01:04.860264] done with segments or bytes [D 20:01:04.860270] of 134217728 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.860276] next loff: 134217728 next poff: 134217728 [D 20:01:04.860283] (return value) 2097152 [D 20:01:04.860289] finished [D 20:01:04.860295] retval = 2097152 [D 20:01:04.860301] exiting distribute returned less than expected [D 20:01:04.860307] done sg 1 sm 64 by 2097152 bm 2097152 ta 100663296 to 134217728 fo 134217728 eof 0 [D 20:01:04.860314] ========================================================= [D 20:01:04.860324] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.860331] dbpf_bstream_rw_list: mem_offset: 0x2aaac081a010, mem_size: 2097152 [D 20:01:04.860338] dbpf_bstream_rw_list: stream_offset: 132120576, stream_size: 2097152 [D 20:01:04.860345] dbpf_open_cache_get: called [D 20:01:04.860351] dbpf_open_cache_get: found bstream entry in unused list. [D 20:01:04.860357] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.860370] DBPF I/O ops in progress: 1 [D 20:01:04.860376] lio_listio called with the following aiocbs: [D 20:01:04.860382] aiocb_ptr_array[0]: fd: 13, off: 132120576, bytes: 2097152, buf: 0x2aaac081a010, type: 0 [D 20:01:04.860395] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3024f0 (handle 2863311531, ret 0) [D 20:01:04.860404] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1505890. [D 20:01:04.860412] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.860420] ========================================================= [D 20:01:04.860426] PINT_process_request [D 20:01:04.860432] tiling 10 copies [D 20:01:04.860438] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.860445] lvl 0 el 0 blk 0 by 287309824 [D 20:01:04.860452] to 287309824 ta 268435456 fi 301989888 [D 20:01:04.860459] basic type or contiguous data [D 20:01:04.860466] Request disp [D 20:01:04.860472] contig_offset = 287309824 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.860479] calling distribute [D 20:01:04.860485] PINT_distribute [D 20:01:04.860491] of 287309824 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.860498] begin iteration loff: 287309824 [D 20:01:04.860505] segment exceeds byte limit [D 20:01:04.860511] process a segment [D 20:01:04.860517] of 287309824 sz 2097152 [D 20:01:04.860523] add a segment 0 [D 20:01:04.860530] end iteration [D 20:01:04.860535] done with segments or bytes [D 20:01:04.860541] of 289406976 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.860549] next loff: 289406976 next poff: 289406976 [D 20:01:04.860555] return value 2097152 [D 20:01:04.860562] finished [D 20:01:04.860568] retval = 2097152 [D 20:01:04.860574] exiting distribute returned less than expected [D 20:01:04.860580] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 289406976 fo 301989888 eof 0 [D 20:01:04.860595] ========================================================= [D 20:01:04.860603] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.860610] dbpf_bstream_rw_list: mem_offset: 0x2aaab8e3f010, mem_size: 2097152 [D 20:01:04.860616] dbpf_bstream_rw_list: stream_offset: 287309824, stream_size: 2097152 [D 20:01:04.860623] dbpf_open_cache_get: called [D 20:01:04.860629] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.860636] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.860657] DBPF I/O ops in progress: 2 [D 20:01:04.860665] lio_listio called with the following aiocbs: [D 20:01:04.860670] aiocb_ptr_array[0]: fd: 13, off: 287309824, bytes: 2097152, buf: 0x2aaab8e3f010, type: 0 [D 20:01:04.860679] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304880 (handle 2863311531, ret 0) [D 20:01:04.861345] openib_check_cq: completed recv wr_id 683de0. [D 20:01:04.861368] openib_post_sr_rdmaw: ibv_post_send wr_id 66aea0 to 10.1.4.56:39990 remote addr 9d44b000 rkey 108f27 nic_wr_credit 5. [D 20:01:04.861384] openib_post_rr: 10.1.4.56:39990 bh 13 wr_id 683de0. [D 20:01:04.862299] openib_check_cq: completed rdma wr_id 2aaabd303ee0, nic_wr_credit 4. [D 20:01:04.862328] post_sr: 10.1.4.56:39990 bh 54 len 16 wr_id 6832c0. [D 20:01:04.862338] openib_check_cq: completed send wr_id 683080. [D 20:01:04.862346] openib_check_cq: completed send wr_id 6830b0. [D 20:01:04.862354] openib_check_cq: completed send wr_id 6830e0. [D 20:01:04.862362] openib_check_cq: completed send wr_id 683110. [D 20:01:04.862369] openib_check_cq: completed send wr_id 683140. [D 20:01:04.862399] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1518130. [D 20:01:04.862413] (send callback time) ini posts: 0, pending: 0, last: 1, src_list emtpy: yes [D 20:01:04.862421] flowproto completing 0x2aaab1518130 [D 20:01:04.862446] Job flows in progress (callback time): 12 [D 20:01:04.862506] [SM Entering]: (0x660710) io_sm:send_completion_ack (status: 0) [D 20:01:04.862528] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.862538] [SM Exiting]: (0x660710) io_sm:send_completion_ack (error code: 0), (action: COMPLETE) [D 20:01:04.862547] [SM Entering]: (0x660710) io_sm:release (status: 0) [D 20:01:04.862554] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.862567] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x2aaab1501060 [D 20:01:04.862579] [SM Exiting]: (0x660710) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.862586] [SM Entering]: (0x660710) io_sm:cleanup (status: 0) [D 20:01:04.862593] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.862958] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:04.862978] (send callback time) ini posts: 0, pending: 1, last: 1, src_list emtpy: yes [D 20:01:04.862992] openib_check_cq: completed recv wr_id 683e10. [D 20:01:04.863009] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd302650 to 10.1.4.56:39990 remote addr 92841000 rkey 109626 nic_wr_credit 5. [D 20:01:04.863020] openib_post_rr: 10.1.4.56:39990 bh 14 wr_id 683e10. [D 20:01:04.864056] openib_check_cq: completed recv wr_id 683e40. [D 20:01:04.864112] openib_post_sr_rdmaw: ibv_post_send wr_id 6c79e0 to 10.1.4.56:39990 remote addr a024d000 rkey 108e27 nic_wr_credit 6. [D 20:01:04.864125] openib_post_rr: 10.1.4.56:39990 bh 15 wr_id 683e40. [D 20:01:04.864533] openib_check_cq: completed rdma wr_id 2aaabd31d890, nic_wr_credit 5. [D 20:01:04.864551] post_sr: 10.1.4.56:39990 bh 55 len 16 wr_id 6832f0. [D 20:01:04.864561] openib_check_cq: completed send wr_id 683170. [D 20:01:04.864569] openib_check_cq: completed send wr_id 6831a0. [D 20:01:04.864576] openib_check_cq: completed send wr_id 6831d0. [D 20:01:04.864605] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:04.864622] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.864633] ========================================================= [D 20:01:04.864640] PINT_process_request [D 20:01:04.864649] tiling 6 copies [D 20:01:04.864657] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.864665] lvl 0 el 0 blk 0 by 157286400 [D 20:01:04.864672] to 157286400 ta 134217728 fi 167772160 [D 20:01:04.864679] basic type or contiguous data [D 20:01:04.864686] Request disp [D 20:01:04.864693] contig_offset = 157286400 contig_size = 44040192 lvl_flag = 1 [D 20:01:04.864702] calling distribute [D 20:01:04.864708] PINT_distribute [D 20:01:04.864715] of 157286400 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.864724] begin iteration loff: 157286400 [D 20:01:04.864731] segment exceeds byte limit [D 20:01:04.864739] process a segment [D 20:01:04.864745] of 157286400 sz 2097152 [D 20:01:04.864752] add a segment 0 [D 20:01:04.864759] end iteration [D 20:01:04.864772] done with segments or bytes [D 20:01:04.864779] of 159383552 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.864786] next loff: 159383552 next poff: 159383552 [D 20:01:04.864793] return value 2097152 [D 20:01:04.864800] finished [D 20:01:04.864806] retval = 2097152 [D 20:01:04.864813] exiting distribute returned less than expected [D 20:01:04.864820] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 159383552 fo 167772160 eof 0 [D 20:01:04.864827] ========================================================= [D 20:01:04.864849] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.864858] dbpf_bstream_rw_list: mem_offset: 0x2aaab5e26010, mem_size: 2097152 [D 20:01:04.864865] dbpf_bstream_rw_list: stream_offset: 157286400, stream_size: 2097152 [D 20:01:04.864873] dbpf_open_cache_get: called [D 20:01:04.864882] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.864889] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.864919] DBPF I/O ops in progress: 3 [D 20:01:04.864926] lio_listio called with the following aiocbs: [D 20:01:04.864934] aiocb_ptr_array[0]: fd: 13, off: 157286400, bytes: 2097152, buf: 0x2aaab5e26010, type: 0 [D 20:01:04.864949] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3049d0 (handle 2863311531, ret 0) [D 20:01:04.864960] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c750. [D 20:01:04.864969] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.864977] ========================================================= [D 20:01:04.864983] PINT_process_request [D 20:01:04.864990] tiling 7 copies [D 20:01:04.864996] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.865003] lvl 0 el 0 blk 0 by 188743680 [D 20:01:04.865010] to 188743680 ta 167772160 fi 201326592 [D 20:01:04.865017] basic type or contiguous data [D 20:01:04.865024] Request disp [D 20:01:04.865030] contig_offset = 188743680 contig_size = 46137344 lvl_flag = 1 [D 20:01:04.865037] calling distribute [D 20:01:04.865043] PINT_distribute [D 20:01:04.865049] of 188743680 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.865056] begin iteration loff: 188743680 [D 20:01:04.865063] segment exceeds byte limit [D 20:01:04.865069] process a segment [D 20:01:04.865075] of 188743680 sz 2097152 [D 20:01:04.865082] add a segment 0 [D 20:01:04.865088] end iteration [D 20:01:04.865094] done with segments or bytes [D 20:01:04.865100] of 190840832 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.865107] next loff: 190840832 next poff: 190840832 [D 20:01:04.865113] return value 2097152 [D 20:01:04.865120] finished [D 20:01:04.865126] retval = 2097152 [D 20:01:04.865132] exiting distribute returned less than expected [D 20:01:04.865138] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 190840832 fo 201326592 eof 0 [D 20:01:04.865146] ========================================================= [D 20:01:04.865154] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.865161] dbpf_bstream_rw_list: mem_offset: 0x2aaab0add010, mem_size: 2097152 [D 20:01:04.865168] dbpf_bstream_rw_list: stream_offset: 188743680, stream_size: 2097152 [D 20:01:04.865175] dbpf_open_cache_get: called [D 20:01:04.865181] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.865188] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.865210] DBPF I/O ops in progress: 4 [D 20:01:04.865217] lio_listio called with the following aiocbs: [D 20:01:04.865223] aiocb_ptr_array[0]: fd: 13, off: 188743680, bytes: 2097152, buf: 0x2aaab0add010, type: 0 [D 20:01:04.865232] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304b20 (handle 2863311531, ret 0) [D 20:01:04.866770] openib_check_cq: completed rdma wr_id 2aaabd31e390, nic_wr_credit 4. [D 20:01:04.866821] post_sr: 10.1.4.56:39990 bh 56 len 16 wr_id 683320. [D 20:01:04.866835] openib_check_cq: completed send wr_id 683200. [D 20:01:04.866853] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517400. [D 20:01:04.866865] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.866881] ========================================================= [D 20:01:04.866889] PINT_process_request [D 20:01:04.866896] tiling 8 copies [D 20:01:04.866904] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.866913] lvl 0 el 0 blk 0 by 222298112 [D 20:01:04.866919] to 222298112 ta 201326592 fi 234881024 [D 20:01:04.866926] basic type or contiguous data [D 20:01:04.866934] Request disp [D 20:01:04.866940] contig_offset = 222298112 contig_size = 46137344 lvl_flag = 1 [D 20:01:04.866948] calling distribute [D 20:01:04.866955] PINT_distribute [D 20:01:04.866962] of 222298112 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.866971] begin iteration loff: 222298112 [D 20:01:04.866978] segment exceeds byte limit [D 20:01:04.866985] process a segment [D 20:01:04.866991] of 222298112 sz 2097152 [D 20:01:04.866998] add a segment 0 [D 20:01:04.867005] end iteration [D 20:01:04.867012] done with segments or bytes [D 20:01:04.867018] of 224395264 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.867025] next loff: 224395264 next poff: 224395264 [D 20:01:04.867033] return value 2097152 [D 20:01:04.867040] finished [D 20:01:04.867046] retval = 2097152 [D 20:01:04.867052] exiting distribute returned less than expected [D 20:01:04.867058] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 224395264 fo 234881024 eof 0 [D 20:01:04.867066] ========================================================= [D 20:01:04.867083] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.867091] dbpf_bstream_rw_list: mem_offset: 0x2aaab2c0b010, mem_size: 2097152 [D 20:01:04.867097] dbpf_bstream_rw_list: stream_offset: 222298112, stream_size: 2097152 [D 20:01:04.867105] dbpf_open_cache_get: called [D 20:01:04.867112] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.867119] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.867145] DBPF I/O ops in progress: 5 [D 20:01:04.867153] lio_listio called with the following aiocbs: [D 20:01:04.867160] aiocb_ptr_array[0]: fd: 13, off: 222298112, bytes: 2097152, buf: 0x2aaab2c0b010, type: 0 [D 20:01:04.867175] issue_or_delay_io_operation: lio_listio posted 0x2aaabd301840 (handle 2863311531, ret 0) [D 20:01:04.868611] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3024f0) [D 20:01:04.868649] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.868660] dbpf_open_cache_put: cached entry. [D 20:01:04.868667] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.868674] op_queue add: 0x2aaabd3024f0 [D 20:01:04.868701] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1522750. [D 20:01:04.868712] BMI_post_send: addr: 6843072, offset: 0x2aaac081a010, size: 2097152, tag: 26 [D 20:01:04.868729] post_sr: 10.1.4.56:39990 bh 57 len 32 wr_id 683350. [D 20:01:04.868760] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 1 [D 20:01:04.868776] DBPF I/O ops in progress: 4 [D 20:01:04.868812] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304880) [D 20:01:04.868823] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.868833] dbpf_open_cache_put: cached entry. [D 20:01:04.868839] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.868845] op_queue add: 0x2aaabd304880 [D 20:01:04.868860] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.868869] BMI_post_send: addr: 6843072, offset: 0x2aaab8e3f010, size: 2097152, tag: 31 [D 20:01:04.868879] DBPF I/O ops in progress: 3 [D 20:01:04.868895] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3049d0) [D 20:01:04.868899] post_sr: 10.1.4.56:39990 bh 58 len 32 wr_id 683380. [D 20:01:04.868923] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [A 20:01:04.869143] troy.scl@10.1.4.56:39990 H=2863311531 S=0x660810: io: finish (Success) [D 20:01:04.869158] PINT_encode_release [D 20:01:04.869167] lebf_encode_rel [D 20:01:04.869177] [SM frame get]: (0x660710) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.869185] server_state_machine_complete 0x660710 [D 20:01:04.869193] PINT_decode_release [D 20:01:04.869200] lebf_decode_rel [D 20:01:04.869209] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.869217] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.869224] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 12 [D 20:01:04.869235] [SM Exiting]: (0x660710) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.869247] server_state_machine_terminate 0x660710 [D 20:01:04.868983] openib_check_cq: completed rdma wr_id 2aaabd307790, nic_wr_credit 3. [D 20:01:04.869272] post_sr: 10.1.4.56:39990 bh 59 len 16 wr_id 6833b0. [D 20:01:04.869283] openib_check_cq: completed send wr_id 683230. [D 20:01:04.869291] openib_check_cq: completed send wr_id 683260. [D 20:01:04.869298] openib_check_cq: completed send wr_id 683290. [D 20:01:04.869313] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c210. [D 20:01:04.869323] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.869332] ========================================================= [D 20:01:04.869339] PINT_process_request [D 20:01:04.869347] tiling 12 copies [D 20:01:04.869355] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.869364] lvl 0 el 0 blk 0 by 354418688 [D 20:01:04.869370] to 354418688 ta 335544320 fi 369098752 [D 20:01:04.869378] basic type or contiguous data [D 20:01:04.869385] Request disp [D 20:01:04.869392] contig_offset = 354418688 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.869400] calling distribute [D 20:01:04.869406] PINT_distribute [D 20:01:04.869413] of 354418688 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.869424] begin iteration loff: 354418688 [D 20:01:04.869431] segment exceeds byte limit [D 20:01:04.869438] process a segment [D 20:01:04.869444] of 354418688 sz 2097152 [D 20:01:04.869451] add a segment 0 [D 20:01:04.869458] end iteration [D 20:01:04.869464] done with segments or bytes [D 20:01:04.869470] of 356515840 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.869478] next loff: 356515840 next poff: 356515840 [D 20:01:04.869485] return value 2097152 [D 20:01:04.869492] finished [D 20:01:04.869498] retval = 2097152 [D 20:01:04.869505] exiting distribute returned less than expected [D 20:01:04.869512] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 356515840 fo 369098752 eof 0 [D 20:01:04.869519] ========================================================= [D 20:01:04.869532] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.869541] dbpf_bstream_rw_list: mem_offset: 0x2aaab501e010, mem_size: 2097152 [D 20:01:04.869547] dbpf_bstream_rw_list: stream_offset: 354418688, stream_size: 2097152 [D 20:01:04.869556] dbpf_open_cache_get: called [D 20:01:04.869563] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.869570] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.869594] DBPF I/O ops in progress: 4 [D 20:01:04.869601] lio_listio called with the following aiocbs: [D 20:01:04.869607] aiocb_ptr_array[0]: fd: 13, off: 354418688, bytes: 2097152, buf: 0x2aaab501e010, type: 0 [D 20:01:04.869622] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3013f0 (handle 2863311531, ret 0) [D 20:01:04.869632] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1506e50. [D 20:01:04.869640] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.869650] ========================================================= [D 20:01:04.869657] PINT_process_request [D 20:01:04.869663] tiling 13 copies [D 20:01:04.869670] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.869677] lvl 0 el 0 blk 0 by 387973120 [D 20:01:04.869684] to 387973120 ta 369098752 fi 402653184 [D 20:01:04.869690] basic type or contiguous data [D 20:01:04.869696] Request disp [D 20:01:04.869702] contig_offset = 387973120 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.869709] calling distribute [D 20:01:04.869715] PINT_distribute [D 20:01:04.869721] of 387973120 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.869729] begin iteration loff: 387973120 [D 20:01:04.869735] segment exceeds byte limit [D 20:01:04.869741] process a segment [D 20:01:04.869747] of 387973120 sz 2097152 [D 20:01:04.869753] add a segment 0 [D 20:01:04.869759] end iteration [D 20:01:04.869764] done with segments or bytes [D 20:01:04.869771] of 390070272 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.869778] next loff: 390070272 next poff: 390070272 [D 20:01:04.869784] return value 2097152 [D 20:01:04.869790] finished [D 20:01:04.869796] retval = 2097152 [D 20:01:04.869802] exiting distribute returned less than expected [D 20:01:04.869808] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 390070272 fo 402653184 eof 0 [D 20:01:04.869816] ========================================================= [D 20:01:04.869824] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.869831] dbpf_bstream_rw_list: mem_offset: 0x2aaab4e1d010, mem_size: 2097152 [D 20:01:04.869837] dbpf_bstream_rw_list: stream_offset: 387973120, stream_size: 2097152 [D 20:01:04.869844] dbpf_open_cache_get: called [D 20:01:04.869850] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.869856] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.869873] DBPF I/O ops in progress: 5 [D 20:01:04.869881] lio_listio called with the following aiocbs: [D 20:01:04.869887] aiocb_ptr_array[0]: fd: 13, off: 387973120, bytes: 2097152, buf: 0x2aaab4e1d010, type: 0 [D 20:01:04.869896] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304c70 (handle 2863311531, ret 0) [D 20:01:04.871218] openib_check_cq: completed rdma wr_id 66aea0, nic_wr_credit 2. [D 20:01:04.871277] post_sr: 10.1.4.56:39990 bh 60 len 16 wr_id 6833e0. [D 20:01:04.871292] openib_check_cq: completed send wr_id 6832c0. [D 20:01:04.871330] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b470. [D 20:01:04.871346] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.871355] ========================================================= [D 20:01:04.871363] PINT_process_request [D 20:01:04.871371] tiling 11 copies [D 20:01:04.871378] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.871387] lvl 0 el 0 blk 0 by 320864256 [D 20:01:04.871393] to 320864256 ta 301989888 fi 335544320 [D 20:01:04.871401] basic type or contiguous data [D 20:01:04.871407] Request disp [D 20:01:04.871413] contig_offset = 320864256 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.871422] calling distribute [D 20:01:04.871429] PINT_distribute [D 20:01:04.871436] of 320864256 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.871445] begin iteration loff: 320864256 [D 20:01:04.871453] segment exceeds byte limit [D 20:01:04.871459] process a segment [D 20:01:04.871465] of 320864256 sz 2097152 [D 20:01:04.871472] add a segment 0 [D 20:01:04.871478] end iteration [D 20:01:04.871485] done with segments or bytes [D 20:01:04.871491] of 322961408 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.871498] next loff: 322961408 next poff: 322961408 [D 20:01:04.871505] return value 2097152 [D 20:01:04.871512] finished [D 20:01:04.871518] retval = 2097152 [D 20:01:04.871531] exiting distribute returned less than expected [D 20:01:04.871538] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 322961408 fo 335544320 eof 0 [D 20:01:04.871547] ========================================================= [D 20:01:04.871565] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.871574] dbpf_bstream_rw_list: mem_offset: 0x2aaab5420010, mem_size: 2097152 [D 20:01:04.871581] dbpf_bstream_rw_list: stream_offset: 320864256, stream_size: 2097152 [D 20:01:04.871589] dbpf_open_cache_get: called [D 20:01:04.871597] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.871603] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.871629] DBPF I/O ops in progress: 6 [D 20:01:04.871638] lio_listio called with the following aiocbs: [D 20:01:04.871646] aiocb_ptr_array[0]: fd: 13, off: 320864256, bytes: 2097152, buf: 0x2aaab5420010, type: 0 [D 20:01:04.871660] issue_or_delay_io_operation: lio_listio posted 0x2aaabd303ee0 (handle 2863311531, ret 0) [D 20:01:04.873440] openib_check_cq: completed rdma wr_id 2aaabd302650, nic_wr_credit 1. [D 20:01:04.873511] post_sr: 10.1.4.56:39990 bh 61 len 16 wr_id 683410. [D 20:01:04.875658] openib_check_cq: completed rdma wr_id 6c79e0, nic_wr_credit 0. [D 20:01:04.875714] post_sr: 10.1.4.56:39990 bh 62 len 16 wr_id 683440. [D 20:01:04.875728] openib_check_cq: completed send wr_id 6832f0. [D 20:01:04.875737] openib_check_cq: completed send wr_id 683320. [D 20:01:04.875746] openib_check_cq: completed send wr_id 683350. [D 20:01:04.875752] openib_check_cq: completed send wr_id 683380. [D 20:01:04.875760] openib_check_cq: completed send wr_id 6833b0. [D 20:01:04.875767] openib_check_cq: completed send wr_id 6833e0. [D 20:01:04.875775] openib_check_cq: completed send wr_id 683410. [D 20:01:04.875793] openib_check_cq: completed send wr_id 683440. [D 20:01:04.875837] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:04.875853] (send callback time) ini posts: 0, pending: 7, last: 1, src_list emtpy: yes [D 20:01:04.875863] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b630. [D 20:01:04.875871] (send callback time) ini posts: 0, pending: 0, last: 1, src_list emtpy: yes [D 20:01:04.875878] flowproto completing 0x2aaab152b630 [D 20:01:04.875902] Job flows in progress (callback time): 11 [D 20:01:04.875955] [SM Entering]: (0x6619d0) io_sm:send_completion_ack (status: 0) [D 20:01:04.875970] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.875980] [SM Exiting]: (0x6619d0) io_sm:send_completion_ack (error code: 0), (action: COMPLETE) [D 20:01:04.875988] [SM Entering]: (0x6619d0) io_sm:release (status: 0) [D 20:01:04.875995] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.876009] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x2aaab1501c60 [D 20:01:04.876019] [SM Exiting]: (0x6619d0) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:04.876027] [SM Entering]: (0x6619d0) io_sm:cleanup (status: 0) [D 20:01:04.876034] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.876365] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152db60. [D 20:01:04.876382] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.876392] ========================================================= [D 20:01:04.876399] PINT_process_request [D 20:01:04.876406] tiling 9 copies [D 20:01:04.876414] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.876421] lvl 0 el 0 blk 0 by 255852544 [D 20:01:04.876428] to 255852544 ta 234881024 fi 268435456 [D 20:01:04.876435] basic type or contiguous data [D 20:01:04.876441] Request disp [D 20:01:04.876448] contig_offset = 255852544 contig_size = 46137344 lvl_flag = 1 [D 20:01:04.876456] calling distribute [D 20:01:04.876462] PINT_distribute [D 20:01:04.876469] of 255852544 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:04.876482] begin iteration loff: 255852544 [D 20:01:04.876507] segment exceeds byte limit [D 20:01:04.876515] process a segment [D 20:01:04.876521] of 255852544 sz 2097152 [D 20:01:04.876528] add a segment 0 [D 20:01:04.876534] end iteration [D 20:01:04.876540] done with segments or bytes [D 20:01:04.876547] of 257949696 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.876554] next loff: 257949696 next poff: 257949696 [D 20:01:04.876561] return value 2097152 [D 20:01:04.876568] finished [D 20:01:04.876574] retval = 2097152 [D 20:01:04.876580] exiting distribute returned less than expected [D 20:01:04.876587] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 257949696 fo 268435456 eof 0 [D 20:01:04.876604] ========================================================= [D 20:01:04.876624] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.876633] dbpf_bstream_rw_list: mem_offset: 0x2aaab300e010, mem_size: 2097152 [D 20:01:04.876640] dbpf_bstream_rw_list: stream_offset: 255852544, stream_size: 2097152 [D 20:01:04.876648] dbpf_open_cache_get: called [D 20:01:04.876656] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.876663] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.876687] DBPF I/O ops in progress: 7 [D 20:01:04.876694] lio_listio called with the following aiocbs: [D 20:01:04.876701] aiocb_ptr_array[0]: fd: 13, off: 255852544, bytes: 2097152, buf: 0x2aaab300e010, type: 0 [D 20:01:04.876716] issue_or_delay_io_operation: lio_listio posted 0x2aaabd302630 (handle 2863311531, ret 0) [D 20:01:04.876725] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1505890. [D 20:01:04.876733] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.876740] ========================================================= [D 20:01:04.876747] PINT_process_request [D 20:01:04.876753] tiling 10 copies [D 20:01:04.876759] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.876767] lvl 0 el 0 blk 0 by 289406976 [D 20:01:04.876773] to 289406976 ta 268435456 fi 301989888 [D 20:01:04.876780] basic type or contiguous data [D 20:01:04.876787] Request disp [D 20:01:04.876793] contig_offset = 289406976 contig_size = 46137344 lvl_flag = 1 [D 20:01:04.876800] calling distribute [D 20:01:04.876806] PINT_distribute [D 20:01:04.876811] of 289406976 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:04.876819] begin iteration loff: 289406976 [D 20:01:04.876825] segment exceeds byte limit [D 20:01:04.876831] process a segment [D 20:01:04.876838] of 289406976 sz 2097152 [D 20:01:04.876844] add a segment 0 [D 20:01:04.876850] end iteration [D 20:01:04.876856] done with segments or bytes [D 20:01:04.876862] of 291504128 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.876869] next loff: 291504128 next poff: 291504128 [D 20:01:04.876875] return value 2097152 [D 20:01:04.876882] finished [D 20:01:04.876887] retval = 2097152 [D 20:01:04.876893] exiting distribute returned less than expected [D 20:01:04.876900] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 291504128 fo 301989888 eof 0 [D 20:01:04.876907] ========================================================= [D 20:01:04.876915] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.876922] dbpf_bstream_rw_list: mem_offset: 0x2aaab5c25010, mem_size: 2097152 [D 20:01:04.876929] dbpf_bstream_rw_list: stream_offset: 289406976, stream_size: 2097152 [D 20:01:04.876935] dbpf_open_cache_get: called [D 20:01:04.876941] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.876948] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.876961] DBPF I/O ops in progress: 8 [D 20:01:04.876969] lio_listio called with the following aiocbs: [D 20:01:04.876975] aiocb_ptr_array[0]: fd: 13, off: 289406976, bytes: 2097152, buf: 0x2aaab5c25010, type: 0 [D 20:01:04.868907] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.877002] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3013f0) [D 20:01:04.877030] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304b20) [D 20:01:04.877061] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31d890 (handle 2863311531, ret 0) [D 20:01:04.877072] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1507b90. [D 20:01:04.877080] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:04.877087] ========================================================= [D 20:01:04.877094] PINT_process_request [D 20:01:04.877100] tiling 14 copies [D 20:01:04.877106] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:04.877113] lvl 0 el 0 blk 0 by 421527552 [D 20:01:04.877120] to 421527552 ta 402653184 fi 436207616 [D 20:01:04.877127] basic type or contiguous data [D 20:01:04.877133] Request disp [D 20:01:04.877139] contig_offset = 421527552 contig_size = 48234496 lvl_flag = 1 [D 20:01:04.877146] calling distribute [D 20:01:04.877152] PINT_distribute [D 20:01:04.877159] of 421527552 sz 14680064 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:04.877167] begin iteration loff: 421527552 [D 20:01:04.877173] segment exceeds byte limit [D 20:01:04.877180] process a segment [D 20:01:04.877186] of 421527552 sz 2097152 [D 20:01:04.877192] add a segment 0 [D 20:01:04.877198] end iteration [D 20:01:04.877203] done with segments or bytes [D 20:01:04.877210] of 423624704 sz 12582912 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:04.877217] next loff: 423624704 next poff: 423624704 [D 20:01:04.877223] return value 2097152 [D 20:01:04.877230] finished [D 20:01:04.877236] retval = 2097152 [D 20:01:04.877242] exiting distribute returned less than expected [D 20:01:04.877248] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 423624704 fo 436207616 eof 0 [D 20:01:04.877256] ========================================================= [D 20:01:04.877264] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:04.877270] dbpf_bstream_rw_list: mem_offset: 0x2aaab4a1b010, mem_size: 2097152 [D 20:01:04.877277] dbpf_bstream_rw_list: stream_offset: 421527552, stream_size: 2097152 [D 20:01:04.877284] dbpf_open_cache_get: called [D 20:01:04.877034] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.877024] dbpf_open_cache_put: cached entry. [D 20:01:04.877306] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:04.877313] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:04.877334] DBPF I/O ops in progress: 9 [D 20:01:04.877342] lio_listio called with the following aiocbs: [D 20:01:04.877348] aiocb_ptr_array[0]: fd: 13, off: 421527552, bytes: 2097152, buf: 0x2aaab4a1b010, type: 0 [D 20:01:04.877357] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31cad0 (handle 2863311531, ret 0) [D 20:01:04.877399] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab15d5af0. [D 20:01:04.877409] (send callback time) ini posts: 0, pending: 3, last: 1, src_list emtpy: yes [A 20:01:04.882540] troy.scl@10.1.4.56:39990 H=2863311531 S=0x661ad0: io: finish (Success) [D 20:01:04.882614] PINT_encode_release [D 20:01:04.882622] lebf_encode_rel [D 20:01:04.882650] [SM frame get]: (0x6619d0) op-id: 3 index: 0 base-frm: 0 [D 20:01:04.882659] server_state_machine_complete 0x6619d0 [D 20:01:04.882666] PINT_decode_release [D 20:01:04.882672] lebf_decode_rel [D 20:01:04.882680] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:04.882688] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:04.882694] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 11 [D 20:01:04.882704] [SM Exiting]: (0x6619d0) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:04.882717] server_state_machine_terminate 0x6619d0 [D 20:01:04.877049] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301840) [D 20:01:04.885033] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.885044] dbpf_open_cache_put: cached entry. [D 20:01:04.885051] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.885058] op_queue add: 0x2aaabd301840 [D 20:01:04.885069] DBPF I/O ops in progress: 8 [D 20:01:04.885076] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:04.885093] BMI_post_send: addr: 6843072, offset: 0x2aaab2c0b010, size: 2097152, tag: 29 [D 20:01:04.885107] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303ee0) [D 20:01:04.885109] post_sr: 10.1.4.56:39990 bh 63 len 32 wr_id 683470. [D 20:01:04.885129] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.885148] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.885157] --- aio_progress_notification called with handle 2863311531 (0x2aaabd31d890) [D 20:01:04.885168] openib_check_cq: completed send wr_id 683470. [D 20:01:04.885174] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.885196] --- aio_progress_notification called with handle 2863311531 (0x2aaabd31cad0) [D 20:01:04.885212] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.877044] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.885150] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302630) [D 20:01:04.885252] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.888600] dbpf_open_cache_put: cached entry. [D 20:01:04.888610] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.888618] op_queue add: 0x2aaabd3013f0 [D 20:01:04.888626] DBPF I/O ops in progress: 7 [D 20:01:04.888630] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:04.888643] BMI_post_send: addr: 6843072, offset: 0x2aaab501e010, size: 2097152, tag: 33 [D 20:01:04.877013] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304c70) [D 20:01:04.888665] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:04.888668] post_sr: 10.1.4.56:39990 bh 64 len 32 wr_id 6834a0. [D 20:01:04.888683] dbpf_open_cache_put: cached entry. [D 20:01:04.888690] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.888697] dbpf_open_cache_put: cached entry. [D 20:01:04.888705] openib_check_cq: completed send wr_id 6834a0. [D 20:01:04.888714] dbpf_open_cache_put: cached entry. [D 20:01:04.888724] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.888731] op_queue add: 0x2aaabd31d890 [D 20:01:04.888739] DBPF I/O ops in progress: 6 [D 20:01:04.888742] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:04.888708] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.888754] BMI_post_send: addr: 6843072, offset: 0x2aaab5c25010, size: 2097152, tag: 31 [D 20:01:04.888769] dbpf_open_cache_put: cached entry. [D 20:01:04.888780] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.888692] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.888763] op_queue add: 0x2aaabd303ee0 [D 20:01:04.888805] DBPF I/O ops in progress: 5 [D 20:01:04.888817] dbpf_open_cache_put: cached entry. [D 20:01:04.888826] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.888833] op_queue add: 0x2aaabd304b20 [D 20:01:04.888840] DBPF I/O ops in progress: 4 [D 20:01:04.888869] op_queue add: 0x2aaabd31cad0 [D 20:01:04.888879] DBPF I/O ops in progress: 3 [D 20:01:04.888903] dbpf_open_cache_put: cached entry. [D 20:01:04.888912] dbpf_open_cache_put: move to unused list. [D 20:01:04.888919] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.888926] op_queue add: 0x2aaabd302630 [D 20:01:04.888932] DBPF I/O ops in progress: 2 [D 20:01:04.888951] op_queue add: 0x2aaabd304c70 [D 20:01:04.888961] DBPF I/O ops in progress: 1 [D 20:01:04.896591] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:04.896617] op_queue add: 0x2aaabd3049d0 [D 20:01:04.896632] DBPF I/O ops in progress: 0 [D 20:01:04.896670] post_sr: 10.1.4.56:39990 bh 65 len 32 wr_id 6834d0. [D 20:01:04.896684] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.896689] openib_check_cq: completed send wr_id 6834d0. [D 20:01:04.896697] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:04.896707] BMI_post_send: addr: 6843072, offset: 0x2aaab5420010, size: 2097152, tag: 32 [D 20:01:04.896716] post_sr: 10.1.4.56:39990 bh 66 len 32 wr_id 683500. [D 20:01:04.896727] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.896731] openib_check_cq: completed send wr_id 683500. [D 20:01:04.896735] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:04.896750] BMI_post_send: addr: 6843072, offset: 0x2aaab0add010, size: 2097152, tag: 28 [D 20:01:04.900592] post_sr: 10.1.4.56:39990 bh 67 len 32 wr_id 683530. [D 20:01:04.900604] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.900609] openib_check_cq: completed send wr_id 683530. [D 20:01:04.900612] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:04.900627] BMI_post_send: addr: 6843072, offset: 0x2aaab4a1b010, size: 2097152, tag: 35 [D 20:01:04.903787] post_sr: 10.1.4.56:39990 bh 68 len 32 wr_id 683560. [D 20:01:04.903798] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.903803] openib_check_cq: completed send wr_id 683560. [D 20:01:04.903807] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:04.903822] BMI_post_send: addr: 6843072, offset: 0x2aaab300e010, size: 2097152, tag: 30 [D 20:01:04.915787] post_sr: 10.1.4.56:39990 bh 69 len 32 wr_id 683590. [D 20:01:04.915800] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.915802] openib_check_cq: completed send wr_id 683590. [D 20:01:04.915810] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:04.915826] BMI_post_send: addr: 6843072, offset: 0x2aaab4e1d010, size: 2097152, tag: 34 [D 20:01:04.919789] post_sr: 10.1.4.56:39990 bh 70 len 32 wr_id 6835c0. [D 20:01:04.919802] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.919806] openib_check_cq: completed send wr_id 6835c0. [D 20:01:04.919812] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:04.919826] BMI_post_send: addr: 6843072, offset: 0x2aaab5e26010, size: 2097152, tag: 27 [D 20:01:04.920607] post_sr: 10.1.4.56:39990 bh 71 len 32 wr_id 6835f0. [D 20:01:04.920619] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:04.920624] openib_check_cq: completed send wr_id 6835f0. -[D 20:01:05.224093] openib_check_cq: completed recv wr_id 683e70. [D 20:01:05.224105] openib_post_sr_rdmaw: ibv_post_send wr_id 6eadf0 to 10.1.4.56:39990 remote addr 9ac49000 rkey 109126 nic_wr_credit 1. [D 20:01:05.224114] openib_post_rr: 10.1.4.56:39990 bh 16 wr_id 683e70. [D 20:01:05.225242] openib_check_cq: completed recv wr_id 683ea0. [D 20:01:05.225251] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd307630 to 10.1.4.56:39990 remote addr 9d64b000 rkey 108f27 nic_wr_credit 2. [D 20:01:05.225260] openib_post_rr: 10.1.4.56:39990 bh 17 wr_id 683ea0. [D 20:01:05.226349] openib_check_cq: completed rdma wr_id 6eadf0, nic_wr_credit 1. [D 20:01:05.226358] post_sr: 10.1.4.56:39990 bh 72 len 16 wr_id 683620. [D 20:01:05.504618] [SM Entering]: (0x680150) perf_update_sm:do_work (status: 0) [P 20:01:05.788429] Start times (hr:min:sec): 20:01:04.441 20:01:00.143 20:00:55.876 20:00:54.832 20:00:53.796 20:00:52.756 [P 20:01:05.788437] Intervals (hr:min:sec) : 00:00:01.347 00:00:04.298 00:00:04.267 00:00:01.044 00:00:01.036 00:00:01.040 [P 20:01:05.788444] ------------------------------------------------------------------------------------------------------------- [P 20:01:05.788450] bytes read : 83886080 123731968 0 0 0 0 [P 20:01:05.788456] bytes written : 0 245366784 257949696 0 0 0 [P 20:01:05.788463] metadata reads : 42 42 42 26 25 24 [P 20:01:05.788469] metadata writes : 7 7 7 7 7 7 [P 20:01:05.788475] metadata dspace ops : 0 0 0 0 0 0 [P 20:01:05.788481] metadata keyval ops : 0 0 0 0 0 0 [P 20:01:05.788488] request scheduler : 30 30 30 14 13 12 [D 20:01:05.788504] [SM Exiting]: (0x680150) perf_update_sm:do_work (error code: 0), (action: DEFERRED) [D 20:01:05.788513] [SM Entering]: (0x681490) job_timer_sm:do_work (status: 0) [D 20:01:05.788522] [SM Exiting]: (0x681490) job_timer_sm:do_work (error code: 0), (action: DEFERRED) [D 20:01:05.788535] openib_check_cq: completed recv wr_id 683ed0. [D 20:01:05.788544] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea840 to 10.1.4.56:39990 remote addr 98a47000 rkey 109226 nic_wr_credit 2. [D 20:01:05.788552] openib_post_rr: 10.1.4.56:39990 bh 18 wr_id 683ed0. [D 20:01:05.788559] openib_check_cq: completed recv wr_id 683f00. [D 20:01:05.788567] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3019a0 to 10.1.4.56:39990 remote addr 96a45000 rkey 109426 nic_wr_credit 3. [D 20:01:05.788574] openib_post_rr: 10.1.4.56:39990 bh 19 wr_id 683f00. [D 20:01:05.788582] openib_check_cq: completed rdma wr_id 2aaabd307630, nic_wr_credit 2. [D 20:01:05.788589] post_sr: 10.1.4.56:39990 bh 73 len 16 wr_id 683650. [D 20:01:05.788596] openib_check_cq: completed recv wr_id 683f30. [D 20:01:05.788604] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd300a30 to 10.1.4.56:39990 remote addr 94a43000 rkey 109526 nic_wr_credit 3. [D 20:01:05.788614] openib_post_rr: 10.1.4.56:39990 bh 20 wr_id 683f30. [D 20:01:05.788622] openib_check_cq: completed recv wr_id 683f60. [D 20:01:05.788629] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3027b0 to 10.1.4.56:39990 remote addr 92a41000 rkey 109626 nic_wr_credit 4. [D 20:01:05.788637] openib_post_rr: 10.1.4.56:39990 bh 21 wr_id 683f60. [D 20:01:05.788644] openib_check_cq: completed recv wr_id 683f90. [D 20:01:05.788652] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3016e0 to 10.1.4.56:39990 remote addr 8e83d000 rkey 109926 nic_wr_credit 5. [D 20:01:05.788660] openib_post_rr: 10.1.4.56:39990 bh 22 wr_id 683f90. [D 20:01:05.788667] openib_check_cq: completed recv wr_id 683fc0. [D 20:01:05.788674] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31cef0 to 10.1.4.56:39990 remote addr 8c83b000 rkey 109a26 nic_wr_credit 6. [D 20:01:05.788683] openib_post_rr: 10.1.4.56:39990 bh 23 wr_id 683fc0. [D 20:01:05.788690] openib_check_cq: completed recv wr_id 683ff0. [D 20:01:05.788697] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd304040 to 10.1.4.56:39990 remote addr 9083f000 rkey 109826 nic_wr_credit 7. [D 20:01:05.788705] openib_post_rr: 10.1.4.56:39990 bh 24 wr_id 683ff0. [D 20:01:05.788712] openib_check_cq: completed recv wr_id 684020. [D 20:01:05.788720] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31d9f0 to 10.1.4.56:39990 remote addr 8a839000 rkey 109b26 nic_wr_credit 8. [D 20:01:05.788728] openib_post_rr: 10.1.4.56:39990 bh 25 wr_id 684020. [D 20:01:05.788736] openib_check_cq: completed recv wr_id 684050. [D 20:01:05.788744] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31e4f0 to 10.1.4.56:39990 remote addr 86835000 rkey 109d26 nic_wr_credit 9. [D 20:01:05.788752] openib_post_rr: 10.1.4.56:39990 bh 26 wr_id 684050. [D 20:01:05.788759] openib_check_cq: completed send wr_id 683620. [D 20:01:05.788771] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:05.788780] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.788788] ========================================================= [D 20:01:05.788794] PINT_process_request [D 20:01:05.788801] tiling 6 copies [D 20:01:05.788807] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.788815] lvl 0 el 0 blk 0 by 159383552 [D 20:01:05.788821] to 159383552 ta 134217728 fi 167772160 [D 20:01:05.788828] basic type or contiguous data [D 20:01:05.788835] Request disp [D 20:01:05.788841] contig_offset = 159383552 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.788848] calling distribute [D 20:01:05.788854] PINT_distribute [D 20:01:05.788860] of 159383552 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.788868] begin iteration loff: 159383552 [D 20:01:05.788875] segment exceeds byte limit [D 20:01:05.788881] process a segment [D 20:01:05.788887] of 159383552 sz 2097152 [D 20:01:05.788893] add a segment 0 [D 20:01:05.788899] end iteration [D 20:01:05.788905] done with segments or bytes [D 20:01:05.788911] of 161480704 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.788918] next loff: 161480704 next poff: 161480704 [D 20:01:05.788924] return value 2097152 [D 20:01:05.788931] finished [D 20:01:05.788936] retval = 2097152 [D 20:01:05.788942] exiting distribute returned less than expected [D 20:01:05.788948] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 161480704 fo 167772160 eof 0 [D 20:01:05.788956] ========================================================= [D 20:01:05.788966] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.788973] dbpf_bstream_rw_list: mem_offset: 0x2aaab3e15010, mem_size: 2097152 [D 20:01:05.788979] dbpf_bstream_rw_list: stream_offset: 159383552, stream_size: 2097152 [D 20:01:05.788986] dbpf_open_cache_get: called [D 20:01:05.788993] dbpf_open_cache_get: found bstream entry in unused list. [D 20:01:05.788999] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.789014] DBPF I/O ops in progress: 1 [D 20:01:05.789021] lio_listio called with the following aiocbs: [D 20:01:05.789027] aiocb_ptr_array[0]: fd: 13, off: 159383552, bytes: 2097152, buf: 0x2aaab3e15010, type: 0 [D 20:01:05.789039] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3049d0 (handle 2863311531, ret 0) [D 20:01:05.789830] openib_check_cq: completed recv wr_id 684080. [D 20:01:05.789865] openib_post_sr_rdmaw: ibv_post_send wr_id 6ea5e0 to 10.1.4.56:39990 remote addr 9ae49000 rkey 109126 nic_wr_credit 10. [D 20:01:05.789879] openib_post_rr: 10.1.4.56:39990 bh 27 wr_id 684080. [D 20:01:05.791129] openib_check_cq: completed rdma wr_id 6ea840, nic_wr_credit 9. [D 20:01:05.791175] post_sr: 10.1.4.56:39990 bh 74 len 16 wr_id 683680. [D 20:01:05.791619] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3049d0) [D 20:01:05.791634] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.791645] dbpf_open_cache_put: cached entry. [D 20:01:05.791652] dbpf_open_cache_put: move to unused list. [D 20:01:05.791658] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.791665] op_queue add: 0x2aaabd3049d0 [D 20:01:05.791689] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:05.791700] BMI_post_send: addr: 6843072, offset: 0x2aaab3e15010, size: 2097152, tag: 27 [D 20:01:05.791723] post_sr: 10.1.4.56:39990 bh 75 len 32 wr_id 6836b0. [D 20:01:05.791750] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:05.791712] DBPF I/O ops in progress: 0 [D 20:01:05.793006] openib_check_cq: completed rdma wr_id 2aaabd3019a0, nic_wr_credit 8. [D 20:01:05.793018] post_sr: 10.1.4.56:39990 bh 76 len 16 wr_id 6836e0. [D 20:01:05.793026] openib_check_cq: completed send wr_id 683650. [D 20:01:05.793039] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:05.793048] (send callback time) ini posts: 0, pending: 6, last: 1, src_list emtpy: yes [D 20:01:05.794228] openib_check_cq: completed recv wr_id 6840b0. [D 20:01:05.794238] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3078f0 to 10.1.4.56:39990 remote addr 9d84b000 rkey 108f27 nic_wr_credit 9. [D 20:01:05.794246] openib_post_rr: 10.1.4.56:39990 bh 28 wr_id 6840b0. [D 20:01:05.795232] openib_check_cq: completed rdma wr_id 2aaabd300a30, nic_wr_credit 8. [D 20:01:05.795241] post_sr: 10.1.4.56:39990 bh 77 len 16 wr_id 683710. [D 20:01:05.797453] openib_check_cq: completed rdma wr_id 2aaabd3027b0, nic_wr_credit 7. [D 20:01:05.797463] post_sr: 10.1.4.56:39990 bh 78 len 16 wr_id 683740. [D 20:01:05.799675] openib_check_cq: completed rdma wr_id 2aaabd3016e0, nic_wr_credit 6. [D 20:01:05.799683] post_sr: 10.1.4.56:39990 bh 79 len 16 wr_id 683770. [D 20:01:05.801897] openib_check_cq: completed rdma wr_id 2aaabd31cef0, nic_wr_credit 5. [D 20:01:05.801907] post_sr: 10.1.4.56:39990 bh 80 len 16 wr_id 6837a0. [D 20:01:05.804119] openib_check_cq: completed rdma wr_id 2aaabd304040, nic_wr_credit 4. [D 20:01:05.804128] post_sr: 10.1.4.56:39990 bh 81 len 16 wr_id 6837d0. [D 20:01:05.806340] openib_check_cq: completed rdma wr_id 2aaabd31d9f0, nic_wr_credit 3. [D 20:01:05.806348] post_sr: 10.1.4.56:39990 bh 82 len 16 wr_id 683800. [D 20:01:05.808561] openib_check_cq: completed rdma wr_id 2aaabd31e4f0, nic_wr_credit 2. [D 20:01:05.808570] post_sr: 10.1.4.56:39990 bh 83 len 16 wr_id 683830. [D 20:01:05.810782] openib_check_cq: completed rdma wr_id 6ea5e0, nic_wr_credit 1. [D 20:01:05.810791] post_sr: 10.1.4.56:39990 bh 84 len 16 wr_id 683860. [D 20:01:05.810799] openib_check_cq: completed send wr_id 683680. [D 20:01:05.810806] openib_check_cq: completed send wr_id 6836b0. [D 20:01:05.810813] openib_check_cq: completed send wr_id 6836e0. [D 20:01:05.810824] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c750. [D 20:01:05.810833] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.810840] ========================================================= [D 20:01:05.810846] PINT_process_request [D 20:01:05.810853] tiling 7 copies [D 20:01:05.810860] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.810867] lvl 0 el 0 blk 0 by 190840832 [D 20:01:05.810874] to 190840832 ta 167772160 fi 201326592 [D 20:01:05.810881] basic type or contiguous data [D 20:01:05.810887] Request disp [D 20:01:05.810893] contig_offset = 190840832 contig_size = 44040192 lvl_flag = 1 [D 20:01:05.810900] calling distribute [D 20:01:05.810906] PINT_distribute [D 20:01:05.810912] of 190840832 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.810921] begin iteration loff: 190840832 [D 20:01:05.810928] segment exceeds byte limit [D 20:01:05.810935] process a segment [D 20:01:05.810940] of 190840832 sz 2097152 [D 20:01:05.810947] add a segment 0 [D 20:01:05.810953] end iteration [D 20:01:05.810959] done with segments or bytes [D 20:01:05.810965] of 192937984 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.810972] next loff: 192937984 next poff: 192937984 [D 20:01:05.810979] return value 2097152 [D 20:01:05.810985] finished [D 20:01:05.810991] retval = 2097152 [D 20:01:05.810997] exiting distribute returned less than expected [D 20:01:05.811003] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 192937984 fo 201326592 eof 0 [D 20:01:05.811011] ========================================================= [D 20:01:05.811021] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.811028] dbpf_bstream_rw_list: mem_offset: 0x2aaaaeecf010, mem_size: 2097152 [D 20:01:05.811035] dbpf_bstream_rw_list: stream_offset: 190840832, stream_size: 2097152 [D 20:01:05.811042] dbpf_open_cache_get: called [D 20:01:05.811049] dbpf_open_cache_get: found bstream entry in unused list. [D 20:01:05.811055] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.811068] DBPF I/O ops in progress: 1 [D 20:01:05.811075] lio_listio called with the following aiocbs: [D 20:01:05.811082] aiocb_ptr_array[0]: fd: 13, off: 190840832, bytes: 2097152, buf: 0x2aaaaeecf010, type: 0 [D 20:01:05.811094] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304aa0 (handle 2863311531, ret 0) [D 20:01:05.811105] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517400. [D 20:01:05.811113] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.811120] ========================================================= [D 20:01:05.811127] PINT_process_request [D 20:01:05.811133] tiling 8 copies [D 20:01:05.811140] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.811147] lvl 0 el 0 blk 0 by 224395264 [D 20:01:05.811154] to 224395264 ta 201326592 fi 234881024 [D 20:01:05.811161] basic type or contiguous data [D 20:01:05.811167] Request disp [D 20:01:05.811174] contig_offset = 224395264 contig_size = 44040192 lvl_flag = 1 [D 20:01:05.811181] calling distribute [D 20:01:05.811187] PINT_distribute [D 20:01:05.811193] of 224395264 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.811200] begin iteration loff: 224395264 [D 20:01:05.811207] segment exceeds byte limit [D 20:01:05.811213] process a segment [D 20:01:05.811219] of 224395264 sz 2097152 [D 20:01:05.811225] add a segment 0 [D 20:01:05.811232] end iteration [D 20:01:05.811238] done with segments or bytes [D 20:01:05.811244] of 226492416 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.811251] next loff: 226492416 next poff: 226492416 [D 20:01:05.811258] return value 2097152 [D 20:01:05.811264] finished [D 20:01:05.811270] retval = 2097152 [D 20:01:05.811276] exiting distribute returned less than expected [D 20:01:05.811282] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 226492416 fo 234881024 eof 0 [D 20:01:05.811290] ========================================================= [D 20:01:05.811297] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.811304] dbpf_bstream_rw_list: mem_offset: 0x2aaab0edf010, mem_size: 2097152 [D 20:01:05.811310] dbpf_bstream_rw_list: stream_offset: 224395264, stream_size: 2097152 [D 20:01:05.811317] dbpf_open_cache_get: called [D 20:01:05.811323] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.811330] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.811348] DBPF I/O ops in progress: 2 [D 20:01:05.811355] lio_listio called with the following aiocbs: [D 20:01:05.811361] aiocb_ptr_array[0]: fd: 13, off: 224395264, bytes: 2097152, buf: 0x2aaab0edf010, type: 0 [D 20:01:05.811371] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3019a0 (handle 2863311531, ret 0) [D 20:01:05.812109] openib_check_cq: completed recv wr_id 6840e0. [D 20:01:05.812141] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec680 to 10.1.4.56:39990 remote addr 98c47000 rkey 109226 nic_wr_credit 2. [D 20:01:05.812152] openib_post_rr: 10.1.4.56:39990 bh 29 wr_id 6840e0. [D 20:01:05.813021] openib_check_cq: completed rdma wr_id 2aaabd3078f0, nic_wr_credit 1. [D 20:01:05.813064] post_sr: 10.1.4.56:39990 bh 85 len 16 wr_id 683890. [D 20:01:05.813074] openib_check_cq: completed send wr_id 683710. [D 20:01:05.813082] openib_check_cq: completed send wr_id 683740. [D 20:01:05.813089] openib_check_cq: completed send wr_id 683770. [D 20:01:05.813096] openib_check_cq: completed send wr_id 6837a0. [D 20:01:05.813103] openib_check_cq: completed send wr_id 6837d0. [D 20:01:05.813111] openib_check_cq: completed send wr_id 683800. [D 20:01:05.813117] openib_check_cq: completed send wr_id 683830. [D 20:01:05.813125] openib_check_cq: completed send wr_id 683860. [D 20:01:05.813152] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:05.813166] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.813176] ========================================================= [D 20:01:05.813184] PINT_process_request [D 20:01:05.813192] tiling 6 copies [D 20:01:05.813200] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.813208] lvl 0 el 0 blk 0 by 161480704 [D 20:01:05.813214] to 161480704 ta 134217728 fi 167772160 [D 20:01:05.813223] basic type or contiguous data [D 20:01:05.813229] Request disp [D 20:01:05.813236] contig_offset = 161480704 contig_size = 39845888 lvl_flag = 1 [D 20:01:05.813244] calling distribute [D 20:01:05.813250] PINT_distribute [D 20:01:05.813256] of 161480704 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.813265] begin iteration loff: 161480704 [D 20:01:05.813272] segment exceeds byte limit [D 20:01:05.813278] process a segment [D 20:01:05.813284] of 161480704 sz 2097152 [D 20:01:05.813291] add a segment 0 [D 20:01:05.813297] end iteration [D 20:01:05.813303] done with segments or bytes [D 20:01:05.813309] of 163577856 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.813316] next loff: 163577856 next poff: 163577856 [D 20:01:05.813323] return value 2097152 [D 20:01:05.813329] finished [D 20:01:05.813335] retval = 2097152 [D 20:01:05.813341] exiting distribute returned less than expected [D 20:01:05.813348] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 163577856 fo 167772160 eof 0 [D 20:01:05.813356] ========================================================= [D 20:01:05.813373] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.813383] dbpf_bstream_rw_list: mem_offset: 0x2aaaba048010, mem_size: 2097152 [D 20:01:05.813390] dbpf_bstream_rw_list: stream_offset: 161480704, stream_size: 2097152 [D 20:01:05.813398] dbpf_open_cache_get: called [D 20:01:05.813405] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.813413] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.813440] DBPF I/O ops in progress: 3 [D 20:01:05.813449] lio_listio called with the following aiocbs: [D 20:01:05.813455] aiocb_ptr_array[0]: fd: 13, off: 161480704, bytes: 2097152, buf: 0x2aaaba048010, type: 0 [D 20:01:05.813470] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3027b0 (handle 2863311531, ret 0) [D 20:01:05.813480] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152db60. [D 20:01:05.813489] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.813496] ========================================================= [D 20:01:05.813502] PINT_process_request [D 20:01:05.813509] tiling 9 copies [D 20:01:05.813516] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.813526] lvl 0 el 0 blk 0 by 257949696 [D 20:01:05.813532] to 257949696 ta 234881024 fi 268435456 [D 20:01:05.813540] basic type or contiguous data [D 20:01:05.813546] Request disp [D 20:01:05.813552] contig_offset = 257949696 contig_size = 44040192 lvl_flag = 1 [D 20:01:05.813559] calling distribute [D 20:01:05.813565] PINT_distribute [D 20:01:05.813570] of 257949696 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.813578] begin iteration loff: 257949696 [D 20:01:05.813584] segment exceeds byte limit [D 20:01:05.813590] process a segment [D 20:01:05.813596] of 257949696 sz 2097152 [D 20:01:05.813603] add a segment 0 [D 20:01:05.813609] end iteration [D 20:01:05.813615] done with segments or bytes [D 20:01:05.813621] of 260046848 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.813628] next loff: 260046848 next poff: 260046848 [D 20:01:05.813634] return value 2097152 [D 20:01:05.813640] finished [D 20:01:05.813646] retval = 2097152 [D 20:01:05.813652] exiting distribute returned less than expected [D 20:01:05.813658] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 260046848 fo 268435456 eof 0 [D 20:01:05.813666] ========================================================= [D 20:01:05.813674] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.813681] dbpf_bstream_rw_list: mem_offset: 0x2aaab12e1010, mem_size: 2097152 [D 20:01:05.813688] dbpf_bstream_rw_list: stream_offset: 257949696, stream_size: 2097152 [D 20:01:05.813695] dbpf_open_cache_get: called [D 20:01:05.813701] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.813708] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.813729] DBPF I/O ops in progress: 4 [D 20:01:05.813736] lio_listio called with the following aiocbs: [D 20:01:05.813742] aiocb_ptr_array[0]: fd: 13, off: 257949696, bytes: 2097152, buf: 0x2aaab12e1010, type: 0 [D 20:01:05.813751] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3016e0 (handle 2863311531, ret 0) [D 20:01:05.813761] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1505890. [D 20:01:05.813770] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.813777] ========================================================= [D 20:01:05.813784] PINT_process_request [D 20:01:05.813791] tiling 10 copies [D 20:01:05.813798] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.813805] lvl 0 el 0 blk 0 by 291504128 [D 20:01:05.813813] to 291504128 ta 268435456 fi 301989888 [D 20:01:05.813820] basic type or contiguous data [D 20:01:05.813826] Request disp [D 20:01:05.813832] contig_offset = 291504128 contig_size = 44040192 lvl_flag = 1 [D 20:01:05.813839] calling distribute [D 20:01:05.813845] PINT_distribute [D 20:01:05.813850] of 291504128 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.813858] begin iteration loff: 291504128 [D 20:01:05.813865] segment exceeds byte limit [D 20:01:05.813871] process a segment [D 20:01:05.813877] of 291504128 sz 2097152 [D 20:01:05.813883] add a segment 0 [D 20:01:05.813889] end iteration [D 20:01:05.813895] done with segments or bytes [D 20:01:05.813901] of 293601280 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.813908] next loff: 293601280 next poff: 293601280 [D 20:01:05.813915] return value 2097152 [D 20:01:05.813921] finished [D 20:01:05.813927] retval = 2097152 [D 20:01:05.813933] exiting distribute returned less than expected [D 20:01:05.813939] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 293601280 fo 301989888 eof 0 [D 20:01:05.813946] ========================================================= [D 20:01:05.813953] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.813960] dbpf_bstream_rw_list: mem_offset: 0x2aaab521f010, mem_size: 2097152 [D 20:01:05.813967] dbpf_bstream_rw_list: stream_offset: 291504128, stream_size: 2097152 [D 20:01:05.813976] dbpf_open_cache_get: called [D 20:01:05.813982] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.813989] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.814010] DBPF I/O ops in progress: 5 [D 20:01:05.814017] lio_listio called with the following aiocbs: [D 20:01:05.814024] aiocb_ptr_array[0]: fd: 13, off: 291504128, bytes: 2097152, buf: 0x2aaab521f010, type: 0 [D 20:01:05.814033] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304040 (handle 2863311531, ret 0) [D 20:01:05.814042] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c210. [D 20:01:05.814050] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.814058] ========================================================= [D 20:01:05.814064] PINT_process_request [D 20:01:05.814070] tiling 12 copies [D 20:01:05.814077] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.814084] lvl 0 el 0 blk 0 by 356515840 [D 20:01:05.814090] to 356515840 ta 335544320 fi 369098752 [D 20:01:05.814097] basic type or contiguous data [D 20:01:05.814104] Request disp [D 20:01:05.814111] contig_offset = 356515840 contig_size = 46137344 lvl_flag = 1 [D 20:01:05.814118] calling distribute [D 20:01:05.814124] PINT_distribute [D 20:01:05.814129] of 356515840 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.814137] begin iteration loff: 356515840 [D 20:01:05.814144] segment exceeds byte limit [D 20:01:05.814150] process a segment [D 20:01:05.814156] of 356515840 sz 2097152 [D 20:01:05.814162] add a segment 0 [D 20:01:05.814168] end iteration [D 20:01:05.814174] done with segments or bytes [D 20:01:05.814180] of 358612992 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.814187] next loff: 358612992 next poff: 358612992 [D 20:01:05.814193] return value 2097152 [D 20:01:05.814200] finished [D 20:01:05.814206] retval = 2097152 [D 20:01:05.814212] exiting distribute returned less than expected [D 20:01:05.814218] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 358612992 fo 369098752 eof 0 [D 20:01:05.814225] ========================================================= [D 20:01:05.814233] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.814240] dbpf_bstream_rw_list: mem_offset: 0x2aaab3410010, mem_size: 2097152 [D 20:01:05.814247] dbpf_bstream_rw_list: stream_offset: 356515840, stream_size: 2097152 [D 20:01:05.814254] dbpf_open_cache_get: called [D 20:01:05.814260] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.814267] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.814285] DBPF I/O ops in progress: 6 [D 20:01:05.814294] lio_listio called with the following aiocbs: [D 20:01:05.814300] aiocb_ptr_array[0]: fd: 13, off: 356515840, bytes: 2097152, buf: 0x2aaab3410010, type: 0 [D 20:01:05.814308] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304bf0 (handle 2863311531, ret 0) [D 20:01:05.814317] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b470. [D 20:01:05.814325] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.814333] ========================================================= [D 20:01:05.814339] PINT_process_request [D 20:01:05.814346] tiling 11 copies [D 20:01:05.814352] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.814360] lvl 0 el 0 blk 0 by 322961408 [D 20:01:05.814367] to 322961408 ta 301989888 fi 335544320 [D 20:01:05.814373] basic type or contiguous data [D 20:01:05.814379] Request disp [D 20:01:05.814385] contig_offset = 322961408 contig_size = 46137344 lvl_flag = 1 [D 20:01:05.814392] calling distribute [D 20:01:05.814398] PINT_distribute [D 20:01:05.814404] of 322961408 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.814411] begin iteration loff: 322961408 [D 20:01:05.814418] segment exceeds byte limit [D 20:01:05.814425] process a segment [D 20:01:05.814431] of 322961408 sz 2097152 [D 20:01:05.814437] add a segment 0 [D 20:01:05.814443] end iteration [D 20:01:05.814449] done with segments or bytes [D 20:01:05.814455] of 325058560 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.814462] next loff: 325058560 next poff: 325058560 [D 20:01:05.814468] return value 2097152 [D 20:01:05.814474] finished [D 20:01:05.814480] retval = 2097152 [D 20:01:05.814486] exiting distribute returned less than expected [D 20:01:05.814492] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 325058560 fo 335544320 eof 0 [D 20:01:05.814499] ========================================================= [D 20:01:05.814508] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.814515] dbpf_bstream_rw_list: mem_offset: 0x2aaab3611010, mem_size: 2097152 [D 20:01:05.814521] dbpf_bstream_rw_list: stream_offset: 322961408, stream_size: 2097152 [D 20:01:05.814528] dbpf_open_cache_get: called [D 20:01:05.814534] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.814541] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.814561] DBPF I/O ops in progress: 7 [D 20:01:05.814569] lio_listio called with the following aiocbs: [D 20:01:05.814575] aiocb_ptr_array[0]: fd: 13, off: 322961408, bytes: 2097152, buf: 0x2aaab3611010, type: 0 [D 20:01:05.814584] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304db0 (handle 2863311531, ret 0) [D 20:01:05.814602] openib_check_cq: completed recv wr_id 684110. [D 20:01:05.814618] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd301b00 to 10.1.4.56:39990 remote addr 96c45000 rkey 109426 nic_wr_credit 2. [D 20:01:05.814630] openib_post_rr: 10.1.4.56:39990 bh 30 wr_id 684110. [D 20:01:05.814667] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1506e50. [D 20:01:05.814682] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.814689] ========================================================= [D 20:01:05.814696] PINT_process_request [D 20:01:05.814703] tiling 13 copies [D 20:01:05.814709] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.814716] lvl 0 el 0 blk 0 by 390070272 [D 20:01:05.814723] to 390070272 ta 369098752 fi 402653184 [D 20:01:05.814730] basic type or contiguous data [D 20:01:05.814736] Request disp [D 20:01:05.814742] contig_offset = 390070272 contig_size = 46137344 lvl_flag = 1 [D 20:01:05.814749] calling distribute [D 20:01:05.814755] PINT_distribute [D 20:01:05.814761] of 390070272 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:05.814769] begin iteration loff: 390070272 [D 20:01:05.814775] segment exceeds byte limit [D 20:01:05.814781] process a segment [D 20:01:05.814788] of 390070272 sz 2097152 [D 20:01:05.814794] add a segment 0 [D 20:01:05.814801] end iteration [D 20:01:05.814806] done with segments or bytes [D 20:01:05.814812] of 392167424 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.814820] next loff: 392167424 next poff: 392167424 [D 20:01:05.814826] return value 2097152 [D 20:01:05.814833] finished [D 20:01:05.814839] retval = 2097152 [D 20:01:05.814845] exiting distribute returned less than expected [D 20:01:05.814851] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 392167424 fo 402653184 eof 0 [D 20:01:05.814859] ========================================================= [D 20:01:05.814866] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.814874] dbpf_bstream_rw_list: mem_offset: 0x2aaab320f010, mem_size: 2097152 [D 20:01:05.814880] dbpf_bstream_rw_list: stream_offset: 390070272, stream_size: 2097152 [D 20:01:05.814887] dbpf_open_cache_get: called [D 20:01:05.814893] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.814900] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.814920] DBPF I/O ops in progress: 8 [D 20:01:05.814929] lio_listio called with the following aiocbs: [D 20:01:05.814936] aiocb_ptr_array[0]: fd: 13, off: 390070272, bytes: 2097152, buf: 0x2aaab320f010, type: 0 [D 20:01:05.814944] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31cef0 (handle 2863311531, ret 0) [D 20:01:05.814954] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1507b90. [D 20:01:05.814962] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.814969] ========================================================= [D 20:01:05.814975] PINT_process_request [D 20:01:05.814982] tiling 14 copies [D 20:01:05.814988] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.814995] lvl 0 el 0 blk 0 by 423624704 [D 20:01:05.815002] to 423624704 ta 402653184 fi 436207616 [D 20:01:05.815009] basic type or contiguous data [D 20:01:05.815015] Request disp [D 20:01:05.815021] contig_offset = 423624704 contig_size = 46137344 lvl_flag = 1 [D 20:01:05.815028] calling distribute [D 20:01:05.815034] PINT_distribute [D 20:01:05.815040] of 423624704 sz 12582912 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:05.815048] begin iteration loff: 423624704 [D 20:01:05.815054] segment exceeds byte limit [D 20:01:05.815060] process a segment [D 20:01:05.815066] of 423624704 sz 2097152 [D 20:01:05.815073] add a segment 0 [D 20:01:05.815079] end iteration [D 20:01:05.815085] done with segments or bytes [D 20:01:05.815091] of 425721856 sz 10485760 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.815098] next loff: 425721856 next poff: 425721856 [D 20:01:05.815104] return value 2097152 [D 20:01:05.815110] finished [D 20:01:05.815116] retval = 2097152 [D 20:01:05.815122] exiting distribute returned less than expected [D 20:01:05.815128] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 425721856 fo 436207616 eof 0 [D 20:01:05.815136] ========================================================= [D 20:01:05.815143] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.815150] dbpf_bstream_rw_list: mem_offset: 0x2aaab2e0c010, mem_size: 2097152 [D 20:01:05.815157] dbpf_bstream_rw_list: stream_offset: 423624704, stream_size: 2097152 [D 20:01:05.815164] dbpf_open_cache_get: called [D 20:01:05.815170] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.815176] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.815198] DBPF I/O ops in progress: 9 [D 20:01:05.815206] lio_listio called with the following aiocbs: [D 20:01:05.815213] aiocb_ptr_array[0]: fd: 13, off: 423624704, bytes: 2097152, buf: 0x2aaab2e0c010, type: 0 [D 20:01:05.815221] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31d9f0 (handle 2863311531, ret 0) [D 20:01:05.815230] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab15d5af0. [D 20:01:05.815238] (send callback time) ini posts: 0, pending: 2, last: 1, src_list emtpy: yes [D 20:01:05.815249] openib_check_cq: completed recv wr_id 684140. [D 20:01:05.815260] openib_post_sr_rdmaw: ibv_post_send wr_id 6bfc80 to 10.1.4.56:39990 remote addr 9b049000 rkey 109126 nic_wr_credit 3. [D 20:01:05.815270] openib_post_rr: 10.1.4.56:39990 bh 31 wr_id 684140. [D 20:01:05.815280] openib_check_cq: completed rdma wr_id 6ec680, nic_wr_credit 2. [D 20:01:05.815290] post_sr: 10.1.4.56:39990 bh 86 len 16 wr_id 6838c0. [D 20:01:05.815298] openib_check_cq: completed send wr_id 683890. [D 20:01:05.815317] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:05.815329] (send callback time) ini posts: 0, pending: 5, last: 1, src_list emtpy: yes [D 20:01:05.816125] openib_check_cq: completed recv wr_id 684170. [D 20:01:05.816160] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd301b90 to 10.1.4.56:39990 remote addr 94c43000 rkey 109526 nic_wr_credit 3. [D 20:01:05.816173] openib_post_rr: 10.1.4.56:39990 bh 32 wr_id 684170. [D 20:01:05.817301] openib_check_cq: completed recv wr_id 6841a0. [D 20:01:05.817368] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd302910 to 10.1.4.56:39990 remote addr 92c41000 rkey 109626 nic_wr_credit 4. [D 20:01:05.817382] openib_post_rr: 10.1.4.56:39990 bh 33 wr_id 6841a0. [D 20:01:05.817487] openib_check_cq: completed rdma wr_id 2aaabd301b00, nic_wr_credit 3. [D 20:01:05.817502] post_sr: 10.1.4.56:39990 bh 87 len 16 wr_id 6838f0. [D 20:01:05.817962] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304aa0) [D 20:01:05.817983] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.817993] dbpf_open_cache_put: cached entry. [D 20:01:05.818000] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.818007] op_queue add: 0x2aaabd304aa0 [D 20:01:05.818032] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:05.818043] BMI_post_send: addr: 6843072, offset: 0x2aaaaeecf010, size: 2097152, tag: 28 [D 20:01:05.818055] DBPF I/O ops in progress: 8 [D 20:01:05.818064] post_sr: 10.1.4.56:39990 bh 88 len 32 wr_id 683920. [D 20:01:05.818088] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3019a0) [D 20:01:05.818094] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:05.818100] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.818126] dbpf_open_cache_put: cached entry. [D 20:01:05.818133] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.818139] op_queue add: 0x2aaabd3019a0 [D 20:01:05.818148] DBPF I/O ops in progress: 7 [D 20:01:05.818104] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3027b0) [D 20:01:05.818165] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.818175] dbpf_open_cache_put: cached entry. [D 20:01:05.818182] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.818188] op_queue add: 0x2aaabd3027b0 [D 20:01:05.818195] DBPF I/O ops in progress: 6 [D 20:01:05.818472] openib_check_cq: completed recv wr_id 6841d0. [D 20:01:05.818487] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3036a0 to 10.1.4.56:39990 remote addr 8ea3d000 rkey 109926 nic_wr_credit 4. [D 20:01:05.818495] openib_post_rr: 10.1.4.56:39990 bh 34 wr_id 6841d0. [D 20:01:05.819663] openib_check_cq: completed recv wr_id 684200. [D 20:01:05.819713] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3041a0 to 10.1.4.56:39990 remote addr 90a3f000 rkey 109826 nic_wr_credit 5. [D 20:01:05.819727] openib_post_rr: 10.1.4.56:39990 bh 35 wr_id 684200. [D 20:01:05.819737] openib_check_cq: completed rdma wr_id 6bfc80, nic_wr_credit 4. [D 20:01:05.819748] post_sr: 10.1.4.56:39990 bh 89 len 16 wr_id 683950. [D 20:01:05.819756] openib_check_cq: completed send wr_id 6838c0. [D 20:01:05.819790] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c750. [D 20:01:05.819806] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.819830] ========================================================= [D 20:01:05.819841] PINT_process_request [D 20:01:05.819849] tiling 7 copies [D 20:01:05.819857] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.819866] lvl 0 el 0 blk 0 by 192937984 [D 20:01:05.819872] to 192937984 ta 167772160 fi 201326592 [D 20:01:05.819881] basic type or contiguous data [D 20:01:05.819888] Request disp [D 20:01:05.819896] contig_offset = 192937984 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.819903] calling distribute [D 20:01:05.819910] PINT_distribute [D 20:01:05.819916] of 192937984 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.819925] begin iteration loff: 192937984 [D 20:01:05.819932] segment exceeds byte limit [D 20:01:05.819939] process a segment [D 20:01:05.819945] of 192937984 sz 2097152 [D 20:01:05.819952] add a segment 0 [D 20:01:05.819958] end iteration [D 20:01:05.819970] done with segments or bytes [D 20:01:05.819977] of 195035136 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.819985] next loff: 195035136 next poff: 195035136 [D 20:01:05.819991] return value 2097152 [D 20:01:05.819998] finished [D 20:01:05.820005] retval = 2097152 [D 20:01:05.820011] exiting distribute returned less than expected [D 20:01:05.820017] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 195035136 fo 201326592 eof 0 [D 20:01:05.820025] ========================================================= [D 20:01:05.820042] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.820052] dbpf_bstream_rw_list: mem_offset: 0x2aaaad2c1010, mem_size: 2097152 [D 20:01:05.820059] dbpf_bstream_rw_list: stream_offset: 192937984, stream_size: 2097152 [D 20:01:05.820067] dbpf_open_cache_get: called [D 20:01:05.820075] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.820082] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.820110] DBPF I/O ops in progress: 7 [D 20:01:05.820118] lio_listio called with the following aiocbs: [D 20:01:05.820125] aiocb_ptr_array[0]: fd: 13, off: 192937984, bytes: 2097152, buf: 0x2aaaad2c1010, type: 0 [D 20:01:05.820141] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31cad0 (handle 2863311531, ret 0) [D 20:01:05.820898] openib_check_cq: completed recv wr_id 684230. [D 20:01:05.820938] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3008d0 to 10.1.4.56:39990 remote addr 9da4b000 rkey 108f27 nic_wr_credit 5. [D 20:01:05.820954] openib_post_rr: 10.1.4.56:39990 bh 36 wr_id 684230. [D 20:01:05.821940] openib_check_cq: completed rdma wr_id 2aaabd301b90, nic_wr_credit 4. [D 20:01:05.821979] post_sr: 10.1.4.56:39990 bh 90 len 16 wr_id 683980. [D 20:01:05.822060] openib_check_cq: completed recv wr_id 684260. [D 20:01:05.822080] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31d050 to 10.1.4.56:39990 remote addr 8ca3b000 rkey 109a26 nic_wr_credit 5. [D 20:01:05.822091] openib_post_rr: 10.1.4.56:39990 bh 37 wr_id 684260. [D 20:01:05.823252] openib_check_cq: completed recv wr_id 684290. [D 20:01:05.823296] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31db50 to 10.1.4.56:39990 remote addr 8aa39000 rkey 109b26 nic_wr_credit 6. [D 20:01:05.823312] openib_post_rr: 10.1.4.56:39990 bh 38 wr_id 684290. [D 20:01:05.824164] openib_check_cq: completed rdma wr_id 2aaabd302910, nic_wr_credit 5. [D 20:01:05.824204] post_sr: 10.1.4.56:39990 bh 91 len 16 wr_id 6839b0. [D 20:01:05.824216] openib_check_cq: completed send wr_id 6838f0. [D 20:01:05.824224] openib_check_cq: completed send wr_id 683920. [D 20:01:05.824250] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517400. [D 20:01:05.824268] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.824279] ========================================================= [D 20:01:05.824286] PINT_process_request [D 20:01:05.824294] tiling 8 copies [D 20:01:05.824302] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.824311] lvl 0 el 0 blk 0 by 226492416 [D 20:01:05.824318] to 226492416 ta 201326592 fi 234881024 [D 20:01:05.824325] basic type or contiguous data [D 20:01:05.824332] Request disp [D 20:01:05.824339] contig_offset = 226492416 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.824346] calling distribute [D 20:01:05.824354] PINT_distribute [D 20:01:05.824360] of 226492416 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.824369] begin iteration loff: 226492416 [D 20:01:05.824377] segment exceeds byte limit [D 20:01:05.824383] process a segment [D 20:01:05.824389] of 226492416 sz 2097152 [D 20:01:05.824396] add a segment 0 [D 20:01:05.824404] end iteration [D 20:01:05.824409] done with segments or bytes [D 20:01:05.824415] of 228589568 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.824423] next loff: 228589568 next poff: 228589568 [D 20:01:05.824430] return value 2097152 [D 20:01:05.824437] finished [D 20:01:05.824449] retval = 2097152 [D 20:01:05.824456] exiting distribute returned less than expected [D 20:01:05.824463] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 228589568 fo 234881024 eof 0 [D 20:01:05.824470] ========================================================= [D 20:01:05.824488] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.824497] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf2d1010, mem_size: 2097152 [D 20:01:05.824504] dbpf_bstream_rw_list: stream_offset: 226492416, stream_size: 2097152 [D 20:01:05.824513] dbpf_open_cache_get: called [D 20:01:05.824521] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.824529] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.824558] DBPF I/O ops in progress: 8 [D 20:01:05.824568] lio_listio called with the following aiocbs: [D 20:01:05.824575] aiocb_ptr_array[0]: fd: 13, off: 226492416, bytes: 2097152, buf: 0x2aaaaf2d1010, type: 0 [D 20:01:05.824593] issue_or_delay_io_operation: lio_listio posted 0x2aaabd30a300 (handle 2863311531, ret 0) [D 20:01:05.824607] openib_check_cq: completed recv wr_id 6842c0. [D 20:01:05.824641] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31e650 to 10.1.4.56:39990 remote addr 86a35000 rkey 109d26 nic_wr_credit 6. [D 20:01:05.824652] openib_post_rr: 10.1.4.56:39990 bh 39 wr_id 6842c0. [D 20:01:05.825748] openib_check_cq: completed recv wr_id 6842f0. [D 20:01:05.825795] openib_post_sr_rdmaw: ibv_post_send wr_id 6aad30 to 10.1.4.56:39990 remote addr 98e47000 rkey 109226 nic_wr_credit 7. [D 20:01:05.825810] openib_post_rr: 10.1.4.56:39990 bh 40 wr_id 6842f0. [D 20:01:05.826391] openib_check_cq: completed rdma wr_id 2aaabd3036a0, nic_wr_credit 6. [D 20:01:05.826406] post_sr: 10.1.4.56:39990 bh 92 len 16 wr_id 6839e0. [D 20:01:05.826930] openib_check_cq: completed recv wr_id 684320. [D 20:01:05.826949] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3070b0 to 10.1.4.56:39990 remote addr 96e45000 rkey 109426 nic_wr_credit 7. [D 20:01:05.826960] openib_post_rr: 10.1.4.56:39990 bh 41 wr_id 684320. [D 20:01:05.828629] openib_check_cq: completed rdma wr_id 2aaabd3041a0, nic_wr_credit 6. [D 20:01:05.828662] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304db0) [D 20:01:05.828684] post_sr: 10.1.4.56:39990 bh 93 len 16 wr_id 683a10. [D 20:01:05.828704] openib_check_cq: completed send wr_id 683950. [D 20:01:05.828739] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:05.828752] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.828762] ========================================================= [D 20:01:05.828769] PINT_process_request [D 20:01:05.828778] tiling 6 copies [D 20:01:05.828786] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.828794] lvl 0 el 0 blk 0 by 163577856 [D 20:01:05.828801] to 163577856 ta 134217728 fi 167772160 [D 20:01:05.828808] basic type or contiguous data [D 20:01:05.828814] Request disp [D 20:01:05.828821] contig_offset = 163577856 contig_size = 37748736 lvl_flag = 1 [D 20:01:05.828829] calling distribute [D 20:01:05.828837] PINT_distribute [D 20:01:05.828843] of 163577856 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.828953] begin iteration loff: 163577856 [D 20:01:05.828962] segment exceeds byte limit [D 20:01:05.828970] process a segment [D 20:01:05.828976] of 163577856 sz 2097152 [D 20:01:05.828982] add a segment 0 [D 20:01:05.828988] end iteration [D 20:01:05.828995] done with segments or bytes [D 20:01:05.829001] of 165675008 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.829008] next loff: 165675008 next poff: 165675008 [D 20:01:05.829014] return value 2097152 [D 20:01:05.829021] finished [D 20:01:05.829027] retval = 2097152 [D 20:01:05.829033] exiting distribute returned less than expected [D 20:01:05.829040] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 165675008 fo 167772160 eof 0 [D 20:01:05.829047] ========================================================= [D 20:01:05.829075] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.829083] dbpf_bstream_rw_list: mem_offset: 0x2aaab8038010, mem_size: 2097152 [D 20:01:05.829090] dbpf_bstream_rw_list: stream_offset: 163577856, stream_size: 2097152 [D 20:01:05.829099] dbpf_open_cache_get: called [D 20:01:05.829107] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.829114] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.829143] DBPF I/O ops in progress: 9 [D 20:01:05.829151] lio_listio called with the following aiocbs: [D 20:01:05.829158] aiocb_ptr_array[0]: fd: 13, off: 163577856, bytes: 2097152, buf: 0x2aaab8038010, type: 0 [D 20:01:05.829173] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304f00 (handle 2863311531, ret 0) [D 20:01:05.829817] openib_check_cq: completed recv wr_id 684350. [D 20:01:05.829839] openib_post_sr_rdmaw: ibv_post_send wr_id 6ec3a0 to 10.1.4.56:39990 remote addr 9b249000 rkey 109126 nic_wr_credit 7. [D 20:01:05.829852] openib_post_rr: 10.1.4.56:39990 bh 42 wr_id 684350. [D 20:01:05.830843] openib_check_cq: completed rdma wr_id 2aaabd3008d0, nic_wr_credit 6. [D 20:01:05.830893] post_sr: 10.1.4.56:39990 bh 94 len 16 wr_id 683a40. [D 20:01:05.830906] openib_check_cq: completed send wr_id 683980. [D 20:01:05.830941] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152db60. [D 20:01:05.830957] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.830969] ========================================================= [D 20:01:05.830976] PINT_process_request [D 20:01:05.830984] tiling 9 copies [D 20:01:05.830992] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.831001] lvl 0 el 0 blk 0 by 260046848 [D 20:01:05.831008] to 260046848 ta 234881024 fi 268435456 [D 20:01:05.831017] basic type or contiguous data [D 20:01:05.831024] Request disp [D 20:01:05.831031] contig_offset = 260046848 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.831038] calling distribute [D 20:01:05.831045] PINT_distribute [D 20:01:05.831053] of 260046848 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.831062] begin iteration loff: 260046848 [D 20:01:05.831070] segment exceeds byte limit [D 20:01:05.831077] process a segment [D 20:01:05.831083] of 260046848 sz 2097152 [D 20:01:05.831090] add a segment 0 [D 20:01:05.831096] end iteration [D 20:01:05.831103] done with segments or bytes [D 20:01:05.831109] of 262144000 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.831116] next loff: 262144000 next poff: 262144000 [D 20:01:05.831124] return value 2097152 [D 20:01:05.831131] finished [D 20:01:05.831137] retval = 2097152 [D 20:01:05.831144] exiting distribute returned less than expected [D 20:01:05.831151] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 262144000 fo 268435456 eof 0 [D 20:01:05.831158] ========================================================= [D 20:01:05.831178] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.831188] dbpf_bstream_rw_list: mem_offset: 0x2aaabc85d010, mem_size: 2097152 [D 20:01:05.831195] dbpf_bstream_rw_list: stream_offset: 260046848, stream_size: 2097152 [D 20:01:05.831204] dbpf_open_cache_get: called [D 20:01:05.831211] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.831218] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.831250] DBPF I/O ops in progress: 10 [D 20:01:05.831260] lio_listio called with the following aiocbs: [D 20:01:05.831266] aiocb_ptr_array[0]: fd: 13, off: 260046848, bytes: 2097152, buf: 0x2aaabc85d010, type: 0 [D 20:01:05.831283] issue_or_delay_io_operation: lio_listio posted 0x2aaabd301b70 (handle 2863311531, ret 0) [D 20:01:05.832052] openib_check_cq: completed recv wr_id 684380. [D 20:01:05.832093] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd300d10 to 10.1.4.56:39990 remote addr 94e43000 rkey 109526 nic_wr_credit 7. [D 20:01:05.832107] openib_post_rr: 10.1.4.56:39990 bh 43 wr_id 684380. [D 20:01:05.833067] openib_check_cq: completed rdma wr_id 2aaabd31d050, nic_wr_credit 6. [D 20:01:05.833120] post_sr: 10.1.4.56:39990 bh 95 len 16 wr_id 683a70. [D 20:01:05.835296] openib_check_cq: completed rdma wr_id 2aaabd31db50, nic_wr_credit 5. [D 20:01:05.835369] post_sr: 10.1.4.56:39990 bh 96 len 16 wr_id 683aa0. [D 20:01:05.835384] openib_check_cq: completed send wr_id 6839b0. [D 20:01:05.835422] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1505890. [D 20:01:05.835440] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.835452] ========================================================= [D 20:01:05.835460] PINT_process_request [D 20:01:05.835469] tiling 10 copies [D 20:01:05.835477] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.835485] lvl 0 el 0 blk 0 by 293601280 [D 20:01:05.835492] to 293601280 ta 268435456 fi 301989888 [D 20:01:05.835501] basic type or contiguous data [D 20:01:05.835509] Request disp [D 20:01:05.835516] contig_offset = 293601280 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.835523] calling distribute [D 20:01:05.835530] PINT_distribute [D 20:01:05.835536] of 293601280 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.835546] begin iteration loff: 293601280 [D 20:01:05.835554] segment exceeds byte limit [D 20:01:05.835561] process a segment [D 20:01:05.835567] of 293601280 sz 2097152 [D 20:01:05.835574] add a segment 0 [D 20:01:05.835581] end iteration [D 20:01:05.835587] done with segments or bytes [D 20:01:05.835593] of 295698432 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.835601] next loff: 295698432 next poff: 295698432 [D 20:01:05.835608] return value 2097152 [D 20:01:05.835614] finished [D 20:01:05.835620] retval = 2097152 [D 20:01:05.835626] exiting distribute returned less than expected [D 20:01:05.835633] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 295698432 fo 301989888 eof 0 [D 20:01:05.835641] ========================================================= [D 20:01:05.835663] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.835673] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf6d3010, mem_size: 2097152 [D 20:01:05.835680] dbpf_bstream_rw_list: stream_offset: 293601280, stream_size: 2097152 [D 20:01:05.835689] dbpf_open_cache_get: called [D 20:01:05.835697] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.835704] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.835734] DBPF I/O ops in progress: 11 [D 20:01:05.835745] lio_listio called with the following aiocbs: [D 20:01:05.835752] aiocb_ptr_array[0]: fd: 13, off: 293601280, bytes: 2097152, buf: 0x2aaaaf6d3010, type: 0 [D 20:01:05.835767] issue_or_delay_io_operation: lio_listio posted 0x2aaabd300a10 (handle 2863311531, ret 0) [D 20:01:05.836500] openib_check_cq: completed recv wr_id 6843b0. [D 20:01:05.836545] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd302a70 to 10.1.4.56:39990 remote addr 92e41000 rkey 109626 nic_wr_credit 6. [D 20:01:05.836556] openib_post_rr: 10.1.4.56:39990 bh 44 wr_id 6843b0. [D 20:01:05.836652] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:05.836681] BMI_post_send: addr: 6843072, offset: 0x2aaab0edf010, size: 2097152, tag: 29 [D 20:01:05.828650] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304040) [D 20:01:05.836896] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.836906] dbpf_open_cache_put: cached entry. [D 20:01:05.836913] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.836920] op_queue add: 0x2aaabd304040 [D 20:01:05.836927] DBPF I/O ops in progress: 10 [D 20:01:05.828657] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304bf0) [D 20:01:05.836965] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.836976] dbpf_open_cache_put: cached entry. [D 20:01:05.836983] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.836990] op_queue add: 0x2aaabd304bf0 [D 20:01:05.836997] DBPF I/O ops in progress: 9 [D 20:01:05.828687] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.837024] dbpf_open_cache_put: cached entry. [D 20:01:05.837031] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.837037] op_queue add: 0x2aaabd304db0 [D 20:01:05.837045] DBPF I/O ops in progress: 8 [D 20:01:05.828638] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3016e0) [D 20:01:05.837070] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.837080] dbpf_open_cache_put: cached entry. [D 20:01:05.837086] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.837093] op_queue add: 0x2aaabd3016e0 [D 20:01:05.837100] DBPF I/O ops in progress: 7 [D 20:01:05.837121] --- aio_progress_notification called with handle 2863311531 (0x2aaabd31cef0) [D 20:01:05.837138] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.837151] dbpf_open_cache_put: cached entry. [D 20:01:05.837158] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.837165] op_queue add: 0x2aaabd31cef0 [D 20:01:05.837171] DBPF I/O ops in progress: 6 [D 20:01:05.837207] --- aio_progress_notification called with handle 2863311531 (0x2aaabd31d9f0) [D 20:01:05.837222] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.837235] dbpf_open_cache_put: cached entry. [D 20:01:05.837242] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.837248] op_queue add: 0x2aaabd31d9f0 [D 20:01:05.837254] DBPF I/O ops in progress: 5 [D 20:01:05.837281] --- aio_progress_notification called with handle 2863311531 (0x2aaabd31cad0) [D 20:01:05.837296] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.837308] dbpf_open_cache_put: cached entry. [D 20:01:05.837315] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.837321] op_queue add: 0x2aaabd31cad0 [D 20:01:05.837328] DBPF I/O ops in progress: 4 [D 20:01:05.837353] --- aio_progress_notification called with handle 2863311531 (0x2aaabd30a300) [D 20:01:05.837367] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.837379] dbpf_open_cache_put: cached entry. [D 20:01:05.837386] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.837392] op_queue add: 0x2aaabd30a300 [D 20:01:05.837399] DBPF I/O ops in progress: 3 [D 20:01:05.837511] openib_check_cq: completed rdma wr_id 2aaabd31e650, nic_wr_credit 5. [D 20:01:05.837526] post_sr: 10.1.4.56:39990 bh 97 len 16 wr_id 683ad0. [D 20:01:05.839737] openib_check_cq: completed rdma wr_id 6aad30, nic_wr_credit 4. [D 20:01:05.839803] post_sr: 10.1.4.56:39990 bh 98 len 16 wr_id 683b00. [D 20:01:05.839834] openib_check_cq: completed send wr_id 6839e0. [D 20:01:05.839879] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c210. [D 20:01:05.839896] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.839909] ========================================================= [D 20:01:05.839917] PINT_process_request [D 20:01:05.839925] tiling 12 copies [D 20:01:05.839935] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.839944] lvl 0 el 0 blk 0 by 358612992 [D 20:01:05.839950] to 358612992 ta 335544320 fi 369098752 [D 20:01:05.839959] basic type or contiguous data [D 20:01:05.839966] Request disp [D 20:01:05.839974] contig_offset = 358612992 contig_size = 44040192 lvl_flag = 1 [D 20:01:05.839983] calling distribute [D 20:01:05.839989] PINT_distribute [D 20:01:05.839999] of 358612992 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.840010] begin iteration loff: 358612992 [D 20:01:05.840019] segment exceeds byte limit [D 20:01:05.840027] process a segment [D 20:01:05.840032] of 358612992 sz 2097152 [D 20:01:05.840039] add a segment 0 [D 20:01:05.840046] end iteration [D 20:01:05.840052] done with segments or bytes [D 20:01:05.840058] of 360710144 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.840066] next loff: 360710144 next poff: 360710144 [D 20:01:05.840073] return value 2097152 [D 20:01:05.840079] finished [D 20:01:05.840085] retval = 2097152 [D 20:01:05.840092] exiting distribute returned less than expected [D 20:01:05.840098] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 360710144 fo 369098752 eof 0 [D 20:01:05.840106] ========================================================= [D 20:01:05.840128] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.840139] dbpf_bstream_rw_list: mem_offset: 0x2aaab1801010, mem_size: 2097152 [D 20:01:05.840146] dbpf_bstream_rw_list: stream_offset: 358612992, stream_size: 2097152 [D 20:01:05.840155] dbpf_open_cache_get: called [D 20:01:05.840162] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.840170] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.840198] DBPF I/O ops in progress: 4 [D 20:01:05.840209] lio_listio called with the following aiocbs: [D 20:01:05.840216] aiocb_ptr_array[0]: fd: 13, off: 358612992, bytes: 2097152, buf: 0x2aaab1801010, type: 0 [D 20:01:05.840230] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3036a0 (handle 2863311531, ret 0) [D 20:01:05.840954] openib_check_cq: completed recv wr_id 6843e0. [D 20:01:05.840992] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd303800 to 10.1.4.56:39990 remote addr 8ec3d000 rkey 109926 nic_wr_credit 5. [D 20:01:05.841005] openib_post_rr: 10.1.4.56:39990 bh 45 wr_id 6843e0. [D 20:01:05.841958] openib_check_cq: completed rdma wr_id 2aaabd3070b0, nic_wr_credit 4. [D 20:01:05.841983] post_sr: 10.1.4.56:39990 bh 99 len 16 wr_id 683b30. [D 20:01:05.841996] openib_check_cq: completed send wr_id 683a10. [D 20:01:05.842023] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b470. [D 20:01:05.842037] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.842047] ========================================================= [D 20:01:05.842053] PINT_process_request [D 20:01:05.842063] tiling 11 copies [D 20:01:05.842072] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.842080] lvl 0 el 0 blk 0 by 325058560 [D 20:01:05.842087] to 325058560 ta 301989888 fi 335544320 [D 20:01:05.842095] basic type or contiguous data [D 20:01:05.842102] Request disp [D 20:01:05.842109] contig_offset = 325058560 contig_size = 44040192 lvl_flag = 1 [D 20:01:05.842117] calling distribute [D 20:01:05.842124] PINT_distribute [D 20:01:05.842130] of 325058560 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.842140] begin iteration loff: 325058560 [D 20:01:05.842147] segment exceeds byte limit [D 20:01:05.842155] process a segment [D 20:01:05.842161] of 325058560 sz 2097152 [D 20:01:05.842168] add a segment 0 [D 20:01:05.842175] end iteration [D 20:01:05.842182] done with segments or bytes [D 20:01:05.842188] of 327155712 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.842195] next loff: 327155712 next poff: 327155712 [D 20:01:05.842202] return value 2097152 [D 20:01:05.842209] finished [D 20:01:05.842215] retval = 2097152 [D 20:01:05.842221] exiting distribute returned less than expected [D 20:01:05.842227] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 327155712 fo 335544320 eof 0 [D 20:01:05.842236] ========================================================= [D 20:01:05.842255] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.842265] dbpf_bstream_rw_list: mem_offset: 0x2aaab1a02010, mem_size: 2097152 [D 20:01:05.842275] dbpf_bstream_rw_list: stream_offset: 325058560, stream_size: 2097152 [D 20:01:05.842283] dbpf_open_cache_get: called [D 20:01:05.842291] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.842298] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.842327] DBPF I/O ops in progress: 5 [D 20:01:05.842336] lio_listio called with the following aiocbs: [D 20:01:05.842342] aiocb_ptr_array[0]: fd: 13, off: 325058560, bytes: 2097152, buf: 0x2aaab1a02010, type: 0 [D 20:01:05.842359] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3019a0 (handle 2863311531, ret 0) [D 20:01:05.843191] openib_check_cq: completed recv wr_id 684410. [D 20:01:05.843238] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd304300 to 10.1.4.56:39990 remote addr 90c3f000 rkey 109826 nic_wr_credit 5. [D 20:01:05.843252] openib_post_rr: 10.1.4.56:39990 bh 46 wr_id 684410. [D 20:01:05.844184] openib_check_cq: completed rdma wr_id 6ec3a0, nic_wr_credit 4. [D 20:01:05.844227] post_sr: 10.1.4.56:39990 bh 0 len 16 wr_id 6828a0. [D 20:01:05.844237] openib_check_cq: completed send wr_id 683a40. [D 20:01:05.844260] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:05.844274] (send callback time) ini posts: 0, pending: 4, last: 1, src_list emtpy: yes [D 20:01:05.845401] openib_check_cq: completed recv wr_id 684440. [D 20:01:05.845463] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd300ff0 to 10.1.4.56:39990 remote addr 9dc4b000 rkey 108f27 nic_wr_credit 5. [D 20:01:05.845480] openib_post_rr: 10.1.4.56:39990 bh 47 wr_id 684440. [D 20:01:05.846412] openib_check_cq: completed rdma wr_id 2aaabd300d10, nic_wr_credit 4. [D 20:01:05.846453] post_sr: 10.1.4.56:39990 bh 1 len 16 wr_id 6828d0. [D 20:01:05.846466] openib_check_cq: completed send wr_id 683a70. [D 20:01:05.846476] openib_check_cq: completed send wr_id 683aa0. [D 20:01:05.846499] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1506e50. [D 20:01:05.846513] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.846524] ========================================================= [D 20:01:05.846531] PINT_process_request [D 20:01:05.846539] tiling 13 copies [D 20:01:05.846548] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.846556] lvl 0 el 0 blk 0 by 392167424 [D 20:01:05.846563] to 392167424 ta 369098752 fi 402653184 [D 20:01:05.846570] basic type or contiguous data [D 20:01:05.846577] Request disp [D 20:01:05.846583] contig_offset = 392167424 contig_size = 44040192 lvl_flag = 1 [D 20:01:05.846590] calling distribute [D 20:01:05.846596] PINT_distribute [D 20:01:05.846599] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304f00) [D 20:01:05.846621] post_sr: 10.1.4.56:39990 bh 2 len 32 wr_id 682900. [D 20:01:05.846632] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.846648] dbpf_open_cache_put: cached entry. [D 20:01:05.846652] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:05.846661] --- aio_progress_notification called with handle 2863311531 (0x2aaabd301b70) [D 20:01:05.846668] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:05.846677] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300a10) [D 20:01:05.846683] BMI_post_send: addr: 6843072, offset: 0x2aaaba048010, size: 2097152, tag: 27 [D 20:01:05.846672] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.846697] post_sr: 10.1.4.56:39990 bh 3 len 32 wr_id 682930. [D 20:01:05.846703] dbpf_open_cache_put: cached entry. [D 20:01:05.846711] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:05.846688] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.846732] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1505890. [D 20:01:05.846738] dbpf_open_cache_put: cached entry. [D 20:01:05.846745] BMI_post_send: addr: 6843072, offset: 0x2aaab521f010, size: 2097152, tag: 31 [D 20:01:05.846759] post_sr: 10.1.4.56:39990 bh 4 len 32 wr_id 682960. [D 20:01:05.846769] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:05.846779] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c210. [D 20:01:05.846788] BMI_post_send: addr: 6843072, offset: 0x2aaab3410010, size: 2097152, tag: 33 [D 20:01:05.846797] post_sr: 10.1.4.56:39990 bh 5 len 32 wr_id 682990. [D 20:01:05.846806] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:05.846816] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152b470. [D 20:01:05.846825] BMI_post_send: addr: 6843072, offset: 0x2aaab3611010, size: 2097152, tag: 32 [D 20:01:05.846833] post_sr: 10.1.4.56:39990 bh 6 len 32 wr_id 6829c0. [D 20:01:05.846842] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:05.846851] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152db60. [D 20:01:05.846860] BMI_post_send: addr: 6843072, offset: 0x2aaab12e1010, size: 2097152, tag: 30 [D 20:01:05.846868] post_sr: 10.1.4.56:39990 bh 7 len 32 wr_id 6829f0. [D 20:01:05.846878] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:05.846603] of 392167424 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:05.846902] begin iteration loff: 392167424 [D 20:01:05.846910] segment exceeds byte limit [D 20:01:05.846917] process a segment [D 20:01:05.846923] of 392167424 sz 2097152 [D 20:01:05.846930] add a segment 0 [D 20:01:05.846936] end iteration [D 20:01:05.846943] done with segments or bytes [D 20:01:05.846949] of 394264576 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.846957] next loff: 394264576 next poff: 394264576 [D 20:01:05.846964] return value 2097152 [D 20:01:05.846970] finished [D 20:01:05.846977] retval = 2097152 [D 20:01:05.846984] exiting distribute returned less than expected [D 20:01:05.846991] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 394264576 fo 402653184 eof 0 [D 20:01:05.846999] ========================================================= [D 20:01:05.847012] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.847020] dbpf_bstream_rw_list: mem_offset: 0x2aaab1600010, mem_size: 2097152 [D 20:01:05.847027] dbpf_bstream_rw_list: stream_offset: 392167424, stream_size: 2097152 [D 20:01:05.847036] dbpf_open_cache_get: called [D 20:01:05.847043] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.847050] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.847078] DBPF I/O ops in progress: 6 [D 20:01:05.847085] lio_listio called with the following aiocbs: [D 20:01:05.847092] aiocb_ptr_array[0]: fd: 13, off: 392167424, bytes: 2097152, buf: 0x2aaab1600010, type: 0 [D 20:01:05.847105] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31cef0 (handle 2863311531, ret 0) [D 20:01:05.847117] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1506e50. [D 20:01:05.847127] BMI_post_send: addr: 6843072, offset: 0x2aaab320f010, size: 2097152, tag: 34 [D 20:01:05.847136] post_sr: 10.1.4.56:39990 bh 8 len 32 wr_id 682a20. [D 20:01:05.847148] trove_read_callback_fn: (post send time) ini posts: 0, pending: 7, last: 0 [D 20:01:05.847164] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:05.847175] BMI_post_send: addr: 6843072, offset: 0x2aaab2e0c010, size: 2097152, tag: 35 [D 20:01:05.847184] post_sr: 10.1.4.56:39990 bh 9 len 32 wr_id 682a50. [D 20:01:05.847193] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:05.847203] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:05.847212] BMI_post_send: addr: 6843072, offset: 0x2aaaad2c1010, size: 2097152, tag: 28 [D 20:01:05.847221] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1507b90. [D 20:01:05.919665] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.919694] ========================================================= [D 20:01:05.919701] PINT_process_request [D 20:01:05.919708] tiling 14 copies [D 20:01:05.919715] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.919723] lvl 0 el 0 blk 0 by 425721856 [D 20:01:05.919729] to 425721856 ta 402653184 fi 436207616 [D 20:01:05.919736] basic type or contiguous data [D 20:01:05.919742] Request disp [D 20:01:05.919749] contig_offset = 425721856 contig_size = 44040192 lvl_flag = 1 [D 20:01:05.919756] calling distribute [D 20:01:05.919761] PINT_distribute [D 20:01:05.919768] of 425721856 sz 10485760 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:05.919776] begin iteration loff: 425721856 [D 20:01:05.919783] segment exceeds byte limit [D 20:01:05.919789] process a segment [D 20:01:05.919795] of 425721856 sz 2097152 [D 20:01:05.919801] add a segment 0 [D 20:01:05.919807] end iteration [D 20:01:05.919813] done with segments or bytes [D 20:01:05.919823] of 427819008 sz 8388608 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.919830] next loff: 427819008 next poff: 427819008 [D 20:01:05.919836] return value 2097152 [D 20:01:05.919843] finished [D 20:01:05.919849] retval = 2097152 [D 20:01:05.919855] exiting distribute returned less than expected [D 20:01:05.919861] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 427819008 fo 436207616 eof 0 [D 20:01:05.919868] ========================================================= [D 20:01:05.919880] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.919888] dbpf_bstream_rw_list: mem_offset: 0x2aaab10e0010, mem_size: 2097152 [D 20:01:05.919895] dbpf_bstream_rw_list: stream_offset: 425721856, stream_size: 2097152 [D 20:01:05.919902] dbpf_open_cache_get: called [D 20:01:05.919909] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.919915] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.919931] DBPF I/O ops in progress: 7 [D 20:01:05.919938] lio_listio called with the following aiocbs: [D 20:01:05.919944] aiocb_ptr_array[0]: fd: 13, off: 425721856, bytes: 2097152, buf: 0x2aaab10e0010, type: 0 [D 20:01:05.919956] issue_or_delay_io_operation: lio_listio posted 0x2aaabd3027b0 (handle 2863311531, ret 0) [D 20:01:05.919978] openib_check_cq: completed recv wr_id 684470. [D 20:01:05.919997] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31d1b0 to 10.1.4.56:39990 remote addr 8cc3b000 rkey 109a26 nic_wr_credit 5. [D 20:01:05.920009] openib_post_rr: 10.1.4.56:39990 bh 48 wr_id 684470. [D 20:01:05.920020] openib_check_cq: completed rdma wr_id 2aaabd302a70, nic_wr_credit 4. [D 20:01:05.920032] post_sr: 10.1.4.56:39990 bh 10 len 16 wr_id 682a80. [D 20:01:05.920040] openib_check_cq: completed send wr_id 683ad0. [D 20:01:05.920048] openib_check_cq: completed send wr_id 683b00. [D 20:01:05.920056] openib_check_cq: completed recv wr_id 6844a0. [D 20:01:05.920066] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31dcb0 to 10.1.4.56:39990 remote addr 8ac39000 rkey 109b26 nic_wr_credit 5. [D 20:01:05.920074] openib_post_rr: 10.1.4.56:39990 bh 49 wr_id 6844a0. [D 20:01:05.920082] openib_check_cq: completed recv wr_id 6844d0. [D 20:01:05.920090] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd300bb0 to 10.1.4.56:39990 remote addr 99047000 rkey 109226 nic_wr_credit 6. [D 20:01:05.920098] openib_post_rr: 10.1.4.56:39990 bh 50 wr_id 6844d0. [D 20:01:05.920106] openib_check_cq: completed rdma wr_id 2aaabd303800, nic_wr_credit 5. [D 20:01:05.920114] post_sr: 10.1.4.56:39990 bh 11 len 16 wr_id 682ab0. [D 20:01:05.920122] openib_check_cq: completed send wr_id 683b30. [D 20:01:05.920130] openib_check_cq: completed recv wr_id 684500. [D 20:01:05.920140] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31e7b0 to 10.1.4.56:39990 remote addr 86c35000 rkey 109d26 nic_wr_credit 6. [D 20:01:05.920150] openib_post_rr: 10.1.4.56:39990 bh 51 wr_id 684500. [D 20:01:05.920158] openib_check_cq: completed recv wr_id 684530. [D 20:01:05.920166] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd307370 to 10.1.4.56:39990 remote addr 97045000 rkey 109426 nic_wr_credit 7. [D 20:01:05.920174] openib_post_rr: 10.1.4.56:39990 bh 52 wr_id 684530. [D 20:01:05.920182] openib_check_cq: completed rdma wr_id 2aaabd304300, nic_wr_credit 6. [D 20:01:05.920190] post_sr: 10.1.4.56:39990 bh 12 len 16 wr_id 682ae0. [D 20:01:05.920198] openib_check_cq: completed send wr_id 6828a0. [D 20:01:05.920205] openib_check_cq: completed recv wr_id 684560. [D 20:01:05.920214] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd300da0 to 10.1.4.56:39990 remote addr 9b449000 rkey 109126 nic_wr_credit 7. [D 20:01:05.920222] openib_post_rr: 10.1.4.56:39990 bh 53 wr_id 684560. [D 20:01:05.920230] openib_check_cq: completed rdma wr_id 2aaabd300ff0, nic_wr_credit 6. [D 20:01:05.920239] post_sr: 10.1.4.56:39990 bh 13 len 16 wr_id 682b10. [D 20:01:05.920246] openib_check_cq: completed send wr_id 6828d0. [D 20:01:05.920253] openib_check_cq: completed send wr_id 682900. [D 20:01:05.920261] openib_check_cq: completed send wr_id 682930. [D 20:01:05.920268] openib_check_cq: completed send wr_id 682960. [D 20:01:05.920275] openib_check_cq: completed send wr_id 682990. [D 20:01:05.920282] openib_check_cq: completed send wr_id 6829c0. [D 20:01:05.920290] openib_check_cq: completed send wr_id 6829f0. [D 20:01:05.920297] openib_check_cq: completed send wr_id 682a20. [D 20:01:05.920305] openib_check_cq: completed send wr_id 682a50. [D 20:01:05.920312] openib_check_cq: completed recv wr_id 684590. [D 20:01:05.920319] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3020e0 to 10.1.4.56:39990 remote addr 95043000 rkey 109526 nic_wr_credit 7. [D 20:01:05.920328] openib_post_rr: 10.1.4.56:39990 bh 54 wr_id 684590. [D 20:01:05.920366] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:05.920377] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.920385] ========================================================= [D 20:01:05.920391] PINT_process_request [D 20:01:05.920398] tiling 6 copies [D 20:01:05.920404] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.920412] lvl 0 el 0 blk 0 by 165675008 [D 20:01:05.920418] to 165675008 ta 134217728 fi 167772160 [D 20:01:05.920425] basic type or contiguous data [D 20:01:05.920431] Request disp [D 20:01:05.920438] contig_offset = 165675008 contig_size = 35651584 lvl_flag = 1 [D 20:01:05.920445] calling distribute [D 20:01:05.920451] PINT_distribute [D 20:01:05.920457] of 165675008 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.920465] begin iteration loff: 165675008 [D 20:01:05.920474] process a segment [D 20:01:05.920480] of 165675008 sz 2097152 [D 20:01:05.920486] add a segment 0 [D 20:01:05.920493] end iteration [D 20:01:05.920499] done with segments or bytes [D 20:01:05.920505] of 167772160 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.920513] next loff: 167772160 next poff: 167772160 [D 20:01:05.920519] (return value) 2097152 [D 20:01:05.920526] finished [D 20:01:05.920532] retval = 2097152 [D 20:01:05.920538] exiting distribute returned less than expected [D 20:01:05.920545] done sg 1 sm 64 by 2097152 bm 2097152 ta 134217728 to 167772160 fo 167772160 eof 0 [D 20:01:05.920553] ========================================================= [D 20:01:05.920562] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.920569] dbpf_bstream_rw_list: mem_offset: 0x2aaab642a010, mem_size: 2097152 [D 20:01:05.920576] dbpf_bstream_rw_list: stream_offset: 165675008, stream_size: 2097152 [D 20:01:05.920583] dbpf_open_cache_get: called [D 20:01:05.920589] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.920597] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.920623] DBPF I/O ops in progress: 8 [D 20:01:05.920632] lio_listio called with the following aiocbs: [D 20:01:05.920638] aiocb_ptr_array[0]: fd: 13, off: 165675008, bytes: 2097152, buf: 0x2aaab642a010, type: 0 [D 20:01:05.920648] issue_or_delay_io_operation: lio_listio posted 0x2aaabd30a300 (handle 2863311531, ret 0) [D 20:01:05.920675] post_sr: 10.1.4.56:39990 bh 14 len 32 wr_id 682b40. [D 20:01:05.920709] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:05.920725] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1517400. [D 20:01:05.920735] BMI_post_send: addr: 6843072, offset: 0x2aaaaf2d1010, size: 2097152, tag: 29 [D 20:01:05.920745] post_sr: 10.1.4.56:39990 bh 15 len 32 wr_id 682b70. [D 20:01:05.920754] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:05.920772] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c750. [D 20:01:05.920782] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.920789] ========================================================= [D 20:01:05.920796] PINT_process_request [D 20:01:05.920802] tiling 7 copies [D 20:01:05.920809] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.920816] lvl 0 el 0 blk 0 by 195035136 [D 20:01:05.920822] to 195035136 ta 167772160 fi 201326592 [D 20:01:05.920829] basic type or contiguous data [D 20:01:05.920835] Request disp [D 20:01:05.920841] contig_offset = 195035136 contig_size = 39845888 lvl_flag = 1 [D 20:01:05.920849] calling distribute [D 20:01:05.920855] PINT_distribute [D 20:01:05.920861] of 195035136 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.920868] begin iteration loff: 195035136 [D 20:01:05.920875] segment exceeds byte limit [D 20:01:05.920881] process a segment [D 20:01:05.920887] of 195035136 sz 2097152 [D 20:01:05.920893] add a segment 0 [D 20:01:05.920899] end iteration [D 20:01:05.920905] done with segments or bytes [D 20:01:05.920911] of 197132288 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.920918] next loff: 197132288 next poff: 197132288 [D 20:01:05.920925] return value 2097152 [D 20:01:05.920931] finished [D 20:01:05.920937] retval = 2097152 [D 20:01:05.920943] exiting distribute returned less than expected [D 20:01:05.920950] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 197132288 fo 201326592 eof 0 [D 20:01:05.920958] ========================================================= [D 20:01:05.920966] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.920973] dbpf_bstream_rw_list: mem_offset: 0x2aaab9e47010, mem_size: 2097152 [D 20:01:05.920980] dbpf_bstream_rw_list: stream_offset: 195035136, stream_size: 2097152 [D 20:01:05.920986] dbpf_open_cache_get: called [D 20:01:05.920993] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.921000] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.921021] DBPF I/O ops in progress: 9 [D 20:01:05.921029] lio_listio called with the following aiocbs: [D 20:01:05.921035] aiocb_ptr_array[0]: fd: 13, off: 195035136, bytes: 2097152, buf: 0x2aaab9e47010, type: 0 [D 20:01:05.921044] issue_or_delay_io_operation: lio_listio posted 0x2aaabd302900 (handle 2863311531, ret 0) [D 20:01:05.921054] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517400. [D 20:01:05.921062] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.921069] ========================================================= [D 20:01:05.921075] PINT_process_request [D 20:01:05.921082] tiling 8 copies [D 20:01:05.921088] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.921095] lvl 0 el 0 blk 0 by 228589568 [D 20:01:05.921102] to 228589568 ta 201326592 fi 234881024 [D 20:01:05.921111] basic type or contiguous data [D 20:01:05.921117] Request disp [D 20:01:05.921123] contig_offset = 228589568 contig_size = 39845888 lvl_flag = 1 [D 20:01:05.921130] calling distribute [D 20:01:05.921136] PINT_distribute [D 20:01:05.921142] of 228589568 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.921150] begin iteration loff: 228589568 [D 20:01:05.921156] segment exceeds byte limit [D 20:01:05.921162] process a segment [D 20:01:05.921168] of 228589568 sz 2097152 [D 20:01:05.921174] add a segment 0 [D 20:01:05.921180] end iteration [D 20:01:05.921187] done with segments or bytes [D 20:01:05.921193] of 230686720 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.921200] next loff: 230686720 next poff: 230686720 [D 20:01:05.921207] return value 2097152 [D 20:01:05.921213] finished [D 20:01:05.921218] retval = 2097152 [D 20:01:05.921225] exiting distribute returned less than expected [D 20:01:05.921231] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 230686720 fo 234881024 eof 0 [D 20:01:05.921238] ========================================================= [D 20:01:05.921247] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.921254] dbpf_bstream_rw_list: mem_offset: 0x2aaaad6c3010, mem_size: 2097152 [D 20:01:05.921260] dbpf_bstream_rw_list: stream_offset: 228589568, stream_size: 2097152 [D 20:01:05.921267] dbpf_open_cache_get: called [D 20:01:05.921273] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.921280] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.921303] DBPF I/O ops in progress: 10 [D 20:01:05.921311] lio_listio called with the following aiocbs: [D 20:01:05.921317] aiocb_ptr_array[0]: fd: 13, off: 228589568, bytes: 2097152, buf: 0x2aaaad6c3010, type: 0 [D 20:01:05.921327] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304bf0 (handle 2863311531, ret 0) [D 20:01:05.921336] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152db60. [D 20:01:05.921345] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.921353] ========================================================= [D 20:01:05.921359] PINT_process_request [D 20:01:05.921366] tiling 9 copies [D 20:01:05.921372] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.921380] lvl 0 el 0 blk 0 by 262144000 [D 20:01:05.921386] to 262144000 ta 234881024 fi 268435456 [D 20:01:05.921393] basic type or contiguous data [D 20:01:05.921400] Request disp [D 20:01:05.921406] contig_offset = 262144000 contig_size = 39845888 lvl_flag = 1 [D 20:01:05.921413] calling distribute [D 20:01:05.921419] PINT_distribute [D 20:01:05.921425] of 262144000 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.921433] begin iteration loff: 262144000 [D 20:01:05.921439] segment exceeds byte limit [D 20:01:05.921445] process a segment [D 20:01:05.921451] of 262144000 sz 2097152 [D 20:01:05.921458] add a segment 0 [D 20:01:05.921464] end iteration [D 20:01:05.921470] done with segments or bytes [D 20:01:05.921476] of 264241152 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.921483] next loff: 264241152 next poff: 264241152 [D 20:01:05.921489] return value 2097152 [D 20:01:05.921495] finished [D 20:01:05.921501] retval = 2097152 [D 20:01:05.921507] exiting distribute returned less than expected [D 20:01:05.921514] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 264241152 fo 268435456 eof 0 [D 20:01:05.921522] ========================================================= [D 20:01:05.921530] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.921537] dbpf_bstream_rw_list: mem_offset: 0x2aaabb453010, mem_size: 2097152 [D 20:01:05.921543] dbpf_bstream_rw_list: stream_offset: 262144000, stream_size: 2097152 [D 20:01:05.921550] dbpf_open_cache_get: called [D 20:01:05.921556] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.921563] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.921584] DBPF I/O ops in progress: 11 [D 20:01:05.921591] lio_listio called with the following aiocbs: [D 20:01:05.921598] aiocb_ptr_array[0]: fd: 13, off: 262144000, bytes: 2097152, buf: 0x2aaabb453010, type: 0 [D 20:01:05.921606] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304d40 (handle 2863311531, ret 0) [D 20:01:05.921616] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab15d5af0. [D 20:01:05.921624] (send callback time) ini posts: 0, pending: 1, last: 1, src_list emtpy: yes [D 20:01:05.922294] openib_check_cq: completed rdma wr_id 2aaabd31d1b0, nic_wr_credit 6. [D 20:01:05.922327] post_sr: 10.1.4.56:39990 bh 16 len 16 wr_id 682ba0. [D 20:01:05.922340] openib_check_cq: completed send wr_id 682a80. [D 20:01:05.922356] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1505890. [D 20:01:05.922368] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.922376] ========================================================= [D 20:01:05.922382] PINT_process_request [D 20:01:05.922391] tiling 10 copies [D 20:01:05.922399] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.922407] lvl 0 el 0 blk 0 by 295698432 [D 20:01:05.922413] to 295698432 ta 268435456 fi 301989888 [D 20:01:05.922438] basic type or contiguous data [D 20:01:05.922446] Request disp [D 20:01:05.922453] contig_offset = 295698432 contig_size = 39845888 lvl_flag = 1 [D 20:01:05.922460] calling distribute [D 20:01:05.922466] PINT_distribute [D 20:01:05.922472] of 295698432 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.922481] begin iteration loff: 295698432 [D 20:01:05.922488] segment exceeds byte limit [D 20:01:05.922495] process a segment [D 20:01:05.922501] of 295698432 sz 2097152 [D 20:01:05.922508] add a segment 0 [D 20:01:05.922514] end iteration [D 20:01:05.922520] done with segments or bytes [D 20:01:05.922526] of 297795584 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.922533] next loff: 297795584 next poff: 297795584 [D 20:01:05.922540] return value 2097152 [D 20:01:05.922563] finished [D 20:01:05.922570] retval = 2097152 [D 20:01:05.922576] exiting distribute returned less than expected [D 20:01:05.922583] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 297795584 fo 301989888 eof 0 [D 20:01:05.922590] ========================================================= [D 20:01:05.922603] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.922610] dbpf_bstream_rw_list: mem_offset: 0x2aaaadac5010, mem_size: 2097152 [D 20:01:05.922617] dbpf_bstream_rw_list: stream_offset: 295698432, stream_size: 2097152 [D 20:01:05.922625] dbpf_open_cache_get: called [D 20:01:05.922631] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.922639] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.922664] DBPF I/O ops in progress: 12 [D 20:01:05.923362] lio_listio called with the following aiocbs: [D 20:01:05.923388] aiocb_ptr_array[0]: fd: 13, off: 295698432, bytes: 2097152, buf: 0x2aaaadac5010, type: 0 [D 20:01:05.923401] issue_or_delay_io_operation: lio_listio posted 0x2aaabd302a50 (handle 2863311531, ret 0) [D 20:01:05.923472] openib_check_cq: completed recv wr_id 6845c0. [D 20:01:05.923487] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd302bd0 to 10.1.4.56:39990 remote addr 93041000 rkey 109626 nic_wr_credit 7. [D 20:01:05.923499] openib_post_rr: 10.1.4.56:39990 bh 55 wr_id 6845c0. [D 20:01:05.924468] openib_check_cq: completed rdma wr_id 2aaabd31dcb0, nic_wr_credit 6. [D 20:01:05.924513] post_sr: 10.1.4.56:39990 bh 17 len 16 wr_id 682bd0. [D 20:01:05.925038] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3027b0) [D 20:01:05.925067] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.925084] dbpf_open_cache_put: cached entry. [D 20:01:05.925092] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.925099] op_queue add: 0x2aaabd3027b0 [D 20:01:05.925113] DBPF I/O ops in progress: 11 [D 20:01:05.925123] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab1507b90. [D 20:01:05.925139] BMI_post_send: addr: 6843072, offset: 0x2aaab10e0010, size: 2097152, tag: 35 [D 20:01:05.925145] --- aio_progress_notification called with handle 2863311531 (0x2aaabd30a300) [D 20:01:05.925161] post_sr: 10.1.4.56:39990 bh 18 len 32 wr_id 682c00. [D 20:01:05.925202] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:05.926694] openib_check_cq: completed rdma wr_id 2aaabd300bb0, nic_wr_credit 5. [D 20:01:05.926767] post_sr: 10.1.4.56:39990 bh 19 len 16 wr_id 682c30. [D 20:01:05.926781] openib_check_cq: completed send wr_id 682ab0. [D 20:01:05.926814] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c210. [D 20:01:05.926828] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.926839] ========================================================= [D 20:01:05.926846] PINT_process_request [D 20:01:05.926854] tiling 12 copies [D 20:01:05.926863] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.926872] lvl 0 el 0 blk 0 by 360710144 [D 20:01:05.926878] to 360710144 ta 335544320 fi 369098752 [D 20:01:05.926886] basic type or contiguous data [D 20:01:05.926893] Request disp [D 20:01:05.926899] contig_offset = 360710144 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.926907] calling distribute [D 20:01:05.926914] PINT_distribute [D 20:01:05.926920] of 360710144 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.926930] begin iteration loff: 360710144 [D 20:01:05.926938] segment exceeds byte limit [D 20:01:05.926945] process a segment [D 20:01:05.926951] of 360710144 sz 2097152 [D 20:01:05.926958] add a segment 0 [D 20:01:05.926964] end iteration [D 20:01:05.926971] done with segments or bytes [D 20:01:05.926977] of 362807296 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.926984] next loff: 362807296 next poff: 362807296 [D 20:01:05.926992] return value 2097152 [D 20:01:05.926998] finished [D 20:01:05.927004] retval = 2097152 [D 20:01:05.927010] exiting distribute returned less than expected [D 20:01:05.927017] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 362807296 fo 369098752 eof 0 [D 20:01:05.927025] ========================================================= [D 20:01:05.927042] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.927051] dbpf_bstream_rw_list: mem_offset: 0x2aaaafad5010, mem_size: 2097152 [D 20:01:05.927058] dbpf_bstream_rw_list: stream_offset: 360710144, stream_size: 2097152 [D 20:01:05.927067] dbpf_open_cache_get: called [D 20:01:05.927074] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.927082] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.927109] DBPF I/O ops in progress: 12 [D 20:01:05.927117] lio_listio called with the following aiocbs: [D 20:01:05.927124] aiocb_ptr_array[0]: fd: 13, off: 360710144, bytes: 2097152, buf: 0x2aaaafad5010, type: 0 [D 20:01:05.927141] issue_or_delay_io_operation: lio_listio posted 0x2aaabd303800 (handle 2863311531, ret 0) [D 20:01:05.927909] openib_check_cq: completed recv wr_id 6845f0. [D 20:01:05.927945] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd303960 to 10.1.4.56:39990 remote addr 8ee3d000 rkey 109926 nic_wr_credit 6. [D 20:01:05.927958] openib_post_rr: 10.1.4.56:39990 bh 56 wr_id 6845f0. [D 20:01:05.928916] openib_check_cq: completed rdma wr_id 2aaabd31e7b0, nic_wr_credit 5. [D 20:01:05.928962] post_sr: 10.1.4.56:39990 bh 20 len 16 wr_id 682c60. [D 20:01:05.931144] openib_check_cq: completed rdma wr_id 2aaabd307370, nic_wr_credit 4. [D 20:01:05.931223] post_sr: 10.1.4.56:39990 bh 21 len 16 wr_id 682c90. [D 20:01:05.931245] openib_check_cq: completed send wr_id 682ae0. [D 20:01:05.931286] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152b470. [D 20:01:05.931303] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.931316] ========================================================= [D 20:01:05.931323] PINT_process_request [D 20:01:05.931332] tiling 11 copies [D 20:01:05.931340] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.931349] lvl 0 el 0 blk 0 by 327155712 [D 20:01:05.931356] to 327155712 ta 301989888 fi 335544320 [D 20:01:05.931363] basic type or contiguous data [D 20:01:05.931371] Request disp [D 20:01:05.931378] contig_offset = 327155712 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.931386] calling distribute [D 20:01:05.931393] PINT_distribute [D 20:01:05.931400] of 327155712 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.931410] begin iteration loff: 327155712 [D 20:01:05.931417] segment exceeds byte limit [D 20:01:05.931424] process a segment [D 20:01:05.931430] of 327155712 sz 2097152 [D 20:01:05.931437] add a segment 0 [D 20:01:05.931444] end iteration [D 20:01:05.931450] done with segments or bytes [D 20:01:05.931457] of 329252864 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.931464] next loff: 329252864 next poff: 329252864 [D 20:01:05.931471] return value 2097152 [D 20:01:05.931478] finished [D 20:01:05.931484] retval = 2097152 [D 20:01:05.931490] exiting distribute returned less than expected [D 20:01:05.931497] done sg 1 sm 64 by 2097152 bm 2097152 ta 301989888 to 329252864 fo 335544320 eof 0 [D 20:01:05.931505] ========================================================= [D 20:01:05.931528] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.931540] dbpf_bstream_rw_list: mem_offset: 0x2aaaafcd6010, mem_size: 2097152 [D 20:01:05.931548] dbpf_bstream_rw_list: stream_offset: 327155712, stream_size: 2097152 [D 20:01:05.931556] dbpf_open_cache_get: called [D 20:01:05.931564] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.931571] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.931603] DBPF I/O ops in progress: 13 [D 20:01:05.931612] lio_listio called with the following aiocbs: [D 20:01:05.931619] aiocb_ptr_array[0]: fd: 13, off: 327155712, bytes: 2097152, buf: 0x2aaaafcd6010, type: 0 [D 20:01:05.931636] issue_or_delay_io_operation: lio_listio posted 0x2aaabd304300 (handle 2863311531, ret 0) [D 20:01:05.932349] openib_check_cq: completed recv wr_id 684620. [D 20:01:05.932386] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd304460 to 10.1.4.56:39990 remote addr 90e3f000 rkey 109826 nic_wr_credit 5. [D 20:01:05.932400] openib_post_rr: 10.1.4.56:39990 bh 57 wr_id 684620. [D 20:01:05.933365] openib_check_cq: completed rdma wr_id 2aaabd300da0, nic_wr_credit 4. [D 20:01:05.933415] post_sr: 10.1.4.56:39990 bh 22 len 16 wr_id 682cc0. [D 20:01:05.933428] openib_check_cq: completed send wr_id 682b10. [D 20:01:05.933455] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1522750. [D 20:01:05.933470] (send callback time) ini posts: 0, pending: 3, last: 1, src_list emtpy: yes [D 20:01:05.925164] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.934456] dbpf_open_cache_put: cached entry. [D 20:01:05.934464] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.934471] op_queue add: 0x2aaabd30a300 [D 20:01:05.934484] DBPF I/O ops in progress: 12 [D 20:01:05.934497] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab151fe50. [D 20:01:05.934520] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302900) [D 20:01:05.934534] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.934543] dbpf_open_cache_put: cached entry. [D 20:01:05.934550] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.934557] op_queue add: 0x2aaabd302900 [D 20:01:05.934566] DBPF I/O ops in progress: 11 [D 20:01:05.934571] openib_check_cq: completed recv wr_id 684650. [D 20:01:05.934592] flowproto-multiqueue trove_read_callback_fn, error_code: 0, flow: 0x2aaab152c750. [D 20:01:05.934585] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304bf0) [D 20:01:05.934604] BMI_post_send: addr: 6843072, offset: 0x2aaab9e47010, size: 2097152, tag: 28 [D 20:01:05.846655] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.846718] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3036a0) [D 20:01:05.934640] --- aio_progress_notification called with handle 2863311531 (0x2aaabd302a50) [D 20:01:05.934642] op_queue add: 0x2aaabd304f00 [D 20:01:05.846713] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.934660] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.934663] DBPF I/O ops in progress: 10 [D 20:01:05.934680] dbpf_open_cache_put: cached entry. [D 20:01:05.846747] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.934724] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.934740] dbpf_open_cache_put: cached entry. [D 20:01:05.934747] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.851368] --- aio_progress_notification called with handle 2863311531 (0x2aaabd3019a0) [D 20:01:05.936667] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.936760] dbpf_open_cache_put: cached entry. [D 20:01:05.936775] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.851417] --- aio_progress_notification called with handle 2863311531 (0x2aaabd31cef0) [D 20:01:05.936807] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.936831] dbpf_open_cache_put: cached entry. [D 20:01:05.936841] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.934613] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304d40) [D 20:01:05.934587] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd301cf0 to 10.1.4.56:39990 remote addr 9de4b000 rkey 108f27 nic_wr_credit 5. [D 20:01:05.936871] openib_post_rr: 10.1.4.56:39990 bh 58 wr_id 684650. [D 20:01:05.936891] openib_check_cq: completed rdma wr_id 2aaabd3020e0, nic_wr_credit 4. [D 20:01:05.936906] post_sr: 10.1.4.56:39990 bh 23 len 16 wr_id 682cf0. [D 20:01:05.936915] openib_check_cq: completed send wr_id 682b40. [D 20:01:05.936922] openib_check_cq: completed send wr_id 682b70. [D 20:01:05.936930] openib_check_cq: completed send wr_id 682ba0. [D 20:01:05.936939] openib_check_cq: completed recv wr_id 684680. [D 20:01:05.936954] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31d310 to 10.1.4.56:39990 remote addr 8ce3b000 rkey 109a26 nic_wr_credit 5. [D 20:01:05.936963] openib_post_rr: 10.1.4.56:39990 bh 59 wr_id 684680. [D 20:01:05.936989] post_sr: 10.1.4.56:39990 bh 24 len 32 wr_id 682d20. [D 20:01:05.937025] trove_read_callback_fn: (post send time) ini posts: 0, pending: 8, last: 0 [D 20:01:05.937069] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1506e50. [D 20:01:05.937087] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.937097] ========================================================= [D 20:01:05.937104] PINT_process_request [D 20:01:05.937111] tiling 13 copies [D 20:01:05.937120] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.937128] lvl 0 el 0 blk 0 by 394264576 [D 20:01:05.937134] to 394264576 ta 369098752 fi 402653184 [D 20:01:05.937143] basic type or contiguous data [D 20:01:05.937149] Request disp [D 20:01:05.937156] contig_offset = 394264576 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.937166] calling distribute [D 20:01:05.937172] PINT_distribute [D 20:01:05.937179] of 394264576 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:05.937187] begin iteration loff: 394264576 [D 20:01:05.937195] segment exceeds byte limit [D 20:01:05.937202] process a segment [D 20:01:05.937208] of 394264576 sz 2097152 [D 20:01:05.937214] add a segment 0 [D 20:01:05.937221] end iteration [D 20:01:05.937227] done with segments or bytes [D 20:01:05.937233] of 396361728 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.937241] next loff: 396361728 next poff: 396361728 [D 20:01:05.937247] return value 2097152 [D 20:01:05.937254] finished [D 20:01:05.937260] retval = 2097152 [D 20:01:05.937266] exiting distribute returned less than expected [D 20:01:05.937273] done sg 1 sm 64 by 2097152 bm 2097152 ta 369098752 to 396361728 fo 402653184 eof 0 [D 20:01:05.937280] ========================================================= [D 20:01:05.937297] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.937305] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf8d4010, mem_size: 2097152 [D 20:01:05.937312] dbpf_bstream_rw_list: stream_offset: 394264576, stream_size: 2097152 [D 20:01:05.937320] dbpf_open_cache_get: called [D 20:01:05.937327] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.937335] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.937363] DBPF I/O ops in progress: 11 [D 20:01:05.937371] lio_listio called with the following aiocbs: [D 20:01:05.937377] aiocb_ptr_array[0]: fd: 13, off: 394264576, bytes: 2097152, buf: 0x2aaaaf8d4010, type: 0 [D 20:01:05.937391] issue_or_delay_io_operation: lio_listio posted 0x2aaabd300ff0 (handle 2863311531, ret 0) [D 20:01:05.937815] openib_check_cq: completed rdma wr_id 2aaabd302bd0, nic_wr_credit 4. [D 20:01:05.937834] post_sr: 10.1.4.56:39990 bh 25 len 16 wr_id 682d50. [D 20:01:05.937843] openib_check_cq: completed send wr_id 682bd0. [D 20:01:05.937851] openib_check_cq: completed send wr_id 682c00. [D 20:01:05.937858] openib_check_cq: completed send wr_id 682c30. [D 20:01:05.937870] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c750. [D 20:01:05.937883] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.937893] ========================================================= [D 20:01:05.937899] PINT_process_request [D 20:01:05.937906] tiling 7 copies [D 20:01:05.937913] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.937921] lvl 0 el 0 blk 0 by 197132288 [D 20:01:05.937928] to 197132288 ta 167772160 fi 201326592 [D 20:01:05.937936] basic type or contiguous data [D 20:01:05.937942] Request disp [D 20:01:05.937949] contig_offset = 197132288 contig_size = 37748736 lvl_flag = 1 [D 20:01:05.937955] calling distribute [D 20:01:05.937962] PINT_distribute [D 20:01:05.937968] of 197132288 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.937976] begin iteration loff: 197132288 [D 20:01:05.937982] segment exceeds byte limit [D 20:01:05.937989] process a segment [D 20:01:05.937995] of 197132288 sz 2097152 [D 20:01:05.938001] add a segment 0 [D 20:01:05.938008] end iteration [D 20:01:05.938014] done with segments or bytes [D 20:01:05.938020] of 199229440 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.938027] next loff: 199229440 next poff: 199229440 [D 20:01:05.938034] return value 2097152 [D 20:01:05.938040] finished [D 20:01:05.938046] retval = 2097152 [D 20:01:05.938052] exiting distribute returned less than expected [D 20:01:05.938058] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 199229440 fo 201326592 eof 0 [D 20:01:05.938066] ========================================================= [D 20:01:05.938079] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.938086] dbpf_bstream_rw_list: mem_offset: 0x2aaaba44a010, mem_size: 2097152 [D 20:01:05.938095] dbpf_bstream_rw_list: stream_offset: 197132288, stream_size: 2097152 [D 20:01:05.938105] dbpf_open_cache_get: called [D 20:01:05.938111] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.938118] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.938142] DBPF I/O ops in progress: 12 [D 20:01:05.938151] lio_listio called with the following aiocbs: [D 20:01:05.938157] aiocb_ptr_array[0]: fd: 13, off: 197132288, bytes: 2097152, buf: 0x2aaaba44a010, type: 0 [D 20:01:05.938169] issue_or_delay_io_operation: lio_listio posted 0x2aaabd30a300 (handle 2863311531, ret 0) [D 20:01:05.938179] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1507b90. [D 20:01:05.938187] (send callback time) ini posts: 0, pending: 7, last: 0, src_list emtpy: yes [D 20:01:05.938194] ========================================================= [D 20:01:05.938200] PINT_process_request [D 20:01:05.938207] tiling 14 copies [D 20:01:05.938213] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.938220] lvl 0 el 0 blk 0 by 427819008 [D 20:01:05.938226] to 427819008 ta 402653184 fi 436207616 [D 20:01:05.938234] basic type or contiguous data [D 20:01:05.938240] Request disp [D 20:01:05.938246] contig_offset = 427819008 contig_size = 41943040 lvl_flag = 1 [D 20:01:05.938253] calling distribute [D 20:01:05.938259] PINT_distribute [D 20:01:05.938265] of 427819008 sz 8388608 ix 0 sm 64 by 0 bm 2097152 fsz 478150656 exfl 0 [D 20:01:05.938273] begin iteration loff: 427819008 [D 20:01:05.938279] segment exceeds byte limit [D 20:01:05.938286] process a segment [D 20:01:05.938291] of 427819008 sz 2097152 [D 20:01:05.938298] add a segment 0 [D 20:01:05.938304] end iteration [D 20:01:05.938310] done with segments or bytes [D 20:01:05.938316] of 429916160 sz 6291456 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.938323] next loff: 429916160 next poff: 429916160 [D 20:01:05.938329] return value 2097152 [D 20:01:05.938335] finished [D 20:01:05.938341] retval = 2097152 [D 20:01:05.938348] exiting distribute returned less than expected [D 20:01:05.938354] done sg 1 sm 64 by 2097152 bm 2097152 ta 402653184 to 429916160 fo 436207616 eof 0 [D 20:01:05.938362] ========================================================= [D 20:01:05.938369] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.938376] dbpf_bstream_rw_list: mem_offset: 0x2aaaaf4d2010, mem_size: 2097152 [D 20:01:05.938383] dbpf_bstream_rw_list: stream_offset: 427819008, stream_size: 2097152 [D 20:01:05.938391] dbpf_open_cache_get: called [D 20:01:05.938397] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.938403] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.938422] DBPF I/O ops in progress: 13 [D 20:01:05.938430] lio_listio called with the following aiocbs: [D 20:01:05.938436] aiocb_ptr_array[0]: fd: 13, off: 427819008, bytes: 2097152, buf: 0x2aaaaf4d2010, type: 0 [D 20:01:05.938444] issue_or_delay_io_operation: lio_listio posted 0x2aaabd300bb0 (handle 2863311531, ret 0) [D 20:01:05.939152] openib_check_cq: completed recv wr_id 6846b0. [D 20:01:05.939186] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd301580 to 10.1.4.56:39990 remote addr 99247000 rkey 109226 nic_wr_credit 5. [D 20:01:05.939200] openib_post_rr: 10.1.4.56:39990 bh 60 wr_id 6846b0. [D 20:01:05.940044] openib_check_cq: completed rdma wr_id 2aaabd303960, nic_wr_credit 4. [D 20:01:05.940078] post_sr: 10.1.4.56:39990 bh 26 len 16 wr_id 682d80. [D 20:01:05.940091] openib_check_cq: completed send wr_id 682c60. [D 20:01:05.940099] openib_check_cq: completed send wr_id 682c90. [D 20:01:05.940122] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1517400. [D 20:01:05.940137] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.940147] ========================================================= [D 20:01:05.940157] PINT_process_request [D 20:01:05.940166] tiling 8 copies [D 20:01:05.940174] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.940182] lvl 0 el 0 blk 0 by 230686720 [D 20:01:05.940189] to 230686720 ta 201326592 fi 234881024 [D 20:01:05.940197] basic type or contiguous data [D 20:01:05.940204] Request disp [D 20:01:05.940211] contig_offset = 230686720 contig_size = 37748736 lvl_flag = 1 [D 20:01:05.940218] calling distribute [D 20:01:05.940225] PINT_distribute [D 20:01:05.940231] of 230686720 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.940240] begin iteration loff: 230686720 [D 20:01:05.940248] segment exceeds byte limit [D 20:01:05.940254] process a segment [D 20:01:05.940260] of 230686720 sz 2097152 [D 20:01:05.940267] add a segment 0 [D 20:01:05.940274] end iteration [D 20:01:05.940281] done with segments or bytes [D 20:01:05.940287] of 232783872 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.940295] next loff: 232783872 next poff: 232783872 [D 20:01:05.940301] return value 2097152 [D 20:01:05.940308] finished [D 20:01:05.940314] retval = 2097152 [D 20:01:05.940320] exiting distribute returned less than expected [D 20:01:05.940327] done sg 1 sm 64 by 2097152 bm 2097152 ta 201326592 to 232783872 fo 234881024 eof 0 [D 20:01:05.940335] ========================================================= [D 20:01:05.940349] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.940358] dbpf_bstream_rw_list: mem_offset: 0x2aaaba84c010, mem_size: 2097152 [D 20:01:05.940365] dbpf_bstream_rw_list: stream_offset: 230686720, stream_size: 2097152 [D 20:01:05.940373] dbpf_open_cache_get: called [D 20:01:05.940381] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.940388] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.941115] DBPF I/O ops in progress: 14 [D 20:01:05.941159] lio_listio called with the following aiocbs: [D 20:01:05.941186] aiocb_ptr_array[0]: fd: 13, off: 230686720, bytes: 2097152, buf: 0x2aaaba84c010, type: 0 [D 20:01:05.941201] issue_or_delay_io_operation: lio_listio posted 0x2aaabd307370 (handle 2863311531, ret 0) [D 20:01:05.941214] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab15d5af0. [D 20:01:05.941223] (send callback time) ini posts: 0, pending: 0, last: 1, src_list emtpy: yes [D 20:01:05.941230] flowproto completing 0x2aaab15d5af0 [D 20:01:05.941280] Job flows in progress (callback time): 10 [D 20:01:05.941332] [SM Entering]: (0x66fad0) io_sm:send_completion_ack (status: 0) [D 20:01:05.941348] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:05.941358] [SM Exiting]: (0x66fad0) io_sm:send_completion_ack (error code: 0), (action: COMPLETE) [D 20:01:05.941366] [SM Entering]: (0x66fad0) io_sm:release (status: 0) [D 20:01:05.941373] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:05.941388] REQ SCHED RELEASING, handle: 2863311531, queue_element: 0x2aaab15160f0 [D 20:01:05.941399] [SM Exiting]: (0x66fad0) io_sm:release (error code: 0), (action: COMPLETE) [D 20:01:05.941406] [SM Entering]: (0x66fad0) io_sm:cleanup (status: 0) [D 20:01:05.941413] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:05.941761] openib_check_cq: completed recv wr_id 6846e0. [D 20:01:05.941783] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd31de10 to 10.1.4.56:39990 remote addr 8ae39000 rkey 109b26 nic_wr_credit 5. [D 20:01:05.941797] openib_post_rr: 10.1.4.56:39990 bh 61 wr_id 6846e0. [D 20:01:05.941807] openib_check_cq: completed recv wr_id 684710. [D 20:01:05.941814] openib_post_sr_rdmaw: ibv_post_send wr_id 2aaabd3074d0 to 10.1.4.56:39990 remote addr 97245000 rkey 109426 nic_wr_credit 6. [D 20:01:05.941822] openib_post_rr: 10.1.4.56:39990 bh 62 wr_id 684710. [D 20:01:05.942276] openib_check_cq: completed rdma wr_id 2aaabd304460, nic_wr_credit 5. [D 20:01:05.942297] post_sr: 10.1.4.56:39990 bh 27 len 16 wr_id 682db0. [D 20:01:05.942309] openib_check_cq: completed send wr_id 682cc0. [D 20:01:05.942325] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab151fe50. [D 20:01:05.942338] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:05.944509] openib_check_cq: completed rdma wr_id 2aaabd301cf0, nic_wr_credit 4. [D 20:01:05.944586] post_sr: 10.1.4.56:39990 bh 28 len 16 wr_id 682de0. [D 20:01:05.944604] openib_check_cq: completed send wr_id 682cf0. [D 20:01:05.934605] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.944692] dbpf_open_cache_put: cached entry. [D 20:01:05.944701] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.934690] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.944726] --- aio_progress_notification called with handle 2863311531 (0x2aaabd303800) [D 20:01:05.944738] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.944748] dbpf_open_cache_put: cached entry. [D 20:01:05.944754] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.944765] --- aio_progress_notification called with handle 2863311531 (0x2aaabd304300) [D 20:01:05.944774] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.944784] dbpf_open_cache_put: cached entry. [D 20:01:05.944790] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.944802] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300ff0) [D 20:01:05.944811] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return() says 2097152 [fd = 13] [D 20:01:05.944821] dbpf_open_cache_put: cached entry. [D 20:01:05.944828] *** starting delayed ops if any (state is LIST_PROC_ALLPOSTED) [D 20:01:05.944650] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152db60. [D 20:01:05.945065] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:05.945076] ========================================================= [D 20:01:05.945083] PINT_process_request [D 20:01:05.945091] tiling 9 copies [D 20:01:05.945098] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.945107] lvl 0 el 0 blk 0 by 264241152 [D 20:01:05.945114] to 264241152 ta 234881024 fi 268435456 [D 20:01:05.945122] basic type or contiguous data [D 20:01:05.945129] Request disp [D 20:01:05.945136] contig_offset = 264241152 contig_size = 37748736 lvl_flag = 1 [D 20:01:05.945144] calling distribute [D 20:01:05.945151] PINT_distribute [D 20:01:05.945522] of 264241152 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.945539] begin iteration loff: 264241152 [D 20:01:05.945548] segment exceeds byte limit [D 20:01:05.945554] process a segment [D 20:01:05.945561] of 264241152 sz 2097152 [D 20:01:05.945567] add a segment 0 [D 20:01:05.945575] end iteration [D 20:01:05.945581] done with segments or bytes [D 20:01:05.945588] of 266338304 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.945595] next loff: 266338304 next poff: 266338304 [D 20:01:05.945602] return value 2097152 [D 20:01:05.945609] finished [D 20:01:05.945615] retval = 2097152 [D 20:01:05.945622] exiting distribute returned less than expected [D 20:01:05.945629] done sg 1 sm 64 by 2097152 bm 2097152 ta 234881024 to 266338304 fo 268435456 eof 0 [D 20:01:05.945636] ========================================================= [D 20:01:05.945656] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.945673] dbpf_bstream_rw_list: mem_offset: 0x2aaabb252010, mem_size: 2097152 [D 20:01:05.945680] dbpf_bstream_rw_list: stream_offset: 264241152, stream_size: 2097152 [D 20:01:05.945688] dbpf_open_cache_get: called [D 20:01:05.945696] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.945704] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.945736] DBPF I/O ops in progress: 15 [D 20:01:05.945744] lio_listio called with the following aiocbs: [D 20:01:05.945750] aiocb_ptr_array[0]: fd: 13, off: 264241152, bytes: 2097152, buf: 0x2aaabb252010, type: 0 [D 20:01:05.945765] issue_or_delay_io_operation: lio_listio posted 0x2aaabd300da0 (handle 2863311531, ret 0) [D 20:01:05.946725] openib_check_cq: completed rdma wr_id 2aaabd31d310, nic_wr_credit 3. [D 20:01:05.946760] post_sr: 10.1.4.56:39990 bh 29 len 16 wr_id 682e10. [D 20:01:05.946774] openib_check_cq: completed send wr_id 682d20. [D 20:01:05.946783] openib_check_cq: completed send wr_id 682d50. [D 20:01:05.946813] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab1505890. [D 20:01:05.946827] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:05.946837] ========================================================= [D 20:01:05.946845] PINT_process_request [D 20:01:05.946853] tiling 10 copies [D 20:01:05.946861] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.946869] lvl 0 el 0 blk 0 by 297795584 [D 20:01:05.946876] to 297795584 ta 268435456 fi 301989888 [D 20:01:05.946884] basic type or contiguous data [D 20:01:05.946890] Request disp [D 20:01:05.946897] contig_offset = 297795584 contig_size = 37748736 lvl_flag = 1 [D 20:01:05.946904] calling distribute [D 20:01:05.946911] PINT_distribute [D 20:01:05.946917] of 297795584 sz 4194304 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.946926] begin iteration loff: 297795584 [D 20:01:05.946932] segment exceeds byte limit [D 20:01:05.946939] process a segment [D 20:01:05.946945] of 297795584 sz 2097152 [D 20:01:05.946952] add a segment 0 [D 20:01:05.946958] end iteration [D 20:01:05.946965] done with segments or bytes [D 20:01:05.946971] of 299892736 sz 2097152 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.946978] next loff: 299892736 next poff: 299892736 [D 20:01:05.946984] return value 2097152 [D 20:01:05.946991] finished [D 20:01:05.946997] retval = 2097152 [D 20:01:05.947004] exiting distribute returned less than expected [D 20:01:05.947011] done sg 1 sm 64 by 2097152 bm 2097152 ta 268435456 to 299892736 fo 301989888 eof 0 [D 20:01:05.947018] ========================================================= [D 20:01:05.947036] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.947044] dbpf_bstream_rw_list: mem_offset: 0x2aaabae50010, mem_size: 2097152 [D 20:01:05.947051] dbpf_bstream_rw_list: stream_offset: 297795584, stream_size: 2097152 [D 20:01:05.947059] dbpf_open_cache_get: called [D 20:01:05.947067] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.947074] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.947102] DBPF I/O ops in progress: 16 [D 20:01:05.947110] lio_listio called with the following aiocbs: [D 20:01:05.947118] aiocb_ptr_array[0]: fd: 13, off: 297795584, bytes: 2097152, buf: 0x2aaabae50010, type: 0 [D 20:01:05.947130] issue_or_delay_io_operation: lio_listio posted 0x2aaabd31d180 (handle 2863311531, ret 0) [A 20:01:05.948087] troy.scl@10.1.4.56:39990 H=2863311531 S=0x66fbd0: io: finish (Success) [D 20:01:05.948182] PINT_encode_release [D 20:01:05.948194] lebf_encode_rel [D 20:01:05.948224] [SM frame get]: (0x66fad0) op-id: 3 index: 0 base-frm: 0 [D 20:01:05.948232] server_state_machine_complete 0x66fad0 [D 20:01:05.948240] PINT_decode_release [D 20:01:05.948246] lebf_decode_rel [D 20:01:05.948254] [BMI CONTROL]: BMI_set_info: set_info: 6843072 option: 6 [D 20:01:05.948262] [BMI CONTROL]: BMI_set_info: searching for ref 6843072 [D 20:01:05.948269] [BMI CONTROL]: BMI_set_info: decremented ref 6843072 to: 10 [D 20:01:05.948279] [SM Exiting]: (0x66fad0) io_sm:cleanup (error code: 0), (action: TERMINATE) [D 20:01:05.948290] server_state_machine_terminate 0x66fad0 [D 20:01:05.949074] openib_check_cq: completed rdma wr_id 2aaabd301580, nic_wr_credit 2. [D 20:01:05.949126] post_sr: 10.1.4.56:39990 bh 30 len 16 wr_id 682e40. [D 20:01:05.949139] openib_check_cq: completed send wr_id 682d80. [D 20:01:05.949160] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c210. [D 20:01:05.949174] (send callback time) ini posts: 0, pending: 5, last: 0, src_list emtpy: no [D 20:01:05.949183] ========================================================= [D 20:01:05.949190] PINT_process_request [D 20:01:05.949197] tiling 12 copies [D 20:01:05.949205] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.949212] lvl 0 el 0 blk 0 by 362807296 [D 20:01:05.949219] to 362807296 ta 335544320 fi 369098752 [D 20:01:05.949227] basic type or contiguous data [D 20:01:05.949235] Request disp [D 20:01:05.949242] contig_offset = 362807296 contig_size = 39845888 lvl_flag = 1 [D 20:01:05.949249] calling distribute [D 20:01:05.949255] PINT_distribute [D 20:01:05.949262] of 362807296 sz 6291456 ix 0 sm 64 by 0 bm 2097152 fsz 476053504 exfl 0 [D 20:01:05.949272] begin iteration loff: 362807296 [D 20:01:05.949279] segment exceeds byte limit [D 20:01:05.949286] process a segment [D 20:01:05.949292] of 362807296 sz 2097152 [D 20:01:05.949298] add a segment 0 [D 20:01:05.949305] end iteration [D 20:01:05.949311] done with segments or bytes [D 20:01:05.949317] of 364904448 sz 4194304 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.949324] next loff: 364904448 next poff: 364904448 [D 20:01:05.949331] return value 2097152 [D 20:01:05.949337] finished [D 20:01:05.949343] retval = 2097152 [D 20:01:05.949349] exiting distribute returned less than expected [D 20:01:05.949356] done sg 1 sm 64 by 2097152 bm 2097152 ta 335544320 to 364904448 fo 369098752 eof 0 [D 20:01:05.949363] ========================================================= [D 20:01:05.949381] dbpf_bstream_rw_list: mem_count: 1, stream_count: 1 [D 20:01:05.949388] dbpf_bstream_rw_list: mem_offset: 0x2aaaadec7010, mem_size: 2097152 [D 20:01:05.949395] dbpf_bstream_rw_list: stream_offset: 362807296, stream_size: 2097152 [D 20:01:05.949403] dbpf_open_cache_get: called [D 20:01:05.949410] dbpf_open_cache_get: found bstream entry in used list. [D 20:01:05.949417] dbpf_open_cache_get: moving to (or reordering in) used list. [D 20:01:05.949445] op_queue add: 0x2aaabd303960 [D 20:01:05.949453] delayed I/O operation 0x2aaabd303960 (16 already in progress) [D 20:01:05.949460] DBPF I/O ops in progress: 16 [D 20:01:05.951175] openib_check_cq: completed rdma wr_id 2aaabd31de10, nic_wr_credit 1. [D 20:01:05.951242] post_sr: 10.1.4.56:39990 bh 31 len 16 wr_id 682e70. [D 20:01:05.953400] openib_check_cq: completed rdma wr_id 2aaabd3074d0, nic_wr_credit 0. [D 20:01:05.953472] post_sr: 10.1.4.56:39990 bh 32 len 16 wr_id 682ea0. [D 20:01:05.953485] openib_check_cq: completed send wr_id 682db0. [D 20:01:05.953493] openib_check_cq: completed send wr_id 682de0. [D 20:01:05.953501] openib_check_cq: completed send wr_id 682e10. [D 20:01:05.953507] openib_check_cq: completed send wr_id 682e40. [D 20:01:05.953515] openib_check_cq: completed send wr_id 682e70. [D 20:01:05.953521] openib_check_cq: completed send wr_id 682ea0. [D 20:01:05.953551] flowproto-multiqueue bmi_send_callback_fn, error_code: 0, initial_call_flag: 0, flow: 0x2aaab152c750. [D 20:01:05.953566] (send callback time) ini posts: 0, pending: 6, last: 0, src_list emtpy: no [D 20:01:05.953577] ========================================================= [D 20:01:05.953585] PINT_process_request [D 20:01:05.953592] tiling 7 copies [D 20:01:05.953600] Do seq of 0 ne 33554432 st 0 nb 1 ub 33554432 lb 0 as 33554432 co 0 [D 20:01:05.953608] lvl 0 el 0 blk 0 by 199229440 [D 20:01:05.953614] to 199229440 ta 167772160 fi 201326592 [D 20:01:05.953622] basic type or contiguous data [D 20:01:05.953628] Request disp [D 20:01:05.953635] contig_offset = 199229440 contig_size = 35651584 lvl_flag = 1 [D 20:01:05.953643] calling distribute [D 20:01:05.953649] PINT_distribute [D 20:01:05.953656] of 199229440 sz 2097152 ix 0 sm 64 by 0 bm 2097152 fsz 471859200 exfl 0 [D 20:01:05.953671] begin iteration loff: 199229440 [D 20:01:05.953680] process a segment [D 20:01:05.953686] of 199229440 sz 2097152 [D 20:01:05.953693] add a segment 0 [D 20:01:05.953700] end iteration [D 20:01:05.953706] done with segments or bytes [D 20:01:05.953712] of 201326592 sz 0 sg 1 sm 64 by 2097152 bm 2097152 [D 20:01:05.953719] next loff: 201326592 next poff: 201326592 [D 20:01:05.953725] (return value) 2097152 [D 20:01:05.953732] finished [D 20:01:05.953740] retval = 2097152 [D 20:01:05.953747] exiting distribute returned less than expected [D 20:01:05.953753] done sg 1 sm 64 by 2097152 bm 2097152 ta 167772160 to 201326592 fo 201326592 eof 0 [D 20:01:05.953760] ========================================================= [D 20:01:05.953761] --- aio_progress_notification called with handle 2863311531 (0x2aaabd300bb0) [D 20:01:05.953798] aio_progress_notification: BSTREAM_READ_LIST complete: aio_return()