{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":612354784,"defaultBranch":"master","name":"llama.cpp","ownerLogin":"ggerganov","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-03-10T18:58:00.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/1991296?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1726742480.0","currentOid":""},"activityList":{"items":[{"before":"f03bcd84e7026936775e1a4efe0133834c609dba","after":"5f95dccea8562750e765ae1de6b843c9615dd2aa","ref":"refs/heads/gg/rerank","pushedAt":"2024-09-19T13:19:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"server : add rerank endpoint\n\nggml-ci","shortMessageHtmlLink":"server : add rerank endpoint"}},{"before":"152e90331e03699454af8b54c3a7423f42d16e11","after":"f03bcd84e7026936775e1a4efe0133834c609dba","ref":"refs/heads/gg/rerank","pushedAt":"2024-09-19T10:21:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"llama : add \"rank\" pooling type\n\nggml-ci","shortMessageHtmlLink":"llama : add \"rank\" pooling type"}},{"before":"e01cdda1686e9409b25f3c37c9bc1d58fdf5c328","after":null,"ref":"refs/heads/gg/server-remove-waiting","pushedAt":"2024-09-19T09:44:54.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"}},{"before":"eca0fab44eb449ed34e17a9b651ae7398b494117","after":"6026da52d6942b253df835070619775d849d0258","ref":"refs/heads/master","pushedAt":"2024-09-19T09:44:53.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"server : clean-up completed tasks from waiting list (#9531)\n\nggml-ci","shortMessageHtmlLink":"server : clean-up completed tasks from waiting list (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2532935402\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9531\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9531/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9531\">#9531</a>)"}},{"before":"64c6af3195c3cd4aa3328a1282d29cd2635c34c9","after":"eca0fab44eb449ed34e17a9b651ae7398b494117","ref":"refs/heads/master","pushedAt":"2024-09-19T07:58:14.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"imatrix : disable prompt escape by default (#9543)","shortMessageHtmlLink":"imatrix : disable prompt escape by default (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2534950924\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9543\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9543/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9543\">#9543</a>)"}},{"before":null,"after":"ff231de5533d23780dd4f399cd44c86ad7fe2684","ref":"refs/heads/gg/ttfb","pushedAt":"2024-09-19T06:16:04.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"llama-bench : add time-to-first-byte stat","shortMessageHtmlLink":"llama-bench : add time-to-first-byte stat"}},{"before":"00f40ae0efe3a1c1f35ead39c3425af43ae01a6a","after":"152e90331e03699454af8b54c3a7423f42d16e11","ref":"refs/heads/gg/rerank","pushedAt":"2024-09-18T18:20:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"llama : add classigication head (wip) [no ci]","shortMessageHtmlLink":"llama : add classigication head (wip) [no ci]"}},{"before":"0d2f22e45c3c3b6f8222acb6284d0c8c93443ba1","after":"64c6af3195c3cd4aa3328a1282d29cd2635c34c9","ref":"refs/heads/master","pushedAt":"2024-09-18T17:13:08.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"max-krasnyansky","name":"Max Krasnyansky","path":"/max-krasnyansky","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1380796?s=80&v=4"},"commit":{"message":"ggml : fix n_threads_cur initialization with one thread (#9538)\n\n* ggml : fix n_threads_cur initialization with one thread\r\n\r\n* Update ggml/src/ggml.c\r\n\r\n---------\r\n\r\nCo-authored-by: Max Krasnyansky <quic_maxk@quicinc.com>","shortMessageHtmlLink":"ggml : fix n_threads_cur initialization with one thread (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2533680539\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9538\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9538/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9538\">#9538</a>)"}},{"before":"f9196c917400435f62fe6f10252d2a8e97a2cb15","after":"6b0248c29a224c767ae29551cfd6b40ae10ffa19","ref":"refs/heads/sl/fix-omp-one-thread","pushedAt":"2024-09-18T16:00:26.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"max-krasnyansky","name":"Max Krasnyansky","path":"/max-krasnyansky","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1380796?s=80&v=4"},"commit":{"message":"Update ggml/src/ggml.c","shortMessageHtmlLink":"Update ggml/src/ggml.c"}},{"before":"cbfa2fcbdc27dfcbe0c5b43d0b51057168ae5c33","after":null,"ref":"refs/heads/gg/compare-check","pushedAt":"2024-09-18T15:34:34.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"}},{"before":"6443ddd98576a9da904ef9f07df4e4398bb6a01a","after":"0d2f22e45c3c3b6f8222acb6284d0c8c93443ba1","ref":"refs/heads/master","pushedAt":"2024-09-18T15:34:32.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"scripts : verify py deps at the start of compare (#9520)","shortMessageHtmlLink":"scripts : verify py deps at the start of compare (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2530364014\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9520\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9520/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9520\">#9520</a>)"}},{"before":null,"after":"f9196c917400435f62fe6f10252d2a8e97a2cb15","ref":"refs/heads/sl/fix-omp-one-thread","pushedAt":"2024-09-18T12:59:00.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"slaren","name":null,"path":"/slaren","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2141330?s=80&v=4"},"commit":{"message":"ggml : fix n_threads_cur initialization with one thread","shortMessageHtmlLink":"ggml : fix n_threads_cur initialization with one thread"}},{"before":"8a308354f6520df6bea851b435bd8054ee5617b4","after":"6443ddd98576a9da904ef9f07df4e4398bb6a01a","ref":"refs/heads/master","pushedAt":"2024-09-18T11:42:37.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"llama : use reserve/emplace_back in sampler_sample (#9534)\n\nThis commit updates the llama_sampler_sample function to use reserve and\r\nemplace_back for the vector of llama_token_data structs.\r\n\r\nThe motivation for this change is to avoid the creation of n_vocab\r\ndefault-constructed llama_token_data structs which are then\r\nimmediately overwritten.","shortMessageHtmlLink":"llama : use reserve/emplace_back in sampler_sample (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2533259556\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9534\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9534/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9534\">#9534</a>)"}},{"before":"38b955cf367f31d6e8d7230f012eed669725b9a5","after":"c90a43a2370134c634edc54457fcf1b352689db7","ref":"refs/heads/pr_add_intel_amx_support","pushedAt":"2024-09-18T07:31:22.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"mingfeima","name":"Ma Mingfei","path":"/mingfeima","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/20233731?s=80&v=4"},"commit":{"message":"minor change","shortMessageHtmlLink":"minor change"}},{"before":"385bc22fdb0b6fa627a4eeb7ed3429510ad6e691","after":"e01cdda1686e9409b25f3c37c9bc1d58fdf5c328","ref":"refs/heads/gg/server-remove-waiting","pushedAt":"2024-09-18T07:21:50.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"server : clean-up completed tasks from waiting list\n\nggml-ci","shortMessageHtmlLink":"server : clean-up completed tasks from waiting list"}},{"before":null,"after":"385bc22fdb0b6fa627a4eeb7ed3429510ad6e691","ref":"refs/heads/gg/server-remove-waiting","pushedAt":"2024-09-18T07:21:03.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"server : clean-up completed tasks from waiting list\n\nggml-ci","shortMessageHtmlLink":"server : clean-up completed tasks from waiting list"}},{"before":"9504b9f9ae3b273cd152d86300092719e96b399f","after":"38b955cf367f31d6e8d7230f012eed669725b9a5","ref":"refs/heads/pr_add_intel_amx_support","pushedAt":"2024-09-18T07:12:19.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"mingfeima","name":"Ma Mingfei","path":"/mingfeima","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/20233731?s=80&v=4"},"commit":{"message":"update CMakeLists.txt","shortMessageHtmlLink":"update CMakeLists.txt"}},{"before":"f799155ab8279cfa668086ce584aa52ecc05f5e5","after":"8a308354f6520df6bea851b435bd8054ee5617b4","ref":"refs/heads/master","pushedAt":"2024-09-18T06:50:34.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"server : match OAI structured output response (#9527)","shortMessageHtmlLink":"server : match OAI structured output response (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2532541199\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9527\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9527/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9527\">#9527</a>)"}},{"before":"faf67b3de4688f47c3b1019c89df255df2fd59b4","after":"f799155ab8279cfa668086ce584aa52ecc05f5e5","ref":"refs/heads/master","pushedAt":"2024-09-18T06:28:20.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"server : fix OpenSSL build (remove obsolete `LOG_INFO`) (#9529)","shortMessageHtmlLink":"server : fix OpenSSL build (remove obsolete <code>LOG_INFO</code>) (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2532587129\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9529\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9529/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9529\">#9529</a>)"}},{"before":"47b1a743b8db3a2e29d59cde3989125f1db16836","after":"9504b9f9ae3b273cd152d86300092719e96b399f","ref":"refs/heads/pr_add_intel_amx_support","pushedAt":"2024-09-18T06:20:18.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"mingfeima","name":"Ma Mingfei","path":"/mingfeima","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/20233731?s=80&v=4"},"commit":{"message":"minor change","shortMessageHtmlLink":"minor change"}},{"before":"a43f8e0089acc29d5f55eed6c99730e3eedb6c8b","after":"47b1a743b8db3a2e29d59cde3989125f1db16836","ref":"refs/heads/pr_add_intel_amx_support","pushedAt":"2024-09-18T06:16:29.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"mingfeima","name":"Ma Mingfei","path":"/mingfeima","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/20233731?s=80&v=4"},"commit":{"message":"update header file, the old path for immintrin.h has changed to ggml-cpu-impl.h","shortMessageHtmlLink":"update header file, the old path for immintrin.h has changed to ggml-…"}},{"before":"7be099fa817e9c53ffb4c3ed7d063e1cffcd675a","after":"faf67b3de4688f47c3b1019c89df255df2fd59b4","ref":"refs/heads/master","pushedAt":"2024-09-18T00:30:31.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"NeoZhangJianyu","name":"Neo Zhang Jianyu","path":"/NeoZhangJianyu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/46982523?s=80&v=4"},"commit":{"message":"[SYCL]set context default value to avoid memory issue, update guide (#9476)\n\n* set context default to avoid memory issue, update guide\r\n\r\n* Update docs/backend/SYCL.md\r\n\r\nCo-authored-by: Meng, Hengyu <hengyu.meng@intel.com>\r\n\r\n---------\r\n\r\nCo-authored-by: arthw <14088817+arthw@users.noreply.github.com>\r\nCo-authored-by: Meng, Hengyu <hengyu.meng@intel.com>","shortMessageHtmlLink":"[SYCL]set context default value to avoid memory issue, update guide (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2525912185\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9476\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9476/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9476\">#…</a>"}},{"before":"8b836ae731bbb2c5640bc47df5b0a78ffcb129cb","after":"7be099fa817e9c53ffb4c3ed7d063e1cffcd675a","ref":"refs/heads/master","pushedAt":"2024-09-17T20:41:38.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"slaren","name":null,"path":"/slaren","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2141330?s=80&v=4"},"commit":{"message":"llama-bench: correct argument parsing error message (#9524)","shortMessageHtmlLink":"llama-bench: correct argument parsing error message (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2532078076\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9524\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9524/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9524\">#9524</a>)"}},{"before":"a5307f5acff748cbdd9644e00be69834b3db8cc5","after":"00f40ae0efe3a1c1f35ead39c3425af43ae01a6a","ref":"refs/heads/gg/rerank","pushedAt":"2024-09-17T13:38:55.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"llama : read new cls tensors [no ci]","shortMessageHtmlLink":"llama : read new cls tensors [no ci]"}},{"before":"8344ef58f8cdb8ebd6faf4463ca32ae91c374c81","after":"8b836ae731bbb2c5640bc47df5b0a78ffcb129cb","ref":"refs/heads/master","pushedAt":"2024-09-17T13:35:39.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"arg : add env variable for parallel (#9513)\n\n* add env variable for parallel\r\n\r\n* Update README.md with env:  LLAMA_ARG_N_PARALLEL","shortMessageHtmlLink":"arg : add env variable for parallel (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2529251995\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9513\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9513/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9513\">#9513</a>)"}},{"before":"fbbb64fffed4cc042badf0bb5daf59f14b5a4640","after":"a5307f5acff748cbdd9644e00be69834b3db8cc5","ref":"refs/heads/gg/rerank","pushedAt":"2024-09-17T10:53:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"py : fix position embeddings chop [no ci]","shortMessageHtmlLink":"py : fix position embeddings chop [no ci]"}},{"before":"13e6d732a01e869eb6425730e67b75834f8413d3","after":"fbbb64fffed4cc042badf0bb5daf59f14b5a4640","ref":"refs/heads/gg/rerank","pushedAt":"2024-09-17T10:41:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"py : fix scalar-tensor conversion [no ci]","shortMessageHtmlLink":"py : fix scalar-tensor conversion [no ci]"}},{"before":"0226613853133c081b55bb892a41bb5eacc0bc94","after":"8344ef58f8cdb8ebd6faf4463ca32ae91c374c81","ref":"refs/heads/master","pushedAt":"2024-09-17T10:18:22.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"llama : fix n_vocab init for 'no_vocab' case (#9511)\n\n* llama: fixed n_vocab for `no_vocab` models\r\n\r\n* llama: updated error output for `llama_decode_internal` and `llama_encode_internal`\r\n\r\n* llama: log warning if there's no vocab_size in metadata\r\n\r\n* llama: correct vocab size for logging\r\n\r\nCo-authored-by: Georgi Gerganov <ggerganov@gmail.com>\r\n\r\n---------\r\n\r\nCo-authored-by: Georgi Gerganov <ggerganov@gmail.com>","shortMessageHtmlLink":"llama : fix n_vocab init for 'no_vocab' case (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2528945060\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9511\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9511/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9511\">#9511</a>)"}},{"before":"8241151f1645a7dd6087c2c5d409bfb69de7d28c","after":"a6a8f8d09c8e27a70926d8a52016ce8f49d954dd","ref":"refs/heads/fix_ctx_default","pushedAt":"2024-09-17T08:25:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"NeoZhangJianyu","name":"Neo Zhang Jianyu","path":"/NeoZhangJianyu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/46982523?s=80&v=4"},"commit":{"message":"Update docs/backend/SYCL.md\n\nCo-authored-by: Meng, Hengyu <hengyu.meng@intel.com>","shortMessageHtmlLink":"Update docs/backend/SYCL.md"}},{"before":"503147a9f9d195d6a14e7c998df23b6eb61f2bae","after":"0226613853133c081b55bb892a41bb5eacc0bc94","ref":"refs/heads/master","pushedAt":"2024-09-17T08:19:47.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ggerganov","name":"Georgi Gerganov","path":"/ggerganov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1991296?s=80&v=4"},"commit":{"message":"threadpool : skip polling for unused threads (#9461)\n\n* threadpool: skip polling for unused threads\r\n\r\nCurrently all threads do N polling rounds even if only 1 thread is active (n_threads_cur == 1).\r\nThis commit adds a check to skip the polling for unused threads (ith >= n_threads_cur).\r\n\r\nn_threads_cur is now an atomic_int to explicitly tell thread sanitizer that it is written\r\nfrom one thread and read from other threads (not a race conditions).\r\n\r\n* threadpool: further simplify and improve ggml_barrier\r\n\r\nAvoid using strict memory order while polling, yet make sure that all threads go through\r\nfull memory barrier (memory fence) on ggml_barrier entrace and exit.\r\n\r\n* threads: add simple barrier test\r\n\r\nThis test does lots of small, parallel matmul ops where the barriers in between dominate the overhead.\r\n\r\n* threadpool: improve thread sync for new-graphs\r\n\r\nUsing the same tricks as ggml_barrier. All the polling is done with relaxed memory order\r\nto keep it efficient, once the new graph is detected we do full fence using read-modify-write\r\nwith strict memory order.\r\n\r\n* threadpool: improve abort handling\r\n\r\nDo not use threadpool->ec (exit code) to decide whether to exit the compute loop.\r\nthreadpool->ec is not atomic which makes thread-sanitizer rightfully unhappy about it.\r\n\r\nInstead introduce atomic threadpool->abort flag used for this. This is consistent with\r\nhow we handle threadpool->stop or pause.\r\n\r\nWhile at it add an explicit atomic_load for n_threads_cur for consistency.\r\n\r\n* test-barrier: release threadpool before releasing the context\r\n\r\nfixes use-after-free detected by gcc thread-sanitizer on x86-64\r\nfor some reason llvm sanitizer is not detecting this issue.","shortMessageHtmlLink":"threadpool : skip polling for unused threads (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2523899749\" data-permission-text=\"Title is private\" data-url=\"https://github.com/ggerganov/llama.cpp/issues/9461\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/ggerganov/llama.cpp/pull/9461/hovercard\" href=\"https://github.com/ggerganov/llama.cpp/pull/9461\">#9461</a>)"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEuuNf9AA","startCursor":null,"endCursor":null}},"title":"Activity · ggerganov/llama.cpp"}