Anoncheg<p><a href="https://techhub.social/tags/dailyreport" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dailyreport</span></a> <a href="https://techhub.social/tags/tokenizers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>tokenizers</span></a> <a href="https://techhub.social/tags/huggingface" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>huggingface</span></a> <a href="https://techhub.social/tags/rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rust</span></a> <a href="https://techhub.social/tags/gentoo" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>gentoo</span></a><br> <a href="https://techhub.social/tags/ebuild" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ebuild</span></a> <a href="https://techhub.social/tags/secops" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>secops</span></a> <a href="https://techhub.social/tags/cargo" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>cargo</span></a><br>I compiled the HF 🤗 tokenizers library from sources and<br> enhanced Gentoo ebuild file to allow reproducible<br> installation from sources.<br>I removed optional dependencies and disabled HTTP<br> requirements to enhance security.</p><p>I wrote very simple tests for tokenizers, safetensors,<br> transformers and integration test for them, because<br> tokenizers require HF hub for testing, that I disabled.</p><p>It was a hard but good experience with Cargo package<br> manager of Rust. The main problems was due to strange<br> cfg flags that Gentoo should have set automaticly, for<br> ex. target_os=linux was not set. "cfg" is an<br> abomination that you can't add change this safely.</p><p>I didn't find a working solution to manage "cfg" and, so<br> I just patched the Cargo.toml files of dependencies by<br> commenting out lines.<br>(∠・ω )⌒</p>