#dailyreport #tokenizers #huggingface #rust #gentoo
#ebuild #secops #cargo
I compiled the HF tokenizers library from sources and
enhanced Gentoo ebuild file to allow reproducible
installation from sources.
I removed optional dependencies and disabled HTTP
requirements to enhance security.

I wrote very simple tests for tokenizers, safetensors,
transformers and integration test for them, because
tokenizers require HF hub for testing, that I disabled.

It was a hard but good experience with Cargo package
manager of Rust. The main problems was due to strange
cfg flags that Gentoo should have set automaticly, for
ex. target_os=linux was not set. "cfg" is an
abomination that you can't add change this safely.

I didn't find a working solution to manage "cfg" and, so
I just patched the Cargo.toml files of dependencies by
commenting out lines.
(∠・ω )⌒

#dailyreport #tokenizers #huggingface #rust #gentoo
#ebuild #secops #cargo
I compiled the HF tokenizers library from sources and
enhanced Gentoo ebuild file to allow reproducible
installation from sources.
I removed optional dependencies and disabled HTT
requirements to enhance security.

I wrote very simple tests for tokenizers, safetensors,
transformers and integration test for them.

It was a hard but good experience with Cargo package
manager of Rust. The main problems was due to strange
cfg flags that Gentoo should have set automaticly, for
ex. target_os=linux was not set. "cfg" is an
abomination that you can't add change this safely.

I didn't find a working solution to manage "cfg" and, so
I just patched the Cargo.toml files of dependencies by
commenting out lines.
(∠・ω )⌒

#dailyreport #tokenizers #huggingface #rust #gentoo
#ebuild #secops #cargo
I compiled HF tokenizers library from sources and
enhanced Gentoo ebuild file that allow reproducable
installation from sources.
I removed optional dependencies and disabled http
requirements to enhance security.

I wrote very simple tests for tokenizers, safetensors,
transformers and integration test for them.

It was hard but good experience with Cargo package
manager of Rust. Main problems was because of strange
cfg flags that Gentoo should set automaticly:
target_os=linux was not set. "cfg" is an abomination you
can't add change this safely.

I didn't found working solution to manage "cfg" and just
patched Cargo.toml files of dependencies. by commenting
lines.
(∠・ω )⌒

Part2: #dailyreport #rust #gentoo #ebuild #safetensors
again. Some packages require “std” some not, remove
“std” if they not require.
- Generate Cargo.toml dependencies from “cargo tree”
8) Copy output to [dependencies] in Cargo.toml, in
original ones change version only.
9) (script) Remove “features = ["std"],”
10) check that count of dependencies not reduced, you may
repeat step (7)
11) (script to create) CRATES= for Gentoo Ebuild: from
“cargo tree”.
12) try to build/emerge, some dependencies still may be
missed, but downloaded with “cargo vendor”.
- may help: rustup update nightly, cargo +nightly -Z
unstable-options build --release --build-plan

I removed Cargo.lock from bindings/python. For me
optimization has greater priority than stability.

Part1: #dailyreport #rust #gentoo #ebuild #safetensors
#huggingface #sci-ml
Today I updated HuggingFace "safetensors" package one of
core dependencies for HF "transformers" ML-software.

I manually checked all dependencies, with bash scripts
for one reason: exclude defaults and optional
dependendies from "so called" transitive dependencies
(subdependencies) from fxxxing Rust.

Steps:
1) wget https://github.com/.../v0.5.3.tar.gz
2) tar xpf v0.5.3.tar.gz
3) cd safetensors-0.5.3/safetensors
4) comment [dev-dependencies] section in Cargo.toml
5) “$ cargo tree -e normal,build”
6) specify subdependencies in Cargo.toml from script
7) (script) run “$ cargo tree -e normal,build”

Since KiCad still does not work with Wayland, I decided to give Horizon EDA a try (https://horizon-eda.org/).

Now, that I shaved a yak by writing a little package for installing Horizon on Gentoo (https://codeberg.org/smeikx/horizon-eda-gentoo), I can first start figuring out how to design circuit boards, and then finally get back to designing my very own keyboard.