[SOLVED] Packaging issues on NixOS #562

onny · 2021-12-10T08:42:38Z

onny
Dec 10, 2021

Hey :)
there were several attempts trying to package the new opensnitch version for NixOS but we have trouble with following build error:

[...]
Building subPackage .
go: inconsistent vendoring in /build/source/daemon:
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod

        To ignore the vendor directory, use -mod=readonly or -mod=mod.
        To sync the vendor directory, run:
                go mod vendor

see pull request here and community forum discussion here.

I'm not familiar with Go or Go packaging so maybe someone can give us an hint what we could try to get it to build :)

Thank you and best regards
Jonas

gustavo-iniguez-goya · 2021-12-10T10:15:28Z

gustavo-iniguez-goya
Dec 10, 2021
Collaborator

Hi Jonas,

First of all, thank you for packaging it for Nix!

According to chuahou the daemon builds fine with go build, but no with nix build. Could you share the complete log of nix build process?

Also, before go mod vendor, could you run go mod tidy and try again?

0 replies

onny · 2021-12-13T12:35:37Z

onny
Dec 13, 2021
Author

Thank you @gustavo-iniguez-goya for your reposone!

This is my current build file:

{ buildGoModule
, fetchFromGitHub
, fetchpatch
, protobuf
, go-protobuf
, pkg-config
, libnetfilter_queue
, libnfnetlink
, lib
, coreutils
, iptables
}:

buildGoModule rec {
  pname = "opensnitch";
  version = "1.4.3";

  src = fetchFromGitHub {
    owner = "evilsocket";
    repo = "opensnitch";
    rev = "v${version}";
    sha256 = "1c2v2x8hfqk524sa42vry74lda4lg6ii40ljk2qx9j2f69446sva";
  };

  patches = [
    # https://github.com/evilsocket/opensnitch/pull/384 don't require
    # a configuration file in /etc
    (fetchpatch {
      name = "dont-require-config-in-etc.patch";
      url = "https://github.com/evilsocket/opensnitch/commit/8a3f63f36aa92658217bbbf46d39e6d20b2c0791.patch";
      sha256 = "sha256-WkwjKTQZppR0nqvRO4xiQoKZ307NvuUwoRx+boIpuTg=";
    })
  ];

  modRoot = "daemon";

  vendorSha256 = "1mawy8bamzwzj8hq3vl7qi40j3xkykk0w0cafg2s8l6p4mnd441a";

  #proxyVendor = true;

  buildInputs = [ libnetfilter_queue libnfnetlink ];

  nativeBuildInputs = [ pkg-config protobuf go-protobuf ];

  preBuild = ''
    make -C ../proto ../daemon/ui/protocol/ui.pb.go
  '';

  buildPhase = ''
    export GOPROXY=https://goproxy.io
    go mod tidy
    go mod vendor
  #  #export GOFLAGS="-buildmode=pie -trimpath -ldflags=-linkmode=external -mod=mod"
    go build -o opensnitchd .
  '';

  postBuild = ''
    mv $GOPATH/bin/daemon $GOPATH/bin/opensnitchd
    mkdir -p $out/lib/systemd/system
    substitute opensnitchd.service $out/lib/systemd/system/opensnitchd.service \
      --replace "/usr/local/bin/opensnitchd" "$out/bin/opensnitchd" \
      --replace "/etc/opensnitchd/rules" "/var/lib/opensnitch/rules" \
      --replace "/bin/mkdir" "${coreutils}/bin/mkdir"
    sed -i '/\[Service\]/a Environment=PATH=${iptables}/bin' $out/lib/systemd/system/opensnitchd.service
  '';


  meta = with lib; {
    description = "An application firewall";
    homepage = "https://github.com/evilsocket/opensnitch/wiki";
    license = licenses.gpl3Only;
    maintainers = [ maintainers.raboof ];
    platforms = platforms.linux;
  };
}

I tried to use go mod tidy && go mod vendor and also set GOPROXY because usually it is disabled by nix-build as one can see here in the default build and configure phases.

These are the build errors I get:

nix-env -f /home/onny/projects/nixpkgs -iA opensnitch
installing 'opensnitch-1.4.3'
this derivation will be built:
  /nix/store/qhlfc99k026ahgx8jhk80fknv9d8mz6n-opensnitch-1.4.3.drv
building '/nix/store/qhlfc99k026ahgx8jhk80fknv9d8mz6n-opensnitch-1.4.3.drv'...
unpacking sources
unpacking source archive /nix/store/iqkixdyhv15fq93f25c71qfhl9m6fbyb-source
source root is source
patching sources
applying patch /nix/store/z3ikhcsg0hc60sczdc5y0wr8c24055m7-dont-require-config-in-etc.patch
patching file daemon/ui/config.go
configuring
building
go: github.com/evilsocket/[email protected]: Get "https://goproxy.io/github.com/evilsocket/ftrace/@v/v1.2.0.mod": dial tcp: lookup goproxy.io on [::1]:53: read udp [::1]:37061->[::1]:53: read: connection refused
error: builder for '/nix/store/qhlfc99k026ahgx8jhk80fknv9d8mz6n-opensnitch-1.4.3.drv' failed with exit code 1;
       last 9 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/iqkixdyhv15fq93f25c71qfhl9m6fbyb-source
       > source root is source
       > patching sources
       > applying patch /nix/store/z3ikhcsg0hc60sczdc5y0wr8c24055m7-dont-require-config-in-etc.patch
       > patching file daemon/ui/config.go
       > configuring
       > building
       > go: github.com/evilsocket/[email protected]: Get "https://goproxy.io/github.com/evilsocket/ftrace/@v/v1.2.0.mod": dial tcp: lookup goproxy.io on [::1]:53: read udp [::1]:37061->[::1]:53: read: connection refused
       For full logs, run 'nix log /nix/store/qhlfc99k026ahgx8jhk80fknv9d8mz6n-opensnitch-1.4.3.drv'.
``

1 reply

gustavo-iniguez-goya Dec 13, 2021
Collaborator

Thank you @onny . Unfortunately I also ran into this problem when packaging the app on build.opensuse.org, which is also a restricted environment. I didn't manage to solve it.

However @munix9 managed to create the packages on build.opensuse.org (#480 ) so maybe s/he can help you here, @munix9?

munix9 · 2021-12-14T13:16:12Z

munix9
Dec 14, 2021

Can I help anything?

1 reply

onny Dec 14, 2021
Author

@munix9
I'm getting following error when trying to build the opensnitch package in a restricted build environment in NixOS

[...]
Building subPackage .
go: inconsistent vendoring in /build/source/daemon:
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod

        To ignore the vendor directory, use -mod=readonly or -mod=mod.
        To sync the vendor directory, run:
                go mod vendor

Is it possible to patch go.mod or modules.txt to avoid this?

munix9 · 2021-12-14T18:40:16Z

munix9
Dec 14, 2021

Ok, I can describe the procedure how I implemented this for openSUSE.

See also the project on OBS: https://build.opensuse.org/package/show/home:munix9/opensnitch
(maybe also https://build.opensuse.org/package/show/home:munix9/opensnitch-ebpf-module)

local script with internet access (eg. prep.sh)

This creates the files vendor.tar.gz and fix-go_mod-require.patch

#!/bin/sh

set -x

ar="$1"
wd=".1_prep"

test -n "$ar" || { echo "*** archive parameter empty ***"; exit; }
test -f "$ar" || { echo "*** archive '$ar' not found ***"; exit; }

echo "*** prepare working dir ... ***"
rm -rf "$wd"
mkdir -p "$wd"
tar -xf "$ar" -C "$wd" --strip-components=1

pc="opensnitch.changes"
test -f "$pc" || { echo "*** file '$pc' not found ***"; exit; }
export SOURCE_DATE_EPOCH=$(stat -c "%Y" "$pc")

gm="$wd/daemon/go.mod"
test -s "$gm" || { echo "*** file '$gm' not found ***"; exit; }
test -d "$wd/proto" || { echo "*** dir '$wd/proto' not found ***"; exit; }

echo "*** create go vendor archive ... ***"
ocd="$(pwd)"
pushd "$wd"
cp -p daemon/go.mod daemon/go.mod.orig
cd proto
make ../daemon/ui/protocol/ui.pb.go
cd ../daemon
go mod vendor
rm -f "$ocd/vendor.tar.gz"
tar cf "$ocd/vendor.tar.gz" -I 'gzip -9 -n' --format=gnu \
        --sort=name --mtime="@${SOURCE_DATE_EPOCH}" \
        --owner=0 --group=0 --numeric-owner \
        vendor
popd

echo "*** create/update fix-go_mod-require.patch ... ***"
diff -ruN "$wd/daemon/go.mod.orig" "$wd/daemon/go.mod" > fix-go_mod-require.patch
sed -e '1d; 2s|.*|--- a/daemon/go.mod\n+++ b/daemon/go.mod|' -i fix-go_mod-require.patch

echo "*** all done ***"

To be called with the corresponding source archive:
./prep.sh opensnitch-1.4.3.tar.gz

configuration of the build script

Somewhere in your build script the following has to happen:
a) roll out the patch fix-go_mod-require.patch to the sources accordingly
b) additionally use the patch fix-disable-ui-Makefile-deps.patch - deps are not needed/do not work
c) the archive vendor.tar.gz must be unpacked in the daemon subdirectory
d) then do an export GOFLAGS="-mod=vendor -buildmode=pie -trimpath -ldflags=-linkmode=external -ldflags=-buildid=" and make

In the rpm-spec-file you can follow this reasonably well, if necessary ask again.

Maybe the build log files at the OBS are also interesting, eg.:
https://build.opensuse.org/package/live_build_log/home:munix9/opensnitch/openSUSE_Tumbleweed/x86_64
https://build.opensuse.org/package/live_build_log/home:munix9/opensnitch-ebpf-module/openSUSE_Tumbleweed/x86_64

0 replies

onny · 2021-12-14T19:08:49Z

onny
Dec 14, 2021
Author

Thank you, that gave me some helpful hints! Fixed it and described the solution here NixOS/nixpkgs#148612 (comment)

0 replies

gustavo-iniguez-goya · 2021-12-14T20:31:17Z

gustavo-iniguez-goya
Dec 14, 2021
Collaborator

Thank you @munix9 !

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SOLVED] Packaging issues on NixOS #562

{{title}}

Replies: 6 comments 2 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[SOLVED] Packaging issues on NixOS #562

onny Dec 10, 2021

Replies: 6 comments · 2 replies

gustavo-iniguez-goya Dec 10, 2021 Collaborator

onny Dec 13, 2021 Author

gustavo-iniguez-goya Dec 13, 2021 Collaborator

munix9 Dec 14, 2021

onny Dec 14, 2021 Author

munix9 Dec 14, 2021

onny Dec 14, 2021 Author

gustavo-iniguez-goya Dec 14, 2021 Collaborator

onny
Dec 10, 2021

Replies: 6 comments 2 replies

gustavo-iniguez-goya
Dec 10, 2021
Collaborator

onny
Dec 13, 2021
Author

gustavo-iniguez-goya Dec 13, 2021
Collaborator

munix9
Dec 14, 2021

onny Dec 14, 2021
Author

munix9
Dec 14, 2021

onny
Dec 14, 2021
Author

gustavo-iniguez-goya
Dec 14, 2021
Collaborator