14:00:29 <achow101> #startmeeting 14:00:36 <josie> hi 14:00:38 <dergoegge> hi 14:00:39 <glozow> hi 14:00:40 <achow101> #bitcoin -core-dev Meeting: achow101 _aj_ amiti ariard aureleoules b10c BlueMatt brunoerg cfields darosior dergoegge dongcarl fanquake fjahr furszy gleb glozow hebasto instagibbs jamesob jarolrod jonatack josibake kallewoof kanzure kouloumos kvaciral laanwj LarryRuane lightlike luke-jr MacroFake Murch phantomcircuit pinheadmz promag provoostenator ryanofsky sdaftuar S3RK stickies-v sipa theStack TheCharlatan vasild 14:00:41 <theStack> hi 14:00:42 <pinheadmz> hi 14:00:46 <lightlike> hi 14:00:49 <stickies-v> hi 14:00:49 <hebasto> hi 14:00:49 <brunoerg> hi 14:00:51 <instagibbs> hi 14:00:51 <TheCharlatan> hi 14:00:52 <cfields> hi 14:00:55 <abubakarsadiq> hi 14:01:02 <furszy> hi 14:01:03 <_aj_> hi 14:01:06 <achow101> #topic package relay updates (glozow) 14:01:28 <glozow> For #26711, added a bench from instagibbs, will push with a few fixups soon. Positive feedback on approach so far. 14:01:31 <gribble> https://github.com/bitcoin/bitcoin/issues/26711 | validate package transactions with their in-package ancestor sets by glozow ÷ Pull Request #26711 ÷ bitcoin/bitcoin ÷ GitHub 14:01:41 <glozow> For #28031, I've peeled a test off into #28199. I think it would be good to merge that first to help with reviewing the refactoring commits 14:01:45 <gribble> https://github.com/bitcoin/bitcoin/issues/28031 | Package Relay 1/3: Introduce TxPackageTracker as Orphan Resolution Module by glozow ÷ Pull Request #28031 ÷ bitcoin/bitcoin ÷ GitHub 14:01:46 <gribble> https://github.com/bitcoin/bitcoin/issues/28199 | test: tx orphan handling by glozow ÷ Pull Request #28199 ÷ bitcoin/bitcoin ÷ GitHub 14:02:01 <sipa> hi 14:02:55 <_aj_> 28031 has CI failures and needs rebase? 14:03:00 <willcl-ark> hi 14:04:03 <_aj_> oh, 28199 is fine so do it first? 14:04:19 <instagibbs> ð 14:04:29 <glozow> yeah. I was going to push my rebase but now I'm basically rewriting it ð 14:05:30 <achow101> do you want to put 28199 on the board instead of 28031? 14:05:45 <glozow> oh yes please thanks 14:06:15 <glozow> forgot to do that 14:06:21 <achow101> don 14:06:25 <glozow> thanks! 14:06:39 <achow101> #topic BIP 324 updates (sipa) 14:07:23 <sipa> hi 14:07:33 <sipa> We're getting there. 14:07:53 <sipa> There are currently 3 PRs open. 14:08:18 <sipa> The ciphersuite one, the transport abstraction, and then the v2 transport itself 14:09:03 <sipa> With the last one, you can experiment with bip324 connections, though in a test-only form only (using -bip324=ip). 14:09:39 <achow101> ciphersuite seems to be getting review, looks close to merge 14:09:47 <darosior> (hi) 14:09:50 <sipa> bitcoin.sipa.be runs this code, if you need a remote to connect to (-addnode=NAME -bip324=NAME should suffice). 14:10:22 <sipa> Going forward, I think I'm going to wait for the ciphersuite and transport abstraction to make progress first. 14:10:23 <instagibbs> session_id happens regardless of peer's behavior? 14:10:41 <instagibbs> (assuming within the ranges specified) 14:11:50 <sipa> After that, I'll probably drop the test-only -bip324 option from the v2 transport stuf, and implement actual service bit based decisions, maybe in a separate PR. 14:12:07 <sipa> @instagibbs It appears as soon as we know the other side has the same key we do. 14:12:22 <instagibbs> ill ask more offline :) 14:12:25 <sipa> It's empty ("") otherwise, regardless of v1 or v2. 14:12:39 <sipa> So if you see a non-empty session_id, the other side should show the same. 14:13:26 <sipa> sg 14:13:49 <achow101> #topic libbitcoinkernel updates (TheCharlatan) 14:14:04 <TheCharlatan> Opened PR #28186 for pruning the leveldb headers. 14:14:07 <gribble> https://github.com/bitcoin/bitcoin/issues/28186 | kernel: Prune leveldb headers by TheCharlatan ÷ Pull Request #28186 ÷ bitcoin/bitcoin ÷ GitHub 14:14:13 <TheCharlatan> Besides that, no updates. 14:14:54 <cfields> ð 14:14:56 <cfields> nice :) 14:15:29 <achow101> #topic assumeutxo updates (jamesob) 14:16:34 <achow101> #27746 was merged, so I think we're on the big one now #27596 14:16:37 <gribble> https://github.com/bitcoin/bitcoin/issues/27746 | Rework validation logic for assumeutxo by sdaftuar ÷ Pull Request #27746 ÷ bitcoin/bitcoin ÷ GitHub 14:16:39 <gribble> https://github.com/bitcoin/bitcoin/issues/27596 | assumeutxo (2) by jamesob ÷ Pull Request #27596 ÷ bitcoin/bitcoin ÷ GitHub 14:18:26 <achow101> #topic Ad-hoc high priority for review 14:18:43 <achow101> anything to add or remove from https://github.com/orgs/bitcoin/projects/1/views/4 14:18:47 <MacroFake> Can I have #28087 It is a blocker for a bunch of CI stuff (also a bugfix in a strict sense) 14:18:49 <gribble> https://github.com/bitcoin/bitcoin/issues/28087 | ci: Use qemu-user through container engine by MarcoFalke ÷ Pull Request #28087 ÷ bitcoin/bitcoin ÷ GitHub 14:19:10 <achow101> MacroFake: added 14:19:38 <MacroFake> (For example it would allow to run all CI tasks on a riscv machine) 14:19:38 <achow101> Or any other topics to discuss today? 14:20:03 <MacroFake> Thanks 14:21:12 <sipa> @MacroFake What is the status of CI, with cirrus changes? 14:21:33 <MacroFake> sipa: I think hebasto is moving macOS and Windows to Github CI 14:22:03 <MacroFake> The macOS seems ready, apart from the feedback I left 14:22:33 <MacroFake> Also, it would need a maintainer to enable Github CI (if we want to run it on pull requests) 14:22:52 <fanquake> Yea the PR description in that PR needs updating to actually explain whatâÂÂs its doing (dropping support for a release platform from CI), rather than making it look like just swapping infra 14:22:58 <sipa> So will nothing be on cirrus anymore? 14:23:09 <dergoegge> macOS and Windows on persistent workers is not an option? 14:23:18 <sipa> Or only self-hosted things? 14:23:21 <MacroFake> sipa: I think we'll keep Linux on Cirrus, but with persistent workers 14:23:53 <achow101> aren't persistent workers self hosted? 14:24:07 <MacroFake> yes, it is the same 14:24:18 <achow101> who's hosting them? 14:24:50 <sipa> Who is sponsoring those? 14:25:00 <MacroFake> Sponsors exist 14:25:07 <sipa> I guess I used the wrong terminology; they're running on our hardware, but it still goes through the Cirrus system for managing. 14:25:28 <sipa> Right? 14:25:46 <MacroFake> yeah, it is called cirrus-cli (or so) 14:25:51 <fanquake> dergoegge: itâÂÂs not clear if thatâÂÂs possible, or just hasnâÂÂt been investigated before picking GitHub CI 14:26:11 <sipa> One question that came up in the libsecp meeting was what to do there, and if these cirrus persistent workers would also be available for libsecp256k1. 14:26:21 <MacroFake> dergoegge: macOS is possible, but I don't think anyone wants to work on that. Do you? 14:26:35 <MacroFake> sipa: They will 14:26:39 <dergoegge> MacroFake: no 14:26:44 <MacroFake> dergoegge: heh 14:27:23 <sipa> Ideally libsecp can follow whatever Bitcoin Core does, but only if that doesn't add costs/burden to people running it. 14:27:24 <MacroFake> sipa: Will libsecp use Github CI for Windows and macOS? 14:28:01 <sipa> I guess we'll have to. 14:28:05 <luke-jr> does this break CI for non-Core repos? 14:28:23 <MacroFake> luke-jr: If you want to use Github CI, you'll have to enable it 14:28:46 <dergoegge> It would be good to have a migration plan, should the current host of the persistent workers no longer be willing to host 14:28:48 <luke-jr> I was thinking the persistent-workers thing 14:28:51 <achow101> persistent workers don't work for forks 14:29:09 <MacroFake> luke-jr: Ah, if you want to use a persistent worker, you'll have to add it to your Cirrus organization 14:29:45 <MacroFake> luke-jr: An alternative would be to create a pull request to use Github CI Linux boxes, where possible 14:29:54 <fanquake> I guess libsecp will also set up/run its own persistent workers for Linux too? 14:29:55 <luke-jr> MacroFake: does it no longer work with normal Cirrus CI workers? 14:29:56 <MacroFake> (but it is not possible for all tasks) 14:30:09 <sipa> I'm not super happy about relying (even) more on GitHub for infrastructure, but as Cirrus basically only works for GitHub anyway, it's not much of a change, and it seems there really aren't many alternatives (except very expensive ones). 14:30:30 <achow101> luke-jr: I think cirrus will cut you off or otherwise limit your ci usage 14:30:30 <MacroFake> luke-jr: That would work, with conditional tasks 14:30:49 <MacroFake> You have 40$/mo free capacity 14:31:03 <MacroFake> If you only do 4 pushes per month, it should be enough 14:31:18 <MacroFake> But I am not sure if this is something we need/should support 14:31:42 <achow101> it's nice for people to be able to run ci on their own branches before opening a pr 14:32:21 <MacroFake> achow101: FWIW, I have CI disabled on my branches in my fork 14:32:34 <sipa> Is setting up a (linux) CI persistent worker hard? If not, we can probably manage our own for libsecp. 14:33:07 <achow101> MacroFake: what's the reason for not migrating the linux tasks to github ci? 14:33:10 <luke-jr> achow101: yes, especially if we don't want people pushing to PRs for testing 14:33:26 <MacroFake> achow101: For some it is not possible 14:33:39 <MacroFake> achow101: For others: No one has opened a pull request to do it yet 14:33:40 <luke-jr> (IIRC someone was asked not to do that recently) 14:34:44 <fanquake> thatâÂÂs mostly only when itâÂÂs clear the person isnâÂÂt even bothering to compile locally 14:34:57 <MacroFake> sipa: It is one line of bash: "cirrus worker run --labels type=lunar --token todo_get_the_correct_token" 14:34:59 <fanquake> and just keeps for pushing the the PR to see what works 14:36:02 <MacroFake> (and "curl -L -o cirrus https://github.com/cirruslabs/cirrus-cli/releases/latest/download/cirrus-$(uname | tr '[:upper:]' '[:lower:]')-amd64 && mv cirrus /usr/local/bin/cirrus && chmod +x /usr/local/bin/cirrus" 14:36:10 <sipa> FWIW, I personally use CI to run tests for my own PRs, for everything beyond what make check / test_runner doesn't do (sanitizers, linters, ...). 14:36:10 <MacroFake> ) 14:36:49 <MacroFake> I think it is fine to use CI for debugging, as long as you do a local make check && test_runner.py 14:37:26 <luke-jr> in any case, it would be easier for people to do that if they don't need to spam PRs with updates ;) 14:37:46 <sipa> I'd like to run linters locally too, but haven't actually ever looked into what is needed for that. 14:38:07 <MacroFake> sipa: You'll need docker for the linters, but I don't run the either locally 14:38:22 <MacroFake> You can push and get a result from CI in 120 seconds or less 14:38:40 <fanquake> you can also skip Docker and use a venv, and just run them directly 14:39:02 <sipa> Right. 14:39:21 <MacroFake> fanquake: I wouldn't trust them. They are random pip packages from the internet and even raw x86 binaries (shellcheck) 14:39:36 <cfields> I think we're in kinda a bad state if most day-to-day devs aren't running these locally :\ 14:39:45 <cfields> local-first tests used to be an explicit goal 14:39:48 <MacroFake> (Ok, I guess docker doesn't help if they are viruses) 14:40:08 <cfields> (I'm guilty as well) 14:40:37 <josie> also guilty of using CI to run the linters / sanitizers. been meaning to look into what it would take to run all this locally, but last time I tried, it wasn't super straightforward 14:40:40 <fanquake> cfields: unit and functional tests should be run locally. Linters and cross-compilation (CI) IâÂÂm sure are much less likely 14:40:43 <MacroFake> cfields: There is some stuff that just can't be run locally. Are you going to compile santizers+fuzz+clang-tidy for every commit locally? 14:41:24 <sipa> @cfields I agree it'd be better if more we easily/automatically doable locally. 14:41:25 <sipa> But also, if it's just linters, there isn't much trust involved. 14:41:39 <fanquake> I think itâÂÂs also a case of time/resource restriction. With Docker, running any of the CI jobs locally is very straightforward 14:41:55 <MacroFake> If you trust docker, all you need to run is "DOCKER_BUILDKIT=1 docker build -t bitcoin-linter --file "./ci/lint_imagefile" ./ && docker run --rm -v $(pwd):/bitcoin -it bitcoin-linter" 14:41:56 <cfields> MacroFake: I think it should be _possible_ for me to do all of that stuff on a commit that I consider scary and worthy of testing before pushing. Of course not for every commit. 14:42:11 <sipa> I should learn docker. 14:42:15 <MacroFake> cfields: It is possible 14:42:16 <cfields> (And of course it's possible. It's just currently cumbersome, which is the topic) 14:42:19 <fanquake> cfields: definitely entirely possible to do that locally 14:42:39 <fanquake> cumbersome at this point is install Docker and run 1 bash script 14:42:40 <MacroFake> cfields: I don't think it can be made easier or take less time. Compiling takes time 14:43:13 <josie> fanquake: the struggle for me was figuring out how to run *just* the jobs I want e.g. linting 14:43:39 <MacroFake> sipa: I mean with trust that the linters can corrupt your .git folder, for example 14:44:01 <josie> I'm sure it just involves some docker-fu involved, but I couldn't figure it out last time I tried. A few examples in the docs would probably go a long way 14:44:13 <fanquake> josie: the linters are a bit of an outlier yea, for all other CI jobs, you just pick the one you want, and call the bash script (with Docker installed) and it'll ru 14:44:14 <cfields> Ok, ranting is beyond the scope here. But personally, I'm hoping after CMake we can get some more checks integrated into the buildsystem for easier testing. Multi-config builds for example, would allow for sanitizer builds in parallel. 14:44:24 <MacroFake> josie: It is a single line of bash: "DOCKER_BUILDKIT=1 docker build -t bitcoin-linter --file "./ci/lint_imagefile" ./ && docker run --rm -v $(pwd):/bitcoin -it bitcoin-linter" 14:44:29 <MacroFake> (taken from the docs) 14:45:03 <josie> MacroFake: ah, nice, I haven't checked the docs recently (or might have just missed this before) 14:45:04 <fanquake> cfields: I can't wait for CMake to magically give everyone more time and CPU heh 14:45:09 <sipa> @cfields That sounds great. 14:45:33 <MacroFake> I can convert the linter to the same syntax as the other tests, but probably not before next month 14:45:35 <sipa> (even if just aspirational) 14:45:56 <cfields> fanquake: for me: less docker == more time. Maybe I'm just old :) 14:46:17 <fanquake> josie: https://github.com/bitcoin/bitcoin/tree/master/ci 14:46:29 <MacroFake> fanquake: My laptop could use a new CPU 14:46:38 <MacroFake> apt install cmake 14:47:54 <sipa> looking forward to downloading some extra CPU 14:48:11 <luke-jr> XD 14:48:14 <sipa> Ok, I think that's enough for the CI (what turned out to be) meeting topic. 14:48:35 <hebasto> an cmake just integrates clang-tidy and iwyu... 14:48:48 <achow101> any other topics to discuss? 14:49:26 <MacroFake> pizza 14:49:35 <luke-jr> too early for pizza <.< 14:49:48 <MacroFake> It 5pm in Malmoe 14:49:48 <sipa> @cfields I'm just a simple software engineer, all these dockers and virtual environments daze and confuse me. 14:49:55 <achow101> #endmeeting