16:00:15 <achow101> #startmeeting 
16:00:15 <corebot`> achow101: Meeting started at 2025-06-12T16:00+0000
16:00:16 <corebot`> achow101: Current chairs: achow101
16:00:17 <hodlinator> hi
16:00:18 <corebot`> achow101: Useful commands: #action #info #idea #link #topic #motion #vote #close #endmeeting
16:00:19 <corebot`> achow101: See also: https://hcoop-meetbot.readthedocs.io/en/stable/
16:00:20 <corebot`> achow101: Participants should now identify themselves with '#here' or with an alias like '#here FirstLast'
16:00:21 <TheCharlatan> hi
16:00:24 <achow101> #bitcoin -core-dev Meeting: abubakarsadiq achow101 _aj_ ajonas b10c brunoerg cfields darosior dergoegge fanquake fjahr furszy gleb glozow hebasto hodlinator instagibbs jarolrod jonatack josibake kanzure laanwj LarryRuane lightlike luke-jr maflcko marcofleon maxedw Murch pinheadmz provoostenator ryanofsky sdaftuar S3RK stickies-v sipa sr_gi tdb3 theStack TheCharlatan vasild willcl-ark
16:00:25 <maxedw> hi
16:00:27 <hebasto> hi
16:00:30 <furszy> hi
16:00:30 <johnny9dev> hi
16:00:32 <dergoegge> hi
16:00:34 <pinheadmz> hi
16:00:36 <fjahr> hi
16:00:39 <brunoerg_> hi
16:00:41 <vasild> hi (I am back on time)
16:00:42 <lightlike> hi
16:00:51 <instagibbs> hi
16:00:52 <achow101> There are 2 preproposed meeting topics this week. Any last minute ones to add?
16:01:05 <eugenesiegel> hi
16:01:15 <stickies-v> hi
16:01:31 <l0rinc> hi
16:01:45 <cfields> hi
16:01:48 <jonatack> hi
16:01:55 <achow101> #topic Fuzzing WG Update (dergoegge)
16:02:09 <dergoegge> Fuzzamoto (https://github.com/dergoegge/fuzzamoto) has been my primary focus the last couple months. Wrote some notes on what I’ve been up to here: https://gist.github.com/dergoegge/46c7a5fd2468b5b108f5c34ac1f88e45. Quick summary:
16:02:21 <dergoegge> The project now includes a custom fuzzing engine for structure aware fuzzing of p2p protocol message sequences. Coverage report for that as of jun 10: https://dergoegge.github.io/bitcoin-coverage/fuzzamoto/ir-jun10/coverage/bitcoin/index.html
16:02:32 <dergoegge> Highlights from the cov report:
16:02:39 <dergoegge> More net_processing coverage than the in-repo fuzz tests, in all coverage metrics (line, function, region, branch) coverage (e.g. ~70% vs ~ 64% line coverage). Also coverage for reorgs went from nothing to something (i.e. Chainstate::DisconnectTip, Chainstate::MaybeUpdateMempoolForReorg)
16:03:00 <dergoegge> I’m still working on extending the fuzzer to increase coverage (e.g. no compact block support atm) and brainstorming what kind of bug oracles are feasible (please lmk if you have any ideas, join the fuzzing wg)
16:03:08 <purpleKarrot> hi
16:03:19 <dergoegge> Lastly, it would be interesting to have a collection of real p2p/validation bugs to test the fuzzer against (beyond what’s list ed on https://bitcoincore.org/en/security-advisories/). If you know of or have found bugs in your own or other PRs please let me know! Any kind of bug (logic, crash, …) is fine.
16:03:39 <marcofleon> hype
16:03:48 <instagibbs> nice re:reorgs
16:03:55 <rkrux> hi
16:03:55 <eugenesiegel> awesome about reorgs
16:04:15 <brunoerg_> nice
16:04:41 <sipa> hi
16:05:35 <sipa> dergoegge: cool, will think about bugs that can be included
16:06:38 <achow101> #topic Kernel WG Update (TheCharlatan)
16:06:43 <TheCharlatan> I pushed a new version of #32427 including a write ahead log and crc32c checksums for each entry. Performance still seems to be a bit better compared to leveldb. This should make writes atomic and provide detection for potential corruption.
16:06:45 <corebot`> https://github.com/bitcoin/bitcoin/issues/32427 | (RFC) kernel: Replace leveldb-based BlockTreeDB with flat-file based store by TheCharlatan · Pull Request #32427 · bitcoin/bitcoin · GitHub
16:07:12 <cfields> 🚀
16:07:41 <TheCharlatan> also looking for review on #32317
16:07:44 <corebot`> https://github.com/bitcoin/bitcoin/issues/32317 | kernel: Separate UTXO set access from validation functions by TheCharlatan · Pull Request #32317 · bitcoin/bitcoin · GitHub
16:08:13 <TheCharlatan> that's all for now
16:08:13 <sipa> TheCharlatan: (sorry for being lazy and not looking at code), what path structure are you using?
16:08:14 <cfields> 32427 spawned a discussion about a new on-disk block layout that is a bit of a distraction...
16:08:17 <sr_gi[m]1> hi
16:08:30 <abubakarsadiq> hi
16:08:40 <sipa> cfields: jinx, ok...
16:08:45 <cfields> I think it might be worth having that discussion here a bit..
16:08:46 <TheCharlatan> path structures?
16:08:47 <cfields> sipa: hah
16:09:00 <sipa> TheCharlatan: directory/file names
16:09:22 <l0rinc> TheCharlatan: "Performance still seems to be a bit better compared to leveldb" - I have measured a reindex until 888888 blocks for your PR and the performance seemed roughly the same
16:09:47 <TheCharlatan> oh, yeah, the PR does not touch the actual block storage hierarchy.
16:09:59 <sipa> ah, ok
16:10:32 <cfields> tldr: in the review for 32427, several reviewers brought up the idea of changing the on-disk block format while we're at it. One suggested example is one-file-per-block...
16:10:54 <dergoegge> the on disk block index storage performance should not affect ibd at all, we keep all of it in memory no?
16:11:30 <sipa> dergoegge: flushes to utxo set trigger flushes of block data, which could take some marginal amount of time?
16:11:37 <TheCharlatan> I also think that given that we have some flexibility with the existing pair of 64bit numbers to point to a location on disk, we might be able to use the encoding for single block files.
16:11:47 <cfields> for 32427 to move forward, we either need to agree to punt on that discussion and bring it up again later, or bikeshed some layout changes. For the sake of not grinding the flat index changes to a halt, I prefer the former.
16:11:53 <dergoegge> sipa: ah right, ty
16:11:56 <vasild> "one-file-per-block" ... some filesystems may not handle well 1M files, is there any benefit from doing that?
16:12:00 <l0rinc> won't we have problems with so many open files on Windows - like we had with LevelDB?
16:12:27 <sipa> vasild: they can be grouped into subdirectories too, either by hash, or (my suggestion, i think) by height
16:12:31 <cfields> vasild: you don't really need an index anymore if the blocks can be located using the filesystem alone.
16:12:47 <vasild> sure, I do not mean 1M files in a single directory
16:12:53 <cfields> er, a much simpler index at least.
16:12:59 <sipa> yes, agree with that, don't put 1M files in a single directory :)
16:13:00 <l0rinc> would be really cool to try it out though - let me know if I can help with benchmarking anything
16:13:13 <TheCharlatan> dergoegge yes, in ibd time it is just good to know there are no regressions. The speedup during block index loading when starting a node is more significant though.
16:13:14 <lightlike> people tend to copy datadirs for backups etc. years ago that would take very long with millions of files, did that change?
16:13:55 <sipa> cfields: no need for the block data structure change (if any) to be made in the same PR, it suffices if it's in the same release
16:15:00 <sipa> cfields: well "index" isn't the right word anymore then, but we still have a block database with things like validation status and so on (though, arguably, that belongs more in validation then block storage)
16:15:02 <vasild> cfields: that would be like outsouring the indexing to the OS/filesystem, e.g. blocks/1/15/153/153764 (for block 153764). The question is can we do better than the worst OS/FS that is going to use Bitcoin Core, or even, better than the best FS?
16:15:03 <l0rinc> would that mean that leveldb indexes can point to files and indexes now instead of duplicating any data? Or is that still problematic for pruned nodes?
16:15:30 <sipa> it'd be pretty awesome if the block index can go away, and its remaining fields moved into the chainstate
16:15:56 <vasild> I guess it has to be tested...
16:15:57 <cfields> sipa: right
16:16:07 <sipa> though maybe not more than an aesthetical advantage
16:16:26 <sipa> it's just ugly that we store validity of a block in the block index, it's not a storage-related thing but a validation-related one
16:16:33 <cfields> sipa: much easier for non-core software to read blocks. definitely not just aesthetic.
16:16:52 <sipa> cfields: i'm talking about the block index going away, independent of block storage structure used
16:17:41 <sipa> lightlike: yeah, i worry about that too - maybe in these days with fast SSDs that's less of an issue, but i don't actually know
16:18:19 <cfields> sipa: not sure what you mean. how does it go away if we still need to map them to a non-addressable location?
16:19:21 <TheCharlatan> also seems useful though to have block validity included for as long as we can't read the chainstate easily from external sources.
16:20:00 <sipa> cfields: sorry, i'm being confusing. I'm talking about the advantage of the block index going away, which is something that's only possible with a change to directory structure - but i'm just talking that that index going away itself is nice, ignoring the advantage that the directory structure may bring without that
16:20:06 <sipa> TheCharlatan: i disagree!
16:20:29 <sipa> TheCharlatan: it's pretty counterintuitive that someone just copying the block data, but not chainstate, would inherit the validity status from that block data
16:20:39 <cfields> sipa: ah, yes, completely agree.
16:22:45 <TheCharlatan> mmh, so you couldn't recreate the  block tree anymore without re-validating?
16:23:27 <sipa> TheCharlatan: i just wouldn't think of the validity status as part of the block data
16:23:47 <sipa> you can recreate it from just disk, but without a chainstate to fill in validity status, every block would be marked as non-validated
16:24:10 <sipa> this is probably getting a bit into the weeds for this meeting
16:24:19 <TheCharlatan> yes :)
16:24:25 <lightlike> i feel like the discussion of this idea deserves a separate issue, it's not that related to ThCharlatan's PR.
16:25:40 <TheCharlatan> sooo... next topic?
16:25:54 <achow101> next topic
16:25:55 <achow101> #topic Benchmarking WG Update (josie, l0rinc)
16:26:10 <l0rinc> I have opened the obfuscation PR in Oct 2024, split off other smaller optimizations since, some of those are merged already - the XOR block obfuscation one is next, reproducers and reviewers are welcome
16:26:35 <l0rinc> I've received 2 raspberry pi servers recently, I will be able to do benchmarks on those as well (already measured Sipa's #32545) - since many people are complaining they're IBDs are taking weeks
16:26:37 <corebot`> https://github.com/bitcoin/bitcoin/issues/32545 | Replace cluster linearization algorithm with SFL by sipa · Pull Request #32545 · bitcoin/bitcoin · GitHub
16:26:57 <l0rinc> And lately I'm measuring memory usages as well besides performance with massif and heaptrack as well - see #28531
16:27:00 <corebot`> https://github.com/bitcoin/bitcoin/issues/28531 | improve MallocUsage() accuracy by LarryRuane · Pull Request #28531 · bitcoin/bitcoin · GitHub
16:27:15 <jonatack> nice
16:27:48 <l0rinc> I noticed that GCC and Clang optimize differently - is it important to benchmark both?
16:28:01 <sipa> whatever is used for releases is what matters most
16:28:11 <fanquake> GCC 13.3.0
16:28:13 <sipa> which is IIRC clang for macOS and gcc for windows and linux
16:28:21 <fanquake> clang 18 for macos
16:29:30 <l0rinc> the tracking PR for speeding up Initial Block Download, where I have collected the results of the benchmarks I've done is in #32043
16:29:32 <corebot`> https://github.com/bitcoin/bitcoin/issues/32043 | [IBD] Tracking PR for speeding up Initial Block Download by l0rinc · Pull Request #32043 · bitcoin/bitcoin · GitHub
16:29:43 <jonatack> maybe you are already doing so, but benchmarking with slow internet connections for ibd/p2p might reveal additional issues in how a change under consideration impacts nodes under more hostile conditions like that
16:30:42 <l0rinc> the PI server are also on slightly slower network and lower memory (4 & 8GB)
16:31:28 <dzxzg> it's challenging to create realistic bad network conditions in a way that is also reproducible
16:31:59 <l0rinc> if you think it's important, I can simulate slow connections as well to see what that reveals (the Hetzner servers I have are very fast for normal IBD as well, I guess the nodes it find are probably also from Hetnzer)
16:32:36 <jonatack> benchmarking how bitcoind holds up under tough conditions seems valuable (i find it does very well currently, apart from ibd time)
16:33:01 <jonatack> (and the stalling/timeout->disconnect logic)
16:33:13 <sipa> yeah i think those are very different things to measure
16:33:20 <l0rinc> I've also checked swiftsync with very simpmple xor instead of muhash - and locally on my laptop it finished a reindex until 888888 blocks in 29 minutes
16:33:31 <sipa> pure computation/disk/io based effects
16:33:39 <abubakarsadiq> not sure whether it will help, but having a node with few peers as well? like doing ibd without inbounds, someone complained to me that their ibd was slow.
16:33:39 <abubakarsadiq> when I checked it due to having very few peers, basically no inbounds at all.
16:33:40 <sipa> vs network block IBD fetching effects
16:33:48 <eugenesiegel> I have heard that netem can be used to simulate latency
16:34:42 <achow101> sorry, going to try to keep things moving since we have 2 actual topics this week
16:34:51 <achow101> we can have further discussion after the meeting, or in an issue on github
16:34:52 <l0rinc> k, thanks
16:34:57 <achow101> #topic Cluster Mempool WG Update (sdaftuar, sipa)
16:35:01 <sipa> not much to report
16:35:24 <sipa> next to review remains reorg support in txgraph, #31553
16:35:28 <corebot`> https://github.com/bitcoin/bitcoin/issues/31553 | cluster mempool: add TxGraph reorg functionality by sipa · Pull Request #31553 · bitcoin/bitcoin · GitHub
16:36:00 <sipa> had some activity on a test-only but non-blocking PR, #30605, which had a review club, and grew a bit in more test improvements since
16:36:02 <corebot`> https://github.com/bitcoin/bitcoin/issues/30605 | Cluster linearization: separate tests from tests-of-tests by sipa · Pull Request #30605 · bitcoin/bitcoin · GitHub
16:36:35 <sipa> i posted instructions for more intensive benchmarking of the new linearization algorithm in #32545, if people want to participate in that
16:36:37 <corebot`> https://github.com/bitcoin/bitcoin/issues/32545 | Replace cluster linearization algorithm with SFL by sipa · Pull Request #32545 · bitcoin/bitcoin · GitHub
16:36:46 <sipa> that's it from me
16:36:52 <achow101> #topic MuSig2 WG Update (achow101, rkrux)
16:37:02 <achow101> Not much to update, #31244 has gotten some review which I've been addressing
16:37:06 <corebot`> https://github.com/bitcoin/bitcoin/issues/31244 | descriptors: MuSig2 by achow101 · Pull Request #31244 · bitcoin/bitcoin · GitHub
16:37:11 <achow101> The change to the BIP is still waiting to see if anyone has an opinion on the duplicate participants
16:37:25 <achow101> #topic QML GUI WG Update (jarolrod, johnny9dev)
16:37:26 <sipa> achow101: link to BIP change?
16:37:43 <sipa> ah, or that's on the ML?
16:37:47 <achow101> sipa: ML
16:37:49 <jonatack> https://github.com/bitcoin/bips/pull/1867
16:37:51 <johnny9dev> Significant progress was made this week on rebasing the project and moving it to CMake+Qt6. Both hebasto and pinheadmz have PRs now to test and fix issues with the update. bitcoin-core/gui-qml#470 bitcoin-core/gui-qml#472. Both PRs are now able to load up the QML app and get to the main navigation and views.
16:37:51 <johnny9dev> For myself, I've been focused on the current Send and Recieve PRs that are in review. I've also been making updates to how address inputs are handled, bitcoin amount label formatting, and address label formatting to get closer to the design spec.
16:37:56 <corebot`> https://github.com/bitcoin-core/gui-qml/issues/470 | [WIP] Sync with main repo: CMake + Qt 6 by hebasto · Pull Request #470 · bitcoin-core/gui-qml · GitHub
16:37:59 <sipa> achow101: thanks
16:38:00 <corebot`> https://github.com/bitcoin-core/gui-qml/issues/472 | Rebase gui-qml on bitcoin/bitcoin, include qt6 and cmake by pinheadmz · Pull Request #472 · bitcoin-core/gui-qml · GitHub
16:38:37 <hebasto> bitcoin-core/gui-qml#472 is preferred
16:38:41 <corebot`> https://github.com/bitcoin-core/gui-qml/issues/472 | Rebase gui-qml on bitcoin/bitcoin, include qt6 and cmake by pinheadmz · Pull Request #472 · bitcoin-core/gui-qml · GitHub
16:39:57 <achow101> #topic Script Validation WG Update (fjahr)
16:40:01 <fjahr> Nothing really noteworthy this week, more time for the *actual* topics ;)
16:40:15 <achow101> #topic Erlay WG Update (sr_gi, gleb)
16:41:03 <sr_gi[m]1> Nothing much to report on my end either, I've been off a couple of weeks. Currently picking warnet sims where I left them before leaving. Will report once I'm finished
16:41:16 <achow101> #topic Rough poll - how many people use GCC and how many of those do generate code coverage with GCC? (vasild)
16:41:25 <vasild> Trying to get an estimate of how many people are using the current GCC-oriented coverage we have in CMake.
16:41:55 <eugenesiegel> I use gcc for code coverage
16:41:58 <achow101> I use gcc, but I haven't generated coverage with it in ~5 years (or any other coverage really)
16:42:08 <sipa> i use GCC for creating benchmarking builds, and clang for fuzzing builds; i haven't used coverage tools for a long time (maybe i should?)
16:42:27 <fjahr> Only have been using clang for anything cov related since that was what was recommended to me, but definitely not a heavy user lately
16:42:34 <eugenesiegel> coverage reports are nice for telling you where the fuzzer is stuck
16:43:01 <fanquake> eugenesiegel: are you using GCC for fuzzing? Or just switching to it for coverage?
16:44:19 <eugenesiegel> fanquake: I use libfuzzer to fuzz on a VM, and then run the corpus through with gcc coverage on my mac
16:44:56 <fanquake> gcc on a mac is clang though? So brew installed GCC?
16:44:59 <l0rinc> I usually use test code coverage in cLion which automatically adds the `-DCMAKE_CXX_FLAGS="-fcoverage-mapping -fprofile-instr-generate" -DCMAKE_C_FLAGS="-fcoverage-mapping -fprofile-instr-generate"` options
16:45:53 <dergoegge> l0rinc: and then you (or cLion) uses the llvm tools to create the actual report?
16:46:55 <l0rinc> there's a built-in report, haven't checked how it generates it
16:46:55 <eugenesiegel> fanquake: oh, I didn't know that. I use gcov so I assumed it was gcc
16:47:22 <fanquake> What is the output of gcc --version
16:47:33 <dergoegge> eugenesiegel: to clarify, your reports look like this: https://maflcko.github.io/b-c-cov/fuzz.coverage/index.html ?
16:47:41 <vasild> for the records, the context of this topic is https://github.com/bitcoin/bitcoin/issues/31047
16:48:05 <eugenesiegel> fanquake: clang 16
16:48:50 <fanquake> Right, so that'll probably be Clang in compat mode
16:49:23 <eugenesiegel> dergoegge: no they don't, they look like this https://crypt-iq.github.io/fuzz_coverage_reports/cmpctblock-nyx-cov-06012025/
16:49:24 <jonatack> brew says gcc 15.1, but currently for me gcc --version is Apple clang version 17.0.0 (clang-1700.0.13.5)
16:49:58 <achow101> I'm fine with replacing the gcc based coverage with clang based. IIRC, the current instructions don't even work?
16:50:35 <achow101> already have to switch compilers to use libfuzzer, and to get things like thread-safety warnings
16:50:55 <marcofleon> I think we have proper clang coverage instructions in the docs?
16:50:59 <dergoegge> eugenesiegel: got it! "Generated by llvm-cov", so you're using the clang and the llvm tooling
16:51:15 <vasild> achow101: current instructions work for me: https://github.com/bitcoin/bitcoin/issues/31047#issuecomment-2658930755
16:51:18 <dergoegge> (that's what it says at the bottom)
16:51:21 <Sjors[m]1> #31047 
16:51:22 <corebot`> https://github.com/bitcoin/bitcoin/issues/31047 | build: RFC Coverage build type · Issue #31047 · bitcoin/bitcoin · GitHub
16:51:39 <dzxzg> Isn't it possible to compile with gcc and make a coverage report with lcov?
16:52:05 <achow101> vasild: hmm ok. I did try a while ago and it didn't work, but didn't try to debug what went wrong
16:52:37 <eugenesiegel> dergoegge: oops. I think I used to use gcc but I guess not anymore
16:52:41 <fanquake> dzxzg: it depends on the system, the verison of the tools, might require additional options etc
16:53:05 <fanquake> i.e there are breaking changes between lcov 1.6 and 2.0
16:53:22 <dergoegge> I think the llvm reports are also just more readable (see my comment here: https://github.com/bitcoin/bitcoin/issues/31047#issuecomment-2396661558)
16:53:37 <fanquake> using clang / llvm, unless you're installation is somehow broken, everything will just work in harmony
16:53:52 <fanquake> and you get working sanitizers, fuzzing, thread safety etc
16:54:08 <fanquake> (hence why all of our important infra is using it)
16:54:18 <achow101> i think further discussion can happen in the issue
16:54:40 <dergoegge> so no one is using gcc is the conclusion?
16:54:47 <sipa> except release builds
16:54:50 <sipa> :p
16:54:53 <achow101> dergoegge: seems like it
16:54:59 <vasild> :)
16:55:07 <achow101> #topic [IBD] multi-byte block obfuscation (l0rinc)
16:55:09 <dergoegge> :D
16:55:56 <l0rinc> I have already talked about this, thanks
16:56:17 <achow101> ah, it was just asking for more review?
16:56:39 <achow101> Any other topics to discuss?
16:56:40 <l0rinc> and I haven't talked about this before, just decided to join IRC, so wanted to give some context :)
16:58:07 <achow101> #endmeeting