16:02:39 <abubakarsadiq> #startmeeting 
16:02:39 <corebot> abubakarsadiq: Meeting started at 2025-11-20T16:02+0000
16:02:40 <corebot> abubakarsadiq: Current chairs: abubakarsadiq
16:02:41 <corebot> abubakarsadiq: Useful commands: #action #info #idea #link #topic #motion #vote #close #endmeeting
16:02:42 <corebot> abubakarsadiq: See also: https://hcoop-meetbot.readthedocs.io/en/stable/
16:02:43 <corebot> abubakarsadiq: Participants should now identify themselves with '#here' or with an alias like '#here FirstLast'
16:02:51 <TheCharlatan> hi :)
16:02:55 <fjahr> hi
16:02:57 <eugenesiegel> re-hi
16:03:04 <abubakarsadiq> #bitcoin -core-dev Meeting: abubakarsadiq achow101 _aj_ ajonas b10c brunoerg cfields darosior dergoegge dzxzg eugenesiegel fanquake fjahr furszy gleb glozow hebasto hodlinator instagibbs janb84 jarolrod jonatack josibake kanzure kevkevin laanwj LarryRuane lightlike l0rinc luke-jr maflcko marcofleon maxedw Murch pinheadmz provoostenator ryanofsky sdaftuar S3RK stickies-v sipa sr_gi tdb3 theStack TheCharlatan vasild
16:03:05 <abubakarsadiq> willcl-ark
16:03:06 <hebasto> hi
16:03:07 <pinheadmz> hi
16:03:20 <sipa> i for one welcome our new overlord
16:03:28 <abubakarsadiq> There are no pre-proposed meeting topics this week. Any last minute ones to add?
16:03:48 <laanwj> hii
16:03:57 <abubakarsadiq> sorry for being late :) first time thanks sipa
16:03:57 <vasild> hi
16:04:00 <lightlike> hi
16:04:15 <fjahr> I will give a super short asmap update at the end
16:04:32 <abubakarsadiq> #topic Kernel WG Update (TheCharlatan)
16:04:40 <TheCharlatan> Opened an RFC issue to figure out versioning of the kernel header: https://github.com/bitcoin/bitcoin/issues/33911
16:04:47 <TheCharlatan> Please leave some comments :)
16:04:56 <sr_gi[m]1> hi
16:05:06 <TheCharlatan> That's all.
16:05:23 <abubakarsadiq> #topic Benchmarking WG Update (l0rinc)
16:05:31 <l0rinc> We've finished the article about benchmarking and performance optimization for Bitcoin Magazine: Outrunning Entropy: Why Bitcoin Can't Stand Still
16:05:39 <l0rinc> Did a few full reindex-chainstate measurements for different parallelism levels for the InputFetcher to see how the cpu affects the performance: https://github.com/bitcoin/bitcoin/pull/31132#issuecomment-3532063730 - it seems beyond ~4 threads the performance gains are usually negligible.
16:05:46 <l0rinc> Looking for reviewers for two PRs I had to rebase recently: #33738 and #30442 - that's it from me
16:05:49 <corebot> https://github.com/bitcoin/bitcoin/issues/33738 | log: avoid collecting `GetSerializeSize` data when compact block logging is disabled by l0rinc · Pull Request #33738 · bitcoin/bitcoin · GitHub
16:05:53 <corebot> https://github.com/bitcoin/bitcoin/issues/30442 | precalculate SipHash constant salt XORs by l0rinc · Pull Request #30442 · bitcoin/bitcoin · GitHub
16:06:28 <sipa> i plan to review those, but i have a few other things on my plate too
16:06:51 <l0rinc> thank you
16:07:00 <abubakarsadiq> I will also review
16:07:02 <abubakarsadiq> #topic Cluster Mempool WG Update (sdaftuar, sipa)
16:07:44 <sipa> review on #33629 seems to be going well, as the things being addressed seem to be becoming more nitty
16:07:48 <corebot> https://github.com/bitcoin/bitcoin/issues/33629 | Cluster mempool by sdaftuar · Pull Request #33629 · bitcoin/bitcoin · GitHub
16:08:28 <abubakarsadiq> #topic Stratum v2 WG Update (sjors)
16:08:36 <sipa> sdaftuar has been benchmarking the final resulting performance impact of the invocations of linearization, hoping to tune the logic for determining how much budget to give it
16:08:45 <sipa> (that's it for me, continue)
16:09:05 <Sjors[m]1> Some smal interface changes are in PRs.
16:09:43 <Sjors[m]1> And the SRI team is working a Rust client that consumes our IPC interface, which is nice.
16:09:44 <abubakarsadiq> share the links @sjors[m]1 ?
16:09:56 <Sjors[m]1> #33819 
16:10:00 <corebot> https://github.com/bitcoin/bitcoin/issues/33819 | mining: add getCoinbase() by Sjors · Pull Request #33819 · bitcoin/bitcoin · GitHub
16:10:06 <Sjors[m]1> Maybe #33890
16:10:08 <corebot> https://github.com/bitcoin/bitcoin/issues/33890 | mining: add requestedOutputs field, e.g. for merged mining by Sjors · Pull Request #33890 · bitcoin/bitcoin · GitHub
16:10:19 <Sjors[m]1> But it's unclear if merge mining is actually a goal for sv2.
16:10:37 <Sjors[m]1> BlueMatt or someone else should clarify
16:10:43 <yancy> https://github.com/bitcoin/bitcoin/issues/33890
16:10:47 <yancy> oops
16:10:53 <yancy> link to rust client for ipc?
16:10:55 <Sjors[m]1> Or maybe someone who worked on that can chime in on the latter PR.
16:11:16 <Sjors[m]1> yancy: https://github.com/stratum-mining/sv2-apps/pull/59
16:11:18 <BlueMatt[m]> yea, merged mining is definitely a goal
16:11:22 <yancy> thanks
16:11:25 <fanquake> Sjors: can you elaborate on #33899, and why we might need to start doing memory management for external clients?
16:11:27 <corebot> https://github.com/bitcoin/bitcoin/issues/33899 | Block template memory management (for IPC clients) · Issue #33899 · bitcoin/bitcoin · GitHub
16:11:28 <BlueMatt[m]> various miners have it as a requirement
16:11:57 <Sjors[m]1> ^ fanquake it's our memory that holds the transactions, we'll crash long before the client does.
16:12:09 <Sjors[m]1> It's because multiprocess holds shared pointers on behalf of the client
16:12:11 <cfields> Sjors[m]1: while changing the api, maybe fix the missing context param as discussed at coredev? Or has that been addressed somewhere already?
16:12:27 <BlueMatt[m]> presumably bitcoin core should just discard them after a minute?
16:12:34 <Sjors[m]1> cfields: yes, I have a PR with a bunch of small breaking changes that include the context fix.
16:12:43 <cfields> Sjors[m]1: ack, thanks.
16:12:43 <BlueMatt[m]> and if its only generating new templates once every NN seconds at max that would trivially remove the issue?
16:12:45 <Sjors[m]1> cfields: but the changes I listed above are non-breaking
16:12:56 <cfields> 👍
16:13:17 <fanquake> sjors: ok, reading the issue its not super clear, which direction do you want the memory querying to go?
16:13:27 <Sjors[m]1> BlueMatt: ideally i'd like the client to decide when to drop things
16:13:46 <BlueMatt[m]> @sjors:sprovoost.nl that is rather incompatible with the goal of being able to expose this over TCP eventually?
16:13:52 <BlueMatt[m]> but also I'm not clear on why that's important?
16:14:06 <BlueMatt[m]> like, yea, client deciding when to drop things in the short term seems reasonable
16:14:19 <cfields> is that a goal??
16:14:33 <BlueMatt[m]> but if a template has been sitting around for several minutes, then it can presumably be dropped
16:14:36 <BlueMatt[m]> cfields: yes?
16:14:43 <abubakarsadiq> imo changes at this stage for the interface that is beneficial should go in we should  discuss how not break compatibility when we expose a stable interface
16:14:54 <sipa> i wouldn't expect to expose the IPC interface over TCP; it'll incur latency you don't want i think?
16:15:15 <Sjors[m]1> I don't think the IPC interface needs to be exposed. The Template Provider would.
16:15:16 <sipa> run an sv2-ipv protocol adapter locally
16:15:23 <sipa> *sv2-ipc
16:15:41 <BlueMatt[m]> I do not see why you'd want to run a proxy?
16:15:55 <BlueMatt[m]> I mean you'll need to run a proxy if bitcoin core doesn't support listening on tcp, but that can be socat
16:15:55 <Sjors[m]1> And a public facing TP can simply tell clients: don't bother me about templates more than 60 seconds old.
16:16:15 <fanquake> abubakar: I agree. This is so new, has so few users, and we've given 0 expectations, there shouldn't be an issue making any changes at this point
16:16:19 <BlueMatt[m]> Sjors[m]1: I guess my question is why do you want to support templates older than a few minutes at all?
16:16:38 <Sjors[m]1> BlueMatt: maybe because miner ignored the fee bump templates?
16:16:39 <BlueMatt[m]> I mean after ten-ish minutes the templates are pretty likely to be useless :p
16:16:52 <sipa> BlueMatt[m]: my understanding is that they need to be kept for the case a hasher still finds a nonce for one
16:17:16 <BlueMatt[m]> right, but my point is only that after some reasonably long duration it seems like an incredibly safe assumption that no one is still mining on it
16:17:21 <BlueMatt[m]> and thus they wont find a nonce for it
16:17:34 <BlueMatt[m]> maybe a related question - is ten minutes sufficient?
16:17:40 <Sjors[m]1> My point is that stratum v2 software knows what miners need, and if it thinks it's safe to drop a template then it can do that. On our side we don't have to reason about it.
16:18:05 <BlueMatt[m]> like if all 10-minute-old templates are discarded even if the client hasn't marked it explicitly safe to discard does that sufficiently limit memory to not be a concern (I imagine it would?)
16:18:08 <Sjors[m]1> Currently the Template Provider holds on to all templates until 30 seconds after a tip change.
16:18:23 <BlueMatt[m]> even that presumably suffices to not run out of memory?
16:19:10 <Sjors[m]1> If we push a new template every second for 2 hours that's 28 GB assuming maximum mempool churn.
16:19:15 <BlueMatt[m]> as long as bitcoin core is deciding when to build a new template, and its only doing so every 30 seconds, worst-case you have like 40 templates?
16:19:32 <BlueMatt[m]> right, but presumably we aren't sending a new template every second ever?
16:19:35 <BlueMatt[m]> bitcoin core should refuse to do that
16:19:39 <Sjors[m]1> We do if the client asks
16:19:48 <BlueMatt[m]> we should stop doing that :)
16:19:56 <Sjors[m]1> And I also think we shouldn't decide what the safe minimum frequency is.
16:20:00 <BlueMatt[m]> well, if a client asks we can send the one we just generated :)
16:20:03 <sipa> the IPC interface is trivially DoS vulnerable anyway - it inherently assumes a client that isn't going to DoS it
16:20:27 <abubakarsadiq> fanquake: we would want to handle memory internally because currently we hold on the generated block templates, waitnext create templates after after second when their is fee increase. their is a scenario where there is lots of inflow that improve mempool fees continuously and you generate lots of templates and keep in memory. If you don't manage and have a limit it thats a potential problem I think
16:20:31 <Sjors[m]1> Currently the IPC takes an argument which sets that minimum update frequency (in seconds).
16:20:37 <BlueMatt[m]> sipa: if you assume the protocol is more strictly checked so that the messages aren't invalid is that still true?
16:20:53 <BlueMatt[m]> Sjors[m]1: why? shouldn't that be something bitcoin core decides?
16:20:55 <sipa> BlueMatt[m]: yes
16:21:02 <BlueMatt[m]> bitcoin core knows what the feerate difference is between different templates
16:21:15 <Sjors[m]1> BlueMatt[m]: No because we can do it 10x per second but node software of ASIC firmware might crash if it can't keep up.
16:21:20 <BlueMatt[m]> so it seems like it should always be bitcoin core deciding when to issue new templates (maybe based on config, but certainly not based on when clients request)
16:21:35 <Sjors[m]1> We send a new templates when fees rise by N sats, but only if it's been at least M seconds.
16:22:02 <BlueMatt[m]> right, but presumably if the client is limited to N updates per second on their asic they will simply not push the new work and ignore it?
16:22:13 <BlueMatt[m]> that's what ultimately will happen in the sv1/2 client on the device anyway
16:22:42 <BlueMatt[m]> like even if the translator feeds templates in 100 times a second the firmware will just not switch fast (unless you set the flush flag, ofc)
16:23:16 <BlueMatt[m]> anyway, all I'm saying is fewer config knobs for the template IPC client means less effort thinking about all this stuff, and I'm quite skeptical the config knobs are useful :)
16:23:35 <Sjors[m]1> Those are all things miners / pools should be experts in, and we have no idea what values to pick.
16:23:50 <BlueMatt[m]> sipa: I'll follow up and ask why later, but that seems like something we should resolve (at least if we limit the interface to just the mining IPC and not the various other IPC things)
16:23:54 <Sjors[m]1> The Template Provider can hardcode values.
16:24:14 <BlueMatt[m]> hmm? I'm really unclear on which of the above are an issue?
16:24:29 <BlueMatt[m]> my point is that it doesn't matter how fast bitcoin core generates templates, the mining device will rate-limit it itself
16:25:00 <sipa> so it should tell bitcoin core / template provider / something sv2, what that rate limit is?
16:25:02 <BlueMatt[m]> so istm bitcoin core should be configurable (via bitcoin.conf) to generate templates based on fee increases or time as needed, with reasonable defaults (seems like that's already a thing?) and from there we don't need to worry about individual clients
16:25:11 <Sjors[m]1> Bitcoin Core can ship only one release per 6 months. I'd rather have a few too many IPC parameters that the TP doesn't use, then having to argue about changing something in the codebase because new miner suddenly wants N or M different.
16:25:27 <BlueMatt[m]> sipa: no, I dont think so because you may have 1000 devices with different limits hanging off of one template generation coming out of bitcoin core
16:25:44 <BlueMatt[m]> the templates themselves shouldn't be rate-limited coming out of bitcoin core based on hardware limitations
16:26:01 <sipa> i don't follow at all anymore
16:26:08 <BlueMatt[m]> Sjors[m]1: my related comment was that the config knobs for this should be in bitcoin.conf, not the IPC client :p
16:26:08 <sipa> and i don't think this is a good use of this meeting time
16:26:24 <BlueMatt[m]> okay, well clearly this needs more discussion, maybe we schedule a followup call
16:26:29 <BlueMatt[m]> so we can do it live rather than over irc
16:26:30 <Sjors[m]1> Right, seems better to discuss in one of the above issues.
16:26:41 <abubakarsadiq> moving on
16:26:44 <abubakarsadiq> #topic Net Split WG Update (cfields)
16:26:59 <cfields> Very sorry to all who are waiting... still nothing to report. Working through a few other things before shifting my focus 99% to this.
16:27:31 <abubakarsadiq> #topic asmap update (fjahr)
16:27:46 <fjahr> Hi! The original embedding PR has been slip up into a triplet. The first part is already merged by now. The second part contains necessary refactorings and extensive added documentation in the implementation which was briefly discussed at CoreDev and should hopefully help getting people through this toughest part of the review. The third adds the build stuff (and that is the PR that used to be the main thing).
16:27:52 <fjahr> There have also been some spin-off discussions on how the -asmap arg should work and this resulted in two separate PRs that are currently open. All this can be found through the freshly rebooted tracking issue: #33879.
16:27:53 <corebot> https://github.com/bitcoin/bitcoin/issues/33879 | Embedded ASMap data Tracking Issue (Part 2) · Issue #33879 · bitcoin/bitcoin · GitHub
16:27:58 <fjahr> Special thanks to hodlinator who has given very valuable feedback in his detailed reviews in the last two weeks ❤️
16:28:10 <fjahr> That’s it unless there are questions.
16:28:33 <fanquake> fjahr: Wanted to follow up with one of mine
16:28:40 <sipa> fjahr: i have been very hard to refrain myself from suggesting yet another "how -asmap argument should work", because i really don't care and want people to agree on something :p
16:28:59 <sipa> +trying
16:29:27 <fjahr> hehe, yeah, seems like everyone has their favorite -asmap :)
16:29:34 <fanquake> In regards to recreating the files in https://github.com/asmap/asmap-data. How does someone go about that
16:29:56 <fanquake> That was my question in regards to how would someone build from source, without having to take some binary blob as an input
16:30:21 <sipa> it's time sensitive, so if you repeat the gathering process at a different time, you'll get a different result
16:30:55 <fanquake> Yes, but is the raw data used to create those blobs saved somewhere?
16:31:03 <BlueMatt[m]> best you can do is re-run it and do some kind of diff telling you total IPs different
16:31:11 <fjahr> The runs we do with multiple people are coordinated in issues such as this: https://github.com/asmap/asmap-data/issues/34 If there is agreement between enough people about the included data, that is shared there.
16:31:12 <sipa> BlueMatt[m]: we have a diff tool
16:31:18 <fanquake> Otherwise we are going to end up in a situation where you can't actually build a release bin from source, without taking some binary blob
16:31:29 <fanquake> which some group, and some time, generated
16:31:32 <fanquake> *at
16:31:43 <fjahr> Then the file is processed and merged into the repo with a PR such as this: https://github.com/asmap/asmap-data/pull/35
16:31:47 <fanquake> it'd be nice if that didn't become mandatory
16:32:03 <sipa> so that would imply storing all the data fetched by kartograf in the repo?
16:33:00 <fanquake> I assume so. Asking somewhat in the context of wether we have a build option to disable this feature. I imagine we might want that, if someone want to be able to compile with having to use this data, which they can't actually recreate
16:33:01 <fjahr> The raw data that is downloaded by each participant could be shared somewhere too, this is about 2G and it will be slightly different for each participant so we would have again decide who should upload it and where.
16:34:06 <fjahr> It's a snapshot of the internet routing table, I think to some degree it will be that we are working with data that some group at some point generated.
16:34:34 <fjahr> The process we are doing is trying to match the level of scrutiny of a core PR.
16:34:50 <fanquake> Sure, I am just trying to keep things trust minimized, and introducing new blobs, which can't be recreated after the fact, is moving in the other direction
16:35:15 <sipa> i think that's inevitable because it's ultimately based on non-verifiable data
16:36:27 <sipa> even if we save the source data gathered during the process, to enable re-creating the asmap.dat determinstically from that data, it's just moving the trust question elsewhere: was this dump created honestly?
16:37:08 <vasild> is the dump easier to inspect than asmap.dat?
16:37:17 <darosior> Could it be something that is shipped separately from the binary?
16:37:20 <fjahr> We can inspect the data and do checks to make sure the data is reasonably realistic but fully validating it would be unresable
16:38:09 <sipa> i doubt it's easier
16:38:23 <vasild> if I do fetch myself and create my own asmap.dat from my dump, can I diff it against the shipped asmap.dat with the diff tools or do they run on the dumps?
16:38:25 <fanquake> Yea, I agree, but having the data at least lets anyone that wants too, not have to take a blob, that they can re-create it themselves. Maybe if we have more of the group involved in the creation of these inputs, it's also better
16:38:31 <fjahr> A manipulated map where all peers are in one bucket is easy to detect for example
16:39:31 <sipa> fanquake: but in what way is the kartograf fetched data dump different from "have to take a blob", it's still a blob you have to trust, just a much bigger one, in multiple formats, with a ton of information that's irrelevant for us
16:39:32 <TheCharlatan> darosior I don't think that would significantly improve things.
16:39:34 <l0rinc> fjahr: are the automatic (statistical?) validations for that?
16:40:14 <instagibbs> sipa having regular process with more than 1 person is good to stave off process rot if not a security enhancement
16:40:15 <sipa> l0rinc: we report the asn variety in addrman, according to the provided asmap data
16:40:26 <sipa> instagibbs: there is
16:40:33 <fjahr> You mean the health check? Yeah, that's information that is printed though there is no threshold where it sets of an alarm or so
16:40:35 <fanquake> sipa: sure, but if everyone who signed off on the smaller blob, also dumps there data, I can atleast recreate the thing they are all attestign too, using the same tools
16:41:01 <instagibbs> sipa 👍 (it tried to have me pick ship emoji, telling)
16:41:45 <darosior> TheCharlatan: why? It's separating concerns. Keep the binary entirely reproducible. Have the semi-reproducible data alongside, which can be discarded if desired (but still shipped and enabled by default).
16:42:01 <sipa> darosior: 2G per participant is a lot of data
16:43:26 <darosior> sipa: not sure i follow. I'm just asking why not shipping the compressed data alongside the binary. For instance in /usr/share, in the same way we ship the multiprocess binaries in /usr/libexec
16:44:23 <sipa> darosior: as i understand it, the source data is gigabytes per participant, and it won't be the same for everything - it's just the resulting asmap data that doesn't change
16:44:28 <sipa> (mostly)
16:44:38 <sipa> *for everyone
16:44:49 <darosior> Yes i'm talking about the data that would be embedded in the binary
16:44:56 <TheCharlatan> darosior, not sure that is really relevant, I would treat this similarly to any other chainparams data. But thinking aloud here, it might still be nice to do that in case people would like to udpate it without having to either change their configs, or their binary.
16:45:29 <sipa> darosior: the asmap.dat file? you can just extract it from bitcoind (i think there was a plan of adding an RPC to dump it out)
16:45:54 <sipa> i think the rest of the discussion is here about the source material from which the asmap.dat file is built
16:45:59 <sipa> which is routing table dumps
16:48:51 <vasild> is it easier to compare two routing table dumps VS compare two asmap.dat?
16:49:21 <vasild> compare = see how many % difference there is
16:49:37 <fjahr> the two asmap, the dump has even more data we don't care about so we don't need to look at that because we don't use it anyway
16:50:06 <fanquake> (I'll follow up in https://github.com/asmap/asmap-data / the relevant PRs)
16:50:21 <abubakarsadiq> Anything else to discuss?
16:50:47 <fjahr> The tracking issue could also be a venue for discussion, fwiw.
16:50:47 <vasild> ok, so if I do not trust the binary blob asmap.dat and if I want to "verify" it, I can generate my own and diff it and if the diff % is below some number, then it is ok.
16:51:24 <abubakarsadiq> thanks fjahr link here https://github.com/bitcoin/bitcoin/issues/33879
16:52:17 <BlueMatt[m]> ha cool my flight got diverted and it reset the wifi and made me buy it again, sorry if some messages got delivered late
16:52:53 <fjahr> vasild: maybe :) a low diff is a good indication there is nothing completely wrong but there would still be manipulation which targets a specific subset of nodes. So we need to look at coverage of nodes and that there asn distribution is still feasible at least.
16:54:03 <vasild> fjahr: have a built in asmap.dat diff tool which does that against the own addrman ;)
16:54:24 <fjahr> Happy to answer more questions after the meeting or in the tracking issue or whereever
16:54:35 <abubakarsadiq> novo__: mentioned to me that he has no update this week on silent payments wg
16:54:41 <abubakarsadiq> with that
16:54:46 <abubakarsadiq> #endmeeting