14:00:33 <achow101> #startmeeting 
14:00:36 <tdb3> hi
14:00:38 <achow101> #bitcoin -core-dev Meeting: achow101 _aj_ amiti ariard aureleoules b10c BlueMatt brunoerg cfields darosior dergoegge dongcarl fanquake fjahr furszy gleb glozow hebasto instagibbs jamesob jarolrod jonatack josibake kallewoof kanzure kouloumos kvaciral laanwj LarryRuane lightlike luke-jr MacroFake Murch phantomcircuit pinheadmz promag provoostenator ryanofsky sdaftuar S3RK stickies-v sipa sr_gi theStack TheCharlatan vasild
14:00:46 <pinheadmz> hi
14:00:47 <willcl-ark> hi
14:00:47 <vasild> hi
14:00:48 <kanzure> hi
14:00:52 <achow101> There are no pre-proposed meeting topics this week. Any last minute ones to add?
14:00:56 <bitcoin-git> [bitcoin] marcofleon opened pull request #30980: fuzz: fix bug in p2p_headers_presync harness (master...2024/09/headers-presync-fuzz-bugfix) https://github.com/bitcoin/bitcoin/pull/30980
14:01:02 <dergoegge> hi
14:01:05 <fjahr> hi
14:01:40 <hebasto> hi
14:01:52 <achow101> #topic Ad-hoc high priority for review
14:01:53 <achow101> Anything to add or remove from https://github.com/orgs/bitcoin/projects/1/views/4
14:03:18 <achow101> #topic 28.0
14:03:30 <achow101> Any new issues in 28.0rc2?
14:04:31 <achow101> Current backports PR is #30959. Since the 2 fixes being backported are trivial, I'm plannin on skipping rc3 and tagging final once that is merged
14:04:32 <gribble> https://github.com/bitcoin/bitcoin/issues/30959 | [28.x] backports and finalize (or rc3) by achow101 · Pull Request #30959 · bitcoin/bitcoin · GitHub
14:04:56 <willcl-ark> That sounds reasonable :)
14:05:19 <achow101> Please also review the draft release notes https://github.com/bitcoin-core/bitcoin-devwiki/wiki/28.0-Release-Notes-Draft
14:06:02 <sipa> hi
14:06:11 <achow101> anything else to discuss?
14:06:15 <fjahr> Just a quick PSA: I have moved all my ASMap related repos to a dedicated org (https://github.com/asmap) to make collaboration easier. Old links should still work via redirect but if you see an outdated link please update because GH says they don't maintain those redirects forever.
14:06:20 <willcl-ark> (reading first bit of that): perhaps we should discuss macos notarization? i.e. #15774
14:06:21 <gribble> https://github.com/bitcoin/bitcoin/issues/15774 | macOS App Notarization & Stapling · Issue #15774 · bitcoin/bitcoin · GitHub
14:06:46 <willcl-ark> is the plan to give in to Apple's requirements and fully Notarize and staple?
14:07:14 <pinheadmz> piggy back #29749 on that >
14:07:14 <gribble> https://github.com/bitcoin/bitcoin/issues/29749 | release: ship codesigned MacOS arm64 binaries · Issue #29749 · bitcoin/bitcoin · GitHub
14:07:36 <achow101> fjahr: good to know. IIRC github does keep redirects for a long time, at least I've had a redirect that's been working for 5 years now
14:07:46 <achow101> #topic MacOS notarization
14:08:04 <achow101> willcl-ark: I don't think there's really been a plan yet?
14:08:13 <achow101> but it does seem like we should probably figure it out for the next release
14:08:36 <fjahr> achow101: yeah, that's just what I read, I guess they want to have the option to remove them at some point
14:08:47 <sipa> if macos is phoning home regardless of stapling/notarization, there seems to be little downside to doing it
14:08:48 <willcl-ark> Exactly THe current requirement is that a user must generate a codesigning cert, then codesign, and remove the quarantine bit. We can't really get around it AFAIK
14:10:10 <Chris_Stewart_5> Seems like a note about this in the release notes would be a good first step? I've run into this multiple times at this point and its a PITA finding the exact commands that need to be run to make bitcoind work on mac
14:11:08 <Chris_Stewart_5> a quick glance of the 28.0 release notes doesn't contain anything AFAICT. Seems like putting this in the 'How to upgrade' section would make sense? :man_shrugging:
14:11:08 <gmaxwell> If OSX phones home the software users run it might be worth a persistant note on that too in the docs for running on OSX.
14:11:22 <achow101> I've moved #15774 to the 29.0 milestone
14:11:24 <gribble> https://github.com/bitcoin/bitcoin/issues/15774 | macOS App Notarization & Stapling · Issue #15774 · bitcoin/bitcoin · GitHub
14:11:34 <sipa> gmaxwell: right
14:11:45 <achow101> Chris_Stewart_5: if you know what the commands are, feel free to add it yourself
14:11:57 <Chris_Stewart_5> Can do assuming everyone is on board
14:12:34 <sipa> Chris_Stewart_5: does the software actually not work if you don't?
14:12:38 <fanquake> My understanding is that our binaries just wont work at all on the latest macos release, and there's no-longer workarounds available
14:12:43 <willcl-ark> It does not work at all
14:12:44 <sipa> oh wow
14:12:46 <Chris_Stewart_5> sipa: it doesn't start and gives a cryptic error
14:13:01 <sipa> i assumed it was just a "warning unverified software" popup or so
14:13:03 <fanquake> so we are basically shipping no-ops for the latest version of macos
14:13:07 <Chris_Stewart_5> I don't have it handy atm.... i'll dig up the commands again. I haven't done it since the 27.1 release iirc.
14:13:23 <hebasto> is ad-hoc self signing no longer works as a workaround?
14:13:24 <Chris_Stewart_5> sipa: This also applies to bitcoind, bitcoin-cli
14:13:36 <willcl-ark> I had a think about adding an "auto-sign" script for macos: https://github.com/bitcoin/bitcoin/compare/master...willcl-ark:bitcoin:codesign-script but you still need a self-signed signing certificate
14:13:39 <achow101> sipa: it started that way
14:13:43 <vasild> well, if macos is phoning home and bitcoind does not work on macos, then even better. Why do we need to do anything?
14:13:44 <Chris_Stewart_5> hebasto: it does, but where are the commands? How are people suppose to find them?
14:13:45 <emzy> AFAIK it is this: xattr -rd com.apple.quarantine /path/to/applicationname.app
14:13:56 <willcl-ark> fanquake: no, I'm pretty sure the self signing and removing quarantine bit still works?
14:14:11 <Chris_Stewart_5> to be clear, this is probably a stop gap, things should probably just work :tm: ? But giving people instructions as a workaround for now seems appropriate?
14:14:19 <sipa> Chris_Stewart_5: agreed
14:14:23 <achow101> Chris_Stewart_5: yes
14:14:34 <fanquake> willcl-ark: not sure, but anyone using bitcoin-qt, which is primarily what we ship on macos, isn't doing any of that
14:14:43 <willcl-ark> fanquake: agree
14:14:48 <sipa> vasild: the world would be a lot better place if all humans were reasonable, but if that were the case, we probably wouldn't need bitcoin at all?
14:15:17 <willcl-ark> (but I have for sure run bitcoind 28rc2 after self-signing)
14:15:19 <vasild> sipa: yeah, I was half-joking ;-)
14:16:15 <sipa> to self-sign, you need a self-signing certificate? can you do that without apple's involvement?
14:16:16 <achow101> Any other topics to discuss?
14:16:22 <sipa> are there workarounds besides that?
14:16:38 <achow101> sipa: there's "ad-hoc signing" which doesn't actually require a certificate
14:16:38 <willcl-ark> sipa: not that I'm aware of. {Perhaps disable the entire SIP mechanism?
14:16:40 <Chris_Stewart_5> sipa: No need to involve apple, you self sign and the cert gets stored locally iirc
14:16:48 <glozow> re topics, I was wondering about cluster mempool / TxGraph status? If sipa has an update on that?
14:17:07 <sipa> glozow: happy to give an update
14:17:27 <achow101> #topic cluster mempool update
14:17:32 <glozow> :D
14:18:01 <sipa> so, we're currently imagining something like 3 layers for the cluster mempool code
14:18:26 <Chris_Stewart_5> One last comment on the notarization problem on mac, here is a bitcoin stack exchange question detailing the problem and solution: https://bitcoin.stackexchange.com/a/117101
14:18:40 <gmaxwell> vasild: considering how easy it is to enumerate people running bitcoin from the p2p network the fact that it phones home isn't hard incompatible with bitcoin I think.  But perhaps it would be a very unwelcome surprise to people using bitcoin behind tor.
14:19:21 <sipa> * the bottom layer (depgraph) is pretty much done, apart from 30857 (which i don't plan to change anymore barring significant review comments). It deals with individual transaction clusters in a very abstract way, but contains all the computationally-hard stuff.
14:20:51 <vasild> gmaxwell: yeah, and it depends on what it phones home - maybe it would automatically backup important files to the cloud (like wallet.dat). A friend of mine complained that some anti virus moved his wallet.dat (!) I wouldn't be surprised if it sent it to the anti virus company for an analysis, if it treated it as a virus.
14:20:54 <sipa> * the middle layer (txgraph) is sort of a barebones mempool, knowing just about all transaction fees/sizes and dependencies between them (but has no concept of txids, outputs, inputs, prioritization, ...), clusters of them, and linearizations for them...; i have been working on fleshing the design for this out, but i don't have anything to show yet, sorry
14:21:58 <sipa> txgraph would also have a notion of "changesets", proposed sets of transaction additions/removals/dependencies to add to the mempool, so a rbfs can be staged (create a changeset for a proposed RBF, inspect the feerate diagram changes it would make, and then decide to throw it away or commit it)
14:22:27 <sipa> * the top layer is the mempool/validation code, which would use txgraph but add actual transactions, policy rules, validation
14:22:47 <sipa> the point of txgraph is that being more abstract it can be tested in isolation much more easily
14:23:48 <sipa> suhas' current PR (28676) contains a prototype for txgraph, but i'm working on rewriting it as a separate PR, after which 28676 would be rebased on that
14:24:17 <glozow> This sounds really cool!
14:25:32 <glozow> Thanks for the update sipa
14:25:49 <tdb3> thanks sipa!
14:26:34 <achow101> anything else to discuss?
14:26:49 <Chris_Stewart_5> sipa: Perhaps cart before the horse, but what do we think the deployment story looks like?
14:27:21 <Chris_Stewart_5> last I heard, the idea would be the totally replace existing mempool code with the new cluster mempool code, is that still the plan (presumably in 29.0)?
14:27:55 <sipa> Chris_Stewart_5: i think everything up to txgraph (which may well end up becoming multiple PRs) can be staged in, as in merged without actually changing any observable behavior, as it's just new data structures and algorithms that can be tested in isolation
14:28:49 <sipa> but the introduction of cluster-based mempool will be a hard switch; i don't think we realistically want both the old and new code ever simultaneously in production (it'd involve the worst of both worlds: the memory and CPU costs related to having both, plus the policy rules that are necessitated by both).
14:29:21 <sipa> i could imagine the introduction of cluster mempool, and the removal of the old mempool data structures to be in separate PRs only if the intent is for both to go in in the same release.
14:29:54 <Chris_Stewart_5> I guess I was thinking more along the lines of a -clustermempool flag in 29.0 and then assuming everything goes good, default to -clustermempool in 30.0 and remove old code in 31.0? Perhaps this a bit too conservative though?
14:30:06 <sipa> there is no way we can do that
14:30:11 <bitcoin-git> [bitcoin] willcl-ark opened pull request #30981: ci: add timestamps to cirrus jobs (master...cirrus-timestamps) https://github.com/bitcoin/bitcoin/pull/30981
14:30:47 <Chris_Stewart_5> Is this written about anywhere why its not possible? Or just have to read through code?
14:30:53 <sipa> i just told you :)
14:31:07 <sipa> it would involve the worst of both worlds
14:31:35 <Chris_Stewart_5> Yeah, and I guess I assumed only 1 of the mempool implementations would be running (-clustermempool or legacy mempool). Not both.
14:31:56 <sipa> they're just data structures that need to be maintained - if they're compiled in, they will be used
14:32:09 <sipa> at the very least you'd get the memory usage from both
14:32:30 <sipa> it's not like we're creating a completely new mempool and can decide at runtime which of the two you're using
14:32:59 <_aj_> if you're nervous about the new mempool code in 29.x you'd just keep running 28.x anyway?
14:33:21 <Chris_Stewart_5> Runtime performance and code maintenance are different topics. I agree there will be maintenance burden for awhile. Idk if i'm so convinced on runtime performance, but you have more experience than i do on the topic so i'll defer to you
14:33:24 <sipa> we do have simulations based on replays of all mempool data to get an idea of what would be impacted by the policy changes that cluster mempool entails
14:33:58 <Chris_Stewart_5> _aj_: Yeah, and then we give people another reason not to upgrade with the latest and greatest stuff and get them stuck on 28.x.
14:34:26 <_aj_> Chris_Stewart_5: if they don't upgrade from 28.x until 30.x is out, that's already fine and supported
14:34:38 <Chris_Stewart_5> Any way, perhaps i'm concern trolling at this point so i'll keep quiet. I am excited about clustermempool and think its great!
14:35:30 <sipa> Chris_Stewart_5: i think the biggest issue is the policy rules... the current mempool has certain policy rules that are necessitated by its implementation (primarily the ancestor and descendant set limits); cluster mempool has different policy rules (cluster size limit, which isn't exactly the same as ancestor or descendant limits, but supports higher numbers)
14:35:42 <sipa> if we keep the old mempool data structures, we need to keep the old policy rules
14:35:55 <sipa> if we introduce the new mempool data structures, we need to introduce the new policy rules
14:36:28 <sipa> so having a release that supports both would effectively entail having the combination of both policy rules (neither ancestor/descendant limits can be violated, nor cluster size limits).
14:36:56 <glozow> I imagine it would be even more invasive to add code everywhere (validation, policy, mining, etc) to have 2 mempools. And they'd have different transactions in them... sounds like a nightmare
14:37:30 <glozow> (And I can't think of another way to support both)
14:37:46 <Chris_Stewart_5> I'll think about this more. We can move on
14:37:46 <sipa> Chris_Stewart_5: yeah i understand the concern; it will be a nontrivial and abrubt change - but i don't think the alternative is very realistic
14:37:55 <vasild> "I guess I assumed only 1 of the mempool implementations would be running"
14:38:25 <sipa> a compile-time switch between them may be vaguely possible
14:38:34 <sipa> but even that is pretty hard
14:39:36 <achow101> anything else to discuss?
14:39:54 <tdb3> Sounds like the probability of bugs would be higher in a conjoined/transition implementation than with a clean update. And an end goal is minimizing bugs
14:42:01 <achow101> #endmeeting