Make Transactions Count (PR 3248)

I’ve been quiet in this blog for a few months because I’m working on a book tentatively titled Dogecoin Tricks. While that hasn’t exactly slowed my development activities, it hasn’t entirely sped them up either.

On the other hand, several tips I’ve written for that book have taken me into the source code to ask “what’s going on” or “does this exist” or “why doesn’t this exist”.

Some Questions Should Be Easy to Answer

For example, the question “How many transactions are on the Dogecoin blockchain?” seems like an easy question to answer. The entire point of a blockchain is to keep track of every transaction, in a public form, in a dataset distributed among many, many computers around the globe. This should be an easy question to answer!

My first stop when facing any question like this is to ask “Is there an RPC command to give me this information”, so I cycled through all of the Dogecoin RPC commands and couldn’t find the right information.

“Cool,” I said. “That means someone needs to add this feature.” I figured that if I wanted this information, other people wanted it too. Sure, you can go to an API like the Blockchair Dogecoin explorer and browse around for the information, but part of the point of a distributed cryptocurrency (and some of the motivation for me writing the book) is to have this information available on your own where you don’t have to depend on other people to answer your question and you can verify things on your own.

Obviously Dogecoin Core should provide a way to get this information. That’s what PR 3248 tries to do.

A Blocking Strategy

To understand all of my attempts in code, you have to understand a little bit about how the Core represents transactions. When you send or receive Dogecoin and the network accepts that transaction, miners collect valid transactions into something called blocks. The act of mining gathers these transactions together, calculates a signature representing the contents of the transactions in the block, the signature of the previous block, and a random number that makes the block’s new signature conform to a mathematical puzzle, and then BAM the complete block propagates through the network and reaches every node.

If you’re running a node, your node will get this new block and look at its contents and write its contents to disk with information such as the signature of the block (its hex, as we sometimes call it) and the number of the block (its height, we often call it).

If you’re running a full node with the entire Dogecoin blockchain history, you have every block saved on your disk somewhere in your .dogecoin directory, wherever that is.

Good Idea, Bad Implementation, I’m the One with the Pun

“Aha,” I said. “All I have to do is look at every block, count the number of transactions in each one, and add all those numbers together.”

In programming terms, this means I wrote a loop.

When I tried the code, I thought I’d crashed my system. For a long time, nothing happened. I sent my RPC command to the core and waited. Then I waited longer. Then I waited some more. Eventually I decided I’d waited long enough and stopped the program.

Then I added some debugging information to count how many times I could run through the loop in a second. The numbers were not good.

There are about 4.678 million blocks right now, so my loop had to look at each one of those. That’s not a lot for a modern computer to do, except that to look at each block, I asked the core “Look for the file on disk containing this block, open the file, find the block in the file, read the block, then give me the transaction count.”

This isn’t wrong. It’s accurate. It’s also incredibly slow. I could get the right answer in maybe a couple of hours on a good machine. That’s way too slow.

Maybe You Should Remember Harder

I opened a pull request with that code anyhow, expecting that I’d get feedback like “You should do this some other way”. I did; Patrick didn’t care for the implementation.

This is actually a good strategy; I like getting feedback early and often. I had enough code to show the outcome I wanted, so we could debate two things. First, is this an outcome anyone else understands and wants (or at least understands that other people might also want it). Second, are there better ways to accomplish things.

One of the non-obvious problems of opening every file on disk to look for each block is that your node may not store blocks in order. Disk files can contain lots of blocks; there’s not one file per block. You could end up opening and closing the same couple of thousand files 4.678 million times, which is a lot of wasted work.

For my second attempt, I used an internal Dogecoin Core data structure called a block index. This is an in-memory representation of a block, and it contains data on the number of transactions in each block. The loop is more or less the same except there’s no (obvious) file I/O needed. The job that took a couple of hours to run by reading a bunch of files from disk over and over took less than a second with this change.

Are there circumstances where this approach doesn’t work? I’m not sure.

That Thing I Keep Seeing Looks Familiar

Then the other day, I had to reboot a cloud VM with a full node running, so I was watching the debug log while doing some other work. I noticed something suspicious in the log output for every new block received.

    2023-04-15 21:37:17 UpdateTip: new best=... height=4678125 ... tx=95260806

Do you see it too? What does tx mean? It sure seems like a transaction count, and it matches transaction counts I can find elsewhere for the same block. I must have seen this particular line in log output many, many times and yet this is the first time it really jumped out at me.

“Self,” I said. “If the Core can print this number in debug logs, surely it has this number somewhere.”

I searched the code for the logging of UpdateTip and found code named UpdateTip that uses something called nChainTx in a CBlockIndex object.

This CBlockIndex is the same internal object I used in my second loop). The field called nChainTx represents the number of transactions in the entire chain through the current block represented by the CBlockIndex.

I don’t even need the loop, if this value is present.

Does it Work?

Will this work?

I’m not sure. There may be failure conditions for a pruned node or a node that has yet to sync fully. It may be worthwhile to fall back from the simple one-element lookup to the index loop. I’m not sure yet.

The good news is that it looks like I have an easy way to get my answer, and I have working code that connects all the pieces together.

The decent news is that writing these loops was actually very quick; it took longer to remember how to make RPC arguments interpret as integers instead of strings.

The embarrassing news is that I did things first the difficult and expensive way, then an easier way, and only after I looked at something entirely different did I realize that there was an easy way.

That’s how it goes sometimes. When you see code that actually gets accepted into free and open source projects like this, you don’t see all of the experiments that went into making things smart and effective. You don’t see the dead ends we go down sometimes or the false starts or the ways we make our lives more difficult than they should be because we don’t always see or realize the easy things that are already there.