I’m Tom. I’ve been working on the Zap desktop wallet for the last six months or so. If you’re not familiar, we’re trying to build consumer facing Lightning applications. We’ve currently got an iOS app and a desktop wallet. They’re pretty similar, the interface is pretty similar between them but they’ve got a slightly different focus. I’ve been working on the desktop wallet. Can I just get an idea of how many people here have used Zap? Quite a lot. That’s probably about 80-90%, that’s awesome.

I want to take you through a couple of different things. First I want to talk about my experience of building on Lightning and what I found is involved in that. Then I’m going to take a look under the hood of Zap. I’m going to get a little technical but I’m not going to be showing code samples or anything like that. I want to give you a sense of how it’s interacting with Lightning and what the communication stream is like and what we’re doing underneath the interface that you guys see. I’m going to show you some of the new features and things we’ve been working on. I’m going to touch on some of the challenges that I’ve faced and some of the insights that I’ve got from working on this stuff over the last few months.

So starting off, building on Lightning. The first thing I would say is that it is basically a web app at its core. Technically it is a React application and it is running in Electron. If you’ve done web development before and you’re familiar with client-server architecture it is pretty similar. When you’re building websites you’re maybe using Stripe API or PayPal API to make payments, we’re just using lnd under the hood. Everything you’re doing in the interface is literally sending messages over to lnd and lnd is doing all the interfacing with the Lightning Network. We’re getting realtime data back from lnd and we’re presenting that and allowing people to interact with the network without really knowing that’s what they’re doing. The tools and processes that we’re using to build out the app are really similar to if you’ve done any web development. It’s literally a Javascript application. You guys will probably be familiar with that sort of thing. We’re running continuous integration with Travis and AppVeyor. We’re using Jest for our automated testing. We have a pretty much zero touch deployment process whereby when we commit everything to GitHub, every commit triggers a build which triggers the test suite which triggers packaging up the application into the Electron package. That gets uploaded to GitHub. We always have the latest version of Zap available on GitHub for download. We actually just publish, when we’re ready to do a release, we publish the most recent version. We’ve always got those builds available in the background.

Essentially it is really just a wrapper around lnd. We’ve built into it support for running lnd locally on your machine so you can run lnd on here via zap-desktop. You can also connect it to a remote node as well. In both cases it is just working as a remote control. The difference is that when you’re running lnd locally you’re communicating with gRPC locally on that node. We basically have two different Javascript processes running. We have the browser environment which is basically a Chromium environment that runs inside Electron. That’s giving the browser window, essentially what you’re seeing is a web browser with the React application running in it. We’ve got lnd running in the background and we’re using the gRPC API to communicate between the two. Because we’re managing lnd as an external process so we have the lnd binary packaged up in that Electron app, we literally use Node spawn so we have full Node.js facilities within there. We use Node, we spawn that process. We have a finite state machine that sits on top of that lnd process so that we can more easily manage transitioning it from one state to another. When we… the app we set the state machine in a ready state and then when you access a wallet we transition to a running state as we start the process. We attach some event listeners to the lnd log output and we monitor the process. We can stop it and restart it and things like that. We’ve also got some supporting infrastructure, more backend stuff. Specifically we have a cluster of btcd nodes which are our Bitcoin full nodes. If anyone is not familiar with btcd, it is a full node implementation similar to Bitcoin Core. We have a few instances of that running. We currently run them on the Google Cloud environment so we have Docker containers. If you go to our GitHub we’ve got two different Docker containers. We’ve got a lnd one and a btcd one. We use Kubernetes to deploy those out to Google Cloud. The actual full blockchain data is stored in Google persistent drives which we attach the stateful drives to the node instances up on Google Cloud. We have those because when you’re running lnd locally on the machine, lnd connects to our btcd nodes. It connects to a couple of them in parallel and it synchronizes the blockchain data from those nodes. If you’re using the remote node it doesn’t touch our btcd nodes at all because you’re connecting to your own setup where you’ve got your own full node.

Diving a little further under the hood. When you start the app, the first thing we do is we go through some sanity checks. We check the filesystem, we check that your wallet is aware. We expect it to be so when we start lnd we create a wallet which creates a lnd directory in a specific location. It actually gets stored in the user data directory from Electron. It will be in a slightly different place depending on which environment you’re running on, Mac or Windows or whatever. It’s essentially application state data. We also use IndexedDB. Within the Electron browser environment we’re using IndexedDB which is a native browser based database. In that we store some persistent data. We don’t store a lot but we store your wallet configuration. When you spin up a remote connection you’ve got to tell lnd where’s your macaroon, where’s your certificate, things like that. That configuration we store in IndexedDB. We store some user preference data, what theme you’re using whether it is the light theme or the dark theme. What language you’re using or what currency you’ve selected. There’s only a small amount of data that we’re using currently. Previously we were using files on the filesystem, we had a JSON file with a little bit of config in there. We found that as we started to build it out more we needed something with a bit more of a rich API than just accessing a file. IndexedDB is really lightweight and built into the Chromium environment so we’re using that for our persistent data. As we start to build things out more there are probably some nice things we can do by having access to that database.

The next thing once we’ve booted up the app is we send the user into this onboarding process. I think this screen embodies what we’re trying to do with Zap which is providing something simple and easy for end users who aren’t so technical. I think we’re partly the way there. There are a lot of things that can be done to continue to make it more user friendly. There are a couple of basic options whether you want to create a wallet, import a wallet or connect to a remote node. We have a form wizard to collect up that data. We collect up the data, we store it in Redux within React. We store that data in Redux which is the persistent data in the browser while we’re going through this process. Once we get through this process, we spawn the lnd. If you choose to select the local node… this is where we start to spin up lnd. We spawn a lnd process which basically gets gRPC running up on port… We actually select a random port. It will be somewhere around the 10009 range which is the default port. We have a range of ports that we use. We scan for a free port, start up lnd which gives us that gRPC interface. We make an API call into it and we tell it to generate a seed. We spit that back to the renderer process and we present that on the screen for the user and obviously ask them to write it down. Then we actually kill that lnd process. That is a new thing we’ve started doing because some of the stuff I’ll show you in a bit, some of the new stuff, you can move in and out of this part of the app. It is something I’ve been working on recently. I’m really interested if someone has a better way of doing this. If we just want to generate a seed, spinning up a binary process and getting access to an API and connecting to gRPC and making an API call, it seems like overkill just to generate a seed. If there was a native Javascript library we could use…

Q - There’s a BIP 39 Javascript library. BitcoinJS can probably do it or the other one?

A - The seed format is slightly different for lnd I believe. They have an aezeed. It’s not standard BIP 39. These things will come

Q - Then you’re out of luck. It’s probably just one little Go file that maybe could be converted to Javascript. To WebAssembly? That’s a bit overkill maybe too but sounds like fun.

A - If someone wants to pick it up we’ll happily accept it.

That’s what is happening here. You’re connecting to a remote server. Currently it works and is functional. We get a lot of support requests in Slack for people trying to figure out how to do this. If you ever try to connect to a remote node you’ve got to get three key pieces of information. You need to know the hostname and the IP address of the machine where lnd is running. You need to know the port where lnd is running. You need to get a TLS certificate and an admin macaroon file from that server. You need to copy those to your machine and you need to tell Zap where those files are. Even in saying that you can hear the friction that is involved in that. We’re trying to build something that is consumer facing, that’s an easy to use Lightning wallet. It’s really not easy to use.

Q - If it runs on your own machine, I assume these are all…

A - This is the point. This is the case where you are connecting to a remote node. Imagine you’ve got a piece of hardware at home that is running a full node for you and from your laptop wherever you are, you just want to be able to connect to that. Or you’ve got a BTCPay Server instance or anywhere where you’ve got a remote node. In order to do this you need to shell into that node. You need to navigate into the lnd directory. You need to find these two specific files. You’ve got to copy them over to your local machine. You’ve now got to store them somewhere on your local machine. You’ve got to enter in the path to where they are. We could have a button where you click browse and you could navigate to the file. You can see the friction in that. Also that’s inherently insecure because if you think about it if you’ve got those files: the TLS file and the admin macaroon, you basically have full control over that Lightning node. Whoever has those files can control the node. They are your keys essentially in a sense. We’re asking users to copy them across a network and that’s going to be pretty insecure in most situations and we’re asking people to store them somewhere. Where do they put them? There’s no standard place. This is the current situation and there are definitely things we can do to try to improve that process.

Q - The whole point of macaroons is that you can give partial permissions? The lnd server itself should be able to generate some thing that you download or some QR code?

A - It can. But in order to have a wallet like Zap you need a macaroon file which gives you permission to send money. Anyone who has got that macaroon now has permission to send money. There’s no way around that. If you want a wallet you’ve got to be able to send right? It is the most basic functionality.

Q - Why are you saving them as files?

A - That’s what we do currently. We can certainly use the operating system’s private store which would be a better solution. But the user has also still got to get that data to us somehow. We could rather than ask you to enter in a path to a file, we could have somewhere they just paste in something. It gets a little more complex because the macaroon is actually a binary file so you can’t copy and paste binary data into a field in a web browser. You can base64 encode it. This library here zapconnect is a library we wrote to help this process. zapconnect is a little Go application that’s compiled into a binary. You can install zapconnect on your lnd node where lnd is running. When you run zapconnect it will generate a QR code for you that is a representation of the macaroon, the certificate and the connection: the hostname, port, all these bits of information. They’re encoded in the QR code. With the iOS app you can just scan that. In the iOS app it is pretty frictionless. You install zapconnect on your Lightning node, you run zapconnect, you get a QR code and you go connect with your phone and you just scan it.

Q - How do they do hole punching on iOS because you have to have an open port? We had this problem in the 90s, it resurfaces…

A - It’s another of the questions we get all the time in Slack. There are a few different pitfalls here. One is people have got to open up their port. They’ve got to make sure their firewall is forwarding packets. It is definitely fraught with difficulty.

Q - I don’t know if macaroons support this but what you could do is generate a private key on the desktop app and then give the public key to the lnd server and then make it generate a macaroon that says only the signer of this public key can communicate with me. At least then you’re not sending any private keys over the wire, just public keys.

A - That’s an interesting idea.

So that connect works really well with the mobile app. There’s two ways you can run it. You can generate a QR code. You can also generate a JSON snippet as well which you can copy and paste. That JSON snippet contains the macaroon in it as well. That’s a really interesting idea, maybe we can chat about that afterwards.

Q - I read the macaroon paper like ten years ago so don’t interrogate me on it but I remember that they said you can have your Flickr photo album and then you can give somebody a macaroon that’s as if you’re logged in with your Yahoo account or your mom’s account. They had some fancy ideas about permissions to solve this type of problem.

One of the final things that we built into the app in terms of connectivity is we built an integration with BTCPay Server. This is another way around this issue. We worked with Nicolas who is the developer of BTCPay Server to build out this configuration format. He now has in BTCPay… BTCPay is like a store that merchants can use to really easily spin up a Lightning node and have a store and point-of-sale device and things like that. We’re seeing a lot of merchants using that. In BTCPay you can go in and say “show me my connection string”. It will show you a QR code or something that you can copy and paste. You can just copy that, paste it directly into here and hit next and you’re done. It’s a bit more frictionless but that only works because on BTCPay they’ve already got the infrastructure to generate this… It’s basically doing what the zapconnect library is doing, something very similar, generating something that you can copy and paste over.

So once you’ve made your connection the next thing we do is we start synchronizing. If you’re running lnd locally then we go into lnd synchronization process. It is something that was really tricky to build. Our implementation of this is hacky to say the best. The problem is that with lnd you’ve got no idea how far through the synchronization process it really is. The way that we do this is we attach a listener. We’ve got the lnd process running locally on the machine and we attach a listener to the standard output and the standard error of the lnd process. We run regex on every single line of log output and we’re basically listening out for things that give us some kind of hints on what’s happening in the sync process. We’ve got a list of probably 15 or 20 log messages that we know are interesting to us and give us a hint as to whether a new block header has just been downloaded or some commitment filters have been downloaded. It would be nice if lnd had a stream that you could just connect to. lnd has bidirectional streaming like WebSockets where you can get data fed to you in realtime. They have it for some stuff like transactions and invoices and channel data I think you can get data to stream for you. You can’t get any of this state data streamed to you currently. PR 1355 is trying to build support for Prometheus which is a monitoring system which allows you to tap in and get all kinds of metrics from lnd about memory usage. I think they are talking about including things like sync data and stuff like that. If there was something like that and that was exposed over the gRPC interface we’d get a lot more insights into what’s going on in lnd. At the moment it is pretty rudimentary and it’s limited because we can only do that on a local process we’re running. If we’ve connected to a remote node we’ve got no way of knowing what’s going on in that. lnd has an API call that’s called getinfo and when you run that it has a flag in there that says “synced to chain”. It’s true or it’s false. When the synchronization process takes several minutes you need to give the user some feedback as to what is going on. Another shortcoming that we found as well is that even just knowing what the current block height is, what’s the latest block on the blockchain, what’s the current block height number? When you connect to lnd and it starts synchronizing, lnd will tell you what it thinks is the latest block. It will tell you the latest block that it has. But if lnd itself is still in the process of synchronizing with the blockchain, the latest block that it has might not be the actual latest block. So we have to make a call out to a block explorer. I think we use blockchain.info and we figure out what the latest block is so we can show the progress incrementing. Even that is fraught with challenges because we found cases where blockchain.info would go down. So we actually now make three different API calls out to three separate block explorers. Whichever one gives us a result back first we use that to assume that’s the current block height and we start pulling…

Q - Why are you not just pinging the full node?

A - We don’t have access to the full node. We’re only connected directly to lnd and lnd might be talking to a full node that’s on a different machine somewhere.

Q - lnd cannot relay that? It sounds like a simple pull request

A - When we’re running lnd on here, we’re running it in Neutrino mode and that’s basically talking to our btcd servers.

Q - You said you had several btcd running on Google Cloud. You can connect to those?

A - Yeah we can. There are different places we can get that data from but it would be nice if lnd… Because lnd is our primary interface for everything Lightning we can do 90% of stuff but it’s that extra 10% where we have to figure out workarounds whether it’s calling btcd…

Q - Where is the Neutrino protocol coming in? The lnd node on your local machine is connected through Neutrino to the rest of the Bitcoin network?

A - When you’re running it locally you’re running lnd locally and lnd is running as a Neutrino client which is connected to our btcd full nodes. Neutrino is the light client implementation between lnd and the full nodes.

Q - Can lnd also be run with a local full node? It doesn’t use any of the Neutrino stuff?

A - That’s right but we don’t want to be downloading gigabytes of data. When you use Neutrino with testnet, most of my lnd directories are 400MB or something like that. You can download that pretty quickly. This currently takes about 5 minutes on testnet. If I’d have done this a couple of months ago that would have taken about 15 minutes. There’s a lot of improvements coming. What they’ve done is they’ve now made it so that Neutrino can pull the block headers and the commitment filters in parallel and it can pull them from multiple btcd nodes at the same time. So we’re getting 4x what we used to get. They’re starting to work more actively on the Neutrino support so I think there will be some more improvements. That’s testnet as well so mainnet is much smaller and would be faster again.

Another thing to point out here is that on the right you’ve got the channel management piece. The idea here is that once lnd has built up a picture of the network which as I said can take some time, then you can go and start establishing channels with nodes. If you turn on autopilot it will do it for you. By default we have it set, I think we just use the lnd defaults which allocate 60% of your funds over a maximum of 5 channels or something like that. It should theoretically give you a good route into the network but in practice that doesn’t really happen at the moment. The channel management piece - you can click on one of these channels and you can see how much balance is in each channel and things like that. Ultimately we’d like that to disappear into the background. Maybe we can collapse it or something. Maybe eventually we can hide it away in an advanced settings page. Currently there’s literally no way you can use the wallet without being able to inspect your channels and manage them manually. That speaks to a bigger point which is it is difficult for you to use the wallet without knowing actually what a channel is. We’re talking about building things for mainstream. If you need to know what channels are we’re not there yet.

I wanted to show you a couple of new bits and pieces that we’ve been working on. Some of this is already in the master branch, some of this is going through review at the moment. The first thing is the payment and the request forms, if you’re familiar with Zap currently you might see some differences here. We started building out support for fee estimation which we can get about 50% of the way there currently. In terms of making offchain payments like Lightning payments what we do is we send the payment request to lnd, we use the queryroutes API call. We ask lnd to give us a list of up to ten routes and it will return those to us in order of lowest fee to highest fee. We can then present the user with the lowest fee that we’ve got and the highest fee that we’ve got. We give them the range of what we think it is going to be. When they hit send and we send the payment we take the maximum fee that we got from our list of ten routes and we apply that as a fee limit. In the current version of Zap that you guys might have, when you actually send the payment we just blindly spit it out to the network. If it finds a route it will send on that route regardless of what the fee is. Here we’re showing the user upfront what’s the possible range. We generally see that this is between 1 and 3 satoshis, something like that. They’re pretty low but it is important if we’re showing the user it is between 1 and 3 satoshis, we’ve got to cap it at 3. When we make the sendpayment call into lnd we tell it the maximum fee that we’re going to apply which is the highest of the fees that we got back from queryroutes. If it manages to send it over one of those ten routes then fine. When you tell lnd to send a payment it will keep trying routes until it finds one that works or it times out. So we make sure we put in a hard stop in if the fee limit has been reached. In terms of the onchain fees we’ve had some challenges here because lnd doesn’t currently provide a fee estimation interface on REST or gRPC or whatever. So there is no way for us to know upfront what transaction fee is going to be applied. More specifically what we do currently in this latest implementation is we call out to 21’s fee estimation API. We find out what the current estimated fee is to get something in within a few blocks. We can get that per byte fee rate whether it is 5 satoshis per byte or something like that. We can find out what that is but what we can’t do is generate the transaction upfront. We can’t know how big the transaction is going to be and therefore how much the satoshi per byte fee extrapolates out to the full cost of the transaction.

Q - Why can’t you know the size of the transaction?

A - Because lnd doesn’t let you create a transaction upfront currently. With lnd you can just say “make payment”. Inside lnd it creates the transaction, it does the fee estimation, it applies the fee rate and then it sends out that transaction. But the API doesn’t give you a way to say “Hey lnd. Generate a transaction to send this amount to this address and give me back the transaction so I can look at it and see how big is the transaction.” It is kind of all or nothing at the moment.

Q - So you want some kind of dry run payment?

A - Yeah exactly. We want a dry run. We want to be able to create a dry run transaction so we can see how big it is. Then we can take a fee estimate and we can apply that and we give the user an option of whether they want next block or next six blocks or whatever. Then we can extrapolate that out and show them the cost that they will actually pay. The best thing you can do currently is say that it is 3 sats per byte and what does that mean?

Q - I’ve done this with an application using bitcoinjs-lib. I think you referenced this also. You can generate your own transaction, count the bytes and then use the fee estimation per kilobyte.

A - But how can you do that if your Bitcoin wallet is running on a remote lnd node? How are you going to generate a transaction from that? You need access to the full node to generate the transaction.

Q - You need to be able to get the UTXO to find the inputs you’re going to spend.

A - That’s it. All we can get is what lnd gives us and it doesn’t unfortunately give us that. Maybe someone knows a way of doing it but we haven’t found a way to do that yet. This PR #1228 is a pull request to expose… because lnd does have fee estimation inside it and this PR here is looking to expose that as a gRPC endpoint so you can just say “give me a fee estimate for this transaction.” So that would solve for that.

So that’s some of the new payment request forms. They’re more interactive now. You can basically paste in an onchain address or an offchain address. It will determine what it is, it will decode the invoice and show the users the details and they can get the fee estimate and hit send. One of the things that I found when building this as well that is kind of interesting is that when you have a payment request we basically want to take that payment request and decode it. What we were doing previously is we were sending that payment request to lnd and having it decode the invoice. It spits out a representation of that invoice data.

As I started to build this out I took this as an opportunity to do some more refactoring on the app and this snowballed into building out a storybook for our app. If any of you are familiar with React development or that sort of world, storybooks are a tool that allow you to have a catalog of all the React components that we’ve created. We can see how they look, we can interact with them, we can see how they look and feel and how we can build them together. I’ve gone through the app, I’ve extracted all the components and we’ve got a full catalog of all the different bits and pieces. We have everything from the color palettes, the typography that we’re using, the different button states, page layouts that we’re using within the app and things like that. It gives us a catalog of all the different components that we’ve got to build with and we use those to build out new sections of the app. One of the really cool things about the storybook is that this allows us to build stuff without using lnd or even starting up Zap. What I’ve found is that since I’ve built the storybook I probably spend 80% of my time when I’m building stuff on zap-desktop building in the storybook environment just using Chrome, using my web browser. From here we also have full page mockups which you can interact with. What I’ve found was that when we’re building stuff of lnd we’re trying to build a UI for some particular state. For example, let’s say you’re trying to make a payment and it has failed routing. lnd gives you back an error that says “routing failed” and we want to show the user an error explaining to them the routing failed and what they can do about it. In order to build that into the UI that relies on us having lnd running, we’ve got to have lnd synced, we’ve got to have some testnet coins, we’ve got to have some connections to the network. If you want to do that in a real world scenario it is work to get the app into that state so that you can even test that failure case that you’re trying to build for. In the storybook we basically mock and stub the lnd calls…. For example, when you click on something within the storybook, whereas within the app it would actually call into lnd and ask lnd to do something, in the storybook environment because we don’t have lnd running we just call into a mock function. Then we have that function returned back to us the data we’re expecting to see. Either the success case or one of several failure cases. We can simulate in here success and failures of all sorts of different things that lnd might throw at us without even having to have lnd running. This lets us be really productive in terms of building out UI and we can build out new features in here and work with graphic designers to get the UI fine-tuned. Then only at the last minute do we integrate that into the app and start using it with actual lnd. If we’ve done it properly and we’ve mocked out those API results from lnd properly it is pretty straightforward to integrate that. Some of the new stuff we’ve been building, I probably spend 80% of my time in storybook. Then it’s just the last little hurdle to integrate it into the app when you’re actually running the app and connecting to the network. We haven’t done a lot of work with regtest. We haven’t really been using that, we’ve been building against testnet. We could probably simplify some of that or speed up some of that process by not having to wait for things to be mined on testnet.

Q - Bcoin, I think that’s the Javascript Bitcoin library. There is a Bcoin regtest instance in the browser and then maybe a complete lnd node in the browser.

A - In the storybook environment I don’t really want to have anything to do with anything not Javascript. I know Bcoin is a Javascript implementation but it is something that is pretty heavyweight to run in here and would add some complexities. If we’re just basically calling a function that gives us back data instantly, we can craft those to represent what we see lnd typically gives you.

Q - A mock lnd makes more sense

A - We don’t have a full mock lnd implementation. It would be a nice thing to work towards. Currently we just stub the specific requests we’re trying to build against.

In the storybook, we’re building out a new feature. It is going through some testing at the moment. What we’ve seen is that lots of people who are using Zap are trying to connect to multiple wallets. Certainly I am as a developer building on it. I have got multiple testnet wallets and mainnet wallets. People have got Litecoin wallets, they’ve got wallets up on BTCPay. If you’re a merchant and you’ve got five stores. With zap-desktop currently you can just connect to one node. When you quit the app you basically start from scratch again. You have to go right through the onboarding process every single time. Here we’re storing the save wallet configurations. You’ve got a top level home area where you can access your different wallets. You only access this section if you need to. If you’re just using one wallet you don’t ever need to see this. When you start up Zap now it will take you to wherever you were last when you quit the app. If you were in your wallet and you quit Zap the next time you start up Zap you’ll be back in your wallet. If your wallet is locked and needs a password to unlock then sure it will allow you to unlock it. We’ve removed some of that friction of having to go through the onboarding process every time. You can have your different save wallet configs and you can tune your settings. It is pretty basic at the moment but it is already useful. You can give your wallet a name for organizational purposes in Zap, you can configure the alias, you can configure autopilot. We’ve actually currently got some of the fine-grained autopilot settings in there. We’re trying to strike a balance between what is useful and what is complex. Are we building for power users or not so technical users? Currently we’re probably building for slightly more technical users because that’s the current user base.

I want to finish off with some general challenges and insights I’ve had whilst working through this whole thing. The challenges in terms of the Lightning Network are slow sync times. I talked about that before. If you’re expecting it it is fine. It is fine to start up your app and wait five minutes and have it sync. But it’s not something that you could say is consumer ready. Sync times is a pain and it slows down your development flow as well. If you need to spin up a new node and get it into a ready state you’re going to be waiting some time before you can be productive again. Sync times are painful. The general stability… I guess I’m talking specifically about lnd because that’s where our focus has been. I can’t really talk to some of the other implementations. The general stability of lnd had been a problem. WIth the latest release, I’m really impressed, it’s really nice to work with. Previously we were getting random seg faults and memory overflows. It would just crash randomly and give us no log output. It was tough to work with and to know what was going on. That’s really improved quite a lot. Upgrading as well between the different versions of lnd. Upgrades of the earlier versions, you might find that a migration script didn’t work properly, you’d end up with a corrupt database. If you’re technical you can probably fix it and get your money back. Now with the latest versions, even with 0.4.2 and the latest 0.5.0 and 0.5.1, these problems seem to have gone away. I would say that particularly in lnd, there is a lot of development happening. It is really fast paced. There are PRs going in all the time. The rate I see stuff going in there makes me question how thoroughly this stuff is being reviewed and how stable is it and are these problems going to start reoccurring. It has definitely improved massively from six months ago when I started working with it. Just keeping up is tough. There is just so much stuff happening. I think Rene is going to be talking about some of the stuff that is going on in the wider ecosystem. Whether you’re talking about splicing or AMP or all these terms that sound really cool and would be interesting. How much time can you dedicate to researching these ideas and actually trying to implement them within your project? You’ve got to pick and choose. Just keeping up with this space is definitely a challenge. On some of these points, the slow sync times are rapidly improving. The general stability is getting better but it is not ready for mainstream because mainstream apps shouldn’t crash. Upgrade pains are getting better. In terms of keeping up, it’s really interesting. If you’re interested in this stuff and you’re looking for something to get involved with, there’s no shortage of problems. Whether you’re a protocol developer or you’re someone like me that is coming from a web development background. I’m not a cryptographer. There’s a lot of different angles you can take with it and that’s kind of cool.

In the ecosystem in general, some documentation issues. We’ve seen incorrect documentation or documentation inconsistencies. Last week or the week before, even within lnd. lnd provides you with a REST API, a gRPC API and a CLI interface. They’re basically all the same thing, they’re just three ways of doing the same thing depending on what you’re trying to accomplish. You’d think if you wanted to run the queryroutes command, it would work the same on CLI versus gRPC but that’s not the case sometimes. I was trying to make a queryroutes call and there’s two things you have to supply. One is the payment request of the invoice you’re trying to pay and the other one is the maximum number of routes you want it to return. With the CLI you just have to provide the invoice and by default it will give you ten routes back. With the gRPC one if you don’t specify how many routes you want to get back, you get an error. When you’re reading the documentation and you think you know how the API works, currently it’s not necessarily the case that if it works that way in one interface that it’s going to work the same way on the other one. You have to cross check these things. Errors and reporting I touched on earlier. The error messages that lnd spits out are very technical. It makes sense because the people that are using lnd are going to be technical. That’s a lower level piece of software that people like us are using to build more high level stuff. The messages that lnd gives out are useful for us but really not helpful for anyone using Zap. We get a lot of questions in Slack, “I’ve got this error when trying to make a payment. What does it mean?” That can be tough. We built a wrapper around the error reporting. We feed errors that we get from lnd into a translation list so we can translate them into something a bit more human friendly. We’ve only covered some of the error messages. As we see these error messages coming up we’re trying to translate them into something that’s a bit more understandable for end users. Some of the tools and libraries, I touched on. Things like seed generation libraries. Where is it? If I want to do BOLT 11 invoice parsing where is it? It’s there. There’s actually a couple of different implementations. One of the challenges with that is that if you give an invoice to lnd and you say “decode this invoice” it gives you a representation. There’s a Javascript library we’re using called bolt11 and you can give that a payment request and it will decode it. It will give you a representation of the same data but they’re not the same. There’s no standard as to how to display that decoded data or what order things should be in, where they should be. When we’re working in the storybook environment where we’re mocking lnd and we want to be able to take a payment request and decode it without touching lnd, we’re using a different Javascript library that’s giving us different output. Same data but different output format. When we try to integrate with the app we find that there’s some differences and spotting that can be difficult sometimes. The ecosystem is definitely immature. There’s huge scope for tools and libraries. I’ve talked about most of the points here. With the documentation issues, definitely cross check across different implementations. Whether it’s inside lnd itself or whether it’s lnd versus eclair or whatever. For errors and reporting, source code… Take those error messages, go into GitHub and paste in the error messages and try to find where in the code it’s coming from. At least you can have some context and you can take that into lnd Slack and hopefully somebody there can help you with what’s going on. With the tools and libraries, there’s huge scope for new projects. If anyone is looking for how you can build something that’s useful. There’s a lot of things that you could just build. Little utilities that would just be really useful and would be a good way to make some inroads into building stuff and you can snowball out from there.

With user experience, bootstrapping the network is slow, bad user experience. Routing basically doesn’t work, bad user experience. Some of the core concepts within the wallet… inside Zap you’ve basically got two wallets. You’ve got your onchain balance and then you’ve got a balance that’s in some channels. You’ve got to present the user with these two different balances and help them understand what they mean. If you’ve got money in some channels, you’ve got 0.2 BTC in some channels and 50% of that is in one channel and the other 50% is spread unevenly between six other channels. How do you know what’s the maximum I can send to this person at any point in time? There’s no definitive answer. It depends on when you ask the question and how good your connection is to the network to know how much you can actually send. Trying to decode that and present it to the user in something that they can understand and not be confused by is hard. I don’t think we’ve done it but I think as we iterate on the software we can improve that. It is a real core concept. Channels aren’t going to go away, channel balances aren’t going to go away. Maybe when we’ve got AMP or something like that or we’ve got some kind of automated rebalancing system in place where you can just have the software behind the scenes rebalance everything for you. Then you have one number you can show the user and this is how much you’ve got and how much you can send. Currently that’s not the case. That’s a big user experience challenge. I think the interesting thing here on the bootstrapping the network is that what that says to me is because there’s a real problem with the network bootstrap process, it says to me that the user base currently is technical. If the majority of the people that were using Lightning were non-technical users we’d be hearing about the network bootstrap pains a lot more than we do. The fact that people are using it know that it takes a few minutes to synchronize and that’s expected, it must be the case that most of the people using it are technical currently. With the routing I would just say it is a bit of a concern. I wouldn’t say it is an unsolved problem. I know lots of people are working on different ways to solve this. I think the current solutions are not adequate and they need a lot of work. Until we’ve got 95-99% success rate it is going to be continually presenting problems to users. We’re talking about consumer facing applications, you need something that just doesn’t fail. WIth all of these core concepts like with the wallet balances we just need to keep iterating on the software and keep taking in feedback from our user base. Keep tweaking, keep tuning it until we have something that people can just understand. I think this tells us that when we’re talking about building for power users versus end users, what it tells us is that we’re at real early days. When Jack started the project the goal was to build something that’s for end users and make Lightning simple to use. I think he really succeeded in that before there was Zap the only way you could interact with Lightning was being a techno geek and command lining stuff. We do have applications now that are more end user friendly but the fact is that the vast majority of the user base is technical and so it makes sense at the moment for us to be building for a slightly more technical user base. Things like giving them control over autopilot and their allocations, letting them see the channel management, letting them use those features. We’ve had some people in Slack asking “Is this too complex? How are end users going to use this?”. The answer right now is that they’re not. They’re not the current audience. We’re still building a network, we’re still building the protocol. We can’t really be building for end users yet but we can start that process and that’s what we’re trying to do.

To wrap up, I think some of the next things we’re going to do with Zap is we’re going to keep refactoring, keep improving the codebase, keep improving the UI. We’re going to try to fulfill the mission that Jack set out to do which is to make Lightning Network accessible to people. We’d like to do some deeper integration with our mobile wallet for example or with some of these hardware nodes. I haven’t actually used them myself but the Raspiblitz nodes and the Casa nodes and these little hardware Raspberry Pi devices. Some people have built some integrations that allow you to connect Zap really easily but we’d like to get some deeper integration with those tools so that people who are using those nodes have a really easy way to connect their Zap wallet and use that as their remote control. We’d like to get onto looking at channel rebalancing strategies. I’m not too familiar with what’s going on in the space in terms of the lower level, the protocol level. I think that until we start seeing stuff at that level, maybe we can start building something on a slightly higher level to give users some tools to help them rebalance their channels. Maybe that’s a manual process to start with and eventually that can fade away into the background. Then we want to look at some really cool stuff like submarine swaps, the fancy stuff… For us building on a higher level, we need some of the stuff to be more stable and built into the base protocol before we start using these types of things. We don’t want to be building on new features where the API is going to change out from under us. We do want to keep integrating new cool stuff into the software.

I’m just going to leave you with a couple of resources that I think are really useful if you’re in this space, if you’re trying to keep up. The Optech newsletter is weekly, it’s really awesome. It takes you about five minutes to read it. They summarize the key things that are going on in the space. They focus on Bitcoin Core, lnd, eclair and c-lightning and that sort of Bitcoin onchain, offchain crossover. They just pick out some of the key things that have gone on, some of the key developments, some of the key ideas that are coming through. You should definitely subscribe to that if you haven’t already. Lnd Slack, there’s like thousands of people in there talking about lnd and Lightning and asking questions, helping people out. If you’re building stuff you should hang out in there. For us as we’re building closely on top of lnd specifically keeping track of the GitHub milestones gives us a sense of when the next release is going to be, what’s going to be in that release. We can literally see all the stories that are lined up for the next release. We can make sure that what we’re building starts incorporating some of that stuff before it comes into the release and when it’s in the release as well we can start pushing it out to our users. htlc.me is a really cool site. If you just want to generate an invoice and you don’t have lnd running you can just go to htlc.me and you can say “give me an invoice” and it will give you a Lightning invoice. You can then decode it. If you’re building software on top of Lightning and you don’t want to have a node running so you can get an invoice you can jump over there and it will generate you an invoice. If you want to help out with Zap or you’ve got questions on Zap come and ask us in the Slack. We’re in there 24/7 pretty much. Just to finish up, I would say that if I want you to take something away from this, hopefully I’ve given you a sense on a slightly technical level some of the things that are involved in building out applications of Lightning. If you’ve got some experience with web development or just development then you’re probably qualified to start building stuff in this space. Don’t be scared off by the fact that there is cryptography involved because you can come in at a higher level. You don’t need to be building on the protocol. You can be building.. whether it’s the device that dispenses M&Ms or a software wallet, it’s really just a React Javascript web app that’s making API calls. If you’ve got some development experience you’ve probably got what you need to start building cool stuff on Lightning.

Q - Has anyone worked on c-lightning integration for Zap?

A - Not that I’m aware of, I certainly haven’t. I haven’t seen anything. It does support Litecoin as well, some people made sure that it would work for that.

Q - Is there enough abstraction in there that somebody could even do that? It sounds like there’s a tight coupling with lnd.

A - I’m trying to abstract things out, building out mock interfaces and things like that is a step along that line. Currently we’re pretty tightly coupled but there’s no reason in the future that we couldn’t support c-lightning.

Q - There is a Zap wallet for iOS, a Zap wallet for desktop. What about a Zap wallet for Android? I hear it is almost a web app in a native container deployed to desktop systems. There are also techniques to put this into a mobile container and run this on Android.

A - There are. The lnd team, Lightning Labs, are building out their wallet. I think it is called Lightning App. That is a React app as well, they’re using React native so they have one app that runs in all the different environments: iOS, Android and the desktop. We took a slightly different approach with this. The iOS app is written in Swift and is a native app. There are some inherent advantages with that. We’re focusing on slightly different use cases as well. WIth the desktop app we’re trying to build some slightly more technical tools into it so it’s not like there’s a one-to-one mapping between the functionality we want to provide. In terms of Android, I don’t think anyone on the Zap team uses Android so thus why it hasn’t been built. If someone wants to build it. If there’s enough interest we’ll build it but I haven’t seen that many people ask for it.

Q - So if you want to support the other Lightning implementations I think that would be super useful. There’s already Christian Decker’s Lightning integration repo. He’s already trying to support multiple nodes so he can test channels from c-lightning to lnd and vice versa and all different kinds of implementations. I think it would be so useful to have a proper API that would work against all nodes for doing all these things. All the implementations are supposed to be doing the same thing.

A - That’s right. Under the hood they are compatible with each other at the protocol level. When the app started to be built, lnd seemed to be one of the more popular implementations. I’d say it probably still is but maybe that’s just because I’m in that space. As more and more people start using the different implementations it would be a shame if we’re just tied to lnd and there’s three other implementations and two-thirds of the user base that can’t use Zap because of the particular lnd node instance they’re running. Definitely something I’m keen to look into. In terms of tight coupling, when we were building the BTCPay Server integration, Nicolas wanted to expose the REST API and we wanted to use the gRPC API because that’s how Zap works today. It shouldn’t be too hard to have an abstraction layer within Zap where you can use REST and WebSockets if you want to talk to the REST API or gRPC if you want to use that, you should be able to use both. Currently because Zap evolved out of building for a specific use case there’s a tight coupling with gRPC. There’s nothing in the codebase that would make it difficult to do that, the code that integrates with lnd is in a couple of specific places. There’s not that much we do really. With gRPC, the gRPC gives you an object that you can natively use in Javascript. Currently it’s not done right because we have our own little REST API on top of gRPC and as we communicate between the renderer process and the main process in Electron, we’re basically communicating with the REST API which is then telling gRPC to do stuff. We can definitely change that and there’s not too much code involved to say we could use REST and gRPC. By the same token I don’t think there would be that much involved to say you can use this node implementation or that node implementation. Today it’s not there.

Q - I just wanted to encourage you to make it separate from Zap. Then it could be useful for testing and then you might get other contributors. You might get an interface that supports everything and not just what Zap wants to do. Win-win.

A - Definitely. Everything we do is open source. A lot of it is embedded in Zap at the moment. There are a couple of libraries that we’ve taken out and extracted separately. We have a library called lnd-binary which lets you really easily install lnd. You can do npm install lnd-binary. It will go to GitHub, it will fetch the lnd binary, it’s got a repository of the checksums, it will validate that it’s the right binary and it will extract that and make it available for use in Node. There are things like that where we’re starting to pull out some of that functionality and extract it out into separate libraries. If we get to the point where we’re building an abstraction or an API that can spit out to three different node implementations or however many it would definitely be something that we’d open source outside of the context of Zap.

follow-up: https://www.reddit.com/r/lightningnetwork/comments/ba3kac/tom_kirkpatrick_zap_wallet_dev_on_building_lnd/