Skip to content

"We want to know if htlc 1 is missing (immediate)" #911

Closed

Description

@fabianfabian

lightingd keeps crashing, this is in the log:

2018-02-04T23:36:23.636Z lightningd(4284): Adding block 507509: 0000000000000000003b0c2cc3034ea8d6c71dd77d820df8b24cf6e668ac8892
**2018-02-04T23:36:24.989Z lightningd(4284): Owning output 1 300000 (P2SH) txid 20d8e529408a281f43284e216b182e23aaabbe64c7167b9b384065401ed2281a**
2018-02-04T23:36:25.186Z lightningd(4284): Adding block 507510: 0000000000000000005fdf68e6a860c11cad255157504bde4ca090ac910b59b7
2018-02-04T23:36:25.490Z lightningd(4284): Adding block 507511: 00000000000000000028307274c3176ffe9cfe418ac47f27e515b2b19c51d74e
2018-02-04T23:36:25.828Z lightningd(4284): Adding block 507512: 0000000000000000001406f51c9785fb2b07f879f8279da383eba4044c021ba3
**2018-02-04T23:36:27.153Z lightningd(4284): Owning output 0 135980 (SEGWIT) txid cf3b58d23d9773bdff19955add8d009b3ef9c02b895a2fa2c9b4d4459c855c81**
2018-02-04T23:36:27.295Z lightningd(4284): Adding block 507513: 00000000000000000044dac62e6b26827f7f717eda4dfdd32a740d9022b60af5
2018-02-04T23:36:28.811Z lightningd(4284): Adding block 507514: 0000000000000000003d8ccb167602660c5c2f6aca14344f7bc85ca21aa98e07
2018-02-04T23:36:30.307Z lightningd(4284): Adding block 507515: 0000000000000000004e25bb406c70ac566a73bd73af77b517e4cf7f53f2b365
2018-02-04T23:36:30.643Z lightningd(4284): Adding block 507516: 00000000000000000054cfd968c7b6d986f594b24a90c726c44db1c5f1e50c36
2018-02-04T23:36:30.941Z lightningd(4284): Adding block 507517: 0000000000000000005159fc290104ead48aebef3d49a53d9a05b4167cd8072f
2018-02-04T23:36:31.610Z lightningd(4284): Adding block 507518: 000000000000000000505633dbbc2050b349babf3fb22e79b54a4c635c96d24d
2018-02-04T23:36:32.701Z lightningd(4284): Adding block 507519: 000000000000000000178af595809bba6cdb8af5e599884dccfa14e93ccb783b
2018-02-04T23:36:33.365Z lightningd(4284): Adding block 507520: 000000000000000000293baaad7ae3594291e75e63affc8db928344e19848d21
2018-02-04T23:36:33.511Z lightningd(4284): Adding block 507521: 00000000000000000030a6806d06bb707e38f81f94ac2d4c5911c1c8fe71597a
**2018-02-04T23:36:33.712Z lightningd(4284): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: Got UTXO spend for cf3b58d23d9773bdff19955add8d009b3ef9c02b895a2fa2c9b4d4459c855c81:1: 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2
2018-02-04T23:36:33.712Z lightningd(4284): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: Peer permanent failure in ONCHAIND_OUR_UNILATERAL: Funding transaction spent
2018-02-04T23:36:33.712Z lightningd(4284): 	 (tx 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2)
2018-02-04T23:36:33.712Z lightningd(4284): sendrawtransaction: 02000000000101815c859c45d4b4c9a22f5a892bc0f93e9b008ddd5a9519ffbd73973dd2583bcf01000000007d80ba800249d60000000000001600147dbd4634afc993b70fb5839db0047b84db9cd359c56b0100000000002200200686d8b848eea5268a7886fbcab54518506f7c181c65ddfd54664c457ff219410400473044022048d58c831b5c1735ab2198bb5c91855966d5fcde07e491bcef3fb8190be5775c022075c89477b94a689bc4b12d1464462bfad3105011a58640bee05c24a1e9576ac601483045022100f4f75da84118a80bfe1ddfa813812a84e0dd25b8c8f414a8e4acfdf61f1ee13d02204cc4dc75c7e183e277d9837fbd1be9ff0dc3eff3ba5b531f01a325f0dde7b4c201475221027068cdeb935512faf54477427efe112e081df250509e4a356f164f5b072e673f2103058b89c005ddc30cf995da1e5eac9a360fd915db50d41501e2c0ff45c67e377252ae90382b20
2018-02-04T23:36:33.713Z lightningd(4284): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: State changed from ONCHAIND_OUR_UNILATERAL to FUNDING_SPEND_SEEN
2018-02-04T23:36:33.714Z lightningd(4284): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): pid 4529, msgfd 17
2018-02-04T23:36:33.714Z lightningd(4284): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: We want to know if htlc 1 is missing (immediate)**

I'm not sure when it first crashed, all I did was fund a channel with blockstream store and bought a sticker (successfully), then tried to make donation to https://lnmainnet.gaben.win/?100= which failed I think, then the next day I check the logs and its in this infinite crash loop.

getinfo output

$ lightning-cli getinfo
{ "id" : "03de291f2e3e14cd3875b79bc8948f42e582ce7feda246314dfd282548dd1964a6", "port" : 9735, "address" : 
	[ 
		{ "type" : "ipv4", "address" : "X.X.X.X", "port" : 9735 } ], "version" : "v0.5.2-2016-11-21-1814-g5f2cd7c1", "blockheight" : 507439, "network" : "bitcoin" }

listpeers output

$ lightning-cli listpeers
{ "peers" : 
	[ 
		{ "id" : "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432", "connected" : false, "channels" : 
			[ 
				{ "state" : "ONCHAIND_OUR_UNILATERAL", "owner" : "lightning_onchaind", "short_channel_id" : "507512:2585:1", "funding_txid" : "cf3b58d23d9773bdff19955add8d009b3ef9c02b895a2fa2c9b4d4459c855c81", "msatoshi_to_us" : 95142333, "msatoshi_total" : 150000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 144, "max_accepted_htlcs" : 483 } ] } ] }

listpayments output

$ lightning-cli listpayments
{ "payments" : 
	[ 
		{ "id" : 1, "payment_hash" : "ba8e5352952ab59468d543334d9b2d765cf76be61c38d5644695fcbe48cb3126", "destination" : "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432", "msatoshi" : 54857667, "timestamp" : 1517713156, "created_at" : 1517713156, "status" : "complete", "payment_preimage" : "855dcf08dadd147f4b45745c40e59ea92a3324e8e31c7995f8930f099c0b50d5" }, 
		{ "id" : 2, "payment_hash" : "abb88bee3d98b0615e7856418cf3bf9be5ac4fda1cef678cc9f5097fe0a957a6", "destination" : "035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8", "msatoshi" : 500000, "timestamp" : 1517713524, "created_at" : 1517713524, "status" : "failed" } ] }

Activity

  1. fabianfabian commented on Feb 5, 2018

    @fabianfabian
    Author

    In the console there is some more info that is not in the log file:

    2018-02-05T00:46:38.450Z lightningd(8149): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: State changed from ONCHAIND_OUR_UNILATERAL to FUNDING_SPEND_SEEN
    2018-02-05T00:46:38.452Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): pid 8396, msgfd 16
    2018-02-05T00:46:38.453Z lightningd(8149): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: We want to know if htlc 1 is missing (immediate)
    **2018-02-05T00:46:38.985Z lightningd(8149): sendrawtx exit 27, gave error code: -27
    error message:
    transaction already in block chain**
    
    **2018-02-05T00:46:38.986Z lightning_gossipd(8157): TRACE: req: type WIRE_GOSSIP_DISABLE_CHANNEL len 12
    2018-02-05T00:46:38.986Z lightning_gossipd(8157): TRACE: Unable to find channel 507512:2585:1/0
    2018-02-05T00:46:38.987Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Tracking output 1 of cf3b58d23d9773bdff19955add8d009b3ef9c02b895a2fa2c9b4d4459c855c81: FUNDING_TRANSACTION/FUNDING_OUTPUT
    2018-02-05T00:46:38.987Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Remote per-commit point: 024d247a0d3fecaac73c175767e50d02bc01a77f51160abb9f09bda967e629ab10
    2018-02-05T00:46:38.987Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Old remote per-commit point: 02d2c1e111a835f4d9732f240ced13b6366f5d5e0415bee80cb4790cf81c61bba0
    2018-02-05T00:46:38.988Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: commitnum = 6, revocations_received = 6
    2018-02-05T00:46:38.988Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): UPDATE WIRE_ONCHAIN_INIT_REPLY
    2018-02-05T00:46:38.988Z lightningd(8149): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: State changed from FUNDING_SPEND_SEEN to ONCHAIND_OUR_UNILATERAL
    2018-02-05T00:46:38.992Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Initial feerate 0 to 4167**
    **2018-02-05T00:46:38.992Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Resolved FUNDING_TRANSACTION/FUNDING_OUTPUT by OUR_UNILATERAL (7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2)
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Deconstructing unilateral tx: 6 using keyset:  self_revocation_key: 02043d09322c22b06803f15dfc92dd58692920d9b1ef1de38cb399349066f2892d self_delayed_payment_key: 029d95106bfd470cd571842dc63c1c0f05b58dbc7105eb48df175cdef53d8b9d22 self_payment_key: 0346b6c9f949bcde1b70cdf6c95ecd8ce4546499f6bf5f3e9125d11549aff36ded other_payment_key: 03a32dc47746f8a4238454e5743ac45b53d240e2374a5e99b70b0f82a37a5e6d3a self_htlc_key: 0347ded51baed2e51c8f5034865a268c10b8da9f75f4a64e848bfcb55d5ae04773 other_htlc_key: 025ace4ca690b2f379396b33491367323f3e4931dbebfc68f29ad5d040fa345be6
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Script to-me: 144: 00200686d8b848eea5268a7886fbcab54518506f7c181c65ddfd54664c457ff21941 (632102043d09322c22b06803f15dfc92dd58692920d9b1ef1de38cb399349066f2892d67029000b27521029d95106bfd470cd571842dc63c1c0f05b58dbc7105eb48df175cdef53d8b9d2268ac)
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Script to-them: 00147dbd4634afc993b70fb5839db0047b84db9cd359
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Output 0: 00147dbd4634afc993b70fb5839db0047b84db9cd359
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Output 1: 00200686d8b848eea5268a7886fbcab54518506f7c181c65ddfd54664c457ff21941
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Tracking output 0 of 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2: OUR_UNILATERAL/OUTPUT_TO_THEM
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Ignoring output 0 of 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2: OUR_UNILATERAL/OUTPUT_TO_THEM
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Tracking output 1 of 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2: OUR_UNILATERAL/DELAYED_OUTPUT_TO_US
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Propose handling OUR_UNILATERAL/DELAYED_OUTPUT_TO_US by OUR_DELAYED_RETURN_TO_WALLET (02000000000101b286166339404437ddb10979c911fb922e4a576907f9e997057ed596249b147a01000000009000000001786a010000000000160014f467433419bf5f7ef177c5c52ffc63255eab399a03483045022100927bcbc742c3c15d9e4b624dc022389fb4ca2f0880f3bb1f13a96052bfec1f5c02201157f96c697c8d55a15c3f9b6be773e7679516609a3c007bc4da082875d4854f01004d632102043d09322c22b06803f15dfc92dd58692920d9b1ef1de38cb399349066f2892d67029000b27521029d95106bfd470cd571842dc63c1c0f05b58dbc7105eb48df175cdef53d8b9d2268ac00000000) in 144 blocks
    2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): UPDATE WIRE_ONCHAIN_MISSING_HTLC_OUTPUT
    2018-02-05T00:46:38.993Z lightningd(8149): Reporting route failure to gossipd: 0x4008 (WIRE_PERMANENT_CHANNEL_FAILURE) node 03de291f2e3e14cd3875b79bc8948f42e582ce7feda246314dfd282548dd1964a6 channel 507512:2585:1 update** 
    **2018-02-05T00:46:38.997Z lightning_gossipd(8157): TRACE: req: type WIRE_GOSSIP_ROUTING_FAILURE len 47
    2018-02-05T00:46:38.997Z lightning_gossipd(8157): TRACE: Received routing failure 0x4008 (WIRE_PERMANENT_CHANNEL_FAILURE), erring node 03de291f2e3e14cd3875b79bc8948f42e582ce7feda246314dfd282548dd1964a6, channel 507512:2585:1
    2018-02-05T00:46:38.997Z lightning_gossipd(8157): TRACE: UNUSUAL routing_failure: Erring node 03de291f2e3e14cd3875b79bc8948f42e582ce7feda246314dfd282548dd1964a6 not in map**
    2018-02-05T00:46:39.052Z lightningd(8149): Adding block 507522: 00000000000000000040ed0f2c2d686b75fb486c21f58e7cea4b91d9ab7b39e4
    2018-02-05T00:46:39.405Z lightningd(8149): Adding block 507523: 00000000000000000044aae802ae8175223ce3fe9d07d9d9869d73dd98edadb5
    2018-02-05T00:46:40.175Z lightningd(8149): Adding block 507524: 0000000000000000003a1f24cf071a242ec548f3b37e541cd1d1b177edaeadba
    2018-02-05T00:46:40.723Z lightningd(8149): Adding block 507525: 000000000000000000464da0669b510d81e0465cf227462b3833c216b747a4e5
    
  2. rustyrussell commented on Feb 5, 2018

    @rustyrussell
    Collaborator

    There's no log of the crash? Nothing in here looks particularly bad, it's just following the chain.

    You might find a crash.log file in your ~/.lightning directory?

  3. added
    Status::Needs InfoAdditional information is required from the requester or relevant party before progress can continue
    on Feb 23, 2018
  4. fabianfabian commented on Feb 28, 2018

    @fabianfabian
    Author

    I think this may be a problem with my (custom) systemd service file, when starting lightningd as a service it restarts right after the "We want to know if htlc 1 is missing (immediate)**" message. But when I start lightningd manually it keeps running fine.

    I couldn't find something like https://github.com/bitcoin/bitcoin/tree/master/contrib/init for c-lightning.

    This is mine now:

    root@bunker:~# cat /etc/systemd/system/lightningd.service 
    [Unit]
    Description=Lightning daemon
    After=network.target
    
    [Service]
    ExecStart=/usr/local/bin/lightningd --network bitcoin --log-level debug --log-file debug.log
    RuntimeDirectory=lightningd
    User=root
    Type=forking
    PrivateTmp=true
    
    [Install]
    WantedBy=multi-user.target
    
  5. plachta11b commented on Feb 28, 2018

    @plachta11b

    I create my own working by using Type=simple.

    [Unit]
    Description=Lightning Network implementation in C
    After=network.target
    
    [Service]
    User=myUser
    Group=myUser
    
    Type=simple
    
    ExecStart=/path/to/lightningd \
    --bitcoin-datadir=/path/to/.bitcoin \
    --network=bitcoin \
    --alias=myCoolAlias \
    --ipaddr X.X.X.X \
    --log-file=/path/to/.lightning/log_file \
    --log-level=debug
    
    Restart=on-failure
    PrivateTmp=true
    
    [Install]
    WantedBy=multi-user.target
    

    Look for "Type=" in https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files#types-of-units for more informations

    Is it safe to run lightningd with root user?

  6. fabianfabian commented on Feb 28, 2018

    @fabianfabian
    Author

    @plachta11b I will try your configuration

    Is it safe to run lightningd with root user?

    Probably not, for me its just a testing machine.

  7. fabianfabian commented on Mar 1, 2018

    @fabianfabian
    Author

    @plachta11b your config works!

  8. 7 remaining items

  9. added 2 commits that reference this issue on Sep 12, 2022
  10. added a commit that references this issue on Oct 8, 2022
  11. added a commit that references this issue on Oct 28, 2022
  12. added 2 commits that reference this issue on Nov 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Status::Needs InfoAdditional information is required from the requester or relevant party before progress can continue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions