Librem 14 sudden crash when unplugged

Hardware Librem 14
81

I think there’s been a lot of progress in this regard and things look a lot more mature now, IMO.

Just to add some of my experience, that hopefully it will lead to improvements also in these cases:

On EC v1.9 and everything else updated, I’ve also experienced an instant power off, while being on battery under 50% (47% if I recall right).
I had a 4TB 2.5" USB3 HDD connected to the right side USB-A port, and doing some continuos read test from an 860 Evo 2TB SATA SSD in an M.2 to USB enclosure (left USB-A port).
I then plugged an USB-C charger, booted it up ( and it seemed ok afterwards.

Another case regarding v1.9, while playing aroud with the librem-control app (which is still in beta) - setting charge limits and start/stop charge, WiFi led settings (didn’t seem to work) and temporarily having “purism_ectool console” running, I got a hard lock/freeze. It happened also in previous EC revisions and it’s quite a problem.
The keyboard didn’t register any keys (including power off - keeping >4s doesn’t do anything), the power stays LED stays on and I have to open up the underside panel (not a quick job - whatch your nails) and then unscrew all the battery screws, so I can unplug the battery connector - the topleft tab of the battery blocks the connector from being fully disconnected. I then lost the system time - my L14 (2021) didn’t came with a RTC CMOS battery.

Afterwards, I reduced PL2 to 17W and when I’ll have a similar scenarios, I’ll report back.

82

Oh dear… this TI charge controller is driving me mad…

So… regardung the power offs, yes, I am sorry, I figured that out. I wanted to protect the battery from deep discharge so I query the battery flags and if it signals “stop discharge” I would cut power. But it turns out that these flags are not reliable :frowning: They trigger too often or prematurely so I removed that again and that should work better now.

The charge controller though is another beast and I honestly do not know what is going on there. Just last week I first noticed too that the charge current would drop dramatically during charging, in my case almost reliably when starting to charge below 40%. It would start with 2A, which it should, but some time later drop to like 100mA or so. I have no idea why this happens. What helps here is simply to stop charging and restarting it again, then it usually continues at 2A until the end threshold is reached.

I am working on a new branch right now:

This implements the fixes for the power off but more importantly also implements a new charging strategy. We will then, if this tests good, charge at different currents depending on the charge level - starting slow at low %, increasing toward the middle and then decrease again towards the upper %ages. This should on average speed up charging while still reducing wear on the battery. I hope that this also helps with the charge hickups some see.

In this branch I have also again implemented even more safeguards against false battery readings. The batteries implement a “Smart Battery Controller” (SBS) chip which reports the battery state and also design data, like charge voltage, charge current, design capacity etc. Turns out that sometimes batteries report wrong values! And most often I found the charge current to be 0mA - WTH!? This also causes of course totally absurd charging conditions. In this branch I implemented some safe defaults in case the battery reports non sense. The safe default is not fast but safe and should work good enough.

The battery information is BTW read each (once) when the EC powers up. This either happens when plugging in the charger or, when charger is not connected, when pressing the power button. So if you suspect wrong battery information just power off, disconnect the charger and reconnect / power on again. It also seems that the less charge the battery has the less reliable the readings are.

Finally regarding the ectool console, yes, this can, for reasons I do not know yet, hang the EC. It usually does not happen if you have the console running for just some seconds, but the longer it runs the more likely the hang gets. If that happens the only remedy is to unplug the charger and the main battery :frowning:

So… if you feel a bit adventurous please give this post-1.9 branch a try, it could fix quite some of the problems.

Cheers
nicole

4 Likes
83

Thanks for your hard work on this!

One update form me. I just had another crash (this time more like a freeze, but with a black screen, with some white lines on it). It was not connected to the charger this time. Battery charge was around 40-50%.
Also for me since 1.9 it does not charge at all while powered on, not even for a few minutes after it’s connected to the charger. I’ve even checked the charge_now parameter in /sys/class/power_supply/BAT0 It’s not changing. Only when the laptop is turned off. Could there be a problem for me re-flashing 1.7?

If you tell me how to flash/compile this new version (haven’t really found instructions besides “make BOARD=/ flash_internal”) and that it’s already safe for the hardware, I’m also willing to try the new version.

Edit: It does charge a bit, when turned on, but the display is off and it’s just idle.

84

I am getting to the bottom of this… and I have committed some more updates to the before mentioned post-1.9 branch.

First there was still some power off code in the EC that obviously gets triggered prematurely. The batteries seem to be notoriously unreliable about these flags which is why I have now completely removed checking for them :frowning: So sudden power off should not happen anymore :crossed_fingers:

The charging problem is an interesting one connected to the power budget, which is probably also the reason why I did not catch this earlier, since I am testing different power scenarios than others, it seems.

So what is happening, I think, is that charging at higher currents can exceed the total power budget of the AC adapter if in parallel some other high power consumers (like high CPU load etc.) draw power. If that happens the charge controller falls back to some pretty weird state which causes this super slow, if at all, charging - which clearly is not correct. So in my current work branch I am doing two things now:

  1. set more conservative charging currents again
  2. reduce Intel package PL4 to the battery / DC lower Watt level while charging to prevent the Intel package from high power draw (and set it to AC level again once charging has finished)

This should prevent the charge from brown out and charging should, hopefully, work normally again. I will test this more, now that I know what I need to be watching for.

I hope this works out better now and stable enough to make a new release soonish.

Cheers
nicole

5 Likes
85

For me ec_v1.9 work really good, like better speed up charging when laptop is ON, better battery status info. The only problem I have in v1.9 is that turn OFF when it reaches 10%.

@nicole.faerber some Librem 14 do not come with build-in RTC CMOS battery?

86

Yes, the initial batch we shipped came without. This should not make any difference usually since the main battery should never fully drain and keep the CMOS RAM intact. But to be really sure we added it in the later batches.

Cheers
nicole

2 Likes
87

how can i tell if my librem 14 has a separate battery for the rtc?

1 Like
88

can we get battery specs, and where to connect it ?

1 Like
89

I believe it’s a CR2025 (diameter 20mm), see pictures:

IMG20220825201219
IMG202208252012191560×2104 1.41 MB
IMG20220825200945
IMG202208252009451560×2104 1.34 MB

Sorry for the picture quality, that’s my skill level w/ Millipixels on L5, w/ a halogen desk lamp in the evening.

For connecting to the L14 MB, see this picture:

L14_CMOS
L14_CMOS1157×2315 645 KB

1 Like
90

Hi Nicole

I was wondering if you have some new insights concerning the sudden shutdowns? I updated the EC Firmware to latest versions, but my L14 still keeps crashing when the Battery gets in the range of 20% - 30%.
I really hope you can nail this problem as soon as possible, because this behavior is quite annoying…

Thank you in advance for your commitment to fix this issue.

Best regards,
Giacomo

91

Hello all, I’m a bit confused about how to test out the post-19 branch mentioned above. Would I be compiling the firmware from source in order to do so? I’d rather install it in the expected way. I don’t seem to see a .rom file for it.

92

Nevermind, just found the newer ec_v1.10. Didn’t realize there was a recent release.

93

@nicole.faerber Hi, first of all thanks for all your effort and trying to get this fixed!
Unfortunately I have to report, that my laptop again crashed at around 30-40% with EC version 1.11 today. I’ve installed the new EC version a few days ago.

94

Hmm… so you were running from battery at the time?
Additional devices attached?
CPU load?

Worst case we need to go even more conservative…

It could also be that the gas gauge of the battery is off. Ideally then let the laptop drain as much as possible (maybe just let it sit idle until it reaches 10% or 5%) and then attach the charger and let it charge to 100% in one sitting. That can reset the battery gas gauge.

Cheers
nicole

1 Like
95

@nicole.faerber Unfortunalty, I am experiencing the same problem like @Hologram. It is kind of not predictable when it crashes. Sometimes when I wake it up from a suspend, the battery level is sort of around 10%, and after a restart it has for instance 70%…

I do not know if this behaviour is specific to L14 version 1.

96

Yes, I was running from battery at that time, haven’t had crashes when power input is connected recently. This happened only once I think.
No additional devices attached and I was just doing light web browsing. CPU load shouldn’t been high.

I can try to do that. Although the evening before I have at least charged the laptop to 100% in one sitting. I also noticed sometimes, that the battery percentage jumps from time to time, f.e. when the laptop was in sleep for an hour or so it sometimes loses much more percentage then other times. Also (often after the laptop was in sleep mode for a few hours) sometimes the battery percentage drops much faster then usual (same usage behavior). I think this also correlates a bit with the crashes, but I’m not sure.

Maybe that helps investigating it.
Also EC version 1.7 was the only one so far where I had no crashes for a few weeks. Might just have been lucky, but normally I have a few crashes per week, so it might be worth to look at the changes from version 1.7 to later versions.

97

What I can say so far is. That running the laptop on its own, without external Display, runs relatively stable.
But at home I do attach my Dell 49" Display over DP. And then it depends on if Firefox is generating some load, the laptop may crash with 65 - 70% battery. Or, as @Hologram already mentioned, after a suspend the L14 has some strange battery balance. Like it suddenly drops to say 13% battery and crashes. After reboot I do have 70% battery. Very strange.

And when I put the L14 to suspension, my battery gets also drained. As if it would continue to run.

98

@nicole.faerber could you please clarify what you mean by “battery gas gauge”? thanks!

99

The battery gas gauge is the chip inside the battery that reports the battery percentage. Calculating the % state of charge (SOC) is actually not trivial. You can not simply take a battery and measure it somehow. So what these chips do is they measure the energy going into and out from the battery. But this is of course not 100% precise. Over time error builds up and the reported percentage can be way off from the real energy still in. The best way to recover from that is to discharge the battery pretty much and then let it recharge in one sitting to a full 100%. When the charge current drops to almost zero towards 100% the gas gauge will reset this collected new energy level as new full level.

Cheers
nicole

2 Likes
100

@nicole.faerber Hi again, I tested the full recharge you suggested. It was at 5% when I plugged in the charger. The first weird thing was, that it went from 5% to 9% in just 10 seconds or so. I’ve let it charge to 100% but still had a crash the day after at something around 30%. Also today again 2 crashed at around 55%. It seams to get more frequent… Without my USB-C power bank the laptop would be useless (as a laptop and not desktop PC)…
I’m wondering if the battery could be faulty instead of the EC having problems?