Image enhancing on the Librem 5

Hardware Librem 5
21

on the contrary free software can trim the “fat” better than proprietary can because it is open and thus more prone to critique. “proprietary-professional-apps” have just enjoyed the monetary priviledge for a longer period of time.

example - x individual from the government offers sponsorship to the CEO of a popular proprietary software company in exchange for backdoors and certain information leaks about it’s users because he can’t do that with free software (because he would get exposed in public). the result - the said software company can AFFORD to hire more developers. the state-of-the-art-advances. share-holders flock to the company. profit>profit>more profit>global monopoly. and that was just because with patents and copyright protection hidding is possible while with copyleft and anti-patent laws that isn’t feasible.

Ayn Rand in Atlas Shrugged speaks very clearly about how it’s NOT about money but CONTROL.

2 Likes
22

I suggest to remove the tinfoil hat.
Do you really think Adobe/Apple will put backdoors for governments just because the CEO was asked to?
This is not how it works, such corporations have huge numbers of employees and if it was something
common it would be leaked already.
A much cheaper option is just to find exploits for those applications (see the recent Whatsapp remote code execution) than jumping the hoops to attempt what you suggested.

1 Like
23

hmm. no. i believe there isn’t enough tinfoil to go arround for everyone on the planet.

how does it work then ? how do you know for sure that each employee in the company sees all the code at any given point in time. have you heard of Compartmentalization ? i would imagine it’s not difficult to do in a multi-million-dollar-proprietary-software-company.

same can be said about patent holding hardware manufacturers. each employee works on a different piece of the puzzle not knowing they could be creating a mass-destruction-weapon. but let’s not get too caried away here :wink:

1 Like
24

Let’s assume the CEO wants to put a backdoor in the 1million+ lines of code.
He then has to consult with the CTO about which area can be more suitable for this.
The CTO then has to call the team leader of that particular portion and ask him to commit new
changes. The team leader will ask the developer who wrote the portion of the code around this
area to push commits.
That’s roughly how such processes are done in pretty much every 10+ employees startup to 10k employees
mega corporations, although the bigger it is the harder it gets to keep it in secret.

The implications of such things will be huge once it’s leaked, since it will create a mass scandal and distrust
from all the user base, and can potentially bankrupt the brand. No CEO will attempt it, at least not in democratic
countries where the majority of business software originates from.
There are allegations about Huawei doing malicious things in both software and hardware,
but in China the laws are different and so is the mindset of corporations.
Look how badly it affected the brand, nobody wants to use them in the government sector, in 5G deployments,
etc. And that is only because of an allegation.

25

just to be clear here. we are not debating WHO it is that would agree to something like this in the first place and quite honestly the HOW isn’t much of a problem for people in this line of work. we can speculate ad-infinitum here without proof and we would never reach a consensus. my tin-foil hat remains on when it comes to non-free code (hardware and software alike).

there is so much that has already been discussed elswhere and by more apt people that it is a drag to keep this up.

the question of morality between free and non-free has already been set in stone. now if the state-of-the-art in free software is not yet there compared to non-free software then that is simply a matter of RND and constant innovation as you’ve pointed out above but there is NOTHING inherently superior about non-free code.

2 Likes
26

Nobody would ever attempt to make movies purely with free software.
Certainly, Disney would never release a movie that was done entirely with free 3d software
And if Hollywood studios would ever do such, they certainly would never say

Blender first caught our eye because some of our artists had started trying it out and were surprised over how much faster they could produce models.

It will not catch on.

It’s very important to always be realistic about one’s project goals, like

I’m doing a (free) operating system (just a hobby, won’t be big and professional) … It is NOT protable [sic] … and it probably never will support anything other than AT-harddisks.

Only morons would go for world domination.

I stopped doing the “world domination” joke long ago, because it seemed to become less of a joke as time went on.

1 Like
27

You do understand that finding some rare examples will not actually proof the fact that free software lacks the same level of features for the professional world.
Maybe you will also claim that some top world gamers play games on Linux? Finding some unique cases
is always possible. But good luck trying to convince professional DJs (an industry that is closer to me) use
free software for their mixing and production, for example LMMS instead of Fruity Loops.
Or maybe you are those who claim that GIMP can be a full Photoshop replacement?
Luckily good content creators stick to the real thing, which is actually to create good content, instead of
following some flawed ideas that all software in the world should be free and open source.

28

ah, the change-the-topic trick :slight_smile:

and the Linux kernel is not an app, therefore doesn’t count.
LibreOffice is only preferred by a minority of companies, organizations and administrations, therefore doesn’t count. Also, LibreOffice has its roots in “professional”, proprietary software, therefore even more so doesn’t count.
Just like Blender can never count, as it was started by a company and furthered by paid developers.

And while KDE Plasma’s features have outnumbered the features of any Windows Desktop for decades, that does not count because it makes use of the Qt libraries, which, as we all know, are dual-licensed and also available under a non-free license and developed by “real” professionals. Also, it doesn’t count because <insert random reason>.

Free software developers just don’t have what it takes.

Yes. It’s that simple.
Because it would be unethical to charge for free software.
Therefore, it’s the fate of all those hobby-projects to be maintained by beginners who don’t really know what they’re doing.

1 Like
29

Currently, we’re aiming for taking acceptable snap shots, which is already a whole bunch of work. But later, it all depends on what the community decides to apply its brilliant minds and powerful motivation to. Fcamera is a community-made application after all, and works alongside the builtin N900 camera app.

4 Likes
30

the only problem here is that these tools you use in order to achieve this “good” content OWN you and not the other way arround. there WILL come a time when the only PERMITED way to use these tools will be to implant something in your nervous system which will ALLOW you to stay “competitive”. wake up now and realise that sometimes the only way to maintain freedom is to (temporarily) take a few steps back in terms of state-of-the-art.

so why can’t i wait untill the time of implants arrive ? because if that “THING” takes control of my I/O then i have no means of SELF-EXPRESSION LEFT to oppose EVIL. act now before it is too late !

1 Like
31

Here is a paper about Multi-frame Super-resolution method developed by google:

https://sites.google.com/view/handheld-super-res/

2 Likes
32

Reading that paper gives me more hope that we might eventually have a free/open source camera app that can replicate what Google and Huawei are now doing with their cameras. Unfortunately, it doesn’t look like the Librem 5 will have the kind of hardware capable of taking multiple photos in rapid succession to be able to merge them into one image. The i.MX 8M quad simply isn’t a powerful enough processor and we really don’t have any other option at this point if we want 100% free/open source software. The upcoming i.MX 8 mini will be a little better, but we really need NXP, SiFive or some other company to make a better SoC that can run on 100% FOSS.

1 Like
33

The CPU is not a primary concern. Usually the camera is limited by the bandwidth it is connected to RAM, leading to a max number of raw frames that can be taken per second.

1 Like
34

The CPU is not a primary concern. Usually the camera is limited by the bandwidth it is connected to RAM, leading to a max number of raw frames that can be taken per second.

Yes, and another issue is how fast can you save the image. The eMMC 5.0 storage used by the Librem 5 won’t be nearly as fast as the UFS 3.0 storage used by the flagships. Where the SoC is important is when we are talking about the fancy image processing that paper describes. If it takes the SoC 30 seconds to process the image, then it isn’t usable.

Looking through the spec sheet on the i.MX 8M Quad, I’m not very optimistic. It doesn’t have a dedicated digital signal processor or image signal processor, like Snapdragon processors. It also doesn’t list how many megapixels are supported in a camera like Snapdragon does. The i.MX 8M seems to be designed for video streaming, but not for the kind of image processing done by a comparable Snapdragon. In terms of its CPU, the iMX 8M Quad is basically the equivalent to the Snapdragon 425 released in Q3 2016, but it does not seem to have the same image processing capabilities.

It seems like all the open source implementations use OmniVision image sensors, and those range in resolution up to 32 megapixels. Fairphone offers a camera module with the OmniVision OV12870 with 12 MP, so hopefully we will get something similar in the Librem 5, but the Snapdragon 800 in the Fairphone 2 is capable of processing 12MP images, and I’m not sure that the i.MX 8M Quad can do that.

Oh well, I’m still going to have a blast playing with the Librem 5. It looks like it will have enough processing power to run a full Linux desktop on an external monitor, which is the most important thing in my opinion. What worries me is whether we will have better processors from NXP in the future. NXP might decide to not design anything better than the i.MX 8M mini, since that is all that the automotive market needs, and we don’t really have any other options. Maybe RISC-V will eventually provide an alternative, but it looks like that is at least 5 years in the future.

2 Likes
35

I think Machine Learning for Image Manipulation on the Librem Phone would be very
appropriate.

We are going to get crappy specs let’s be honest. According to Antutu, the CPU is 10x
less powerful than even cheaper current phones, I don’t expect the camera to be anywhere
close to the quality of any other recent phone. But we all know this and we are not
ordering the Librem for its spec, but for Freedom.

Given that, I think that ML can really compensate for the so-so camera we are going
to get. I understand it might be seen as “cheating”, but if you are a purist of photography
you should just buy a camera as you’re not going to go far with the librem 5 camera anyway.

Also there is a whole lot of different ML algos out there, not all hallucinate pixels.
For example you could have ML contrast/color/lighting automatic adjustments and all
it does is just automatize what you would do by hand on GIMP. Similarly you could use
ML for HDR, you take pics at different expositions, but when combining them, instead of
hard coding arbitrary parameters, you let the ML part find the best weight for each pic.
It’s pure HDR, no information was “hallucinated”, we just have ML to find the best parameters
that’s all.

It is especially important on phone. On desktop maybe we are okay with hand fine-tuning
dozens of parameters silders on GIMP, but on the phone both because I am on the go and
because of the size of the screen, I want quick easy one-click features.

I might be biased because I am a CS student with specialization in ML, but don’t
think that those things are that complicated. I could completely re-implement myself most
of the so-called “AI” (I hate this term) features of the Pixel / Huawei cameras if there are
published papers.

Now given the spec of the Librem Phone, we are not going to get this in realtime, but we can
take a picture and then process it in background, no need for dedicated ML chip for that, we can
just leverage the integrated GPU.

Concerning Privacy it very sad what ML is used for. Before it was “Give us your data or terrorists will kill your children”, now it’s “Give us your data or science will not progress and we won’t be able to do ML on it”. It’s absolutely horrible how facial recognition (which should be legally banned imo) is used by FB & others, how
NLP algos run on our private conversation to profile us, etc.
But honestly, like for terrorism, it is yet another excuse for taking more private data. Data can be bought from official hand-annotated dataset, can be anonymized and more importantly we can now use Federated Learning to not only preserve privacy but also distribute the cost of computations.

4 Likes
36

A bit off topic maybe, but I think it is interesting to wonder what Free&OpenSource terminology means when it is applied in the context of ML : If you use a pre-trained model but you don’t have the original data, can
you call it FOSS ?

You have the source and can see every line of code, you know perfectly what are the inputs/ouputs, you can study the architecture, but at the same time you’ll never be able to change/improve the program (the ML part of it) if you cannot re-train.

1 Like
37

They are not FOSS, but here are two ML-powered tools that we can use on a Purism device:

  • Polarr.co: A photo editor with both a web and Linux version.
  • LetsEnhance.io: A web app that does upscaling and other enhancements. The team is working to add various functionality. Since rendering is done on their server, having a low powered device won’t be an issue.
38

Do you mind ratting out your source that told you what camera specs we will use :P?

I’m not sure what makes you think that the camera is going to be “so-so”, considering that we haven’t published the specs yet. Perhaps you mean that we’re not going to immediately provide advanced processing software? ML would be advanced processing software to compensate for the missing advanced processing software, so you surely don’t mean that.

This only matters if you want to save the entire captured data, and only when that exceeds the size of available RAM. I don’t think you should worry about it when your goal is to merge 5 pictures into one.

As Caliga said, CPU speed has nothing to do with the capability to process images; slower CPUs take more time, and the raw data can be stored to process at a later time anyway.

4 Likes
39

If anyone is on the dev-kit cahtroom . I’m working on the exact problem. Building an app with Google’s super solution algorithm.
But I’m lacking the camera parameters. So before that… I think I’ll see what I can do with phosh.

https://sites.google.com/view/handheld-super-res/

2 Likes