Certain websites do not load, but load under a VPN connection? DNS settings?


#1

At first I thought I had a wifi issue, as certain sites just would not load at all, however I noticed that if I switched to a VPN connection those sites would then load, and would continue to load after I disconnected from the vpn.

I get this problem fairly regularly and it affects a bunch of sites. I have tried changing Purebrowser’s DNS settings (from preferences>advanced>change how PB connects) from No Proxy, Automatically Detect Proxy, Use system proxy settings with no change

In wireless network settings all settings under ipv4 are set to auto (routes and dns), so I’m a bit stumped. Any ideas?


#2

What sites do not load (please name som of them). What is the version of the PureBrowser you are using?


#3

Yes, please tell what sites don’t load, and confirm that the sites work from a different computer before concluding that it’s your Purism device and not something else.

If it turns out that none of your devices can connect to these sites, I’d start questioning what country you’re in (censorship) or if there’s a firewall / routing device upstream in your network that could be the culprit.

If it really is the PureOS device though - strange. I’d ensure your network adapter is running normally, then check your Firewall and HOSTS file to ensure those don’t have any rules in place for those domains either.

After that I’d start wondering if the sites in question block browsers that identify as using Linux in their User-Agent for some reason - the remedy would be to spoof it of course, but I’d wonder why anyone would do that.

If it’s none of the above then it must be something technical in PureBrowser that I just don’t know because I’ve never used it yet.


But, by the first sounds of it - the connection works on a VPN. That makes it sound like your country is blocking the website or a device upstream in your network is blocking those websites and your VPN is allowing you to tunnel past it. You’re not attempting to connect to these sites at your workplace are you? Did they work before?

But then you say it works after disconnecting the VPN as well. My assumptions are either that the VPN didn’t actually disconnect when you disconnected it (check your IP please) or that there’s something wonky with the network adapter. Many VPNs mess with the adapter setting and install protocols in order to function. This could’ve possibly messed something up. I wouldn’t be able to tell without sitting down at your computer and having a look-see though.

That’s about all I can come up with off the bat. Mind you these are all ideas I’m coming up with as a Windows user, so take them for what you will, I’m not a Linux user yet so for all I know it could be something in Linux that I just didn’t know exists. This is all just advice that I’d be giving to someone who’s having the problem on Windows.


#4

Another option is DNS server of your internet provider is broken. VPN install its own, and the sites work as long as their resolved hostnames remain in local DNS cache.


#5

Ah yes, definitely this as well OP. It’s just so rare for this to be the case that I didn’t consider it. It’s rather odd for DNS to not work only on select sites like that. Hopefully your ISP isn’t deliberately DNS spoofing / DNS poisoning, but that’s one method that many countries governments and ISPs use to block users from using sites they don’t like.

Fun Fact: There was even once an issue with DNS poisoning from China was beginning to spread to the rest of the world because DNS servers learn from each other and there were so many DNS servers in China like this.

Try testing by changing your DNS to one of many public DNS servers. Not suggesting any of these as a permanent DNS replacement (you need to be choosier about that one, preferably one that’s trusted and supports DNSCrypt), but just to check:

Google: https://developers.google.com/speed/public-dns/
Primary: 8.8.8.8
Secondary: 8.8.4.4

Verisign: https://www.verisign.com/en_US/security-services/public-dns/index.xhtml
Primary: 64.6.64.6
Secondary: 64.6.65.6

OpenDNS: https://www.opendns.com/setupguide/
Primary: 208.67.222.222
Secondary: 208.67.220.220

Level3:
Primary: 209.244.0.3
Secondary: 209.244.0.4

DNS.WATCH: https://dns.watch/index
Primary: 84.200.69.80
Secondary: 84.200.70.40

Comodo: https://www.comodo.com/secure-dns/
Primary: 8.26.56.26
Secondary: 8.20.247.20

Dyn: https://help.dyn.com/internet-guide-setup/
Primary: 216.146.35.35
Secondary: 216.146.36.36

Private Internet Access: https://www.privateinternetaccess.com/pages/client-support/ (under “DNS Leak Protection”)
Primary: 209.222.18.222
Secondary: 209.222.18.218

More: https://www.lifewire.com/free-and-public-dns-servers-2626062
OpenNIC Public Servers: https://servers.opennic.org/

If none of those work, it’s something else.

It’s always best to use your VPN’s DNS though. I mean, I assume you trust your VPN handler if you’re buying their service. Using a different DNS than your VPN’s means you now have to trust two companies as opposed to just one.

I always use my VPNs but on the very rare time something seems wonky with them, I use Veracrypt’s.


#6

I mean, you could also try connecting to the websites using their direct IP address as opposed to using a domain name as well. If you know what their IP address is, try it. It’s how you can test to see if it’s really a DNS issue or something else.

That’s how I connected to TPB when I was in China years ago. Back then all they did was DNS methods of blocking and I could just subvert them by using direct IPs. Their blocking has gotten a lot more sophisticated with deep-packet inspection and such since then though. Pretty sure only VPN-obfuscating technologies like Vypr’s “Chameleon” works without hitches against it now. My dad just go back from China and told me that PIA was extremely spotty, which tells me that 256-bit OpenVPN methods isn’t a cure-all anymore… sucks.

It could still be a hosts file problem though, since you CAN put IPs in that as well.


#7

http://www.aljazeera.com/ will fail every time on a fresh boot, then connecting through a VPN that exits in my country will reload that page just fine.

Also, I’m pretty sure my Purebrowser is up to date, ran apt-update & upgrade (about page in PB doesn’t show version number?)


@Alex & @Dwaff thanks very much for the suggestions, I have a feeling that it might be the ISP, I’ll attempt some of the suggestions for the DNS Alex supplied.


Also @Alex, yes I’m also mainly a Windows user, though have taken the plunge and seeing if this can be a daily drive. Also, thanks for the lengthy post and help, you’re right about just trusting the VPN and not adding an extra company to trust via an external DNS entry.


#8

This is most likely ISP/DNS issue.


#9

Now that you’ve mentioned it’s Aljazeera I’m almost certain it’s your government or ISP that’s blocking it. Lots of countries where censorship is law block Aljazeera.

You’ll probably need to use your VPN to access it, plain and simple. If they’re really only using DNS to block the site then you may be able to access it by just using a different DNS or using direct IPs, but I doubt that’s all they’ve done.


#10

So, bit of an update:

I tried using Cisco’s opendns ip’s in my router’s DNS settings and the Librem’s ipv4 settings, both still didn’t allow aljazeera.com to load on the Librem, however other websites loaded just fine. So I scrubbed them and went back to automatic DNS on both.

My other computers on the same wireless network and even my phone (also on the same wireless network, 4g network disabled) load aljazeera.com just fine without a VPN, so now I’m looking back toward the Librem as the cause somehow?


#11

Are you saying the hardware can be the cause of this? There’s got to be some reasonable explanation, check your /etc/hosts file, disable all the extensions you have installed, compare loading https://www.aljazeera.com and http://www.aljazeera.com (for me, http loads fine, loading https takes forever).

Else you might wanna try a different browser (check Software Center) to see if this is unique for PureBrowser or reproducible for other browsers as well.


#12

Yeah this is a rather strange issue.

Your other devices aren’t also using a VPN are they? They’re able to access the site without a VPN or altered DNS?

Disable all extensions and try other browsers.

… You weren’t possibly IP banned by an admin there for some reason, were you?


#13

Ok, so all the behaviour shown below is without VPN, bear with me:

On a fresh boot, both https://www.aljazeera.com/ and http://www.aljazeera.com/ on both Purebrowser and Firefox don’t load:



Then I installed the Dilla browser and http://www.aljazeera.com/ gave me a DNS error, however https://www.aljazeera.com loaded in somewhat janky fashion:


Once Dilla loaded the https version, then Purebrowser and Firefox happily loaded both http and https versions of the site:


I tried rebooting several times and I would have to load https in Dilla 1st in order for PB and FF to load the page. Apart from one time, when I rebooted and it ‘restored’ the Purebrowser session, after which it loaded fine without the use of Dilla.

I had a look at my hosts file:


127.0.0.1 localhost
127.0.1.1 TLaptop

The following lines are desirable for IPv6 capable hosts

::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters


and I noticed there were hosts.allow and hosts.deny files there too, but they only had # comments in them.

Now for the VPN stuff:
@Alex, my other devices can connect to aljazeera without a VPN.

It’s not just Aljazeera though, I’ve noticed other sites that are missing content without the use of VPN. For example (and there have been several sites, but I’m showing this because I remember), this is what the Adobe Flash Player download site looks like without VPN:

Whereas this is what it looks like if I pop the VPN ON:

I’m ok to keep browsing with the VPN on, but I bring it up because it’s annoying behaviour that shouldn’t be happening.


#14

can you do following - after reboot but before you launch any browser.

  • check content of /etc/resolv.conf after the boot and ensure it contains proper nameserver definitions
  • run ping to each of the mentioned there nameserver
  • run dig www.aljazeera.com
  • run curl -v http://www.aljazeera.com/
  • finally open the browser and ensure behaviour is consistent. Eg if curl works and returns content - browser should show the page as well. If curl gives error - browser may give error as well, or may actually display the page.

#15

I kind-of forgot to mention this, but try using trace route in the command line to the domain.

Should show if you can even send or receive packets from the site without a VPN. If you can then it’s probably a browser issue.

Not a lot of experience using trace route in Linux but in Windows I’ve often used tracert to do test and eliminate possibilities. Depending on where the packet is lost you can kind-of deduce what the point of failure is.


#16

Hey ruff, here are the readouts from the commands you suggested:

From /etc/resolv.conf on fresh boot, don’t know why it’s so large or bold

------------------------------
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.10.1

------------------------------

ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.816 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=1.00 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=0.784 ms
64 bytes from 192.168.10.1: icmp_seq=4 ttl=64 time=0.829 ms
^C
--- 192.168.10.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3015ms
rtt min/avg/max/mdev = 0.784/0.857/1.001/0.089 ms


------------------------------------

dig www.aljazeera.com
;; Truncated, retrying in TCP mode.
;; Connection to 192.168.10.1#53(192.168.10.1) for www.aljazeera.com failed: connection refused.


----------------------------------

curl -v http://www.aljazeera.com/
* Could not resolve host: www.aljazeera.com
* Closing connection 0
curl: (6) Could not resolve host: www.aljazeera.com

-------------------------------------

@Alex: here is the traceroute (had to install traceroute first)

traceroute aljazeera.com
traceroute to aljazeera.com (113.29.17.253), 30 hops max, 60 byte packets
1 _gateway (192.168.10.1) 0.982 ms 2.469 ms 2.470 ms
2 router.asus.com (192.168.1.1) 3.197 ms 3.207 ms 3.190 ms
3 10.0.5.1 (10.0.5.1) 4.734 ms 5.300 ms 5.036 ms
4 103.233.225.193 (103.233.225.193) 8.038 ms 8.055 ms 8.424 ms
5 * * *
6 te2-3.bqueedist02.aapt.net.au (203.131.62.24) 21.355 ms 16.509 ms 16.955 ms
7 te0-3-1-0.bforvcore01.aapt.net.au (202.10.12.21) 18.101 ms 18.520 ms 18.515 ms
8 te0-0-1-0.sclarcore01.aapt.net.au (202.10.10.69) 19.043 ms 19.453 ms 19.455 ms
9 po11.sclardist01.aapt.net.au (202.10.12.2) 18.033 ms 18.360 ms 18.319 ms
10 bu8.sclarbrdr11.aapt.net.au (202.10.14.23) 18.597 ms 18.825 ms 16.785 ms
11 syd-optus.gw.aapt.net.au (203.8.183.45) 16.782 ms 17.364 ms 17.468 ms
12 * * *
13 59.154.142.40 (59.154.142.40) 17.738 ms 17.732 ms 59.154.142.42 (59.154.142.42) 17.937 ms
14 220.101.73.30 (220.101.73.30) 130.903 ms 130.926 ms 131.788 ms
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *


#17

Ok, that clearly points to your home router which

  • does not support edns
  • does not support tcp dns
    hence cannot handle payload bigger than 512 bytes.
    The aljazeera’s answer is indeed bigger than usual however it would still fit 512 bytes if client would not request additional headers:
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 22780
;; flags: qr rd ra ; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; www.aljazeera.com.	IN	A

;; ANSWER SECTION:
www.aljazeera.com.	300	IN	CNAME	2-01-3b91-0003.cdx.cedexis.net.
2-01-3b91-0003.cdx.cedexis.net.	20	IN	CNAME	www.aljazeera.com.edgekey.net.
www.aljazeera.com.edgekey.net.	21600	IN	CNAME	e9106.dscg.akamaiedge.net.
e9106.dscg.akamaiedge.net.	20	IN	A	104.125.24.110

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 560 msec
;; SERVER: 10.0.0.158
;; WHEN: Mon Nov 20 20:23:26 2017
;; MSG SIZE  rcvd: 255

which is 255 bytes.
I’d suggest setting up local dns cache - eg. unbound.


#18

Legend! Thanks so much for taking the time to guide me through to the problem. The only other things I’d ask is:

  1. Why do the other machines (Windows based) not have an issue? (i.e. they can connect without vpn no problems, connecting to the same router)

  2. Can you point me to anywhere that can explain how to setup a local dns cache?


#19

Update FIXED!

Thanks very much Ruff, logged into router and saw that DNS caching was disabled, so enabled it and aljazeera.com now loads on cold boot.

Still, find it odd that the Windows machines were fine, however the Librem was not?

Also I still can’t explain why that Adobe page (and others) have content missing, but don’t through VPN


#20

As I’ve demonstrated the answer itself fits standard DNS datagram size so if windows does not request additional data it will be able to resolve the host. Also windows machine normally runs local DNS cache service therefore it does not need to fetch additional data at once (it might be cached). Only if it’s missing.
In other words - windows does lazy dns recursion, linux does aggressive recursion. If you install local dns cache - it will also do lazy recursion (because it is cache).