Finally found an answer after some creative googling.
There are 2 important pieces of information one needs to understand about why this works. The first is about the boot process and how the initial power on self test (executed by the BIOS) looks for a boot record on the first boot sector of the devices attached. This is a small file which basically passes the boot execution process onto the boot loader/manager such as GRUB2 on newer linux installs. For more information on this, read this very informative post.
The 2nd piece of information is to understand that the ISO downloads for Linux OSes are designed to be installed on a disk (CD/DVD/USB) and not for a partition. These ISOs need to be modified using a small tool which can be found on the syslinux-utils package, called isohybrid. The ISO modification for installation on a partition is done with the --partok flag. I found this information from this post which describes the exact same issue.
I am not sure how the --partok magic works, I searched for some explanation but could not find any.
In summary, here is what I did to get my live PureOS install to boot from a usb partition,
- Partitioned my USB using GNOME disk, I initially formatted the disk using MBR/DOS. I created a 4GB partition (FAT32) followed by a an ext4 linux partition in the remaining space for my backups.
- Next I modified the ISO file I downloaded to make it bootable from the partition. To do this I had to install the syslinux-util from my repo
sudo apt-get install syslinux-utilin which the isohydrid tool resides. I then modified the ISO file,isohybrid --partok pureos.ISO - I used dd to copy this ISO to the above partition. You can get the partition device from the GNOME disk details.
dd if=pureos.ISO of=/dev/sdb1 bs=1M. - from GNOME disk I edited the partiton and checked the Bootable option which installs the required book record.
I was now able to boot from the USB key into the live CD and use it as a recovery solution while storing my system backups on the other partition.