How do I fix boot of Ubuntu 17.04 from external USB HDD after update to Windows 10 Creator Update broke it?

Note: This is subtly different from many apparently similar existing questions.

I have Windows 10 installed on the internal drive of my note PC and Ubuntu 17.04 on an external USB HDD. Everything was working tolerably until the Windows 10 Creators Update came along. By tolerably I mean, if the HDD was plugged in it would boot Ubuntu, if not, grub would complain about not finding Ubuntu. I could type "exit" and eventually get into the EFI menu from where I could select Windows Boot Manager and boot Windows.

Since W10 Creators Update it always boots directly into Windows, even when the USB HDD is connected and even though the EFI boot order menu still shows "Ubuntu" as the first item. I tried moving the USB HDD to the first item but it continues to boot directly into Windows.

I want to take the opportunity of this problem to fix this correctly so if the USB HDD is plugged in it will boot into Ubuntu and if not it will boot into Windows without any interaction.

I booted Ubuntu Live, 16.04 because that's what I already had on a USB stick. Using gparted, I see that the Ubuntu USB HDD drive has

  /dev/sdc1 ext4        139.70 GiB boot
  /dev/sdc2 extended     11.17 GiB
  /dev/sdc5 unknown      11.17 GiB
unallocated unallocated 314.89 GiB

I am looking at the instructions in Windows 10 upgrade kills grub and boot-repair doesn't help attempting to modify them for my situation and have the following questions:

1. The root partition with 17.04 in it is /dev/sdc1. The instructions suggest /dev/sd*1 would be an EFI partition and /dev/sd*2 would be the root partition. How do I determine if I need to have an EFI partition?




2. I thought about trying sudo grub_install /dev/sdc per the above linked instructions, but I'd like some confidence it won't make things worse before I try it. Is this a sensible thing to do?




3. Do I even need Grub for this situation? Since disk has only one OS on it I don't need a way to choose the OS on power-up.

ADDENDUM

The original failure to boot when the USB HDD is unplugged is due to grub-installbug 1173457. This also explains how Windows update was able to break the Ubuntu boot now - it was using grub mistakenly installed on the internal disk and explains why the USB HDD is not "bootable" on its own.

The answer over in Install Ubuntu 17.04 on external usb drive without modifying the internal hard drive has suggestions how to fix the buggy grub-installation. I can't apply those suggestions directly because I do not have a good grub.cfg and because my USB HDD is not using EFI. /boot/efi is empty. Furthermore I need the answer to question 3 before I consider whether to install grub to the USB HDD.

boot-repair Findings

The boot-repair report URL is http://paste.ubuntu.com/25148428/.

/dev/sda is the internal disk with Windows, /dev/sdb the USB stick from which Ubuntu Live was booted and /dev/sdc the USB HDD with my 17.04 installation. From my limited knowledge of boot, it looks like there is indeed no boot file on /dev/sdc and Windows installed an EFI/Boot/bootx64.efi on /dev/sda which I'm guessing is taking precedence over EFI/ubuntu/grubx64.efi.

So I need to know how to install a boot file on /dev/sdc without also writing stuff to /dev/sda.

BTW why isn't boot-repair a standard part of Ubuntu Live?

First Attempt at Repair

I made a new 550MiB fat32 partition at the end of the unallocated space on /dev/sdc. The gparted on my Ubuntu Live USB stick does not support the ESP flag so I set just the boot flag and I added a label "ESP". I then attempted to copy the boot files from the internal disk but I got errors from copy: e.g

cp: error reading '/boot/efi/EFI/ubuntu/shimx64.efi’: Input/output error
cp: failed to extend ‘/mnt/EFI/ubuntu/shimx64.efi’: Input/output error`

Thinking about possible ESP filesystem corruption I booted Windows disabled Fast Start and shutdown Windows. I booted Ubuntu Live again and again tried the copy. Same thing.

So then I followed the instructions in the second answer at Windows 10 upgrade kills grub and boot-repair doesn't help, using /dev/sdc1 for the linux partition on my USB HDD and mounting /dev/sdc3 for the ESP. The core of these instructions is running grub-install /dev/sdc.

When I tried to reboot I got a secure boot error. When I dismissed this, Windows booted, running some disk checks in the process. I shut it down went into EFI, turned off secure boot, and this time Ubuntu booted.

I shutdown, unplugged the USB HDD and rebooted. It booted into Windows without any interaction from me. I then repeated with the USB HDD plugged back in and Ubuntu booted.

Progress indeed.

However I see 2 peculiar things:

1. /dev/sda3 (internal disk ESP) is mounted on /boot/efi. /dev/sdb3 (USB HDD) is mounted as /media/mark/ESP.


2. /media/mark/ESP/EFI/ubuntu has a subdirectory also called ubuntu with files of the same name as those in the parent (grub.cfg, grubx64.efi, mmx64.efi shimx64.efi, plus a directory fw and a file fwupx64. The contents of grub.cfg, grubx64.efi &simx64.efi` differ.

I have not yet done steps 5 & 6 in Rod Smith's answer. Step 6 looks like it should fix no. 1.

UPDATE 1 I have done step 6, edited /etc/fstab and now /dev/sdb3 is mounted on /boot/efi. I still see those extra files, of course.

UPDATE 2 I have now done step 5, copy the ubuntu files to EFI/boot and renaming the copied shimx64.cfg to bootx64.cfg. This has made it compatible with secure boot which is now renabled.

Will it be ok to just delete the extra files in the /dev/sdb ESP? What are fw and fwupx64? Do I need them?