Commit Graph

5807 Commits

Author SHA1 Message Date
Dan Bungert ba42c5eed7
Merge pull request #1794 from dbungert/lunar-security-pocket
lunar: security archive
2023-09-12 07:16:43 -06:00
Olivier Gayot c1166b1e0d mirror: do not let curtin decide the URL of the security archive
When the URL of the security archive is unset, curtin will set it to the
URL of the primary archive.

This is not the behavior we want for Ubuntu installations. On amd64 (and
i386), the URL of the security archive should be set to
http://security.ubuntu.com/ubuntu

On other architectures, it should be set to
http://ports.ubuntu.com/ubuntu-ports

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit 5556313652)
2023-09-11 18:24:53 -06:00
Olivier Gayot efb1bd8ae5 mirror: for mirror-testing, disable the security suite
Mirror testing should focus on testing the primary mirror, not the
security archive - therefore we disable the -security suite.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit 79f2c4c432)
2023-09-11 18:16:31 -06:00
Olivier Gayot 7ce11afd57
Merge pull request #1760 from ogayot/lp2013201-lunar
2023-08-09 - Backport manual size fix to ubuntu/lunar
2023-08-09 17:27:40 +02:00
Olivier Gayot b53731356c filesystem: store the actual size in bytes alongside the human readable size
Currently, the partition form stores the size as a human readable value.
(e.g., 123456K, 1.1G, 1.876G, 100G). When we exit the size field (i.e., upon
losing focus), we convert the value to a number of bytes and then align
it up to the nearest MiB (or whatever the alignment requirement is).

Unfortunately, after computing the aligned value, we turn it back into a
human-readable string and store it as is. It is not okay because the
conversion does not ensure that the alignment requirement is still
honored.

For instance, if the user types in 1.1G, we do the following:

 * convert it to a number of bytes -> 1181116006.4 (surprise, it is not
   even an integer).

 * round it up to the nearest MiB -> 1181745152 (this is the correct
   value)

 * transform it into a human readable string and store it as is -> 1.1G
   - which actually corresponds to the original value.

This leads to an exception later when creating the partition:

    File "subiquity/models/filesystem.py", line 1841, in add_partition
      raise Exception(
   Exception: ('size %s or offset %s not aligned to %s', 1181116006, 1048576, 1048576)

Fixed by storing the actual size as a number of bytes - alongside the
human readable size.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit cda6c54b87)
2023-08-09 17:01:44 +02:00
Dan Bungert 0902246d8b
Merge pull request #1665 from dbungert/lunar-curtin-ntfs
snapcraft: rev for ntfsresize change
2023-04-18 19:22:59 +01:00
Dan Bungert 0b4fc4153d snapcraft: rev for ntfsresize change 2023-04-18 19:12:10 +01:00
Dan Bungert 3bddb3632b
Merge pull request #1663 from dbungert/lunar-no-probert-during-partitioning
Lunar no probert during partitioning
2023-04-18 19:10:06 +01:00
Olivier Gayot f1e24f3580 storage: add API test to cover probert run finishing during partitioning
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit 9e4cd77d9d)
2023-04-18 18:56:41 +01:00
Olivier Gayot 7057f77c49 storage: add unit tests to cover probe data locking
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit d8748c27fe)
2023-04-18 18:56:41 +01:00
Olivier Gayot d7ce2e5048 storage: don't let probing discard storage config in the middle of partitioning
If a probert run finished in the middle of partitioning, the probing
data would be loaded automatically and this would essentially discard
changes made so far by the user. This is not a desirable behavior.

Upon starting partitioning, we now prevent later probert runs from
automatically loading the probe data. Instead, the data is stored and
only loaded after after a reset.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit cd25ec454f)
2023-04-18 18:50:03 +01:00
Dan Bungert bfeee32e48
Merge pull request #1662 from ogayot/lunar-2023-04-17
Lunar 2023 04 17
2023-04-17 02:18:07 -06:00
Olivier Gayot d3b27b116b snapcraft: bump probert to pickup verbosity fix
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit 55a5ebd42b)
2023-04-17 09:21:49 +02:00
Olivier Gayot cc01782fb7 mirror: clean environment before running apt-get
For mirror testing, we run apt-get update on the host. By default,
external commands run by subiquity inherit the environment variables set
by the snap (including LD_LIBRARY_PATH).

We don't ship apt-get or its dependencies in the subiquity snap, so we
want to reset LD_LIBRARY_PATH and other variables when running the
command.

Not doing so leads to the following error when running the subiquity
snap on a focal-based system:

 * apt-get: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not
   found (required by /snap/subiquity/4675/usr/lib/x86_64-linux-gnu/libsystemd.so.0)
 * apt-get: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not
   found (required by /snap/subiquity/4675/usr/lib/x86_64-linux-gnu/libsystemd.so.0)
 * apt-get: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not
   found (required by /snap/subiquity/4675/usr/lib/x86_64-linux-gnu/libsystemd.so.0)
 * apt-get: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not
   found (required by /snap/core22/current/lib/x86_64-linux-gnu/libcap.so.2)

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit 3bb74546e2)
2023-04-17 09:21:38 +02:00
Olivier Gayot 96f09a7004 filesystem: fix crash when editing swap partition in v2
When we create a swap partition using v2_add_partition, the mount
parameter always has the value of None.

We do however, create a fake mountpoint object and assign it the value
of "" (empty string) so we know the swap partition is used although it's
not really mounted anywhere in the filesystem hierarchy.

Subsequently, when editing the swap partition, the mount parameter is
set to the empty string, not None.

This causes trouble in the create_mount function, which normally returns
immediately upon encountering None as the mountpoint.

Fixed by skipping the call to create_mount for a swap partition.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit aa6274c38e)
2023-04-17 09:21:28 +02:00
Dan Bungert 6ba87d99c9
Merge pull request #1658 from dbungert/lunar-2023-04-13-2
Fixed paths for lsc and locales commands
2023-04-13 14:31:46 -06:00
Carlos Nihelton 3f179bb73d Fixed paths for lsc and locales commands
Subiquity snap doesn't stage language-selector-common nor locales
packages.
Attempting to run those commands require respecting the env previously
prepared by the caller, as well as fixing the commands paths
(or changing the PATH env var in the env preparation step).
Since the apt commands were already ran with a fixed path,
I'm applying the same principle in here.
Yet, for now, Jammy doesn't ship the l-s-c package seeded,
so a condition was preserved to avoid future crashes.

(cherry picked from commit 2e67005403)
2023-04-13 14:23:14 -06:00
Dan Bungert c5aebd80b1
Merge pull request #1653 from dbungert/lunar-2023-04-13
Lunar 2023 04 13
2023-04-13 10:36:48 -06:00
Dan Bungert 8f647e0971 errorreport: tag report with snap name
(cherry picked from commit cd03fea2f0)
2023-04-13 10:23:03 -06:00
Dan Bungert ea2fca5cb0 errorreport: add tags capability
Future problem_report has a nice add_tags method, simulate it here.
Should be able to drop this with a core24 move.

(cherry picked from commit 232f2f9d8c)
2023-04-13 10:23:03 -06:00
Olivier Gayot b6dc275817 storage: fix crash when editing encrypted VG created in guided storage
When editing an encrypted VG that was created in the guided storage
screen, the VG information is originating from the server. However, the
server does not send the LUKS key over the wire. Instead it sends the
path to a keyfile which contains the key. The client may or may not have
read access to this keyfile so it does not have a reliable way to
determine the key.

This causes problem when editing the VG because the GUI expects to
receive a key when encryption is enabled.

If the VG object only contains a keyfile, the passphrase is set to None
and this result in the GUI crashing.

This patch fixes the crash by passing an empty passphrase instead of a
None value when the VG object only contains a keyfile.

This means the user gets forced to supply a passphrase again when
editing an encrypted VG that was created in the guided partition screen.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit dcc66fa346)
2023-04-13 10:22:51 -06:00
Dan Bungert d549c2bdbb
Merge pull request #1649 from dbungert/lunar-2023-04-12
Lunar 2023 04 12
2023-04-12 10:00:24 -06:00
Olivier Gayot 4722feba26 api: avoid post-ing to source in autoinstalls
When running autoinstalls, the source model will be marked configured
automatically after apply_autoinstall_config has run. The selected
source will be the one described in the autoinstall section or will
default to the legacy server entry: Ubuntu Server.

Doing an additional POST request can only make things inconsistent in
fully automated installs.

When the POST request is handled, most of the models may already have
applied their autoinstall configuration, and are already relying on the
previous source selected.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit 8298ec17c8)
2023-04-12 17:52:09 +02:00
Olivier Gayot 57741451e2 mirror: bail if source changes after applying autoinstall config
If the source model changes, the mirror model gets a chance to apply a
new apt configuration. However, if this happens during an automated
install, this is a recipe to disaster.

Make sure we raise an exception if a POST request to /source occurs
after the mirror model has started applying its autoinstall
configuration.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit 6aa9452bb0)
2023-04-12 17:52:06 +02:00
Olivier Gayot 85420754cd mirror: avoid assertion error if the source changes at the wrong time
When the source model changes, the mirror model gets notified and
replaces the "apt configurer" instance with a new one. If this happens
in the middle of a "deploy apt configuration + run apt-get update"
operation, this leads to an assertion error.

Fixed by making sure we do the entire operation on the same "apt
configurer" instance.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit 938f2ffc24)
2023-04-12 17:52:04 +02:00
Dan Bungert 4e66da5bbc snapcraft: rev probert for diagnostics
(cherry picked from commit fd1fc1c1d3)
2023-04-12 09:31:40 -06:00
Dan Bungert 0fd2b9c642 errorreport: get udi log
(cherry picked from commit 8a71b7d1ed)
2023-04-12 09:31:38 -06:00
Dan Bungert a92bdd4f2b install: make unattended-upgrades non-fatal
Log the result if failed, ensure that this is also journal-visible.

(cherry picked from commit 800a1e9372)
2023-04-12 09:31:35 -06:00
Dan Bungert 5f7598b3c8 runner: show stdout/stderr on exception
(cherry picked from commit e71a9f98cd)
2023-04-12 09:31:35 -06:00
Dan Bungert 28d127ae20 utils: stdout/stderr logger
(cherry picked from commit 7c5a7f4289)
2023-04-12 09:31:35 -06:00
Dan Bungert dd113c14ab scripts: update slimy to handle u-d-i
If the target snap is ubuntu-desktop-installer, put subiquity changes in
the appropriate location.

(cherry picked from commit b72a5397fb)
2023-04-12 09:31:33 -06:00
Dan Bungert 30e2f184a4
Merge pull request #1645 from dbungert/lunar-pr-1639
network: don't run netplan apply if no autoinstall and nm is enabled
2023-04-12 09:11:39 -06:00
Dan Bungert 79eb2af97f
Merge pull request #1644 from dbungert/lunar-2023-04-11
Lunar 2023 04 11
2023-04-12 07:39:53 -06:00
Olivier Gayot 54d7bb1bf6 network: don't run netplan apply if no autoinstall and nm is enabled
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
Co-authored-by: Dan Bungert <daniel.bungert@canonical.com>
(cherry picked from commit 9d32ded2f9)
2023-04-11 16:56:55 -06:00
Dan Bungert 8297c6a65e test/api: regression test for LP: #2015521
(cherry picked from commit fa81e0981e)
2023-04-11 16:50:28 -06:00
Dan Bungert 8795ce819a gaps: add unittests for gap before part on msdos
(cherry picked from commit dc810414d8)
2023-04-11 16:50:28 -06:00
Dan Bungert 7ae5b3aae1 gaps: fix gap when before a partition in extended
We need the ebr space if we're doing logical partitions, but not for the
primaries.

(cherry picked from commit 5219ecf50c)
2023-04-11 16:50:28 -06:00
Dan Bungert 8caa2cf9c2 locale: switch to localectl set-locale
Move away from localedef to fully set the locale.

(cherry picked from commit fb2c6da2e6)
2023-04-11 16:50:28 -06:00
Dan Bungert d1b1dce0df
Merge pull request #1638 from dbungert/lunar-2023-04-05
Lunar 2023 04 05
2023-04-05 07:50:24 -06:00
Olivier Gayot 5cc01432f3 keyboard: fix use of multi-layout keyboard layout
The XKB configuration guide shows that multi-layout configuration is
possible, as in the following example:

    Option "XkbLayout" "us,cz,de"
    Option "XkbVariant" ",bksl,"

which translates to:
 * layout "us" with no variant
 * layout "cz" with variant "bksl"
 * layout "de" with no variant

The same applies to /etc/default/keyboard.

We do make use of this ability to automatically set an alternative
keyboard layout for non latin keyboard layouts (e.g., Arabic "ara"
becomes "us,ara" so that users can toggle between American and Arabic
keyboard layouts.

In the following commit:

  13cae2488 keyboard: validate layout and variant

.. we started implementing early validation of the keyboard layout, but
we did not consider that multi-layouts are supported.

As a result, subiquity fails at keyboard selection when a non latin
keyboard layout is selected.

Fixed by making sure the validation expects possible multi-layout
setups.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
(cherry picked from commit a91dbfa004)
2023-04-05 07:32:07 -06:00
Dan Bungert 13c2fdff1a storage: restricted probes must gather fs info
In ubuntu-desktop-installer/issues/1772, a user formatted an existing
partition as ext4, which triggered the ESP to be "mounted".  Except this
mount step also formatted the ESP.

Previously, when we ran block probing, the restricted=False version
failed, which caused it to run a restricted=True probe.  The
restricted=True probe looks at block devices, but does not gather
filesystem information.  The esp "mount" check is also willing to format
the partition if it is unformatted.  (We do not have a distinction
between a partition that is unformatted and one for which we have not
obtained filesystem information.)

This user had block probing restricted=False fail due to LP: #2012722,
where Ventoy was in use and we handled the device mapper entry poorly.
While recreating the failure case, simply retesting with a build with
the fix for LP: #2012722 is enough to avoid this problem.  This is
because the restricted=False probe passes, which means filesystem info
is gathered, which means the mount step doesn't attempt to format it.

There will inevitably be other block probing failure cases for
restricted=False.  restricted=True must not leave people so vulnerable
to a partition reformat.

Gather filesystem information during a restricted=True probe.

(cherry picked from commit c921192db3)
2023-04-05 07:31:57 -06:00
Dan Bungert d8f30ed36a
Merge pull request #1635 from dbungert/lunar-2023-04-04
Lunar 2023 04 04
2023-04-04 09:03:47 -06:00
Dan Bungert ef99122a9c storage/v2: remove potential_boot_disks
(cherry picked from commit 965a49f28f)
2023-04-04 08:22:55 -06:00
Dan Bungert 7421bf7114 storage/v2: mark can_be_boot_device on disks
(cherry picked from commit 3a0e9ed425)
2023-04-04 08:22:55 -06:00
Dan Bungert 8533f57e9d
Merge pull request #1633 from dbungert/lunar-sizing-policy
Lunar sizing policy
2023-04-04 08:10:26 -06:00
Dan Bungert d49e37edd0 utils: matching_dicts handle key=None, typing
Co-authored-by: Olivier Gayot <olivier.gayot@canonical.com>
2023-04-03 15:22:30 -06:00
Dan Bungert 6c2e52a028 storage: autoinstall sizing-policy
Co-authored-by: Ryan Mounce <ryan@mounce.com.au>
2023-04-03 15:22:30 -06:00
Dan Bungert 9eeb5669f2 storage: implement unscaled LV sizing for guided 2023-04-03 15:22:30 -06:00
Dan Bungert dcb8fe7920 sizing: move lv scaling logic 2023-04-03 15:22:30 -06:00
Dan Bungert e0e856475c util: add matching_dicts
Move some of this logic to a common util, so it can be used in
unittests.  The curtin version is close but expects to work on a storage
config, which is not a flat list.
2023-04-03 15:22:30 -06:00