Commit Graph

4529 Commits

Author SHA1 Message Date
Dan Bungert fae24f3247 logging: log files 0640 root:adm 2022-02-14 11:14:10 -07:00
Dan Bungert 9b5513f311 file_util: add open_perms and generate_config
Create open_perms context manager for custom or multiple writes.
Create generate_config for a small removal of redundancy.
0640 root:adm the resulting files.
2022-02-14 11:14:10 -07:00
Dan Bungert 7762dfaec0
Merge pull request #1178 from ogayot/FR-1953
Expose endpoint to get list of UA activable services
2022-02-14 09:45:37 -07:00
Dan Bungert 08e41ad745
Merge pull request #1181 from dbungert/lp-1960162
Lp 1960162
2022-02-11 16:30:05 -07:00
Dan Bungert 78fe7ccf7f ubuntu advantage: drop client token logging 2022-02-11 16:19:19 -07:00
Dan Bungert 824f6bbaee endpoints: add body logging
Add body logging for POST methods that accept a Payload for data.
Use repr=False on UbuntuAdvantageForm to not log the token.
2022-02-11 16:19:19 -07:00
Dan Bungert 07b8b50399 api: drop body from context reporting
Some API methods have sensitive information in the body of the call.
Stop logging the body unconditionally.
2022-02-11 16:19:19 -07:00
Dan Bungert f316ea5e06 storage: change guided method to Payload 2022-02-11 16:19:19 -07:00
Dan Bungert 9bd3ea6a6c filesystem: passphrase in answers, failing test 2022-02-11 16:19:19 -07:00
Dan Bungert 7fba591bcf
Merge pull request #1183 from ogayot/geoip-aiohttp
Implement geoip using asyncio
2022-02-11 16:18:55 -07:00
Dan Bungert a6268011f5 test: fix installdeps 2022-02-11 16:08:34 -07:00
Dan Bungert b5f0d1fa73 geoip: fix lints 2022-02-11 16:03:35 -07:00
Olivier Gayot ebdf1db636 Move to aiohttp for GeoIP requests
GeoIP requests used to run on the default executor thread and would
prevent the application from exiting if the GeoIP service would not
respond quickly enough. We witnessed an obvious impact during an
incident on geoip.ubuntu.com.

Move to aiohttp so that the HTTP calls are non blocking.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-02-11 22:10:27 +01:00
Olivier Gayot 58df7d6583 Add dry-run mode for GeoIP info retrieval
We used to make real HTTP calls to geoip.ubuntu.com in dry-run mode.
This might have an impact on the service when running automatic testing.

We now provide an hardcoded value for the GeoIP information in dry-run
mode.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-02-11 22:10:27 +01:00
Olivier Gayot 8f024eb038 Only keep the name and description of activable services
The u-a-c returns a set of information about each service. After
filtering out the services that we don't want using the fields
"entitled" and "available", we now only keep the name and the
description of the service.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-02-09 14:43:22 +01:00
Olivier Gayot 98b887d696 Move UA module from common/ to server/
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-02-09 14:43:22 +01:00
Olivier Gayot 455e98aab9 Make sure to initialize _check_token_task in the UA controller
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-02-09 14:43:22 +01:00
Olivier Gayot c171aac2c7 Expose endpoint to get list of UA activable services
The list of activable UA services was only retrieved from the
client-side of Subiquity (using ubuntu-advantage-tools). Therefore, the
desktop installer would need to reimplement the same logic should they
need access to the list of services ; which is inconvenient.

We now expose a new endpoint in the API that takes the token as
a parameter and returns a status (+ a list of services if the token is
valid and not expired).

  $ curl \
    --unix-socket .subiquity/socket \
    --header 'Content-Type: application/json' \
    http://a/ubuntu_advantage/check_token \
    -d '"C123456"'

The token parameter is expected to be in the body of the request -
rather than in the query string - to avoid ending up in the access logs.

This new endpoint is a read-only GET endpoint. It is not designed as a
replacement for the POST to /a/ubuntu_advantage that the client must
(still) do to pass the token to the model.

We now use this new endpoint internally in Subiquity so that the
retrieval of the activable services is done on the server-side only.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-02-09 14:43:22 +01:00
Michael Hudson-Doyle aa659e8911
Merge pull request #1180 from dbungert/workflow-remove-hirsute
workflow: -hirsute
2022-02-04 10:45:22 +13:00
Dan Bungert c260df2894 workflow: -hirsute 2022-02-03 12:21:04 -07:00
Dan Bungert ffc6efb968
Merge pull request #1177 from dbungert/serializer-list-union
serializer: fix and test for KeyError '$type'
2022-02-02 09:32:50 -07:00
Dan Bungert 37cbfb0224 serializer: fix and test for KeyError '$type' 2022-01-28 16:52:16 -07:00
Dan Bungert d158d4e701
Merge pull request #1174 from ogayot/ua-entitled-services-only
Don't list UA services that are not entitled
2022-01-28 11:13:09 -07:00
Olivier Gayot 7dfe722686 Don't list UA services that are not entitled
Instead of only checking if a given UA service is available, we now also
check if it is entitled.

 - the available field for a service refers to its availability on the
   current machine (e.g. on Focal running on a amd64 CPU) ; whereas
 - the entitled field tells us if the contract covers the service.

Therefore, we need to make sure that we only list the services that are
both "available" and "entitled".

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-28 18:58:39 +01:00
Dan Bungert 3803ae0e80
Merge pull request #1172 from ogayot/validate-output-autoinstall-data
Validate output autoinstall data in integration tests
2022-01-26 10:44:46 -07:00
Dan Bungert 59199548aa
Merge pull request #1175 from dbungert/apitest-parallel
api test: run in parallel
2022-01-26 10:44:24 -07:00
Dan Bungert 72fbbc2868 api test: parallelize
The auto setting will choose based on the number of CPUs, including
virtual cores.  For me this is a 7x speedup.
2022-01-26 10:39:23 -07:00
Dan Bungert 0f9c9eef3f options: add output_base for dryrun use
To enable parallel API testing, add an output_base argument to replace
the default '.subiquity' that is sprinkled everywhere.
2022-01-26 10:39:23 -07:00
Dan Bungert 90a2bd6f7a
Merge pull request #1173 from dbungert/os-prober-arch
snap: fix os-prober related non-amd64 build failures
2022-01-25 07:47:09 -07:00
Dan Bungert ee20eb335a
Merge pull request #1167 from dbungert/lp-1952603-nonet
snaplist: handle cancel error paths
2022-01-25 07:46:46 -07:00
Dan Bungert ee528fe6a7 controller/snaplist: protect GET against timeout 2022-01-24 17:05:30 -07:00
Dan Bungert 853c39bae7 ui/snaplist: fix initialization 2022-01-24 17:03:31 -07:00
Dan Bungert a6901ddc25 kvm-test: allow deadnet simulation with --nets -1 2022-01-24 17:03:00 -07:00
Dan Bungert 4927affbe8 snapcraft: better file selection for os-prober sed 2022-01-24 13:18:49 -07:00
Dan Bungert 54ca91038e snapcraft: show os-prober as available app 2022-01-24 12:37:48 -07:00
Dan Bungert 8318fc4617 Revert "os-prober: patch instead of sed"
This reverts commit 7bdeb6707c.
2022-01-24 12:18:41 -07:00
Dan Bungert 229e364956
Merge pull request #1171 from ogayot/bump-curtin-version
Bump curtin version to fix APT preferences being discarded
2022-01-21 17:17:16 -07:00
Didier Roche 34b621ee66
Merge pull request #1165 from CarlosNihelton/fix-wsl-snap-env
[SystemSetup] Fix L-S-C behavior under WSL snap environment
2022-01-21 15:56:42 +01:00
Olivier Gayot e2216c1d53 Run script to validate autoinstall-user-data in integration tests
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-21 13:48:43 +01:00
Olivier Gayot 2bbd4fffe6 Update example states for Subiquity to cover valid token values
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-21 13:48:43 +01:00
Olivier Gayot 7664258aa4 Make sure SSHController always yields valid autoinstall data
When running an autoinstallation with no ssh: field (which is valid),
the output autoinstall-user-data file would end up with None for the
key ssh:authorized-keys, as shown below:

  ssh:
    authorized-keys: null

Unfortunately, null is not a valid value according to the schema (which
expects a string).

Fixed by initializing the authorized_keys variable to [] instead of None
so that it yields an empty list instead of null when no autoinstall data
is loaded.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-21 13:48:43 +01:00
Olivier Gayot 85958b9404 Don't include UA token in autoinstall if empty
If no UA token is provided, the UbuntuAdvantageController will generate
autoinstall data that contains an empty string:

  ubuntu-advantage:
    token: ""

Unfortunately, this is not valid according to the JSON schema.

Fixed by not returning a token: key if the token is an empty string.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-21 13:48:43 +01:00
Olivier Gayot 535a92dad4 Add script to validate autoinstall data against JSON schema
The script can be used to validate autoinstall user data against the
schema. By default, it expects a #cloud-config header and the user-data
to be under the autoinstall: key.

By passing the --no-expect-cloudconfig, it validates the data directly.
We can use this option to validate the YAML files under
examples/autoinstall-*.yaml

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-21 13:48:43 +01:00
Olivier Gayot 1fce021585 Bump curtin version to fix APT preferences being discarded
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-21 10:07:16 +01:00
Michael Hudson-Doyle 56ddf178e4
Merge pull request #1170 from ogayot/FR-1983
Fix hanging when network is the last postinst model configured
2022-01-21 10:25:54 +13:00
Dan Bungert f0907971ed
Merge pull request #1145 from dbungert/os-prober-snap
snapcraft: add os-prober
2022-01-20 12:34:09 -07:00
Dan Bungert 583659270c
Merge pull request #1168 from ogayot/use-fuse
Add --use-fuse switch to kvm-test --install so it can run as non-root
2022-01-20 10:21:59 -07:00
Dan Bungert c9f389d1ca
Merge pull request #1169 from mwhudson/configure_once-oops
initialize MirrorController.configure_once to False
2022-01-20 10:18:26 -07:00
Olivier Gayot 81bc59019c Remove --use-fuse switch and make it the default
Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-20 12:01:44 +01:00
Olivier Gayot 37f286ac93 Fix hanging when network is the last postinst model configured
When running an autoinstallation with only “network” as an interactive
section, the installation hanged after the following steps:

  finalizing installation
    running 'curtin hook'
      curtin command hook
  executing late commands

where in normal conditions, the next step is:

  final system configuration

Also, querying the meta/status endpoint at this point shows that the app
is in POST_WAIT state.

The problem is:

the network model is declared as both an “install” model and a
“postinstall” model (it is the only one we have AFAIK)
when calling .configured() for a given model, we only trigger /at max/
one event. Either:
 * install_event (if the model is an “install” model - and it is the
   last one configured)
 * postinstall_event (if the model is a “postinstall” model but not an
   “install” model - and it is the last one configured)
 * no event (as in most cases)

So when we call .configured() for the network model, we /can/ trigger
the install_event but can never trigger the postinstall event.
Therefore Subiquity, will wait forever until something triggers the
postinstall_event

Fixed by accepting to trigger the install_event and the
postinstall_event in a single call to .configured.

Signed-off-by: Olivier Gayot <olivier.gayot@canonical.com>
2022-01-20 11:35:11 +01:00