2019-06-01 08:08:42 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0-only */
|
2015-12-03 21:01:51 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright (C) 2012 Red Hat, Inc.
|
|
|
|
|
* Copyright (C) 2015 Google, Inc.
|
|
|
|
|
*
|
|
|
|
|
* Author: Mikulas Patocka <mpatocka@redhat.com>
|
|
|
|
|
*
|
|
|
|
|
* Based on Chromium dm-verity driver (C) 2011 The Chromium OS Authors
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef DM_VERITY_H
|
|
|
|
|
#define DM_VERITY_H
|
|
|
|
|
|
2024-02-19 20:28:09 +00:00
|
|
|
#include <linux/dm-io.h>
|
2018-03-15 20:02:31 +00:00
|
|
|
#include <linux/dm-bufio.h>
|
2015-12-03 21:01:51 +00:00
|
|
|
#include <linux/device-mapper.h>
|
2022-07-22 09:38:23 +00:00
|
|
|
#include <linux/interrupt.h>
|
2015-12-03 21:01:51 +00:00
|
|
|
#include <crypto/hash.h>
|
|
|
|
|
|
|
|
|
|
#define DM_VERITY_MAX_LEVELS 63
|
|
|
|
|
|
|
|
|
|
enum verity_mode {
|
|
|
|
|
DM_VERITY_MODE_EIO,
|
|
|
|
|
DM_VERITY_MODE_LOGGING,
|
2020-06-18 06:56:50 +00:00
|
|
|
DM_VERITY_MODE_RESTART,
|
|
|
|
|
DM_VERITY_MODE_PANIC
|
2015-12-03 21:01:51 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
enum verity_block_type {
|
|
|
|
|
DM_VERITY_BLOCK_TYPE_DATA,
|
|
|
|
|
DM_VERITY_BLOCK_TYPE_METADATA
|
|
|
|
|
};
|
|
|
|
|
|
2015-12-03 14:26:30 +00:00
|
|
|
struct dm_verity_fec;
|
|
|
|
|
|
2015-12-03 21:01:51 +00:00
|
|
|
struct dm_verity {
|
|
|
|
|
struct dm_dev *data_dev;
|
|
|
|
|
struct dm_dev *hash_dev;
|
|
|
|
|
struct dm_target *ti;
|
|
|
|
|
struct dm_bufio_client *bufio;
|
|
|
|
|
char *alg_name;
|
dm-verity: hash blocks with shash import+finup when possible
Currently dm-verity computes the hash of each block by using multiple
calls to the "ahash" crypto API. While the exact sequence depends on
the chosen dm-verity settings, in the vast majority of cases it is:
1. crypto_ahash_init()
2. crypto_ahash_update() [salt]
3. crypto_ahash_update() [data]
4. crypto_ahash_final()
This is inefficient for two main reasons:
- It makes multiple indirect calls, which is expensive on modern CPUs
especially when mitigations for CPU vulnerabilities are enabled.
Since the salt is the same across all blocks on a given dm-verity
device, a much more efficient sequence would be to do an import of the
pre-salted state, then a finup.
- It uses the ahash (asynchronous hash) API, despite the fact that
CPU-based hashing is almost always used in practice, and therefore it
experiences the overhead of the ahash-based wrapper for shash.
Because dm-verity was intentionally converted to ahash to support
off-CPU crypto accelerators, a full reversion to shash might not be
acceptable. Yet, we should still provide a fast path for shash with
the most common dm-verity settings.
Another reason for shash over ahash is that the upcoming multibuffer
hashing support, which is specific to CPU-based hashing, is much
better suited for shash than for ahash. Supporting it via ahash would
add significant complexity and overhead. And it's not possible for
the "same" code to properly support both multibuffer hashing and HW
accelerators at the same time anyway, given the different computation
models. Unfortunately there will always be code specific to each
model needed (for users who want to support both).
Therefore, this patch adds a new shash import+finup based fast path to
dm-verity. It is used automatically when appropriate. This makes
dm-verity optimized for what the vast majority of users want: CPU-based
hashing with the most common settings, while still retaining support for
rarer settings and off-CPU crypto accelerators.
In benchmarks with veritysetup's default parameters (SHA-256, 4K data
and hash block sizes, 32-byte salt), which also match the parameters
that Android currently uses, this patch improves block hashing
performance by about 15% on x86_64 using the SHA-NI instructions, or by
about 5% on arm64 using the ARMv8 SHA2 instructions. On x86_64 roughly
two-thirds of the improvement comes from the use of import and finup,
while the remaining third comes from the switch from ahash to shash.
Note that another benefit of using "import" to handle the salt is that
if the salt size is equal to the input size of the hash algorithm's
compression function, e.g. 64 bytes for SHA-256, then the performance is
exactly the same as no salt. This doesn't seem to be much better than
veritysetup's current default of 32-byte salts, due to the way SHA-256's
finalization padding works, but it should be marginally better.
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
2024-07-02 14:41:08 +00:00
|
|
|
struct crypto_ahash *ahash_tfm; /* either this or shash_tfm is set */
|
|
|
|
|
struct crypto_shash *shash_tfm; /* either this or ahash_tfm is set */
|
2015-12-03 21:01:51 +00:00
|
|
|
u8 *root_digest; /* digest of the root block */
|
|
|
|
|
u8 *salt; /* salt: its size is salt_size */
|
dm-verity: hash blocks with shash import+finup when possible
Currently dm-verity computes the hash of each block by using multiple
calls to the "ahash" crypto API. While the exact sequence depends on
the chosen dm-verity settings, in the vast majority of cases it is:
1. crypto_ahash_init()
2. crypto_ahash_update() [salt]
3. crypto_ahash_update() [data]
4. crypto_ahash_final()
This is inefficient for two main reasons:
- It makes multiple indirect calls, which is expensive on modern CPUs
especially when mitigations for CPU vulnerabilities are enabled.
Since the salt is the same across all blocks on a given dm-verity
device, a much more efficient sequence would be to do an import of the
pre-salted state, then a finup.
- It uses the ahash (asynchronous hash) API, despite the fact that
CPU-based hashing is almost always used in practice, and therefore it
experiences the overhead of the ahash-based wrapper for shash.
Because dm-verity was intentionally converted to ahash to support
off-CPU crypto accelerators, a full reversion to shash might not be
acceptable. Yet, we should still provide a fast path for shash with
the most common dm-verity settings.
Another reason for shash over ahash is that the upcoming multibuffer
hashing support, which is specific to CPU-based hashing, is much
better suited for shash than for ahash. Supporting it via ahash would
add significant complexity and overhead. And it's not possible for
the "same" code to properly support both multibuffer hashing and HW
accelerators at the same time anyway, given the different computation
models. Unfortunately there will always be code specific to each
model needed (for users who want to support both).
Therefore, this patch adds a new shash import+finup based fast path to
dm-verity. It is used automatically when appropriate. This makes
dm-verity optimized for what the vast majority of users want: CPU-based
hashing with the most common settings, while still retaining support for
rarer settings and off-CPU crypto accelerators.
In benchmarks with veritysetup's default parameters (SHA-256, 4K data
and hash block sizes, 32-byte salt), which also match the parameters
that Android currently uses, this patch improves block hashing
performance by about 15% on x86_64 using the SHA-NI instructions, or by
about 5% on arm64 using the ARMv8 SHA2 instructions. On x86_64 roughly
two-thirds of the improvement comes from the use of import and finup,
while the remaining third comes from the switch from ahash to shash.
Note that another benefit of using "import" to handle the salt is that
if the salt size is equal to the input size of the hash algorithm's
compression function, e.g. 64 bytes for SHA-256, then the performance is
exactly the same as no salt. This doesn't seem to be much better than
veritysetup's current default of 32-byte salts, due to the way SHA-256's
finalization padding works, but it should be marginally better.
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
2024-07-02 14:41:08 +00:00
|
|
|
u8 *initial_hashstate; /* salted initial state, if shash_tfm is set */
|
2015-12-03 14:26:31 +00:00
|
|
|
u8 *zero_digest; /* digest for a zero block */
|
dm-verity: expose root hash digest and signature data to LSMs
dm-verity provides a strong guarantee of a block device's integrity. As
a generic way to check the integrity of a block device, it provides
those integrity guarantees to its higher layers, including the filesystem
level.
However, critical security metadata like the dm-verity roothash and its
signing information are not easily accessible to the LSMs.
To address this limitation, this patch introduces a mechanism to store
and manage these essential security details within a newly added LSM blob
in the block_device structure.
This addition allows LSMs to make access control decisions on the integrity
data stored within the block_device, enabling more flexible security
policies. For instance, LSMs can now revoke access to dm-verity devices
based on their roothashes, ensuring that only authorized and verified
content is accessible. Additionally, LSMs can enforce policies to only
allow files from dm-verity devices that have a valid digital signature to
execute, effectively blocking any unsigned files from execution, thus
enhancing security against unauthorized modifications.
The patch includes new hook calls, `security_bdev_setintegrity()`, in
dm-verity to expose the dm-verity roothash and the roothash signature to
LSMs via preresume() callback. By using the preresume() callback, it
ensures that the security metadata is consistently in sync with the
metadata of the dm-verity target in the current active mapping table.
The hook calls are depended on CONFIG_SECURITY.
Signed-off-by: Deven Bowers <deven.desai@linux.microsoft.com>
Signed-off-by: Fan Wu <wufan@linux.microsoft.com>
Reviewed-by: Mikulas Patocka <mpatocka@redhat.com>
[PM: moved sig_size field as discussed]
Signed-off-by: Paul Moore <paul@paul-moore.com>
2024-08-03 06:08:26 +00:00
|
|
|
#ifdef CONFIG_SECURITY
|
|
|
|
|
u8 *root_digest_sig; /* signature of the root digest */
|
|
|
|
|
unsigned int sig_size; /* root digest signature size */
|
|
|
|
|
#endif /* CONFIG_SECURITY */
|
2023-01-25 20:14:58 +00:00
|
|
|
unsigned int salt_size;
|
2015-12-03 21:01:51 +00:00
|
|
|
sector_t data_start; /* data offset in 512-byte sectors */
|
|
|
|
|
sector_t hash_start; /* hash start in blocks */
|
|
|
|
|
sector_t data_blocks; /* the number of data blocks */
|
|
|
|
|
sector_t hash_blocks; /* the number of hash blocks */
|
|
|
|
|
unsigned char data_dev_block_bits; /* log2(data blocksize) */
|
|
|
|
|
unsigned char hash_dev_block_bits; /* log2(hash blocksize) */
|
|
|
|
|
unsigned char hash_per_block_bits; /* log2(hashes in hash block) */
|
|
|
|
|
unsigned char levels; /* the number of tree levels */
|
|
|
|
|
unsigned char version;
|
2022-07-22 09:38:23 +00:00
|
|
|
bool hash_failed:1; /* set if hash of any block failed */
|
2024-01-30 09:11:55 +00:00
|
|
|
bool use_bh_wq:1; /* try to verify in BH wq before normal work-queue */
|
2023-01-25 20:14:58 +00:00
|
|
|
unsigned int digest_size; /* digest size for the current hash algorithm */
|
dm-verity: hash blocks with shash import+finup when possible
Currently dm-verity computes the hash of each block by using multiple
calls to the "ahash" crypto API. While the exact sequence depends on
the chosen dm-verity settings, in the vast majority of cases it is:
1. crypto_ahash_init()
2. crypto_ahash_update() [salt]
3. crypto_ahash_update() [data]
4. crypto_ahash_final()
This is inefficient for two main reasons:
- It makes multiple indirect calls, which is expensive on modern CPUs
especially when mitigations for CPU vulnerabilities are enabled.
Since the salt is the same across all blocks on a given dm-verity
device, a much more efficient sequence would be to do an import of the
pre-salted state, then a finup.
- It uses the ahash (asynchronous hash) API, despite the fact that
CPU-based hashing is almost always used in practice, and therefore it
experiences the overhead of the ahash-based wrapper for shash.
Because dm-verity was intentionally converted to ahash to support
off-CPU crypto accelerators, a full reversion to shash might not be
acceptable. Yet, we should still provide a fast path for shash with
the most common dm-verity settings.
Another reason for shash over ahash is that the upcoming multibuffer
hashing support, which is specific to CPU-based hashing, is much
better suited for shash than for ahash. Supporting it via ahash would
add significant complexity and overhead. And it's not possible for
the "same" code to properly support both multibuffer hashing and HW
accelerators at the same time anyway, given the different computation
models. Unfortunately there will always be code specific to each
model needed (for users who want to support both).
Therefore, this patch adds a new shash import+finup based fast path to
dm-verity. It is used automatically when appropriate. This makes
dm-verity optimized for what the vast majority of users want: CPU-based
hashing with the most common settings, while still retaining support for
rarer settings and off-CPU crypto accelerators.
In benchmarks with veritysetup's default parameters (SHA-256, 4K data
and hash block sizes, 32-byte salt), which also match the parameters
that Android currently uses, this patch improves block hashing
performance by about 15% on x86_64 using the SHA-NI instructions, or by
about 5% on arm64 using the ARMv8 SHA2 instructions. On x86_64 roughly
two-thirds of the improvement comes from the use of import and finup,
while the remaining third comes from the switch from ahash to shash.
Note that another benefit of using "import" to handle the salt is that
if the salt size is equal to the input size of the hash algorithm's
compression function, e.g. 64 bytes for SHA-256, then the performance is
exactly the same as no salt. This doesn't seem to be much better than
veritysetup's current default of 32-byte salts, due to the way SHA-256's
finalization padding works, but it should be marginally better.
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
2024-07-02 14:41:08 +00:00
|
|
|
unsigned int hash_reqsize; /* the size of temporary space for crypto */
|
2015-12-03 21:01:51 +00:00
|
|
|
enum verity_mode mode; /* mode for handling verification errors */
|
2024-10-02 14:03:41 +00:00
|
|
|
enum verity_mode error_mode;/* mode for handling I/O errors */
|
2023-01-25 20:14:58 +00:00
|
|
|
unsigned int corrupted_errs;/* Number of errors for corrupted blocks */
|
2015-12-03 21:01:51 +00:00
|
|
|
|
|
|
|
|
struct workqueue_struct *verify_wq;
|
|
|
|
|
|
|
|
|
|
/* starting blocks for each tree level. 0 is the lowest level. */
|
|
|
|
|
sector_t hash_level_block[DM_VERITY_MAX_LEVELS];
|
2015-12-03 14:26:30 +00:00
|
|
|
|
|
|
|
|
struct dm_verity_fec *fec; /* forward error correction */
|
2018-03-23 01:18:04 +00:00
|
|
|
unsigned long *validated_blocks; /* bitset blocks validated */
|
2019-07-18 00:46:15 +00:00
|
|
|
|
|
|
|
|
char *signature_key_desc; /* signature keyring reference */
|
2024-02-19 20:28:09 +00:00
|
|
|
|
|
|
|
|
struct dm_io_client *io;
|
|
|
|
|
mempool_t recheck_pool;
|
2015-12-03 21:01:51 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct dm_verity_io {
|
|
|
|
|
struct dm_verity *v;
|
|
|
|
|
|
|
|
|
|
/* original value of bio->bi_end_io */
|
|
|
|
|
bio_end_io_t *orig_bi_end_io;
|
|
|
|
|
|
2024-02-20 18:11:51 +00:00
|
|
|
struct bvec_iter iter;
|
|
|
|
|
|
2015-12-03 21:01:51 +00:00
|
|
|
sector_t block;
|
2023-01-25 20:14:58 +00:00
|
|
|
unsigned int n_blocks;
|
2024-01-30 09:11:55 +00:00
|
|
|
bool in_bh;
|
2024-10-29 11:17:13 +00:00
|
|
|
bool had_mismatch;
|
2015-12-03 21:01:51 +00:00
|
|
|
|
|
|
|
|
struct work_struct work;
|
2024-01-30 09:11:55 +00:00
|
|
|
struct work_struct bh_work;
|
2015-12-03 21:01:51 +00:00
|
|
|
|
2024-07-02 14:39:13 +00:00
|
|
|
u8 real_digest[HASH_MAX_DIGESTSIZE];
|
|
|
|
|
u8 want_digest[HASH_MAX_DIGESTSIZE];
|
|
|
|
|
|
2015-12-03 21:01:51 +00:00
|
|
|
/*
|
dm-verity: hash blocks with shash import+finup when possible
Currently dm-verity computes the hash of each block by using multiple
calls to the "ahash" crypto API. While the exact sequence depends on
the chosen dm-verity settings, in the vast majority of cases it is:
1. crypto_ahash_init()
2. crypto_ahash_update() [salt]
3. crypto_ahash_update() [data]
4. crypto_ahash_final()
This is inefficient for two main reasons:
- It makes multiple indirect calls, which is expensive on modern CPUs
especially when mitigations for CPU vulnerabilities are enabled.
Since the salt is the same across all blocks on a given dm-verity
device, a much more efficient sequence would be to do an import of the
pre-salted state, then a finup.
- It uses the ahash (asynchronous hash) API, despite the fact that
CPU-based hashing is almost always used in practice, and therefore it
experiences the overhead of the ahash-based wrapper for shash.
Because dm-verity was intentionally converted to ahash to support
off-CPU crypto accelerators, a full reversion to shash might not be
acceptable. Yet, we should still provide a fast path for shash with
the most common dm-verity settings.
Another reason for shash over ahash is that the upcoming multibuffer
hashing support, which is specific to CPU-based hashing, is much
better suited for shash than for ahash. Supporting it via ahash would
add significant complexity and overhead. And it's not possible for
the "same" code to properly support both multibuffer hashing and HW
accelerators at the same time anyway, given the different computation
models. Unfortunately there will always be code specific to each
model needed (for users who want to support both).
Therefore, this patch adds a new shash import+finup based fast path to
dm-verity. It is used automatically when appropriate. This makes
dm-verity optimized for what the vast majority of users want: CPU-based
hashing with the most common settings, while still retaining support for
rarer settings and off-CPU crypto accelerators.
In benchmarks with veritysetup's default parameters (SHA-256, 4K data
and hash block sizes, 32-byte salt), which also match the parameters
that Android currently uses, this patch improves block hashing
performance by about 15% on x86_64 using the SHA-NI instructions, or by
about 5% on arm64 using the ARMv8 SHA2 instructions. On x86_64 roughly
two-thirds of the improvement comes from the use of import and finup,
while the remaining third comes from the switch from ahash to shash.
Note that another benefit of using "import" to handle the salt is that
if the salt size is equal to the input size of the hash algorithm's
compression function, e.g. 64 bytes for SHA-256, then the performance is
exactly the same as no salt. This doesn't seem to be much better than
veritysetup's current default of 32-byte salts, due to the way SHA-256's
finalization padding works, but it should be marginally better.
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
2024-07-02 14:41:08 +00:00
|
|
|
* This struct is followed by a variable-sized hash request of size
|
|
|
|
|
* v->hash_reqsize, either a struct ahash_request or a struct shash_desc
|
|
|
|
|
* (depending on whether ahash_tfm or shash_tfm is being used). To
|
|
|
|
|
* access it, use verity_io_hash_req().
|
2015-12-03 21:01:51 +00:00
|
|
|
*/
|
|
|
|
|
};
|
|
|
|
|
|
dm-verity: hash blocks with shash import+finup when possible
Currently dm-verity computes the hash of each block by using multiple
calls to the "ahash" crypto API. While the exact sequence depends on
the chosen dm-verity settings, in the vast majority of cases it is:
1. crypto_ahash_init()
2. crypto_ahash_update() [salt]
3. crypto_ahash_update() [data]
4. crypto_ahash_final()
This is inefficient for two main reasons:
- It makes multiple indirect calls, which is expensive on modern CPUs
especially when mitigations for CPU vulnerabilities are enabled.
Since the salt is the same across all blocks on a given dm-verity
device, a much more efficient sequence would be to do an import of the
pre-salted state, then a finup.
- It uses the ahash (asynchronous hash) API, despite the fact that
CPU-based hashing is almost always used in practice, and therefore it
experiences the overhead of the ahash-based wrapper for shash.
Because dm-verity was intentionally converted to ahash to support
off-CPU crypto accelerators, a full reversion to shash might not be
acceptable. Yet, we should still provide a fast path for shash with
the most common dm-verity settings.
Another reason for shash over ahash is that the upcoming multibuffer
hashing support, which is specific to CPU-based hashing, is much
better suited for shash than for ahash. Supporting it via ahash would
add significant complexity and overhead. And it's not possible for
the "same" code to properly support both multibuffer hashing and HW
accelerators at the same time anyway, given the different computation
models. Unfortunately there will always be code specific to each
model needed (for users who want to support both).
Therefore, this patch adds a new shash import+finup based fast path to
dm-verity. It is used automatically when appropriate. This makes
dm-verity optimized for what the vast majority of users want: CPU-based
hashing with the most common settings, while still retaining support for
rarer settings and off-CPU crypto accelerators.
In benchmarks with veritysetup's default parameters (SHA-256, 4K data
and hash block sizes, 32-byte salt), which also match the parameters
that Android currently uses, this patch improves block hashing
performance by about 15% on x86_64 using the SHA-NI instructions, or by
about 5% on arm64 using the ARMv8 SHA2 instructions. On x86_64 roughly
two-thirds of the improvement comes from the use of import and finup,
while the remaining third comes from the switch from ahash to shash.
Note that another benefit of using "import" to handle the salt is that
if the salt size is equal to the input size of the hash algorithm's
compression function, e.g. 64 bytes for SHA-256, then the performance is
exactly the same as no salt. This doesn't seem to be much better than
veritysetup's current default of 32-byte salts, due to the way SHA-256's
finalization padding works, but it should be marginally better.
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
2024-07-02 14:41:08 +00:00
|
|
|
static inline void *verity_io_hash_req(struct dm_verity *v,
|
|
|
|
|
struct dm_verity_io *io)
|
2015-12-03 21:01:51 +00:00
|
|
|
{
|
dm-verity: hash blocks with shash import+finup when possible
Currently dm-verity computes the hash of each block by using multiple
calls to the "ahash" crypto API. While the exact sequence depends on
the chosen dm-verity settings, in the vast majority of cases it is:
1. crypto_ahash_init()
2. crypto_ahash_update() [salt]
3. crypto_ahash_update() [data]
4. crypto_ahash_final()
This is inefficient for two main reasons:
- It makes multiple indirect calls, which is expensive on modern CPUs
especially when mitigations for CPU vulnerabilities are enabled.
Since the salt is the same across all blocks on a given dm-verity
device, a much more efficient sequence would be to do an import of the
pre-salted state, then a finup.
- It uses the ahash (asynchronous hash) API, despite the fact that
CPU-based hashing is almost always used in practice, and therefore it
experiences the overhead of the ahash-based wrapper for shash.
Because dm-verity was intentionally converted to ahash to support
off-CPU crypto accelerators, a full reversion to shash might not be
acceptable. Yet, we should still provide a fast path for shash with
the most common dm-verity settings.
Another reason for shash over ahash is that the upcoming multibuffer
hashing support, which is specific to CPU-based hashing, is much
better suited for shash than for ahash. Supporting it via ahash would
add significant complexity and overhead. And it's not possible for
the "same" code to properly support both multibuffer hashing and HW
accelerators at the same time anyway, given the different computation
models. Unfortunately there will always be code specific to each
model needed (for users who want to support both).
Therefore, this patch adds a new shash import+finup based fast path to
dm-verity. It is used automatically when appropriate. This makes
dm-verity optimized for what the vast majority of users want: CPU-based
hashing with the most common settings, while still retaining support for
rarer settings and off-CPU crypto accelerators.
In benchmarks with veritysetup's default parameters (SHA-256, 4K data
and hash block sizes, 32-byte salt), which also match the parameters
that Android currently uses, this patch improves block hashing
performance by about 15% on x86_64 using the SHA-NI instructions, or by
about 5% on arm64 using the ARMv8 SHA2 instructions. On x86_64 roughly
two-thirds of the improvement comes from the use of import and finup,
while the remaining third comes from the switch from ahash to shash.
Note that another benefit of using "import" to handle the salt is that
if the salt size is equal to the input size of the hash algorithm's
compression function, e.g. 64 bytes for SHA-256, then the performance is
exactly the same as no salt. This doesn't seem to be much better than
veritysetup's current default of 32-byte salts, due to the way SHA-256's
finalization padding works, but it should be marginally better.
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
2024-07-02 14:41:08 +00:00
|
|
|
return io + 1;
|
2015-12-03 21:01:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline u8 *verity_io_real_digest(struct dm_verity *v,
|
|
|
|
|
struct dm_verity_io *io)
|
|
|
|
|
{
|
2024-07-02 14:39:13 +00:00
|
|
|
return io->real_digest;
|
2015-12-03 21:01:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline u8 *verity_io_want_digest(struct dm_verity *v,
|
|
|
|
|
struct dm_verity_io *io)
|
|
|
|
|
{
|
2024-07-02 14:39:13 +00:00
|
|
|
return io->want_digest;
|
2015-12-03 21:01:51 +00:00
|
|
|
}
|
|
|
|
|
|
2024-07-02 14:40:41 +00:00
|
|
|
extern int verity_hash(struct dm_verity *v, struct dm_verity_io *io,
|
dm-verity: don't use blocking calls from tasklets
The commit 5721d4e5a9cd enhanced dm-verity, so that it can verify blocks
from tasklets rather than from workqueues. This reportedly improves
performance significantly.
However, dm-verity was using the flag CRYPTO_TFM_REQ_MAY_SLEEP from
tasklets which resulted in warnings about sleeping function being called
from non-sleeping context.
BUG: sleeping function called from invalid context at crypto/internal.h:206
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 14, name: ksoftirqd/0
preempt_count: 100, expected: 0
RCU nest depth: 0, expected: 0
CPU: 0 PID: 14 Comm: ksoftirqd/0 Tainted: G W 6.7.0-rc1 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x32/0x50
__might_resched+0x110/0x160
crypto_hash_walk_done+0x54/0xb0
shash_ahash_update+0x51/0x60
verity_hash_update.isra.0+0x4a/0x130 [dm_verity]
verity_verify_io+0x165/0x550 [dm_verity]
? free_unref_page+0xdf/0x170
? psi_group_change+0x113/0x390
verity_tasklet+0xd/0x70 [dm_verity]
tasklet_action_common.isra.0+0xb3/0xc0
__do_softirq+0xaf/0x1ec
? smpboot_thread_fn+0x1d/0x200
? sort_range+0x20/0x20
run_ksoftirqd+0x15/0x30
smpboot_thread_fn+0xed/0x200
kthread+0xdc/0x110
? kthread_complete_and_exit+0x20/0x20
ret_from_fork+0x28/0x40
? kthread_complete_and_exit+0x20/0x20
ret_from_fork_asm+0x11/0x20
</TASK>
This commit fixes dm-verity so that it doesn't use the flags
CRYPTO_TFM_REQ_MAY_SLEEP and CRYPTO_TFM_REQ_MAY_BACKLOG from tasklets. The
crypto API would do GFP_ATOMIC allocation instead, it could return -ENOMEM
and we catch -ENOMEM in verity_tasklet and requeue the request to the
workqueue.
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Cc: stable@vger.kernel.org # v6.0+
Fixes: 5721d4e5a9cd ("dm verity: Add optional "try_verify_in_tasklet" feature")
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
2023-11-17 17:37:25 +00:00
|
|
|
const u8 *data, size_t len, u8 *digest, bool may_sleep);
|
2015-12-03 21:01:51 +00:00
|
|
|
|
|
|
|
|
extern int verity_hash_for_block(struct dm_verity *v, struct dm_verity_io *io,
|
2015-12-03 14:26:31 +00:00
|
|
|
sector_t block, u8 *digest, bool *is_zero);
|
2015-12-03 21:01:51 +00:00
|
|
|
|
2022-06-27 15:35:24 +00:00
|
|
|
extern bool dm_is_verity_target(struct dm_target *ti);
|
2022-09-07 20:30:58 +00:00
|
|
|
extern int dm_verity_get_mode(struct dm_target *ti);
|
2022-06-27 15:35:24 +00:00
|
|
|
extern int dm_verity_get_root_digest(struct dm_target *ti, u8 **root_digest,
|
|
|
|
|
unsigned int *digest_size);
|
|
|
|
|
|
2015-12-03 21:01:51 +00:00
|
|
|
#endif /* DM_VERITY_H */
|
