Discussion:
[PATCH 0/4] Hyper-V: Enable Linux guests on Hyper-V on ARM64
k***@linuxonhyperv.com
2018-11-22 02:50:51 UTC
Permalink
From: "K. Y. Srinivasan" <***@microsoft.com>

This series enables Linux guests running on Hyper-V on ARM64
hardware. New ARM64-specific code in arch/arm64/hyperv initializes
Hyper-V, including its synthetic clocks and hypercall mechanism.
Existing architecture independent drivers for Hyper-V's VMbus and
synthetic devices just work when built for ARM64. Hyper-V code is
built and included in the image and modules only if CONFIG_HYPERV
is enabled.

The four patches are organized as follows:
1) Add include files that define the Hyper-V interface as
described in the Hyper-V Top Level Functional Spec (TLFS), plus
additional definitions specific to Linux running on Hyper-V.

2) Add core Hyper-V support on ARM64, including hypercalls,
synthetic clock initialization, and interrupt handlers.

3) Update the existing VMbus driver to generalize interrupt
management across x86/x64 and ARM64.

4) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Some areas of Linux guests on Hyper-V on ARM64 are a work-
in-progress, primarily due to work still being done in Hyper-V:

* Hyper-V on ARM64 currently runs with a 4 Kbyte page size, and only
supports guests with a 4 Kbyte page size. Because Hyper-V uses
shared pages to communicate between the guest and the hypervisor,
there are open design decisions on the page size to use when
the guest is using 16K/64K pages. Once those issues are
resolved and Hyper-V fully supports 16K/64K guest pages, changes
may be needed in the Linux drivers for Hyper-V synthetic devices.

* Hyper-V on ARM64 does not currently support mapping PCI devices
into the guest address space. The Hyper-V PCI driver at
drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is
not being built for ARM64.

In a few cases, terminology from the x86/x64 world has been carried
over into the ARM64 code ("MSR", "TSC"). Hyper-V still uses the
x86/x64 terminology and has not replaced it with something more
generic, so the code uses the Hyper-V terminology. This will be
fixed when Hyper-V updates the usage in the TLFS.


Michael Kelley (4):
arm64: hyperv: Add core Hyper-V include files
arm64: hyperv: Add support for Hyper-V as a hypervisor
Drivers: hv: vmbus: Add hooks for per-CPU IRQ
Drivers: hv: Enable CONFIG_HYPERV on ARM64

MAINTAINERS | 4 +
arch/arm64/Makefile | 1 +
arch/arm64/hyperv/Makefile | 2 +
arch/arm64/hyperv/hv_hvc.S | 54 ++++
arch/arm64/hyperv/hv_init.c | 441 +++++++++++++++++++++++++++
arch/arm64/hyperv/mshyperv.c | 178 +++++++++++
arch/arm64/include/asm/hyperv-tlfs.h | 338 ++++++++++++++++++++
arch/arm64/include/asm/mshyperv.h | 116 +++++++
arch/x86/include/asm/mshyperv.h | 4 +
drivers/hv/Kconfig | 3 +-
drivers/hv/hv.c | 2 +
include/asm-generic/mshyperv.h | 240 +++++++++++++++
12 files changed, 1382 insertions(+), 1 deletion(-)
create mode 100644 arch/arm64/hyperv/Makefile
create mode 100644 arch/arm64/hyperv/hv_hvc.S
create mode 100644 arch/arm64/hyperv/hv_init.c
create mode 100644 arch/arm64/hyperv/mshyperv.c
create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
create mode 100644 arch/arm64/include/asm/mshyperv.h
create mode 100644 include/asm-generic/mshyperv.h
--
2.19.1
k***@linuxonhyperv.com
2018-11-22 03:09:14 UTC
Permalink
From: "K. Y. Srinivasan" <***@microsoft.com>

This series enables Linux guests running on Hyper-V on ARM64
hardware. New ARM64-specific code in arch/arm64/hyperv initializes
Hyper-V, including its synthetic clocks and hypercall mechanism.
Existing architecture independent drivers for Hyper-V's VMbus and
synthetic devices just work when built for ARM64. Hyper-V code is
built and included in the image and modules only if CONFIG_HYPERV
is enabled.

The four patches are organized as follows:
1) Add include files that define the Hyper-V interface as
described in the Hyper-V Top Level Functional Spec (TLFS), plus
additional definitions specific to Linux running on Hyper-V.

2) Add core Hyper-V support on ARM64, including hypercalls,
synthetic clock initialization, and interrupt handlers.

3) Update the existing VMbus driver to generalize interrupt
management across x86/x64 and ARM64.

4) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Some areas of Linux guests on Hyper-V on ARM64 are a work-
in-progress, primarily due to work still being done in Hyper-V:

* Hyper-V on ARM64 currently runs with a 4 Kbyte page size, and only
supports guests with a 4 Kbyte page size. Because Hyper-V uses
shared pages to communicate between the guest and the hypervisor,
there are open design decisions on the page size to use when
the guest is using 16K/64K pages. Once those issues are
resolved and Hyper-V fully supports 16K/64K guest pages, changes
may be needed in the Linux drivers for Hyper-V synthetic devices.

* Hyper-V on ARM64 does not currently support mapping PCI devices
into the guest address space. The Hyper-V PCI driver at
drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is
not being built for ARM64.

In a few cases, terminology from the x86/x64 world has been carried
over into the ARM64 code ("MSR", "TSC"). Hyper-V still uses the
x86/x64 terminology and has not replaced it with something more
generic, so the code uses the Hyper-V terminology. This will be
fixed when Hyper-V updates the usage in the TLFS.


Michael Kelley (4):
arm64: hyperv: Add core Hyper-V include files
arm64: hyperv: Add support for Hyper-V as a hypervisor
Drivers: hv: vmbus: Add hooks for per-CPU IRQ
Drivers: hv: Enable CONFIG_HYPERV on ARM64

MAINTAINERS | 4 +
arch/arm64/Makefile | 1 +
arch/arm64/hyperv/Makefile | 2 +
arch/arm64/hyperv/hv_hvc.S | 54 ++++
arch/arm64/hyperv/hv_init.c | 441 +++++++++++++++++++++++++++
arch/arm64/hyperv/mshyperv.c | 178 +++++++++++
arch/arm64/include/asm/hyperv-tlfs.h | 338 ++++++++++++++++++++
arch/arm64/include/asm/mshyperv.h | 116 +++++++
arch/x86/include/asm/mshyperv.h | 4 +
drivers/hv/Kconfig | 3 +-
drivers/hv/hv.c | 2 +
include/asm-generic/mshyperv.h | 240 +++++++++++++++
12 files changed, 1382 insertions(+), 1 deletion(-)
create mode 100644 arch/arm64/hyperv/Makefile
create mode 100644 arch/arm64/hyperv/hv_hvc.S
create mode 100644 arch/arm64/hyperv/hv_init.c
create mode 100644 arch/arm64/hyperv/mshyperv.c
create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
create mode 100644 arch/arm64/include/asm/mshyperv.h
create mode 100644 include/asm-generic/mshyperv.h
--
2.19.1
k***@linuxonhyperv.com
2018-11-22 03:10:59 UTC
Permalink
From: Michael Kelley <***@microsoft.com>

Update drivers/hv/Kconfig so CONFIG_HYPERV can be selected on ARM64,
causing the Hyper-V specific code to be built.

Signed-off-by: Michael Kelley <***@microsoft.com>
Signed-off-by: K. Y. Srinivasan <***@microsoft.com>
---
drivers/hv/Kconfig | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
index 97954f575c3f..c3e11a2f9c70 100644
--- a/drivers/hv/Kconfig
+++ b/drivers/hv/Kconfig
@@ -4,7 +4,8 @@ menu "Microsoft Hyper-V guest support"

config HYPERV
tristate "Microsoft Hyper-V client drivers"
- depends on X86 && ACPI && PCI && X86_LOCAL_APIC && HYPERVISOR_GUEST
+ depends on ACPI && PCI && \
+ ((X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) || ARM64)
select PARAVIRT
help
Select this option to run Linux as a Hyper-V client operating
--
2.19.1
k***@linuxonhyperv.com
2018-11-22 03:10:58 UTC
Permalink
From: Michael Kelley <***@microsoft.com>

Add hooks to enable/disable a per-CPU IRQ for VMbus. These hooks
are in the architecture independent setup and shutdown paths for
Hyper-V, and are needed by Linux guests on Hyper-V on ARM64. The
x86/x64 implementation is null because VMbus interrupts on x86/x64
don't use an IRQ.

Signed-off-by: Michael Kelley <***@microsoft.com>
Signed-off-by: K. Y. Srinivasan <***@microsoft.com>
---
arch/x86/include/asm/mshyperv.h | 4 ++++
drivers/hv/hv.c | 2 ++
2 files changed, 6 insertions(+)

diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 0d6271cce198..8d97bd3a13a6 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
void hv_setup_vmbus_irq(void (*handler)(void));
void hv_remove_vmbus_irq(void);

+/* On x86/x64, there isn't a real IRQ to be enabled/disable */
+static inline void hv_enable_vmbus_irq(void) {}
+static inline void hv_disable_vmbus_irq(void) {}
+
void hv_setup_kexec_handler(void (*handler)(void));
void hv_remove_kexec_handler(void);
void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
index 166c2501de17..d0bb09a4bd73 100644
--- a/drivers/hv/hv.c
+++ b/drivers/hv/hv.c
@@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
hv_set_siefp(siefp.as_uint64);

/* Setup the shared SINT. */
+ hv_enable_vmbus_irq();
hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);

shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
@@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
/* Disable the global synic bit */
sctrl.enable = 0;
hv_set_synic_state(sctrl.as_uint64);
+ hv_disable_vmbus_irq();

return 0;
}
--
2.19.1
Greg KH
2018-11-26 19:21:01 UTC
Permalink
Post by k***@linuxonhyperv.com
Add hooks to enable/disable a per-CPU IRQ for VMbus. These hooks
are in the architecture independent setup and shutdown paths for
Hyper-V, and are needed by Linux guests on Hyper-V on ARM64. The
x86/x64 implementation is null because VMbus interrupts on x86/x64
don't use an IRQ.
---
arch/x86/include/asm/mshyperv.h | 4 ++++
drivers/hv/hv.c | 2 ++
2 files changed, 6 insertions(+)
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 0d6271cce198..8d97bd3a13a6 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
void hv_setup_vmbus_irq(void (*handler)(void));
void hv_remove_vmbus_irq(void);
+/* On x86/x64, there isn't a real IRQ to be enabled/disable */
+static inline void hv_enable_vmbus_irq(void) {}
+static inline void hv_disable_vmbus_irq(void) {}
+
void hv_setup_kexec_handler(void (*handler)(void));
void hv_remove_kexec_handler(void);
void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
index 166c2501de17..d0bb09a4bd73 100644
--- a/drivers/hv/hv.c
+++ b/drivers/hv/hv.c
@@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
hv_set_siefp(siefp.as_uint64);
/* Setup the shared SINT. */
+ hv_enable_vmbus_irq();
hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
@@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
/* Disable the global synic bit */
sctrl.enable = 0;
hv_set_synic_state(sctrl.as_uint64);
+ hv_disable_vmbus_irq();
return 0;
}
--
2.19.1
You created "null" hooks that do nothing, for no one in this patch
series, why?

Add them when they are needed not now. If I saw this code in the tree,
I would just go delete it as it is because it is not used at all.

thanks,

greg k-h
Greg KH
2018-11-26 19:57:27 UTC
Permalink
Post by Greg KH
Post by k***@linuxonhyperv.com
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 0d6271cce198..8d97bd3a13a6 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -109,6 +109,10 @@ void hyperv_vector_handler(struct pt_regs *regs);
void hv_setup_vmbus_irq(void (*handler)(void));
void hv_remove_vmbus_irq(void);
+/* On x86/x64, there isn't a real IRQ to be enabled/disable */
+static inline void hv_enable_vmbus_irq(void) {}
+static inline void hv_disable_vmbus_irq(void) {}
+
void hv_setup_kexec_handler(void (*handler)(void));
void hv_remove_kexec_handler(void);
void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
index 166c2501de17..d0bb09a4bd73 100644
--- a/drivers/hv/hv.c
+++ b/drivers/hv/hv.c
@@ -307,6 +307,7 @@ int hv_synic_init(unsigned int cpu)
hv_set_siefp(siefp.as_uint64);
/* Setup the shared SINT. */
+ hv_enable_vmbus_irq();
hv_get_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
@@ -434,6 +435,7 @@ int hv_synic_cleanup(unsigned int cpu)
/* Disable the global synic bit */
sctrl.enable = 0;
hv_set_synic_state(sctrl.as_uint64);
+ hv_disable_vmbus_irq();
return 0;
}
--
2.19.1
You created "null" hooks that do nothing, for no one in this patch
series, why?
hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
implementations in the ARM64 code in patch 2 of this series. The
implementations are in the new file arch/arm64/hyperv/mshyperv.c.
Or am I misunderstanding your point?
So you use a hook in an earlier patch and then add it in a later one?

Shouldn't you do it the other way around? As it is, the earlier patch
should not work properly, right?

thanks,

greg k-h
Michael Kelley
2018-11-26 20:56:50 UTC
Permalink
Post by Greg KH
Post by Greg KH
You created "null" hooks that do nothing, for no one in this patch
series, why?
hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
implementations in the ARM64 code in patch 2 of this series. The
implementations are in the new file arch/arm64/hyperv/mshyperv.c.
Or am I misunderstanding your point?
So you use a hook in an earlier patch and then add it in a later one?
Shouldn't you do it the other way around? As it is, the earlier patch
should not work properly, right?
The earlier patch implements the hook on the ARM64 side but it is
unused -- it's not called. The later patch then calls it. Wouldn't the
other way around be backwards?

The general approach is for patches 1 and 2 of the series to provide
all the new code under arch/arm64 to enable Hyper-V. But the code
won't get called (or even built) with just these two patches because
CONFIG_HYPERV can't be selected. Patch 3 is separate because it
applies to architecture independent code and arch/x86 code -- I thought
there might be value in keeping the ARM64 and x86 patches distinct.
Patch 4 applies to architecture independent code, and enables the
ARM64 code in patches 1 and 2 to be compiled and run when
CONFIG_HYPERV is selected.

If combining some of the patches in the series is a better approach, I'm
good with that.

Michael
Greg KH
2018-11-27 06:20:56 UTC
Permalink
Post by Michael Kelley
Post by Greg KH
Post by Greg KH
You created "null" hooks that do nothing, for no one in this patch
series, why?
hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
implementations in the ARM64 code in patch 2 of this series. The
implementations are in the new file arch/arm64/hyperv/mshyperv.c.
Or am I misunderstanding your point?
So you use a hook in an earlier patch and then add it in a later one?
Shouldn't you do it the other way around? As it is, the earlier patch
should not work properly, right?
The earlier patch implements the hook on the ARM64 side but it is
unused -- it's not called. The later patch then calls it. Wouldn't the
other way around be backwards?
Ah, it wasn't obvious that the previous patch added it at all, why not
just make that addition part of this patch?
Post by Michael Kelley
The general approach is for patches 1 and 2 of the series to provide
all the new code under arch/arm64 to enable Hyper-V. But the code
won't get called (or even built) with just these two patches because
CONFIG_HYPERV can't be selected. Patch 3 is separate because it
applies to architecture independent code and arch/x86 code -- I thought
there might be value in keeping the ARM64 and x86 patches distinct.
Patch 4 applies to architecture independent code, and enables the
ARM64 code in patches 1 and 2 to be compiled and run when
CONFIG_HYPERV is selected.
If combining some of the patches in the series is a better approach, I'm
good with that.
Ok, that makes more sense, if it is easier to get the ARM people to
review this, that's fine. Doesn't seem like anyone did that yet :(

sorry for the noise,

greg k-h
Will Deacon
2018-11-27 10:19:05 UTC
Permalink
Post by Greg KH
Post by Michael Kelley
Post by Greg KH
Post by Greg KH
You created "null" hooks that do nothing, for no one in this patch
series, why?
hv_enable_vmbus_irq() and hv_disable_vmbus_irq() have non-null
implementations in the ARM64 code in patch 2 of this series. The
implementations are in the new file arch/arm64/hyperv/mshyperv.c.
Or am I misunderstanding your point?
So you use a hook in an earlier patch and then add it in a later one?
Shouldn't you do it the other way around? As it is, the earlier patch
should not work properly, right?
The earlier patch implements the hook on the ARM64 side but it is
unused -- it's not called. The later patch then calls it. Wouldn't the
other way around be backwards?
Ah, it wasn't obvious that the previous patch added it at all, why not
just make that addition part of this patch?
Post by Michael Kelley
The general approach is for patches 1 and 2 of the series to provide
all the new code under arch/arm64 to enable Hyper-V. But the code
won't get called (or even built) with just these two patches because
CONFIG_HYPERV can't be selected. Patch 3 is separate because it
applies to architecture independent code and arch/x86 code -- I thought
there might be value in keeping the ARM64 and x86 patches distinct.
Patch 4 applies to architecture independent code, and enables the
ARM64 code in patches 1 and 2 to be compiled and run when
CONFIG_HYPERV is selected.
If combining some of the patches in the series is a better approach, I'm
good with that.
Ok, that makes more sense, if it is easier to get the ARM people to
review this, that's fine. Doesn't seem like anyone did that yet :(
It's on the list, but thanks for having a look as well!

Will
Michael Kelley
2018-12-03 01:47:39 UTC
Permalink
Post by Will Deacon
Post by Greg KH
Post by Michael Kelley
The general approach is for patches 1 and 2 of the series to provide
all the new code under arch/arm64 to enable Hyper-V. But the code
won't get called (or even built) with just these two patches because
CONFIG_HYPERV can't be selected. Patch 3 is separate because it
applies to architecture independent code and arch/x86 code -- I thought
there might be value in keeping the ARM64 and x86 patches distinct.
Patch 4 applies to architecture independent code, and enables the
ARM64 code in patches 1 and 2 to be compiled and run when
CONFIG_HYPERV is selected.
If combining some of the patches in the series is a better approach, I'm
good with that.
Ok, that makes more sense, if it is easier to get the ARM people to
review this, that's fine. Doesn't seem like anyone did that yet :(
It's on the list, but thanks for having a look as well!
Will
Will -- I'll hold off on sending a new version, pending comments from
the ARM64 maintainers. Let me know if you prefer that I do otherwise.

Michael
k***@linuxonhyperv.com
2018-11-22 03:10:57 UTC
Permalink
This post might be inappropriate. Click to display it.
Greg KH
2018-11-26 19:19:46 UTC
Permalink
Post by k***@linuxonhyperv.com
--- /dev/null
+++ b/arch/arm64/hyperv/hv_hvc.S
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Microsoft Hyper-V hypervisor invocation routines
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT. See the GNU General Public License for more
+ * details.
No need for these two paragraphs at all. If you add them now, someone
will just come along later and rip them out. Please don't include them
in the first place.

Not to mention the first paragraph doesn't make sense, think about
"which" version 2 the FSF published (hint, there was 5 or 7 or so...)
That's why we have one in the kernel tree for everyone to rely on.

thanks,

greg k-h
Will Deacon
2018-12-07 13:42:40 UTC
Permalink
Post by k***@linuxonhyperv.com
* Detecting Hyper-V and initializing the guest/Hyper-V interface
* Setting up Hyper-V's synthetic clocks
* Making hypercalls using the HVC instruction
* Setting up VMbus and stimer0 interrupts
* Setting up kexec and crash handlers
This code is architecture dependent code and is mostly driven by
architecture independent code in the VMbus driver in drivers/hv/hv.c
and drivers/hv/vmbus_drv.c.
This code is built only when CONFIG_HYPERV is enabled.
---
MAINTAINERS | 1 +
arch/arm64/Makefile | 1 +
arch/arm64/hyperv/Makefile | 2 +
arch/arm64/hyperv/hv_hvc.S | 54 +++++
arch/arm64/hyperv/hv_init.c | 441 +++++++++++++++++++++++++++++++++++
arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
6 files changed, 677 insertions(+)
create mode 100644 arch/arm64/hyperv/Makefile
create mode 100644 arch/arm64/hyperv/hv_hvc.S
create mode 100644 arch/arm64/hyperv/hv_init.c
create mode 100644 arch/arm64/hyperv/mshyperv.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 72f19cef4c48..326eeb32a0cd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6837,6 +6837,7 @@ F: arch/x86/kernel/cpu/mshyperv.c
F: arch/x86/hyperv
F: arch/arm64/include/asm/hyperv-tlfs.h
F: arch/arm64/include/asm/mshyperv.h
+F: arch/arm64/hyperv
F: drivers/hid/hid-hyperv.c
F: drivers/hv/
F: drivers/input/serio/hyperv-keyboard.c
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 6cb9fc7e9382..ad9ec0579553 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -106,6 +106,7 @@ core-y += arch/arm64/kernel/ arch/arm64/mm/
core-$(CONFIG_NET) += arch/arm64/net/
core-$(CONFIG_KVM) += arch/arm64/kvm/
core-$(CONFIG_XEN) += arch/arm64/xen/
+core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
libs-y := arch/arm64/lib/ $(libs-y)
core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
new file mode 100644
index 000000000000..988eda55330c
--- /dev/null
+++ b/arch/arm64/hyperv/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-y := hv_init.o hv_hvc.o mshyperv.o
diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
new file mode 100644
index 000000000000..82636969b4f2
--- /dev/null
+++ b/arch/arm64/hyperv/hv_hvc.S
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Microsoft Hyper-V hypervisor invocation routines
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT. See the GNU General Public License for more
+ * details.
+ */
+
+#include <linux/linkage.h>
+
+ .text
+/*
+ * Do the HVC instruction. For Hyper-V the argument is always 1.
+ * x0 contains the hypercall control value, while additional registers
+ * vary depending on the hypercall, and whether the hypercall arguments
+ * are in memory or in registers (a "fast" hypercall per the Hyper-V
+ * TLFS). When the arguments are in memory x1 is the guest physical
+ * address of the input arguments, and x2 is the guest physical
+ * address of the output arguments. When the arguments are in
+ * registers, the register values depends on the hypercall. Note
+ * that this version cannot return any values in registers.
+ */
+ENTRY(hv_do_hvc)
+ hvc #1
+ ret
+ENDPROC(hv_do_hvc)
+
+/*
+ * This variant of HVC invocation is for hv_get_vpreg and
+ * hv_get_vpreg_128. The input parameters are passed in registers
+ * along with a pointer in x4 to where the output result should
+ * be stored. The output is returned in x15 and x16. x18 is used as
+ * scratch space to avoid buildng a stack frame, as Hyper-V does
+ * not preserve registers x0-x17.
+ */
+ENTRY(hv_do_hvc_fast_get)
+ mov x18, x4
+ hvc #1
+ str x15,[x18]
+ str x16,[x18,#8]
+ ret
+ENDPROC(hv_do_hvc_fast_get)
Why are you not following the ARM SMCCC? It would be /much/ better if you
could just follow the standard, which is already implemented by
include/linux/smccc.h.

Will
Marc Zyngier
2018-12-07 14:43:02 UTC
Permalink
Post by k***@linuxonhyperv.com
* Detecting Hyper-V and initializing the guest/Hyper-V interface
* Setting up Hyper-V's synthetic clocks
* Making hypercalls using the HVC instruction
* Setting up VMbus and stimer0 interrupts
* Setting up kexec and crash handlers
This commit message is a clear indication that this should be split in
at least 5 different patches.
Post by k***@linuxonhyperv.com
This code is architecture dependent code and is mostly driven by
architecture independent code in the VMbus driver in drivers/hv/hv.c
and drivers/hv/vmbus_drv.c.
This code is built only when CONFIG_HYPERV is enabled.
---
MAINTAINERS | 1 +
arch/arm64/Makefile | 1 +
arch/arm64/hyperv/Makefile | 2 +
arch/arm64/hyperv/hv_hvc.S | 54 +++++
arch/arm64/hyperv/hv_init.c | 441 +++++++++++++++++++++++++++++++++++
arch/arm64/hyperv/mshyperv.c | 178 ++++++++++++++
6 files changed, 677 insertions(+)
create mode 100644 arch/arm64/hyperv/Makefile
create mode 100644 arch/arm64/hyperv/hv_hvc.S
create mode 100644 arch/arm64/hyperv/hv_init.c
create mode 100644 arch/arm64/hyperv/mshyperv.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 72f19cef4c48..326eeb32a0cd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6837,6 +6837,7 @@ F: arch/x86/kernel/cpu/mshyperv.c
F: arch/x86/hyperv
F: arch/arm64/include/asm/hyperv-tlfs.h
F: arch/arm64/include/asm/mshyperv.h
+F: arch/arm64/hyperv
F: drivers/hid/hid-hyperv.c
F: drivers/hv/
F: drivers/input/serio/hyperv-keyboard.c
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 6cb9fc7e9382..ad9ec0579553 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -106,6 +106,7 @@ core-y += arch/arm64/kernel/ arch/arm64/mm/
core-$(CONFIG_NET) += arch/arm64/net/
core-$(CONFIG_KVM) += arch/arm64/kvm/
core-$(CONFIG_XEN) += arch/arm64/xen/
+core-$(CONFIG_HYPERV) += arch/arm64/hyperv/
core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
libs-y := arch/arm64/lib/ $(libs-y)
core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
diff --git a/arch/arm64/hyperv/Makefile b/arch/arm64/hyperv/Makefile
new file mode 100644
index 000000000000..988eda55330c
--- /dev/null
+++ b/arch/arm64/hyperv/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-y := hv_init.o hv_hvc.o mshyperv.o
diff --git a/arch/arm64/hyperv/hv_hvc.S b/arch/arm64/hyperv/hv_hvc.S
new file mode 100644
index 000000000000..82636969b4f2
--- /dev/null
+++ b/arch/arm64/hyperv/hv_hvc.S
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Microsoft Hyper-V hypervisor invocation routines
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT. See the GNU General Public License for more
+ * details.
+ */
+
+#include <linux/linkage.h>
+
+ .text
+/*
+ * Do the HVC instruction. For Hyper-V the argument is always 1.
+ * x0 contains the hypercall control value, while additional registers
+ * vary depending on the hypercall, and whether the hypercall arguments
+ * are in memory or in registers (a "fast" hypercall per the Hyper-V
+ * TLFS). When the arguments are in memory x1 is the guest physical
+ * address of the input arguments, and x2 is the guest physical
+ * address of the output arguments. When the arguments are in
+ * registers, the register values depends on the hypercall. Note
+ * that this version cannot return any values in registers.
+ */
+ENTRY(hv_do_hvc)
+ hvc #1
+ ret
+ENDPROC(hv_do_hvc)
+
+/*
+ * This variant of HVC invocation is for hv_get_vpreg and
+ * hv_get_vpreg_128. The input parameters are passed in registers
+ * along with a pointer in x4 to where the output result should
+ * be stored. The output is returned in x15 and x16. x18 is used as
+ * scratch space to avoid buildng a stack frame, as Hyper-V does
+ * not preserve registers x0-x17.
+ */
+ENTRY(hv_do_hvc_fast_get)
+ mov x18, x4
+ hvc #1
+ str x15,[x18]
+ str x16,[x18,#8]
+ ret
+ENDPROC(hv_do_hvc_fast_get)
As Will said, this isn't a viable option. Please follow SMCCC 1.1.
Post by k***@linuxonhyperv.com
diff --git a/arch/arm64/hyperv/hv_init.c b/arch/arm64/hyperv/hv_init.c
new file mode 100644
index 000000000000..aa1a8c09d989
--- /dev/null
+++ b/arch/arm64/hyperv/hv_init.c
@@ -0,0 +1,441 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Initialization of the interface with Microsoft's Hyper-V hypervisor,
+ * and various low level utility routines for interacting with Hyper-V.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT. See the GNU General Public License for more
+ * details.
+ */
+
+
+#include <linux/types.h>
+#include <linux/version.h>
+#include <linux/export.h>
+#include <linux/vmalloc.h>
+#include <linux/mm.h>
+#include <linux/clocksource.h>
+#include <linux/sched_clock.h>
+#include <linux/acpi.h>
+#include <linux/module.h>
+#include <linux/hyperv.h>
+#include <linux/slab.h>
+#include <linux/cpuhotplug.h>
+#include <linux/psci.h>
+#include <asm-generic/bug.h>
+#include <asm/hypervisor.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/mshyperv.h>
+#include <asm/sysreg.h>
+#include <clocksource/arm_arch_timer.h>
+
+static bool hyperv_initialized;
+struct ms_hyperv_info ms_hyperv;
+EXPORT_SYMBOL_GPL(ms_hyperv);
Who are the users of this structure? Should they go via accessors instead?
Post by k***@linuxonhyperv.com
+
+static struct ms_hyperv_tsc_page *tsc_pg;
+
+struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
+{
+ return tsc_pg;
+}
+EXPORT_SYMBOL_GPL(hv_get_tsc_page);
+
+static u64 read_hv_sched_clock_tsc(void)
+{
+ u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+ if (current_tick == U64_MAX)
+ current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+
+ return current_tick;
+}
+
+static u64 read_hv_clock_tsc(struct clocksource *arg)
+{
+ u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+ if (current_tick == U64_MAX)
+ current_tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+
+ return current_tick;
+}
Can't you define one function in terms of the other?
Post by k***@linuxonhyperv.com
+
+static struct clocksource hyperv_cs_tsc = {
+ .name = "hyperv_clocksource_tsc_page",
+ .rating = 400,
+ .read = read_hv_clock_tsc,
+ .mask = CLOCKSOURCE_MASK(64),
+ .flags = CLOCK_SOURCE_IS_CONTINUOUS,
+};
+
+static u64 read_hv_sched_clock_msr(void)
+{
+ return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+}
+
+static u64 read_hv_clock_msr(struct clocksource *arg)
+{
+ return hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT);
+}
+
+static struct clocksource hyperv_cs_msr = {
+ .name = "hyperv_clocksource_msr",
+ .rating = 400,
+ .read = read_hv_clock_msr,
+ .mask = CLOCKSOURCE_MASK(64),
+ .flags = CLOCK_SOURCE_IS_CONTINUOUS,
+};
+
+struct clocksource *hyperv_cs;
+EXPORT_SYMBOL_GPL(hyperv_cs);
Why? Who needs to poke this?
Post by k***@linuxonhyperv.com
+
+u32 *hv_vp_index;
+EXPORT_SYMBOL_GPL(hv_vp_index);
+
+u32 hv_max_vp_index;
Same thing.
Post by k***@linuxonhyperv.com
+
+static int hv_cpu_init(unsigned int cpu)
+{
+ u64 msr_vp_index;
+
+ hv_get_vp_index(msr_vp_index);
+
+ hv_vp_index[smp_processor_id()] = msr_vp_index;
+
+ if (msr_vp_index > hv_max_vp_index)
+ hv_max_vp_index = msr_vp_index;
+
+ return 0;
+}
Is that some new way to describe a CPU topology? If so, why isn't that
exposed via the ACPI tables that the kernel already parses?
Post by k***@linuxonhyperv.com
+
+/*
+ * This function is invoked via the ACPI clocksource probe mechanism. We
+ * don't actually use any values from the ACPI GTDT table, but we set up
This doesn't feel like a good idea at all. Piggy-backing on an existing
mechanism and use it for something completely different is not exactly
future-proof.

Also, if this is supposed to be a clocksource, why isn't that a
clocksource driver on its own right?
Post by k***@linuxonhyperv.com
+ * the Hyper-V synthetic clocksource and do other initialization for
+ * interacting with Hyper-V the first time. Using early_initcall to invoke
+ * this function is too late because interrupts are already enabled at that
+ * point, and sched_clock_register must run before interrupts are enabled.
+ *
+ * 1. Setup the guest ID.
+ * 2. Get features and hints info from Hyper-V
+ * 3. Setup per-cpu VP indices.
+ * 4. Register Hyper-V specific clocksource.
+ * 5. Register the scheduler clock.
+ */
+
+static int __init hyperv_init(struct acpi_table_header *table)
+{
+ struct hv_get_vp_register_output result;
+ u32 a, b, c, d;
+ u64 guest_id;
+ int i;
+
+ /*
+ * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
+ * have the string "MsHyperV".
+ */
+ if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
+ return 1;
+
+ /* Setup the guest ID */
+ guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
+ hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
+
+ /* Get the features and hints from Hyper-V */
+ hv_get_vpreg_128(HV_REGISTER_PRIVILEGES_AND_FEATURES, &result);
+ ms_hyperv.features = lower_32_bits(result.registervaluelow);
+ ms_hyperv.misc_features = upper_32_bits(result.registervaluehigh);
+
+ hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
+ ms_hyperv.hints = lower_32_bits(result.registervaluelow);
+
+ pr_info("Hyper-V: Features 0x%x, hints 0x%x\n",
+ ms_hyperv.features, ms_hyperv.hints);
+
+ /*
+ * Direct mode is the only option for STIMERs provided Hyper-V
+ * on ARM64, so Hyper-V doesn't actually set the flag. But add the
+ * flag so the architecture independent code in drivers/hv/hv.c
+ * will correctly use that mode.
+ */
+ ms_hyperv.misc_features |= HV_STIMER_DIRECT_MODE_AVAILABLE;
+
+ /*
+ * Hyper-V on ARM64 doesn't support AutoEOI. Add the hint
+ * that tells architecture independent code not to use this
+ * feature.
+ */
+ ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
+
+ /* Get information about the Hyper-V host version */
+ hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
+ a = lower_32_bits(result.registervaluelow);
+ b = upper_32_bits(result.registervaluelow);
+ c = lower_32_bits(result.registervaluehigh);
+ d = upper_32_bits(result.registervaluehigh);
+ pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
+ b >> 16, b & 0xFFFF, a, d & 0xFFFFFF, c, d >> 24);
+
+ /* Allocate percpu VP index */
+ hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index),
+ GFP_KERNEL);
Why isn't this a percpu variable?
Post by k***@linuxonhyperv.com
+ if (!hv_vp_index)
+ return 1;
+
+ for (i = 0; i < num_possible_cpus(); i++)
+ hv_vp_index[i] = VP_INVAL;
+
+ if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
+ hv_cpu_init, NULL) < 0)
+ goto free_vp_index;
+
+ /*
+ * Try to set up what Hyper-V calls the "TSC reference page", which
+ * uses the ARM Generic Timer virtual counter with some scaling
+ * information to provide a fast and stable guest VM clocksource.
+ * If the TSC reference page can't be set up, fall back to reading
+ * the guest clock provided by Hyper-V's synthetic reference time
+ * register.
+ */
+ if (ms_hyperv.features & HV_MSR_REFERENCE_TSC_AVAILABLE) {
+
+ u64 tsc_msr;
+ phys_addr_t phys_addr;
+
+ tsc_pg = __vmalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL, PAGE_KERNEL);
And why not vmalloc?
Post by k***@linuxonhyperv.com
+ if (tsc_pg) {
+ phys_addr = page_to_phys(vmalloc_to_page(tsc_pg));
+ tsc_msr = hv_get_vpreg(HV_REGISTER_REFERENCE_TSC);
+ tsc_msr &= GENMASK_ULL(11, 0);
Magic.
Post by k***@linuxonhyperv.com
+ tsc_msr = tsc_msr | 0x1 | (u64)phys_addr;
+ hv_set_vpreg(HV_REGISTER_REFERENCE_TSC, tsc_msr);
+ hyperv_cs = &hyperv_cs_tsc;
+ sched_clock_register(read_hv_sched_clock_tsc,
+ 64, HV_CLOCK_HZ);
+ }
+ }
+
+ if (!hyperv_cs &&
+ (ms_hyperv.features & HV_MSR_TIME_REF_COUNT_AVAILABLE)) {
+ hyperv_cs = &hyperv_cs_msr;
+ sched_clock_register(read_hv_sched_clock_msr,
+ 64, HV_CLOCK_HZ);
+ }
+
+ if (hyperv_cs) {
+ hyperv_cs->archdata.vdso_direct = false;
+ clocksource_register_hz(hyperv_cs, HV_CLOCK_HZ);
+ }
+
+ hyperv_initialized = true;
+ return 0;
+
+ kfree(hv_vp_index);
+ hv_vp_index = NULL;
+ return 1;
????
Post by k***@linuxonhyperv.com
+}
+TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_init);
+
+/*
+ * This routine is called before kexec/kdump, it does the required cleanup.
+ */
+void hyperv_cleanup(void)
+{
+ /* Reset our OS id */
+ hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
+
+}
+EXPORT_SYMBOL_GPL(hyperv_cleanup);
+
+/*
+ * hv_do_hypercall- Invoke the specified hypercall
+ */
+u64 hv_do_hypercall(u64 control, void *input, void *output)
+{
+ u64 input_address;
+ u64 output_address;
+
+ input_address = input ? virt_to_phys(input) : 0;
+ output_address = output ? virt_to_phys(output) : 0;
+ return hv_do_hvc(control, input_address, output_address);
+}
+EXPORT_SYMBOL_GPL(hv_do_hypercall);
+
+/*
+ * hv_do_fast_hypercall8 -- Invoke the specified hypercall
+ * with arguments in registers instead of physical memory.
+ * Avoids the overhead of virt_to_phys for simple hypercalls.
+ */
+
+u64 hv_do_fast_hypercall8(u16 code, u64 input)
+{
+ u64 control;
+
+ control = (u64)code | HV_HYPERCALL_FAST_BIT;
+ return hv_do_hvc(control, input);
+}
+EXPORT_SYMBOL_GPL(hv_do_fast_hypercall8);
+
+
+/*
+ * Set a single VP register to a 64-bit value.
+ */
+void hv_set_vpreg(u32 msr, u64 value)
+{
+ union hv_hypercall_status status;
+
+ status.as_uint64 = hv_do_hvc(
+ HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+ HV_HYPERCALL_REP_COUNT_1,
+ HV_PARTITION_ID_SELF,
+ HV_VP_INDEX_SELF,
+ msr,
+ 0,
+ value,
+ 0);
+
+ /*
+ * Something is fundamentally broken in the hypervisor if
+ * setting a VP register fails. There's really no way to
+ * continue as a guest VM, so panic.
+ */
+ BUG_ON(status.status != HV_STATUS_SUCCESS);
+}
+EXPORT_SYMBOL_GPL(hv_set_vpreg);
+
+
+/*
+ * Get the value of a single VP register, and only the low order 64 bits.
+ */
+u64 hv_get_vpreg(u32 msr)
+{
+ union hv_hypercall_status status;
+ struct hv_get_vp_register_output output;
+
+ status.as_uint64 = hv_do_hvc_fast_get(
+ HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+ HV_HYPERCALL_REP_COUNT_1,
+ HV_PARTITION_ID_SELF,
+ HV_VP_INDEX_SELF,
+ msr,
+ &output);
+
+ /*
+ * Something is fundamentally broken in the hypervisor if
+ * getting a VP register fails. There's really no way to
+ * continue as a guest VM, so panic.
+ */
+ BUG_ON(status.status != HV_STATUS_SUCCESS);
+
+ return output.registervaluelow;
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg);
+
+/*
+ * Get the value of a single VP register that is 128 bits in size. This is a
+ * separate call in order to avoid complicating the calling sequence for
+ * the much more frequently used 64-bit version.
+ */
+void hv_get_vpreg_128(u32 msr, struct hv_get_vp_register_output *result)
+{
+ union hv_hypercall_status status;
+
+ status.as_uint64 = hv_do_hvc_fast_get(
+ HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
+ HV_HYPERCALL_REP_COUNT_1,
+ HV_PARTITION_ID_SELF,
+ HV_VP_INDEX_SELF,
+ msr,
+ result);
+
+ /*
+ * Something is fundamentally broken in the hypervisor if
+ * getting a VP register fails. There's really no way to
+ * continue as a guest VM, so panic.
+ */
+ BUG_ON(status.status != HV_STATUS_SUCCESS);
+
+ return;
+
+}
+EXPORT_SYMBOL_GPL(hv_get_vpreg_128);
+
+void hyperv_report_panic(struct pt_regs *regs, long err)
+{
+ static bool panic_reported;
+ u64 guest_id;
+
+ /*
+ * We prefer to report panic on 'die' chain as we have proper
+ * registers to report, but if we miss it (e.g. on BUG()) we need
+ * to report it on 'panic'.
+ */
+ if (panic_reported)
+ return;
+ panic_reported = true;
+
+ guest_id = hv_get_vpreg(HV_REGISTER_GUEST_OSID);
+
+ /*
+ * Hyper-V provides the ability to store only 5 values.
+ * Pick the passed in error value, the guest_id, and the PC.
+ * The first two general registers are added arbitrarily.
+ */
+ hv_set_vpreg(HV_REGISTER_CRASH_P0, err);
+ hv_set_vpreg(HV_REGISTER_CRASH_P1, guest_id);
+ hv_set_vpreg(HV_REGISTER_CRASH_P2, regs->pc);
+ hv_set_vpreg(HV_REGISTER_CRASH_P3, regs->regs[0]);
+ hv_set_vpreg(HV_REGISTER_CRASH_P4, regs->regs[1]);
+
+ /*
+ * Let Hyper-V know there is crash data available
+ */
+ hv_set_vpreg(HV_REGISTER_CRASH_CTL, HV_CRASH_CTL_CRASH_NOTIFY);
+}
+EXPORT_SYMBOL_GPL(hyperv_report_panic);
+
+/*
+ * hyperv_report_panic_msg - report panic message to Hyper-V
+ */
+void hyperv_report_panic_msg(phys_addr_t pa, size_t size)
+{
+ /*
+ * P3 to contain the physical address of the panic page & P4 to
+ * contain the size of the panic data in that page. Rest of the
+ * registers are no-op when the NOTIFY_MSG flag is set.
+ */
+ hv_set_vpreg(HV_REGISTER_CRASH_P0, 0);
+ hv_set_vpreg(HV_REGISTER_CRASH_P1, 0);
+ hv_set_vpreg(HV_REGISTER_CRASH_P2, 0);
+ hv_set_vpreg(HV_REGISTER_CRASH_P3, pa);
+ hv_set_vpreg(HV_REGISTER_CRASH_P4, size);
+
+ /*
+ * Let Hyper-V know there is crash data available along with
+ * the panic message.
+ */
+ hv_set_vpreg(HV_REGISTER_CRASH_CTL,
+ (HV_CRASH_CTL_CRASH_NOTIFY | HV_CRASH_CTL_CRASH_NOTIFY_MSG));
+}
+EXPORT_SYMBOL_GPL(hyperv_report_panic_msg);
+
+bool hv_is_hyperv_initialized(void)
+{
+ return hyperv_initialized;
+}
+EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
new file mode 100644
index 000000000000..3ef055599412
--- /dev/null
+++ b/arch/arm64/hyperv/mshyperv.c
@@ -0,0 +1,178 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Core routines for interacting with Microsoft's Hyper-V hypervisor,
+ * including setting up VMbus and STIMER interrupts, and handling
+ * crashes and kexecs. These interactions are through a set of
+ * static "handler" variables set by the architecture independent
+ * VMbus and STIMER drivers. This design is used to meet x86/x64
+ * requirements for avoiding direct linkages and allowing the VMbus
+ * and STIMER drivers to be unloaded and reloaded.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT. See the GNU General Public License for more
+ * details.
+ */
+
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/interrupt.h>
+#include <linux/kexec.h>
+#include <linux/acpi.h>
+#include <linux/ptrace.h>
+#include <asm/hyperv-tlfs.h>
+#include <asm/mshyperv.h>
+
+static void (*vmbus_handler)(void);
+static void (*hv_stimer0_handler)(void);
+static void (*hv_kexec_handler)(void);
+static void (*hv_crash_handler)(struct pt_regs *regs);
+
+static int vmbus_irq;
+static long __percpu *vmbus_evt;
+static long __percpu *stimer0_evt;
+
+irqreturn_t hyperv_vector_handler(int irq, void *dev_id)
+{
+ if (vmbus_handler)
+ vmbus_handler();
In which circumstances can this be NULL?
Post by k***@linuxonhyperv.com
+ return IRQ_HANDLED;
+}
+
+/* Must be done just once */
+void hv_setup_vmbus_irq(void (*handler)(void))
+{
+ int result;
+
+ vmbus_handler = handler;
If thismust only be done once, maybe you could check that it hasn't been
done already?
Post by k***@linuxonhyperv.com
+ vmbus_irq = acpi_register_gsi(NULL, HYPERVISOR_CALLBACK_VECTOR,
+ ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
+ if (vmbus_irq <= 0) {
+ pr_err("Can't register Hyper-V VMBus GSI. Error %d",
+ vmbus_irq);
+ vmbus_irq = 0;
+ return;
+ }
+ vmbus_evt = alloc_percpu(long);
+ result = request_percpu_irq(vmbus_irq, hyperv_vector_handler,
+ "Hyper-V VMbus", vmbus_evt);
If this is a per-cpu interrupt, why isn't it signalled as a PPI, in an
architecture compliant way?
Post by k***@linuxonhyperv.com
+ if (result) {
+ pr_err("Can't request Hyper-V VMBus IRQ %d. Error %d",
+ vmbus_irq, result);
+ free_percpu(vmbus_evt);
+ acpi_unregister_gsi(vmbus_irq);
+ vmbus_irq = 0;
+ }
+}
+EXPORT_SYMBOL_GPL(hv_setup_vmbus_irq);
+
+/* Must be done just once */
+void hv_remove_vmbus_irq(void)
+{
+ if (vmbus_irq) {
+ free_percpu_irq(vmbus_irq, vmbus_evt);
+ free_percpu(vmbus_evt);
+ acpi_unregister_gsi(vmbus_irq);
+ }
+}
+EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);
+
+/* Must be done by each CPU */
+void hv_enable_vmbus_irq(void)
+{
+ enable_percpu_irq(vmbus_irq, 0);
+}
+EXPORT_SYMBOL_GPL(hv_enable_vmbus_irq);
+
+/* Must be done by each CPU */
+void hv_disable_vmbus_irq(void)
+{
+ disable_percpu_irq(vmbus_irq);
+}
+EXPORT_SYMBOL_GPL(hv_disable_vmbus_irq);
+
+/* Routines to do per-architecture handling of STIMER0 when in Direct Mode */
+
+static irqreturn_t hv_stimer0_vector_handler(int irq, void *dev_id)
+{
+ if (hv_stimer0_handler)
+ hv_stimer0_handler();
+ return IRQ_HANDLED;
+}
+
+int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void))
+{
+ int localirq;
+ int result;
+
+ localirq = acpi_register_gsi(NULL, HV_STIMER0_IRQNR,
+ ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HIGH);
+ if (localirq <= 0) {
+ pr_err("Can't register Hyper-V stimer0 GSI. Error %d",
+ localirq);
+ *irq = 0;
+ return -1;
+ }
+ stimer0_evt = alloc_percpu(long);
+ result = request_percpu_irq(localirq, hv_stimer0_vector_handler,
+ "Hyper-V stimer0", stimer0_evt);
+ if (result) {
+ pr_err("Can't request Hyper-V stimer0 IRQ %d. Error %d",
+ localirq, result);
+ free_percpu(stimer0_evt);
+ acpi_unregister_gsi(localirq);
+ *irq = 0;
+ return -1;
+ }
+
+ hv_stimer0_handler = handler;
+ *vector = HV_STIMER0_IRQNR;
+ *irq = localirq;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(hv_setup_stimer0_irq);
+
+void hv_remove_stimer0_irq(int irq)
+{
+ hv_stimer0_handler = NULL;
+ if (irq) {
+ free_percpu_irq(irq, stimer0_evt);
+ free_percpu(stimer0_evt);
+ acpi_unregister_gsi(irq);
+ }
+}
+EXPORT_SYMBOL_GPL(hv_remove_stimer0_irq);
+
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+ hv_kexec_handler = handler;
+}
+EXPORT_SYMBOL_GPL(hv_setup_kexec_handler);
+
+void hv_remove_kexec_handler(void)
+{
+ hv_kexec_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(hv_remove_kexec_handler);
+
+void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs))
+{
+ hv_crash_handler = handler;
+}
+EXPORT_SYMBOL_GPL(hv_setup_crash_handler);
+
+void hv_remove_crash_handler(void)
+{
+ hv_crash_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(hv_remove_crash_handler);
Overall, this patch needs a massive split up, clean up, and a good dose
of documentation so that we can understand what you are trying to achieve.

Thanks,

M.
--
Jazz is not dead. It just smells funny...
Greg KH
2018-11-26 19:18:12 UTC
Permalink
Post by k***@linuxonhyperv.com
--- /dev/null
+++ b/arch/arm64/include/asm/hyperv-tlfs.h
@@ -0,0 +1,338 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from the Hyper-V Hypervisor Top-Level
+ * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0
"interesting" url :(

Care to use a real one?

thanks,

greg k-h
Joshua R. Poulson
2018-11-26 20:21:33 UTC
Permalink
That's https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs

Thanks, --jrp
Post by Greg KH
Post by k***@linuxonhyperv.com
--- /dev/null
+++ b/arch/arm64/include/asm/hyperv-tlfs.h
@@ -0,0 +1,338 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from the Hyper-V Hypervisor Top-Level
+ * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0
"interesting" url :(
Care to use a real one?
thanks,
greg k-h
KY Srinivasan
2018-11-27 03:06:55 UTC
Permalink
-----Original Message-----
Sent: Monday, November 26, 2018 11:18 AM
Subject: Re: [PATCH 1/4] arm64: hyperv: Add core Hyper-V include files
Post by k***@linuxonhyperv.com
--- /dev/null
+++ b/arch/arm64/include/asm/hyperv-tlfs.h
@@ -0,0 +1,338 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from the Hyper-V Hypervisor Top-Level
+ *
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.
microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-
windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.co
m%7Cce3a8662cbbc4afe157508d653d3eadf%7C72f988bf86f141af91ab2d7cd0
11db47%7C1%7C0%7C636788566988323167&amp;sdata=VJHHMNbpOdAANd
DNUD3UMuR8SkbBGOew7uy%2B%2BUCS2sE%3D&amp;reserved=0
"interesting" url :(
Care to use a real one?
Will fix and resend.

K. Y
thanks,
greg k-h
Will Deacon
2018-12-07 13:42:35 UTC
Permalink
Hi all,
hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level
Functional Spec (TLFS). The TLFS is distinctly oriented to x86/x64,
and Hyper-V has not separated out the architecture-dependent parts into
x86/x64 vs. ARM64. So hyperv-tlfs.h includes information for ARM64
When do you plan to publish the spec? It's pretty hard to review this stuff
without knowing what it's supposed to look like.
docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
mshyperv.h defines Linux-specific structures and routines for
interacting with Hyper-V. It is split into an ARM64 specific file
and an architecture independent file in include/asm-generic.
---
MAINTAINERS | 3 +
arch/arm64/include/asm/hyperv-tlfs.h | 338 +++++++++++++++++++++++++++
arch/arm64/include/asm/mshyperv.h | 116 +++++++++
include/asm-generic/mshyperv.h | 240 +++++++++++++++++++
4 files changed, 697 insertions(+)
create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
create mode 100644 arch/arm64/include/asm/mshyperv.h
create mode 100644 include/asm-generic/mshyperv.h
diff --git a/MAINTAINERS b/MAINTAINERS
index f4855974f325..72f19cef4c48 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6835,6 +6835,8 @@ F: arch/x86/include/asm/trace/hyperv.h
F: arch/x86/include/asm/hyperv-tlfs.h
F: arch/x86/kernel/cpu/mshyperv.c
F: arch/x86/hyperv
+F: arch/arm64/include/asm/hyperv-tlfs.h
+F: arch/arm64/include/asm/mshyperv.h
F: drivers/hid/hid-hyperv.c
F: drivers/hv/
F: drivers/input/serio/hyperv-keyboard.c
@@ -6846,6 +6848,7 @@ F: drivers/video/fbdev/hyperv_fb.c
F: net/vmw_vsock/hyperv_transport.c
F: include/linux/hyperv.h
F: include/uapi/linux/hyperv.h
+F: include/asm-generic/mshyperv.h
F: tools/hv/
F: Documentation/ABI/stable/sysfs-bus-vmbus
diff --git a/arch/arm64/include/asm/hyperv-tlfs.h b/arch/arm64/include/asm/hyperv-tlfs.h
new file mode 100644
index 000000000000..924e37600e92
--- /dev/null
+++ b/arch/arm64/include/asm/hyperv-tlfs.h
@@ -0,0 +1,338 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from the Hyper-V Hypervisor Top-Level
+ * https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fvirtualization%2Fhyper-v-on-windows%2Freference%2Ftlfs&amp;data=02%7C01%7Ckys%40microsoft.com%7Cc831a45fd63e4a4b083908d641216aa8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636768009113747528&amp;sdata=jRSrs9ZWXdmeS7LQUEpoSyUfBS7a5KLYy%2FolFdE2tI0%3D&amp;reserved=0
As mentioned elsewhere, please use a better link here and drop the license
boilerplate below.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT. See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_ARM64_HYPERV_H
+#define _ASM_ARM64_HYPERV_H
__ASM_HYPER_V_H please
+
+#include <linux/types.h>
+
+/*
+ * These Hyper-V registers provide information equivalent to the CPUID
+ * instruction on x86/x64.
+ */
+#define HV_REGISTER_HYPERVISOR_VERSION 0x00000100 /*CPUID 0x40000002 */
+#define HV_REGISTER_PRIVILEGES_AND_FEATURES 0x00000200 /*CPUID 0x40000003 */
+#define HV_REGISTER_FEATURES 0x00000201 /*CPUID 0x40000004 */
+#define HV_REGISTER_IMPLEMENTATION_LIMITS 0x00000202 /*CPUID 0x40000005 */
+#define HV_ARM64_REGISTER_INTERFACE_VERSION 0x00090006 /*CPUID 0x40000001 */
+
+/*
+ * Feature identification. HvRegisterPrivilegesAndFeaturesInfo returns a
+ * 128-bit value with flags indicating which features are available to the
+ * partition based upon the current partition privileges. The 128-bit
+ * value is broken up with different portions stored in different 32-bit
+ * fields in the ms_hyperv structure.
+ */
+
+/* Partition Reference Counter available*/
+#define HV_MSR_TIME_REF_COUNT_AVAILABLE (1 << 1)
+
+/*
+ * Synthetic Timers available
+ */
+#define HV_MSR_SYNTIMER_AVAILABLE (1 << 3)
+
+/* Frequency MSRs available */
+#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE (1 << 8)
+
+/* Reference TSC available */
+#define HV_MSR_REFERENCE_TSC_AVAILABLE (1 << 9)
+
+/* Crash MSR available */
+#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE (1 << 10)
+
+
+/*
+ * This group of flags is in the high order 64-bits of the returned
+ * 128-bit value.
+ */
+
+/* STIMER direct mode is available */
+#define HV_STIMER_DIRECT_MODE_AVAILABLE (1 << 19)
+
+/*
+ * Implementation recommendations in register
+ * HvRegisterFeaturesInfo. Indicates which behaviors the hypervisor
+ * recommends the OS implement for optimal performance.
+ */
+
+/*
+ * Recommend not using Auto EOI
+ */
+#define HV_DEPRECATING_AEOI_RECOMMENDED (1 << 9)
+
+/*
+ * Synthetic register definitions equivalent to MSRs on x86/x64
+ */
+#define HV_REGISTER_CRASH_P0 0x00000210
+#define HV_REGISTER_CRASH_P1 0x00000211
+#define HV_REGISTER_CRASH_P2 0x00000212
+#define HV_REGISTER_CRASH_P3 0x00000213
+#define HV_REGISTER_CRASH_P4 0x00000214
+#define HV_REGISTER_CRASH_CTL 0x00000215
+
+#define HV_REGISTER_GUEST_OSID 0x00090002
+#define HV_REGISTER_VPINDEX 0x00090003
+#define HV_REGISTER_TIME_REFCOUNT 0x00090004
+#define HV_REGISTER_REFERENCE_TSC 0x00090017
+
+#define HV_REGISTER_SINT0 0x000A0000
+#define HV_REGISTER_SINT1 0x000A0001
+#define HV_REGISTER_SINT2 0x000A0002
+#define HV_REGISTER_SINT3 0x000A0003
+#define HV_REGISTER_SINT4 0x000A0004
+#define HV_REGISTER_SINT5 0x000A0005
+#define HV_REGISTER_SINT6 0x000A0006
+#define HV_REGISTER_SINT7 0x000A0007
+#define HV_REGISTER_SINT8 0x000A0008
+#define HV_REGISTER_SINT9 0x000A0009
+#define HV_REGISTER_SINT10 0x000A000A
+#define HV_REGISTER_SINT11 0x000A000B
+#define HV_REGISTER_SINT12 0x000A000C
+#define HV_REGISTER_SINT13 0x000A000D
+#define HV_REGISTER_SINT14 0x000A000E
+#define HV_REGISTER_SINT15 0x000A000F
+#define HV_REGISTER_SCONTROL 0x000A0010
+#define HV_REGISTER_SVERSION 0x000A0011
+#define HV_REGISTER_SIFP 0x000A0012
+#define HV_REGISTER_SIPP 0x000A0013
+#define HV_REGISTER_EOM 0x000A0014
+#define HV_REGISTER_SIRBP 0x000A0015
+
+#define HV_REGISTER_STIMER0_CONFIG 0x000B0000
+#define HV_REGISTER_STIMER0_COUNT 0x000B0001
+#define HV_REGISTER_STIMER1_CONFIG 0x000B0002
+#define HV_REGISTER_STIMER1_COUNT 0x000B0003
+#define HV_REGISTER_STIMER2_CONFIG 0x000B0004
+#define HV_REGISTER_STIMER2_COUNT 0x000B0005
+#define HV_REGISTER_STIMER3_CONFIG 0x000B0006
+#define HV_REGISTER_STIMER3_COUNT 0x000B0007
+
+/*
+ * Crash notification flags.
+ */
+#define HV_CRASH_CTL_CRASH_NOTIFY_MSG BIT_ULL(62)
+#define HV_CRASH_CTL_CRASH_NOTIFY BIT_ULL(63)
+
+/*
+ * The guest OS needs to register the guest ID with the hypervisor.
+ * The guest ID is a 64 bit entity and the structure of this ID is
+ *
+ * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
Dead link :(
+ *
+ * While the current guideline does not specify how Linux guest ID(s)
+ * need to be generated, our plan is to publish the guidelines for
+ * Linux and other guest operating systems that currently are hosted
+ * on Hyper-V. The implementation here conforms to this yet
+ * unpublished guidelines.
Again, any timeline on publishing this stuff? Right now, this is just a file
full of magic numbers and I'm not sure we're in a good position to maintain
it based on the idea that you have a cunning plan.
+ *
+ *
+ * Bit(s)
+ * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
+ * 62:56 - Os Type; Linux is 0x100
+ * 55:48 - Distro specific identification
+ * 47:16 - Linux kernel version number
+ * 15:0 - Distro specific identification
+ *
+ *
+ */
+#define HV_LINUX_VENDOR_ID 0x8100
+
+/* Declare the various hypercall operations. */
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE 0x0002
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST 0x0003
+#define HVCALL_NOTIFY_LONG_SPIN_WAIT 0x0008
+#define HVCALL_SEND_IPI 0x000b
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX 0x0013
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX 0x0014
+#define HVCALL_SEND_IPI_EX 0x0015
+#define HVCALL_GET_VP_REGISTERS 0x0050
+#define HVCALL_SET_VP_REGISTERS 0x0051
+#define HVCALL_POST_MESSAGE 0x005c
+#define HVCALL_SIGNAL_EVENT 0x005d
+#define HVCALL_RETARGET_INTERRUPT 0x007e
+#define HVCALL_START_VIRTUAL_PROCESSOR 0x0099
+#define HVCALL_GET_VP_INDEX_FROM_APICID 0x009a
This all sounds very x86y...
+/* Declare standard hypercall field values. */
+#define HV_PARTITION_ID_SELF ((u64)-1)
+#define HV_VP_INDEX_SELF ((u32)-2)
+
+#define HV_HYPERCALL_FAST_BIT BIT(16)
+#define HV_HYPERCALL_REP_COUNT_1 BIT_ULL(32)
+#define HV_HYPERCALL_RESULT_MASK GENMASK_ULL(15, 0)
+
+/* Define the hypercall status result */
+
+union hv_hypercall_status {
+ u64 as_uint64;
+ struct {
+ u16 status;
+ u16 reserved;
+ u16 reps_completed; /* Low 12 bits */
+ u16 reserved2;
+ };
+};
+
+/* hypercall status code */
+#define HV_STATUS_SUCCESS 0
+#define HV_STATUS_INVALID_HYPERCALL_CODE 2
+#define HV_STATUS_INVALID_HYPERCALL_INPUT 3
+#define HV_STATUS_INVALID_ALIGNMENT 4
+#define HV_STATUS_INSUFFICIENT_MEMORY 11
+#define HV_STATUS_INVALID_CONNECTION_ID 18
+#define HV_STATUS_INSUFFICIENT_BUFFERS 19
+
+/* Define output layout for Get VP Register hypercall */
+struct hv_get_vp_register_output {
+ u64 registervaluelow;
+ u64 registervaluehigh;
+};
+
+#define HV_FLUSH_ALL_PROCESSORS BIT(0)
+#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES BIT(1)
+#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY BIT(2)
+#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT BIT(3)
+
+enum HV_GENERIC_SET_FORMAT {
+ HV_GENERIC_SET_SPARSE_4K,
+ HV_GENERIC_SET_ALL,
+};
+
+/*
+ * The Hyper-V TimeRefCount register and the TSC
+ * page provide a guest VM clock with 100ns tick rate
+ */
+#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
+
+/*
+ * The fields in this structure are set by Hyper-V and read
+ * by the Linux guest. They should be accessed with READ_ONCE()
+ * so the compiler doesn't optimize in a way that will cause
+ * problems.
+ */
+struct ms_hyperv_tsc_page {
+ u32 tsc_sequence;
+ u32 reserved1;
+ u64 tsc_scale;
+ s64 tsc_offset;
+ u64 reserved2[509];
+};
It might be clearer to express this as a union with the page size:

struct ms_hyperv_tsc_page {
union {
struct {
__u32 tsc_sequence;
__u32 reserved1;
};
__u8 reserved2[HV_HYP_PAGE_SIZE];
};
};

What's the required alignment on this structure?
Also, do you intend for 32-bit guests to use this ABI as well?
+
+/* Define the number of synthetic interrupt sources. */
+#define HV_SYNIC_SINT_COUNT (16)
+/* Define the expected SynIC version. */
+#define HV_SYNIC_VERSION_1 (0x1)
+
+#define HV_SYNIC_CONTROL_ENABLE (1ULL << 0)
+#define HV_SYNIC_SIMP_ENABLE (1ULL << 0)
+#define HV_SYNIC_SIEFP_ENABLE (1ULL << 0)
+#define HV_SYNIC_SINT_MASKED (1ULL << 16)
+#define HV_SYNIC_SINT_AUTO_EOI (1ULL << 17)
+#define HV_SYNIC_SINT_VECTOR_MASK (0xFF)
+
+#define HV_SYNIC_STIMER_COUNT (4)
+
+/* Define synthetic interrupt controller message constants. */
+#define HV_MESSAGE_SIZE (256)
+#define HV_MESSAGE_PAYLOAD_BYTE_COUNT (240)
+#define HV_MESSAGE_PAYLOAD_QWORD_COUNT (30)
+
+/* Define hypervisor message types. */
+enum hv_message_type {
+ HVMSG_NONE = 0x00000000,
+
+ /* Memory access messages. */
+ HVMSG_UNMAPPED_GPA = 0x80000000,
+ HVMSG_GPA_INTERCEPT = 0x80000001,
+
+ /* Timer notification messages. */
+ HVMSG_TIMER_EXPIRED = 0x80000010,
+
+ /* Error messages. */
+ HVMSG_INVALID_VP_REGISTER_VALUE = 0x80000020,
+ HVMSG_UNRECOVERABLE_EXCEPTION = 0x80000021,
+ HVMSG_UNSUPPORTED_FEATURE = 0x80000022,
+
+ /* Trace buffer complete messages. */
+ HVMSG_EVENTLOG_BUFFERCOMPLETE = 0x80000040,
+};
+
+/* Define synthetic interrupt controller message flags. */
+union hv_message_flags {
+ __u8 asu8;
+ struct {
+ __u8 msg_pending:1;
+ __u8 reserved:7;
+ };
+};
+
+/* Define port identifier type. */
+union hv_port_id {
+ __u32 asu32;
+ struct {
+ __u32 id:24;
+ __u32 reserved:8;
+ } u;
+};
Hmm, I thought bitfields weren't a good idea for this sort of thing. Isn't
it up to the compiler how they are laid out in memory?
+
+/* Define synthetic interrupt controller message header. */
+struct hv_message_header {
+ __u32 message_type;
+ __u8 payload_size;
+ union hv_message_flags message_flags;
+ __u8 reserved[2];
+ union {
+ __u64 sender;
+ union hv_port_id port;
+ };
+};
+
+/* Define synthetic interrupt controller message format. */
+struct hv_message {
+ struct hv_message_header header;
+ union {
+ __u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
+ } u;
+};
+
+/* Define the synthetic interrupt message page layout. */
+struct hv_message_page {
+ struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
+};
+
+/* Define timer message payload structure. */
+struct hv_timer_message_payload {
+ __u32 timer_index;
+ __u32 reserved;
+ __u64 expiration_time; /* When the timer expired */
+ __u64 delivery_time; /* When the message was delivered */
+};
+
+#define HV_STIMER_ENABLE (1ULL << 0)
+#define HV_STIMER_PERIODIC (1ULL << 1)
+#define HV_STIMER_LAZY (1ULL << 2)
+#define HV_STIMER_AUTOENABLE (1ULL << 3)
+#define HV_STIMER_SINT(config) (__u8)(((config) >> 16) & 0x0F)
+
+#endif
diff --git a/arch/arm64/include/asm/mshyperv.h b/arch/arm64/include/asm/mshyperv.h
new file mode 100644
index 000000000000..a87c431d58b3
--- /dev/null
+++ b/arch/arm64/include/asm/mshyperv.h
@@ -0,0 +1,116 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Linux-specific definitions for managing interactions with Microsoft's
+ * Hyper-V hypervisor. The definitions in this file are specific to
+ * the ARM64 architecture. See include/asm-generic/mshyperv.h for
+ * definitions are that architecture independent.
+ *
+ * Definitions that are specified in the Hyper-V Top Level Functional
+ * Spec (TLFS) should not go in this file, but should instead go in
+ * hyperv-tlfs.h.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT. See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_ARM64_MSHYPERV_H
+#define _ASM_ARM64_MSHYPERV_H
+
+#include <linux/types.h>
+#include <linux/interrupt.h>
+#include <linux/clocksource.h>
+#include <linux/irq.h>
+#include <linux/irqdesc.h>
+#include <asm/hyperv-tlfs.h>
+
+/*
+ * Define the IRQ numbers/vectors used by Hyper-V VMbus interrupts
+ * and by STIMER0 Direct Mode interrupts. Hyper-V should be supplying
+ * these values through ACPI, but there are no other interrupting
+ * devices in a Hyper-V VM on ARM64, so it's OK to hard code for now.
+ * The "CALLBACK_VECTOR" terminology is a left-over from the x86/x64
+ * world that is used in architecture independent Hyper-V code.
+ */
+#define HYPERVISOR_CALLBACK_VECTOR 16
+#define HV_STIMER0_IRQNR 17
+
+extern u64 hv_do_hvc(u64 control, ...);
+extern u64 hv_do_hvc_fast_get(u64 control, u64 input1, u64 input2, u64 input3,
+ struct hv_get_vp_register_output *output);
+
+/*
+ * Declare calls to get and set Hyper-V VP register values on ARM64, which
+ * requires a hypercall.
+ */
+extern void hv_set_vpreg(u32 reg, u64 value);
+extern u64 hv_get_vpreg(u32 reg);
+extern void hv_get_vpreg_128(u32 reg, struct hv_get_vp_register_output *result);
+
+/*
+ * Use the Hyper-V provided stimer0 as the timer that is made
+ * available to the architecture independent Hyper-V drivers.
+ */
+#define hv_init_timer(timer, tick) \
+ hv_set_vpreg(HV_REGISTER_STIMER0_COUNT + (2*timer), tick)
+#define hv_init_timer_config(timer, val) \
+ hv_set_vpreg(HV_REGISTER_STIMER0_CONFIG + (2*timer), val)
+#define hv_get_current_tick(tick) \
+ (tick = hv_get_vpreg(HV_REGISTER_TIME_REFCOUNT))
+
+#define hv_get_simp(val) (val = hv_get_vpreg(HV_REGISTER_SIPP))
+#define hv_set_simp(val) hv_set_vpreg(HV_REGISTER_SIPP, val)
+
+#define hv_get_siefp(val) (val = hv_get_vpreg(HV_REGISTER_SIFP))
+#define hv_set_siefp(val) hv_set_vpreg(HV_REGISTER_SIFP, val)
+
+#define hv_get_synic_state(val) (val = hv_get_vpreg(HV_REGISTER_SCONTROL))
+#define hv_set_synic_state(val) hv_set_vpreg(HV_REGISTER_SCONTROL, val)
+
+#define hv_get_vp_index(index) (index = hv_get_vpreg(HV_REGISTER_VPINDEX))
+
+#define hv_signal_eom() hv_set_vpreg(HV_REGISTER_EOM, 0)
+
+/*
+ * Hyper-V SINT registers are numbered sequentially, so we can just
+ * add the SINT number to the register number of SINT0
+ */
+#define hv_get_synint_state(sint_num, val) \
+ (val = hv_get_vpreg(HV_REGISTER_SINT0 + sint_num))
+#define hv_set_synint_state(sint_num, val) \
+ hv_set_vpreg(HV_REGISTER_SINT0 + sint_num, val)
+
+#define hv_get_crash_ctl(val) \
+ (val = hv_get_vpreg(HV_REGISTER_CRASH_CTL))
+
+#if IS_ENABLED(CONFIG_HYPERV)
+#define hv_enable_stimer0_percpu_irq(irq) enable_percpu_irq(irq, 0)
+#define hv_disable_stimer0_percpu_irq(irq) disable_percpu_irq(irq)
+#endif
+
+/* ARM64 specific code to read the hardware clock */
+static inline u64 hv_read_hwclock(void)
+{
+ u64 result;
+
+ isb();
+ result = read_sysreg(cntvct_el0);
+ isb();
+
+ return result;
+}
Shouldn't you use arch_timer_read_counter() or arch_counter_get_cntvct()
instead?
+#include <asm-generic/mshyperv.h>
+
+#endif
diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
new file mode 100644
index 000000000000..fbe1ec89c85a
--- /dev/null
+++ b/include/asm-generic/mshyperv.h
@@ -0,0 +1,240 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Linux-specific definitions for managing interactions with Microsoft's
+ * Hyper-V hypervisor. The definitions in this file are architecture
+ * independent. See arch/<arch>/include/asm/mshyperv.h for definitions
+ * that are specific to architecture <arch>.
+ *
+ * Definitions that are specified in the Hyper-V Top Level Functional
+ * Spec (TLFS) should not go in this file, but should instead go in
+ * hyperv-tlfs.h.
+ *
+ * Copyright (C) 2018, Microsoft, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
+ * NON INFRINGEMENT. See the GNU General Public License for more
+ * details.
+ */
+
+#ifndef _ASM_GENERIC_MSHYPERV_H
+#define _ASM_GENERIC_MSHYPERV_H
+
+#include <linux/types.h>
+#include <linux/interrupt.h>
+#include <linux/clocksource.h>
+#include <linux/irq.h>
+#include <linux/irqdesc.h>
+#include <asm/hyperv-tlfs.h>
+
+/*
+ * Hyper-V always runs with a page size of 4096. These definitions
+ * are used when communicating with Hyper-V using guest physical
+ * pages and guest physical page addresses, since the guest page
+ * size may not be 4096 on ARM64.
+ */
+#define HV_HYP_PAGE_SIZE 4096
+#define HV_HYP_PAGE_SHIFT 12
Maybe define the page size in terms of the shift?
+#define HV_HYP_PAGE_MASK (~(HV_HYP_PAGE_SIZE - 1))
+
+
+struct ms_hyperv_info {
+ u32 features;
+ u32 misc_features;
+ u32 hints;
+ u32 max_vp_index;
+ u32 max_lp_index;
+};
What is the define endianness for in-memory structures shared with the
hypervisor?
+extern struct ms_hyperv_info ms_hyperv;
+
+extern u64 hv_do_hypercall(u64 control, void *inputaddr, void *outputaddr);
+extern u64 hv_do_fast_hypercall8(u16 control, u64 input8);
+
+/*
+ * The guest OS needs to register the guest ID with the hypervisor.
+ * The guest ID is a 64 bit entity and the structure of this ID is
+ *
+ * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
+ *
+ * While the current guideline does not specify how Linux guest ID(s)
+ * need to be generated, our plan is to publish the guidelines for
+ * Linux and other guest operating systems that currently are hosted
+ * on Hyper-V. The implementation here conforms to this yet
+ * unpublished guidelines.
+ *
+ *
+ * Bit(s)
+ * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
+ * 62:56 - Os Type; Linux is 0x100
+ * 55:48 - Distro specific identification
+ * 47:16 - Linux kernel version number
+ * 15:0 - Distro specific identification
This looks familiar...
+ * Generate the guest ID based on the guideline described above.
+ */
+
+static inline __u64 generate_guest_id(__u64 d_info1, __u64 kernel_version,
+ __u64 d_info2)
+{
+ __u64 guest_id = 0;
+
+ guest_id = (((__u64)HV_LINUX_VENDOR_ID) << 48);
+ guest_id |= (d_info1 << 48);
+ guest_id |= (kernel_version << 16);
+ guest_id |= d_info2;
+
+ return guest_id;
+}
+
+
+/* Free the message slot and signal end-of-message if required */
+static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)
+{
+ /*
+ * On crash we're reading some other CPU's message page and we need
+ * to be careful: this other CPU may already had cleared the header
+ * and the host may already had delivered some other message there.
+ * In case we blindly write msg->header.message_type we're going
+ * to lose it. We can still lose a message of the same type but
+ * we count on the fact that there can only be one
+ * CHANNELMSG_UNLOAD_RESPONSE and we don't care about other messages
+ * on crash.
+ */
+ if (cmpxchg(&msg->header.message_type, old_msg_type,
+ HVMSG_NONE) != old_msg_type)
+ return;
+
+ /*
+ * Make sure the write to MessageType (ie set to
+ * HVMSG_NONE) happens before we read the
+ * MessagePending and EOMing. Otherwise, the EOMing
+ * will not deliver any more messages since there is
+ * no empty slot
+ */
+ mb();
A successful cmpxchg() already implies full barriers, so this isn't needed.
I also wonder whether cmpxchg_acquire() would be sufficient here.
+
+ if (msg->header.message_flags.msg_pending) {
+ /*
+ * This will cause message queue rescan to
+ * possibly deliver another msg from the
+ * hypervisor
+ */
+ hv_signal_eom();
+ }
+}
+
+void hv_setup_vmbus_irq(void (*handler)(void));
+void hv_remove_vmbus_irq(void);
+void hv_enable_vmbus_irq(void);
+void hv_disable_vmbus_irq(void);
+
+void hv_setup_kexec_handler(void (*handler)(void));
+void hv_remove_kexec_handler(void);
+void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs));
+void hv_remove_crash_handler(void);
+
+#if IS_ENABLED(CONFIG_HYPERV)
+extern struct clocksource *hyperv_cs;
+
+/*
+ * Hypervisor's notion of virtual processor ID is different from
+ * Linux' notion of CPU ID. This information can only be retrieved
+ * in the context of the calling CPU. Setup a map for easy access
+ * to this information.
+ */
+extern u32 *hv_vp_index;
+extern u32 hv_max_vp_index;
+
+/* Sentinel value for an uninitialized entry in hv_vp_index array */
+#define VP_INVAL U32_MAX
+
+/**
+ * hv_cpu_number_to_vp_number() - Map CPU to VP.
+ *
+ * This function returns the mapping between the Linux processor
+ * number and the hypervisor's virtual processor number, useful
+ * in making hypercalls and such that talk about specific
+ * processors.
+ *
+ * Return: Virtual processor number in Hyper-V terms
+ */
+static inline int hv_cpu_number_to_vp_number(int cpu_number)
+{
+ return hv_vp_index[cpu_number];
+}
+
+void hyperv_report_panic(struct pt_regs *regs, long err);
+void hyperv_report_panic_msg(phys_addr_t pa, size_t size);
+bool hv_is_hyperv_initialized(void);
+void hyperv_cleanup(void);
+#else /* CONFIG_HYPERV */
+static inline bool hv_is_hyperv_initialized(void) { return false; }
+static inline void hyperv_cleanup(void) {}
+#endif /* CONFIG_HYPERV */
+
+#if IS_ENABLED(CONFIG_HYPERV)
+extern int hv_setup_stimer0_irq(int *irq, int *vector, void (*handler)(void));
+extern void hv_remove_stimer0_irq(int irq);
+#endif
+
+static inline u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg,
+ u64 *cur_tsc)
+{
+ u64 scale, offset;
+ u32 sequence;
+
+ /*
+ * The protocol for reading Hyper-V TSC page is specified in Hypervisor
+ * Top-Level Functional Specification. To get the reference time we
+ * - READ ReferenceTscSequence
+ * A special '0' value indicates the time source is unreliable and we
+ * need to use something else.
+ * - ReferenceTime =
+ * ((HWclock val) * ReferenceTscScale) >> 64) + ReferenceTscOffset
+ * - READ ReferenceTscSequence again. In case its value has changed
+ * since our first reading we need to discard ReferenceTime and repeat
+ * the whole sequence as the hypervisor was updating the page in
+ * between.
+ */
+ do {
+ sequence = READ_ONCE(tsc_pg->tsc_sequence);
+ /*
+ * Make sure we read sequence before we read other values from
+ * TSC page.
+ */
+ smp_rmb();
+
+ scale = READ_ONCE(tsc_pg->tsc_scale);
+ offset = READ_ONCE(tsc_pg->tsc_offset);
+ *cur_tsc = hv_read_hwclock();
+
+ /*
+ * Make sure we read sequence after we read all other values
+ * from TSC page.
+ */
+ smp_rmb();
+
+ } while (READ_ONCE(tsc_pg->tsc_sequence) != sequence);
Could you explain what the writer side looks like, please? I'm a bit
confused as to why we're not using the bottom bit of the sequence
number to detect a concurrent update.

Will
KY Srinivasan
2018-11-22 03:02:55 UTC
Permalink
-----Original Message-----
Sent: Wednesday, November 21, 2018 6:51 PM
Subject: [PATCH 0/4] Hyper-V: Enable Linux guests on Hyper-V on ARM64
This series enables Linux guests running on Hyper-V on ARM64
hardware. New ARM64-specific code in arch/arm64/hyperv initializes
Hyper-V, including its synthetic clocks and hypercall mechanism.
Existing architecture independent drivers for Hyper-V's VMbus and
synthetic devices just work when built for ARM64. Hyper-V code is
built and included in the image and modules only if CONFIG_HYPERV
is enabled.
1) Add include files that define the Hyper-V interface as
described in the Hyper-V Top Level Functional Spec (TLFS), plus
additional definitions specific to Linux running on Hyper-V.
2) Add core Hyper-V support on ARM64, including hypercalls,
synthetic clock initialization, and interrupt handlers.
3) Update the existing VMbus driver to generalize interrupt
management across x86/x64 and ARM64.
4) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.
Some areas of Linux guests on Hyper-V on ARM64 are a work-
* Hyper-V on ARM64 currently runs with a 4 Kbyte page size, and only
supports guests with a 4 Kbyte page size. Because Hyper-V uses
shared pages to communicate between the guest and the hypervisor,
there are open design decisions on the page size to use when
the guest is using 16K/64K pages. Once those issues are
resolved and Hyper-V fully supports 16K/64K guest pages, changes
may be needed in the Linux drivers for Hyper-V synthetic devices.
* Hyper-V on ARM64 does not currently support mapping PCI devices
into the guest address space. The Hyper-V PCI driver at
drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is
not being built for ARM64.
In a few cases, terminology from the x86/x64 world has been carried
over into the ARM64 code ("MSR", "TSC"). Hyper-V still uses the
x86/x64 terminology and has not replaced it with something more
generic, so the code uses the Hyper-V terminology. This will be
fixed when Hyper-V updates the usage in the TLFS.
arm64: hyperv: Add core Hyper-V include files
arm64: hyperv: Add support for Hyper-V as a hypervisor
Drivers: hv: vmbus: Add hooks for per-CPU IRQ
Drivers: hv: Enable CONFIG_HYPERV on ARM64
MAINTAINERS | 4 +
arch/arm64/Makefile | 1 +
arch/arm64/hyperv/Makefile | 2 +
arch/arm64/hyperv/hv_hvc.S | 54 ++++
arch/arm64/hyperv/hv_init.c | 441 +++++++++++++++++++++++++++
arch/arm64/hyperv/mshyperv.c | 178 +++++++++++
arch/arm64/include/asm/hyperv-tlfs.h | 338 ++++++++++++++++++++
arch/arm64/include/asm/mshyperv.h | 116 +++++++
arch/x86/include/asm/mshyperv.h | 4 +
drivers/hv/Kconfig | 3 +-
drivers/hv/hv.c | 2 +
include/asm-generic/mshyperv.h | 240 +++++++++++++++
12 files changed, 1382 insertions(+), 1 deletion(-)
create mode 100644 arch/arm64/hyperv/Makefile
create mode 100644 arch/arm64/hyperv/hv_hvc.S
create mode 100644 arch/arm64/hyperv/hv_init.c
create mode 100644 arch/arm64/hyperv/mshyperv.c
create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
create mode 100644 arch/arm64/include/asm/mshyperv.h
create mode 100644 include/asm-generic/mshyperv.h
Greg,

Please drop this. I will be resending shortly.

K. Y
--
2.19.1
Loading...