LDMP2v

SPARC Physical-to-Virtual (P2V) Migration

Many talk about virtualization nowadays, and one of real example I face would be retiring SunFire-280R (with Solaris 8) and migrating to LDom.
The problem is that the 280R is running Oracle database version 9 and I am not sure how will it work under Solaris 10 (and Oracle guys seems don't like idea of this migration exercise so no further investigation or migration has been done).
Anyway, I still had decided to test ldmp2v migration tool, and wrote this article for future references.

Main players here are:

The source physical system to be migrated is SunFire-V120 (platform sun4u) running Solaris 9 update 8.
The target LDom is on the SunFire-T2000, running Solaris 10 update 9 with Logical Domain Manager version 1.2

In short, "man ldmp2v" says:
Logical Domains Physical-to-Virtual (P2V) Migration Tool (ver 1.0) automatically converts an existing physical system to a virtual system that runs in a logical domain on a chip multithreading (CMT) system.
The source system can be any sun4u SPARC system that runs at least the Solaris 8 or a non-Logical Domains sun4v system that runs the Solaris 10.

Prerequisites

Some prerequisites are needed before doing actual p2v migration.

The first thing it's needed is actually preparing the T2000 to host LDom. Three services (disk service, networking and console access) have to be created and the primary control domain will provide them to the LDom.
Here I'll create two services for network, since one LDom will be on 192.168.20.x subnet and another on 192.168.28.x

{t2000}/> ldm add-vcc port-range=5000-5010 primary-consrv primary

{t2000}/> ldm add-vsw net-dev=e1000g0 primary-net0_srv primary
{t2000}/> ldm add-vsw net-dev=e1000g1 primary-net1_srv primary

{t2000}/> ldm add-vds primary_vdsksrv_cs2 primary

{t2000}/backup/bank0/var> ldm list-services
VCC
    NAME             LDOM             PORT-RANGE
    primary-consrv   primary          5000-5010

VSW
    NAME             LDOM     MAC               NET-DEV   DEVICE     DEFAULT-VLAN-ID PVID VID   MTU   MODE
    primary-net0_srv primary  00:14:4f:fa:24:1a e1000g0   switch@0   1               1          1500
    primary-net1_srv primary  00:14:4f:fb:49:fb e1000g1   switch@1   1               1          1500

VDS
    NAME             LDOM             VOLUME         OPTIONS          MPGROUP        DEVICE
    primary_vdsksrv_cs2 primary

Create a ZPOOL and ZFS that will be backend for virtual disk.

{t2000}/> zpool status space1
  pool: space1
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        space1      ONLINE       0     0     0
          c0t1d0    ONLINE       0     0     0
errors: No known data errors

{t2000}/> zfs list space1
NAME     USED  AVAIL  REFER  MOUNTPOINT
space1   131K   134G    23K  /space1

{t2000}/> zfs list | grep space1
space1                           132K   134G    24K  /space1
space1/cs2                        21K   134G    21K  /space1/cs2

Check the hardware on the T2000 system.

{t2000}/> ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      32    8064M    0.1%  3h 56m

{t2000}/> ldm list -o crypto primary
NAME
primary
MAU
    ID     CPUSET
    0      (0, 1, 2, 3)
    1      (4, 5, 6, 7)
    2      (8, 9, 10, 11)
    3      (12, 13, 14, 15)
    4      (16, 17, 18, 19)
    5      (20, 21, 22, 23)
    6      (24, 25, 26, 27)
    7      (28, 29, 30, 31)

Now allocate some resources to the control domain and free the rest to be used by LDom.

{t2000}/> ldm set-mau 2 primary
{t2000}/> ldm set-vcpu 8 primary
{t2000}/> ldm set-memory 4G primary
{t2000}/> ldm list -o crypto primary
NAME
primary
MAU
    ID     CPUSET
    0      (0, 1, 2, 3)
    1	   (4, 5, 6, 7)

{t2000}/> ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     4G       0.0%  4h 5m

Save configuration and power off/on the T2000 system.

{t2000}/> ldm list-config
factory-default [current]

{t2000}/> ldm add-config initial

{t2000}/> ldm list-config
factory-default
initial [current]

The primary control domain is now ready on the T2000.

The next thing to be done is creating a NFS share on the T2000, to be used to store source's (SunFire-V120) image.

{t2000}/> showmount -e t2000
export list for t2000:
/space1/p2v-images (everyone)

{t2000}/> zfs get sharenfs space1/p2v-images
NAME               PROPERTY  VALUE      SOURCE
space1/p2v-images  sharenfs  rw,anon=0  local

Copy collect script (/usr/sbin/ldmp2v) from the T2000 to the V120 (/var/tmp/ldmp2v).

And now we are ready for p2v migration that goes in 3 phases: collection, preparation and conversion.

Collection phase

Collection phase runs on the physical source system V120, it creates a file system image of the system.

Note: the hostname of the SunFire-V120 is cs2. The directory /p2v is the mount point of t2000:/space1/p2v-images/cs2/, so image will be placed there.

# ./ldmp2v collect -d /p2v
Collecting system configuration ...
Archiving file systems ...
  DUMP: Date of this level 0 dump: Fri 05 Aug 2011 03:19:05 PM PDT
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping /dev/rdsk/c1t0d0s0 (cs2:/) to /p2v/ufsdump.0.
  DUMP: Mapping (Pass I) [regular files]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Writing 63 Kilobyte records
  DUMP: Estimated 159016 blocks (77.64MB).
  DUMP: Dumping (Pass III) [directories]
  DUMP: Dumping (Pass IV) [regular files]
  DUMP: 159010 blocks (77.64MB) on 1 volume at 5542 KB/sec
  DUMP: DUMP IS DONE
  DUMP: Date of this level 0 dump: Fri 05 Aug 2011 03:19:21 PM PDT
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping /dev/rdsk/c1t0d0s5 (cs2:/.0) to /p2v/ufsdump.1.
  DUMP: Mapping (Pass I) [regular files]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Writing 63 Kilobyte records
  DUMP: Estimated 1438 blocks (719KB).
  DUMP: Dumping (Pass III) [directories]
  DUMP: Dumping (Pass IV) [regular files]
  DUMP: 1384 blocks (692KB) on 1 volume at 5125 KB/sec
  DUMP: DUMP IS DONE
  DUMP: Date of this level 0 dump: Fri 05 Aug 2011 03:19:23 PM PDT
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping /dev/rdsk/c1t0d0s6 (cs2:/backup) to /p2v/ufsdump.2.
  DUMP: Mapping (Pass I) [regular files]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Writing 63 Kilobyte records
  DUMP: Estimated 2206 blocks (1.08MB).
  DUMP: Dumping (Pass III) [directories]
  DUMP: Dumping (Pass IV) [regular files]
  DUMP: 2140 blocks (1.04MB) on 1 volume at 5194 KB/sec
  DUMP: DUMP IS DONE
  DUMP: Date of this level 0 dump: Fri 05 Aug 2011 03:19:27 PM PDT
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping /dev/rdsk/c1t0d0s3 (cs2:/usr) to /p2v/ufsdump.3.
  DUMP: Mapping (Pass I) [regular files]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Writing 63 Kilobyte records
  DUMP: Estimated 882080 blocks (430.70MB).
  DUMP: Dumping (Pass III) [directories]
  DUMP: Dumping (Pass IV) [regular files]
  DUMP: 881998 blocks (430.66MB) on 1 volume at 5440 KB/sec
  DUMP: DUMP IS DONE
  DUMP: Date of this level 0 dump: Fri 05 Aug 2011 03:20:50 PM PDT
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping /dev/rdsk/c1t0d0s4 (cs2:/var) to /p2v/ufsdump.4.
  DUMP: Mapping (Pass I) [regular files]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Writing 63 Kilobyte records
  DUMP: Estimated 12730 blocks (6.22MB).
  DUMP: Dumping (Pass III) [directories]
  DUMP: Dumping (Pass IV) [regular files]
  DUMP: 12724 blocks (6.21MB) on 1 volume at 3827 KB/sec
  DUMP: DUMP IS DONE

Preparation phase

Preparation phase runs on the primary control domain of the target system T2000. It prepares and creates the LDom on the target system based on the configuration information collected in the collect phase. The file system image of V120 is restored to a virtual disk and the image is modified to enable it to run as LDom.

The file ldmp2v.conf is used for creation of new LDom, originally it's placed in the directory /opt/SUNWldmp2v/etc/ but seems it's needed to be in /etc/, so just create the symbolic link /etc/ldmp2v.conf that points to file /opt/SUNWldmp2v/etc/ldmp2v.conf

# ldmp2v.conf - default parameters for LDom P2V Tools
# Virtual switch to use
VSW="primary-net0_srv"
# Virtual disk service to use
VDS="primary_vdsksrv_cs2"
# Virtual console concentrator to use
VCC="primary-consrv"
# Location where vdisk backend devices are stored
BACKEND_PREFIX="space1"
# Default backend type: "zvol" or "file".
BACKEND_TYPE="zvol"
# Create sparse backend devices: "yes" or "no"
BACKEND_SPARSE="no"
# Timeout for Solaris boot in seconds
BOOT_TIMEOUT=60

Now prepare the target, -d = is data directory and last parameter is LDom name.

{t2000}/> ldmp2v prepare -d /space1/p2v-images/cs2 cs2
NOTICE: the size of the /usr file system will be increased to 5120 MB
Creating vdisks ...
Resizing file systems ...
Creating file systems ...
Populating file systems ...
Modifying guest OS image ...
Modifying SVM configuration ...
Unmounting file systems ...
Creating domain ...
Attaching vdisks to domain cs2 ...

Let's see what the preparation phase has done.

{t2000}/> ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     2G       0.2%  2h 18m
cs2              inactive   ------          1     1G

{t2000}/> ldm list-services
VCC
    NAME             LDOM             PORT-RANGE
    primary-consrv   primary          5000-5010

VSW
    NAME             LDOM     MAC               NET-DEV   DEVICE     DEFAULT-VLAN-ID PVID VID  MTU  MODE
    primary-net0_srv primary  00:14:4f:fa:24:1a e1000g0   switch@0   1               1         1500
    primary-net1_srv primary  00:14:4f:fb:49:fb e1000g1   switch@1   1               1         1500

VDS
    NAME             LDOM             VOLUME         OPTIONS  MPGROUP   DEVICE
    primary_vdsksrv_cs2 primary          cs2-vol0                       /dev/zvol/dsk/space1/cs2/disk0

{t2000}/> ldm list-bindings cs2
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
cs2              inactive   ------          1     1G
CONTROL
    failure-policy=ignore

DEPENDENCY
    master=

NETWORK
    NAME             SERVICE                     DEVICE     MAC               MODE   PVID VID      MTU
    vnet0            primary-net0_srv                       00:14:4f:fb:ba:c2        1

DISK
    NAME             VOLUME                      TOUT DEVICE  SERVER         MPGROUP
    disk0            cs2-vol0@primary_vdsksrv_cs2

Conversion phase

Conversion phase runs on the primary control domain of the target system T2000. Here the created LDom is converted into LDom that runs the Solaris 10 by using the standard Solaris upgrade process.

But what if you do not want to upgrade to Solaris 10, what if you want to run Solaris 9 in the LDom. So, let's try to boot in Solaris 9.
First, shut down the source system (V120) and try to boot equivalent Ldom.

{t2000}/> ldm bind cs2

{t2000}/> ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     2G       0.3%  2h 34m
cs2              bound      ------  5000    1     1G

{t2000}/> ldm start cs2
LDom cs2 started

{t2000}/> ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     2G       0.2%  2h 47m
cs2              active     -t----  5000    1     1G       100%  3s

{t2000}/> telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connecting to console "cs2" in group "cs2" ....
Press ~? for control options ..

ok printenv
Variable Name           Value                          Default Value
ttya-rts-dtr-off        false                          false
ttya-ignore-cd          true                           true
keyboard-layout
reboot-command
security-mode           none                           No default
--- truncated ---
use-nvramrc?            false                          false
nvramrc
error-reset-recovery    boot                           boot

ok printenv boot-device
boot-device =           /virtual-devices@100/channel-devices@200/disk@0

ok boot

Sun Fire T200, No Keyboard
Copyright (c) 1998, 2010, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.30.4.b, 1024 MB memory available, Serial #83431980.
Ethernet address 0:14:4f:f9:12:2c, Host ID: 84f9122c.

Boot device: /virtual-devices@100/channel-devices@200/disk@0  File and args:
The file just loaded does not appear to be executable.
ok

The LDom cs2 does not boot. Let's continue with the conversion phase that upgrades from Solaris 9 to Solaris 10.

Do not forget to stop and unbind LDom cs2.

In the ldmp2v convert command use -i to specify path to Solaris 10 ISO image and use -d to specify directory with source image.

{t2000}/> ldmp2v convert -i /space1/iso/sol-10-u9-ga-sparc-dvd.iso -d /space1/p2v-images/cs2 cs2
Testing original system status ...
LDom cs2 started
Waiting for Solaris to come up ...

        Select 'Upgrade' (F2) when prompted for the installation type.
        Disconnect from the console after the Upgrade has finished.

Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.

Connecting to console "cs2" in group "cs2" ....
Press ~? for control options ..
Configuring devices.
Using RPC Bootparams for network configuration information.
Attempting to configure interface vnet0...
Skipped interface vnet0
Reading ZFS config: done.
Setting up Java. Please wait...
Serial console, reverting to text install
Beginning system identification...
Searching for configuration file(s)...
Search complete.
Discovering additional network configuration...

And now follow the process, select a language and answer all questions about networking, name services, root password, etc

Go for Upgrade on the installation type screen.

 -----------------------------------------------------------------
 -Solaris Interactive Installation 

  This system is upgradable, so there are two ways to install the Solaris
  software.

  The Upgrade option updates the Solaris software to the new release, saving
  as many modifications to the previous version of Solaris software as
  possible.  Back up the system before using the Upgrade option.

  The Initial option overwrites the system disks with the new version of
  Solaris software.  This option allows you to preserve any existing file
  systems.  Back up any modifications made to the previous version of Solaris
  software before starting the Initial option.

  After you select an option and complete the tasks that follow, a summary of
  your actions will be displayed.
  -----------------------------------------------------------------
     F2_Upgrade    F3_Go Back    F4_Initial    F5_Exit    F6_Help

Verify the profile screen and continue with Upgrade.

------------------------------------------------------------------
 - Profile -
  The information below is your profile which shows how Solaris software will
  be installed. It is a summary of the choices you've made on previous
  screens.
  =========================================================================
                Installation Option: Upgrade
                        Boot Device:          
              Root File System Type: UFS
                     Upgrade Target: Solaris 9 c0d0s0

                            Locales: Canada-English (ISO8859-1)
                                     U.S.A. (en_US.ISO8859-1)
                      System Locale: C ( C )

                           Software: Solaris 10, Core System Support
  ---------------------------------------------------------
     F2_Upgrade    F4_Change    F5_Exit    F6_Help

The upgrade continues and finishes.

 ----------------------------------------------------
 - Upgrading Solaris Software - Progress 

  The Solaris software is now being upgraded on the system using the profile
  you created. Upgrading Solaris software can take up to 2 hours (may be
  longer on servers) depending on the software you've selected, the
  reallocation of any space if needed, and the speed of the network or local
  CD-ROM.
  When Solaris software is completely upgraded, the message `Upgrade complete'
  will be displayed.

          Removing obsolete files in package: SUNWssad                                                                
	0  |||||||||||||||||||||||||| 100

Preparing system for Solaris upgrade
        - Environment variables (/etc/default/init)

Installation log location
        - /a/var/sadm/system/logs/upgrade_log (before reboot)
        - /var/sadm/system/logs/upgrade_log (after reboot)

Please examine the file:
        - /a/var/sadm/system/data/upgrade_cleanup

It contains a list of actions that may need to be performed to complete
the upgrade. After this system is rebooted, this file can be found at:
        - /var/sadm/system/data/upgrade_cleanup

Upgrade complete
Executing SolStart postinstall phase...
Executing finish script "patch_finish"...

Finish script patch_finish execution completed.
Executing JumpStart postinstall phase...

The begin script log 'begin.log'
is located in /var/sadm/system/logs after reboot.

The finish script log 'finish.log'
is located in /var/sadm/system/logs after reboot.

Launching installer. Please Wait...

Installing Additional Software
|-1%-----------25%------------50%-------------75%------------100%|

   Pausing for 30 seconds at the "Summary" screen. The wizard will continue to
   the next step unless you select "Pause". Enter 'p' to pause. Enter 'c' to
   continue. [c]

Press c for continue. 

Creating boot_archive for /a
updating /a/platform/sun4v/boot_archive
syncing file systems... done
rebooting...
Resetting...

Now the LDom reboots, select the terminal type and configure the keyboard layout.

Sun Fire T200, No Keyboard
Copyright (c) 1998, 2010, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.30.4.b, 1024 MB memory available, Serial #83431980.
Ethernet address 0:14:4f:f9:12:2c, Host ID: 84f9122c.
Boot device: disk0  File and args:
SunOS Release 5.10 Version Generic_142909-17 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
Hostname: cs2
Configuring devices.
Loading smf(5) service descriptions: 135/135
Reading ZFS config: done.

What type of terminal are you using?
 1) ANSI Standard CRT
 2) DEC VT52
 3) DEC VT100
 4) Heathkit 19
 5) Lear Siegler ADM31
 6) PC Console
 7) Sun Command Tool
 8) Sun Workstation
 9) Televideo 910
 10) Televideo 925
 11) Wyse Model 50
 12) X Terminal Emulator (xterms)
 13) CDE Terminal Emulator (dtterm)
 14) Other

Type the number of your choice and press Return: 3

- Configure Keyboard Layout -

  Please specify the keyboard layout from the list below.

  -To make a selection, use the arrow keys to highlight the option and 
   press Return to mark it [X].

      Keyboard Layout
    -------------------------
     [ ] Slovak
     [ ] Srpski
     [ ] Spanish
     [ ] Swedish
     [ ] Swiss-French
     [ ] Swiss-German
     [ ] Taiwanese
     [ ] UK-English
     [X] US-English
------------------------------------
    Esc-2_Continue    Esc-6_Help

And at the end, check that all services are running, and now you have virtual instead of physical.

Back to the main page