Creating Mac OS X Lion Installation media without having purchased Lion @ App store
Summary
It is possible to create a bootable Mac OS X Lion installation disk without having purchased the OS from Apple's online App store. I am _not_ talking about a recovery system, but instead of a USB stick or DVD or partition from which the OS can be fully installed or reinstalled onto a computer with an empty disk and no Internet connection. Skip to the process if you will.
All the guides I found on the Internet about creating Lion installation USB sticks or DVDs relied on the premise that the user has purchased their OS on Apple store and the process involves its re-download. I have not purchased Mac OS X Lion on App Store so I am unable to re-download it from there without buying again. The other alternative, about 3x more expensive, would be to buy Apple's Mac OS X Lion USB drive. Not going to happen either.
Background
Apple's latest Mac OS X Lion has a new model of installation and recovery. They have stopped distributing their OS in physical media. Instead people can now purchase Lion on the Apple store, which is delivered in a download. Because of this, a new mechanism to allow for operating system recoveries or reinstalls has been implemented. This new model of recovery relies initially on a 650MB bootable hidden recovery partition, labelled 'Recovery HD' containing what I would call a 'Recovery Mac OS X'. From here users can use recovery tools such as Disk Utility or a terminal, recovering a time machine backup or reinstalling the OS.
In fairness, creating physical installation media is not necessary because hardware released after Lion came out can boot in recovery mode from the Internet. Then a fully restore or installation into an empty disk can be done entirely from the Internet. Older hardware still compatible with Lion can do the same via a boot volume created by the Lion Recovery Assistant tool (same content as in the recovery partition). Both methods have the downside of requiring an Internet connection and time to download about 4GB of data.
I like their new installation and recovery method. It ends up being more flexible than its predecessors. The ability to install Lion without any type of media is great!
But some people - including me - would like to have some sort of physical media with from which a full OS install could be made. Even though I did not use them more than once, I kept the original install discs of 10.4 Tiger, 10.5 Leopard and 10.6 Snow Leopard.
Creating physical Lion installation media is feasible and fairly easy too. It is likely that the number of guides over the Internet about creating Mac OS X Lion installation media has reached triple digits by now.However all the guides I have read (admittedly not all) assume that the OS has been bought at the Apple App Store. They all rely on extracting that InstallESD.dmg by re-downloading Lion from the App Store, eventually by making use of the command + click modifier to force re-download.
This excludes all Apple buyers that obtained their latest operating system by buying a macbook or imac computer recently. Like me.I legally own a copy of Mac OS X Lion because it was pre-installed on a new laptop, which makes it legal but not purchased at the App Store. When I go to the App Store, Lion does not appear as a 'purchased' product under my apple id (makes sense). Therefore if I wanted to re-download Lion from the App Store I would have to buy it again... Not going to happen. Apple also sells Lion in physical media (a USB stick) but it costs about 3x the price of standard ,online install Lion... Not a good solution either.
Someone somewhere on the Internet has claimed that once the OS jumps a minor version, it would show up to download. I could not reproduce this as the OS got updated from 10.7.2 to 10.7.3.
It just does not seem fair that I do not get the same features as if I had purchased Lion on the App Store.
How to: Create a Mac OS X Lion installation volume without having purchased it from the App Store
The limitation of not having bought Lion on the App Store is not being able to re-download the OS's installer, specifically from where it is possible to extract InstallESD.dmg - the 3.6GB image with the full install tree. This can be achieved by running the Internet recovery process to reinstall OS X Lion on a blank disk.
Step 0. Before beginning
Ensure that what you are doing is legal and under Apple's terms and conditions. My laptop came pre-installed with Mac OS X Lion so I am elligible to perform this bare metal recovery.
It is relevant to keep the operating system installed on the internal disk functional. This process requires no changes to the internal disk. In fact if you don't have a working OS on the internal disk, you'll need a second Mac to get this done.Step 1. Prepare the target disk
Find a 20GB+ external disk, USB or Firewire (Thunderbolt should work too) with disposable data (all data on the external disk will be deleted, of course). Use Disk Utility to create a new GUID partition scheme with one partition (labelled with whatever you like, but preferably different from the internal disk's label) formatted with 'Mac OS Extended (Journaled)' file system. Remember to apply the changes.
This will be your target install disk! I have labelled mine 'Lion Install'.Step 2. Go into Internet Recovery mode
Ensure that you have Internet connectivity via Wi-Fi or Ethernet and reboot (erm, remember to memorise the remaining steps or print them or view them in another device).
Connect the external disk and shutdown.
Hold cmd-r while pressing the power button to startup the computer into Recovery mode. Release cmd-r after the apple symbol appears.
You should have booted into recovery mode which has no user accounts, a grey background and starts with the 'Choose your language' screen.
This simpler OS has been loaded from a hidden partition of the internal disk or directly off the Internet. How cool is that??Step 3. Initiate Lion reinstallation into the target media
- Choose the option 'Reinstall Mac OS X Lion' and select your external disk as the target (in my case labelled 'Lion Install'.
Soon after this you will be asked to accept an EULA and Apple will verify your eligibility to perform this installation. If Apple says you're good to go, which should be guaranteed on any hardware released after Mac OS X Lion, the download process begins.
The recovery program mentions that 'your computer will reboot automatically': this is important, because the reinstallation process requires no interaction and will happily stop after the OS is fully installed on the target media, at which point the files we require will have been deleted. Step 4. Interrupt the installation process
- The installation process must not be allowed to finish. I ensured I was present when the download finished and the computer rebooted. At that point, I hijacked the process and forced the computer to boot into the internal disk's OS instead of the external disk's installer program. Simply disconnecting the external disk from the computer immediately after it reboots should suffice to startup into the internal disk's OS.
If the computer is allowed to reboot into the installer program, that is fine, but a reboot must be forced before the installation ends, because at that point the installer program is deleted, which is exactly what we're after. Step 5. Extract InstallESD.dmg
- Having booted back to a functional OS X, connect the external disk onto which Lion Internet Recovery was initiated and it should have the following files:
$ ls -lR total 0 drwx------ 15 pjvenda staff 510 12 Dec 14:10 Mac OS X Install Data/ ./Mac OS X Install Data: total 7448928 -rw------- 1 pjvenda staff 13324 12 Dec 12:42 InstallESD.chunklist.partial -rw-------@ 1 pjvenda staff 3788832912 12 Dec 14:10 InstallESD.dmg -rw-r--r-- 1 pjvenda staff 916 12 Dec 12:42 InstallESD.dmg.partialState -rw-r--r-- 1 pjvenda staff 182 12 Dec 14:10 MacOSXInstaller.choiceChanges -rw-r--r-- 1 pjvenda staff 10884 22 Jul 05:44 MacOS_10_7_IncompatibleAppList.pkg -rw-r--r-- 1 pjvenda staff 435 12 Dec 14:10 OSInstallAttr.plist -rw-r--r--@ 1 pjvenda staff 863920 6 Oct 14:07 boot.efi -rw-r--r-- 1 pjvenda staff 408 12 Dec 14:10 com.apple.Boot.plist -rw-r--r--@ 1 pjvenda staff 6306 12 Dec 14:10 ia.log -rw-r--r-- 1 pjvenda staff 786 12 Dec 14:10 index.sproduct -rw-r--r--@ 1 pjvenda staff 24087081 6 Oct 14:08 kernelcache -rw-r--r-- 1 pjvenda staff 618 12 Dec 14:10 minstallconfig.xml
Locate and keep the file InstallESD.dmg by copying it to somewhere safe.
InstallESD.dmg holds the complete Mac OS X Lion installation program and getting to it was the reason to execute this process.
Target disk before preparation
Creating the GUID partition scheme on the target disk
Creating the Install partition on the target diskThere it is, we were successful at obtaining InstallESD.dmg legally without having bought Lion at the App Store, simply by initiating an Internet Recovery and interrupting installation after the program had downloaded the OS.
The rest of the process of creating a bootable full Lion installation media is the same as in any guide on the Internet from after the step of re-downloading the OS from the App store.
Burning InstallESD.dmg image into the prepared target disk
Contents of Lion installation media prepared from InstallESD.dmg imageAbout the Internet Recovery process
-
Luckily Apple's implementaion of Internet recovery is simple:
- Download OS X Installer onto target drive;
- Reboot from the target drive and run the installer;
- Delete the installer just before rebooting again into the newly installed OS;
So, between steps 1) and 2), what is left on the target disk is actually the full Mac OS X Lion's installer program.
$ ls -lR total 0 drwx------ 15 pjvenda staff 510 12 Dec 14:10 Mac OS X Install Data/ ./Mac OS X Install Data: total 7448928 -rw------- 1 pjvenda staff 13324 12 Dec 12:42 InstallESD.chunklist.partial -rw-------@ 1 pjvenda staff 3788832912 12 Dec 14:10 InstallESD.dmg -rw-r--r-- 1 pjvenda staff 916 12 Dec 12:42 InstallESD.dmg.partialState -rw-r--r-- 1 pjvenda staff 182 12 Dec 14:10 MacOSXInstaller.choiceChanges -rw-r--r-- 1 pjvenda staff 10884 22 Jul 05:44 MacOS_10_7_IncompatibleAppList.pkg -rw-r--r-- 1 pjvenda staff 435 12 Dec 14:10 OSInstallAttr.plist -rw-r--r--@ 1 pjvenda staff 863920 6 Oct 14:07 boot.efi -rw-r--r-- 1 pjvenda staff 408 12 Dec 14:10 com.apple.Boot.plist -rw-r--r--@ 1 pjvenda staff 6306 12 Dec 14:10 ia.log -rw-r--r-- 1 pjvenda staff 786 12 Dec 14:10 index.sproduct -rw-r--r--@ 1 pjvenda staff 24087081 6 Oct 14:08 kernelcache -rw-r--r-- 1 pjvenda staff 618 12 Dec 14:10 minstallconfig.xml
Problem solved!
My initial plan to obtain installation media from the Internet recovery method was significantly more involved. It consisted of doing an Internet based install onto an empty disk while routing the Internet connection via another host which intercepted all traffic. By analysing this traffic, I would hopefully be able to filter important payloads (hopefully most files would be downloaded via plaintext HTTP, but SSL mitm was also within reach).
At some point I reckoned that a full installation tree would at some point be created that could be reused on bootable media, ideally including the InstallESD.dmg image.
Fortunately, soon after I started analysing data (which provided some very interesting results), I realised that the simplest scenario was that Internet recovery simply downloaded the installer onto the target media and ran it from there. So I did a couple of quick tests and, sure enough that was the case.
Nonetheless I had the chance to analyse the network traffic exchanged between laptop and Apple's servers, which revealed the most interesting insights into the process.
Apple's approach to OS distribuition and installation has always been fairly unrestrictive from a technical point of view. There are no serial keys, no activations, no obvious applications of DRM, etc. I reckon the risk they take in facilitating illegal copying of their OS is far outweighted by hardware limitations and especially the pricing model of Mac OS X. OS X is very cheap by any standard and even more so considering how technically good it is. I also think that their legal customer base is a far better investment into the business than working against pirates: that they already know it is an arms-race, that companies tend to lose consistently. Conclusion / References
All that's left is to provide a number of the references I used from the Internet to do this work and a few final remarks. I hope it's been informative and useful as it was for me.
While researching for this, I came across hundreds of online blog posts, news articles, original and copied howtos, copies of copied howtos, etc. After writing this post (believe it or not) I found one link written by somebody that had the same idea about extracting installation media from Internet recovery. Only one.
Some information about InstallESD.dmg's integrity (mine is below):
$ md5deep InstallESD.dmg 412cee9c4c77c04c9c8489c363a7e2e4 /Volumes/New HD/Mac OS X Install Data/InstallESD.dmg
Resources about Lion recovery disk assistant, Recovery mode and Internet recovery
- Hands on with lion recovery disk assistant
- Hands on with lion recovery mode
- Lion Internet Recovery
- OS X Lion: About Lion Recovery
- Lion Recovery Disk Assistant
And finally, arstechnica provides the best Mac OS X guides. These are the nerdiest most detailed guides I've ever seen about an operating system. About all 7 versions of Mac OS X in fact. And they're great!
