For the first time in 16 years (nearly 17, in fact), I am travelling back to London for a 10-day stay. I shall be doing lots of tourist stuff in London, with side-trips to Dublin and my old home town of Gillingham (plus Chatham/Rochester). I’m looking forward to it very much -hope the cats and wallabies won’t mind putting up with the house-guest animal-sitters we’ve arranged for the interim, though! The flight leaves in about 10 hours (doesn’t time drag when you really don’t want it to!), and I’ll be touching down in London at around 7AM (GMT) on October 1st. I won’t see these sunny shores again for a further three weeks after that.
I will confess that I’ve tended to stick with ye olde exp and imp (export and import) utilities because I know them reasonably well and they do the job. However, the writing has been on the wall for both of them for quite a few years and you’re supposed to have been progressively switching over to using the all-new, all-dancing expdp and impdp replacements. For the stuff I use these utilities for, however, I’ve managed not to need the new versions… until today. We recently upgraded all our databases to 11.2 from 10.2.0.4 -except one, which got converted to 11.1 months ago- and when you try to do a traditional export against the 11.1 database, it terminates with an error:
. exporting materialized views . exporting snapshot logs EXP-00008: ORACLE error 1455 encountered ORA-01455: converting column overflows integer datatype EXP-00000: Export terminated unsuccessfully
Hunting around the subject, it seems there’s a bug in late versions of exp that make this to be expected, which is a bit of a show-stopper. However, those same bug reports all unanimously declare that the new data pump version of export does not suffer from the same problem. So, finally, it’s time to switch to using the new “dp” utilities after all!
The trouble is, however, that we’ve used the traditional utilities to pull production data over to development environments. Whilst logged onto the dev server, for example, I’d do something like:
exp "sys/password@proddb as sysdba" file=prod_exp.dmp full=y direct=y
…and the export file, called “prod_exp.dmp” would be produced in whatever directory I happened to be sitting in when I typed the command on the dev server. Production data was thus dumped out to a dev server directory, thanks to the use of a tns alias in the connection string bit of the command. Export runs locally on my development server, but knows to connect to my production database to get the data.
Unfortunately, this is not how the Data Pump utilities work. They always run on the server and never on the client. Which means that, other things being equal, the output file is always on the server, too -even if the utility is invoked remotely!
So, for example, you could issue this command whilst logged into the development server:
expdp "sys/password@proddb as sysdba" directory=dumpdir file=prod_exp.dmp full=y
…and the thing would work just fine. But when you then navigate the file system of your development server, you will not be able to find a file called “prod_exp.dmp” anywhere! And that’s because the file has been written on the production server, in whatever location the “dumpdir” directory object is pointing to on that server. You can invoke data pump export remotely, in other words, but all the action and all the output will only ever happen on the remote database. If you want the dumpfile to be sitting on your development server’s hard disk at the end of the exercise, this sort of syntax won’t achieve that, I’m afraid!
But this, happily, will:
expdp "sys/password as sysdba" network_link=prodserver directory=dumpdir file=prod_exp.dmp full=y
With this syntax, you’re now connecting locally -that is, to the development database. That therefore means you’re going to be outputting to whatever part of the development server’s file system is referenced by the dev box’s “dumpdir” directory object. But, the new parameter network_link means that the development database doing all the work will know to connect to whatever database is at the end of the “prodserver” database link in order to get its data (so, obviously, for this syntax to work, I must first have issued a command such as create database link prodserver connect to system identified by password using ‘dbprod’). This syntax -a local connection string, but an extra network_link parameter- therefore achieves what the old export command did: production data “pulled” across to my development server and ending up as a dump file on my dev server’s hard disk.
In a word, the trick is not to invoke the data pump utility with an old-fashioned “@somewhere” to achieve a ‘remote connection’, but to use one of the new parameters the new utility makes available to you to have the data ‘pumped’ from a remote location. Easy -though not perhaps entirely obvious!
When I have to use Windows 7, the first thing that really puts me off is the dolls-house-y login screen. It’s so clutzy as to be annoying, and I didn’t realise until today that it can be gotten rid of. Which makes this an essential thing to do on any Windows 7 system you are unfortunate enough to have to use. You can do it by poking around in the registry, or by running this svelte utility.
It’s a trivial one, really, but if you disable Fedora’s Network Manager in order to be able to create KVM’s bridged network connections as mentioned in the last post, you will probably find that every time you now launch Firefox, it will start in ‘Work Offline’ mode. This is a pain if you want it to automatically re-open tabs you had open when you last shut down!
The fix is quite simple, however. Type about:config in Firefox’s URL bar and agree to be careful! Filter for the word toolkit. Find the item called toolkit.networkmanager.disable and double-click it so that it becomes set to ‘true’. Shutdown Firefox and then re-start it: this time, it will correctly identify that you are not working in offline mode.
No matter how much I’d rather not run Windows, there are times I have to -principally because work insists on using Checkpoint’s VPN software for which no Linux client exists. So, when I want to work from home, I have to connect to the office in a Windows 7 VM and use tools like Putty or NX Client to manage the various work PCs and servers (all of which are now, ironically enough, Linux boxes). It’s a pain, and if anyone knows how to use openssl or openvpn to connect to a Checkpoint VPN1 SecuRemote VPN, I’d love to be let in on the secret!
Anyway, a Windows VM is essential -and for years I’ve been using VMware Workstation to run one. I paid my US$189 several years ago (interesting to see that price hasn’t budged a cent since!), and I’ve always found it just a fraction more intuitive and well-behaved than, say, Parallels or VirtualBox. VirtualBox has the distinct advantage of being free, of course -and is now owned by Oracle, which seems to be continuing development efforts quite nicely. But the fact remains, I’ve never really warmed to it: I’m just a VMware Workstation fanboy, I guess! (I stress the Workstation in that product name, however: I’ve never liked the zero-cost VMware Server product, since it seems to require klunky web-based interfaces to achieve anything much. On the other hand, I got VMware’s ESXi bare metal virtualisation installed at work and it’s never missed a beat, running all of our Oracle dev and test environments extremely well. (Though I will point out the irony that ESXi lacks a native Linux client and I am therefore forced to use a VMware Workstation VM running Windows 7 on my Linux-running Work PC just so I can manage the ESXi box, which is running a Linux kernel! Go figure!!)
Anyway, I have dabbled in various virtualization technologies in my time, both hypervisors and host-based ones. Citrix Xen Server, for example, was a good hypervisor, but a little inflexible to manage as compared to VMware’s ESXi similar offering. Microsoft’s Hyper-V was certainly slick, but I had terrible performance issues in the presence of an Nvidia graphics card -and I wasn’t the only one. See, for example, this page of complaints. It’s been a year since I ran any Windows OS natively, either at home or at work, so I’ve not tried Hyper-V since -but according to this Wikipedia article -see the Graphics issues on the host paragraph-, the graphics problems persist (but who trusts Wikipedia?!). Funnily enough, using the Xen virtualization features in Red Hat Enterprise Linux 5.5 is very similar to using Hyper-V: both installations slot ‘underneath’ your physical host’s OS install, turning it, effectively, into a virtualized guest (albeit a “parent” one). The moment Xen goes in, for example, a uname -a command in a terminal will reveal that you’re no longer running a standard linux kernel, but a special “xenified” one (which poses all sorts of problems when you are running proprietary graphics drivers which expect only ever to have to compile against ‘standard’ kernels, for example).
But there’s been one virtualization technology I’ve not used before now: KVM (stands for ‘kernel-based virtual machine’, not ‘keyboard, video, mouse’ as in a KVM switch!). As it’s name suggests, it’s built into the Linux kernel -and has thus been shipping as a standard part of Red Hat Enterprise Linux since 5.4 days (around about this time last year, basically). Fedora 13, too, includes KVM ‘out of the box’ (as do a lot of other distros, including Ubuntu). It’s not installed or enabled by default, but it’s right there, in the repositories, just waiting for a simple one-line installation command. What’s more, when you do install those KVM packages, unlike when you install Xen, you don’t end up altering the host OS’s status: uname -a still outputs exactly the same as it always did, in other words. This is simply because (the clue is in the name!) the hypervisor is already built into your existing kernel, so you don’t need a special kernel to make use of it. Not distrubing the host’s kernel in this way makes installing things like Nvidia graphics cards (see posts passim!) not a drama, and is thus a Very Good Thing™.
Installing KVM on Fedora 13 is simple:
su - root yum install qemu-kvm virt-manager virt-viewer python-virtinst libvirtd
Once the libvirtd daemon is running, you can fire up Applications->System Tools->Virtual Machine Manager. Click the ‘new virtual machine’ icon in the top-left and then, basically, follow the prompts of the ensuing wizard to build your first virtual machine. And that’s about it! It’s really incredibly simple.
The only tricky bit comes if you want your new VM to look like an independent host on your network. That requires “bridged” networking, which doesn’t exist until you manually create it (it would be nice if someone was to develop a graphical tool for achieving this!) Worse, bridged connections don’t work with the fancy new ‘network manager’ way of doing networking that Fedora (and Ubuntu, actually) has adopted. So, if you want bridged connections for your VMs on those distros, here’s what you have to do:
As root, issue the command
Find the eth0 item and click the Edit button. Switch ‘Controlled by NetworkManager’ off, ‘Activate device when computer starts’ on and ‘Allow all users to enable and disable the device’ to on. Click OK and then File->Save to preserve the changes.
Now you’ve just disabled the new-fangled Network Manager, so you have to make sure the old-fashioned network control starts at each reboot:
chkconfig network on
You now create a new bridge network interface by issuing the command:
Add the following lines to the new text file thus created:
DEVICE=br0 TYPE=Bridge BOOTPROTO=dhcp ONBOOT=yes DELAY=0
The typing here has to be precise -it’s very case-sensitive, for example, so ‘bridge’ as a “type” entry won’t work, where ‘Bridge’ will!
You now tell the eth0 interface that it is to be bridged. Do that by issuing the command:
Add the following line to the file’s existing contents:
Now you can re-start the network so the new configuration is activated:
service network restart
Note that your physical PC now connects to the rest of the world via the br0 interface, which happens to know (thanks to the edits above) that the physical eth0 is responsible for handling its traffic. But, as far as your physical PC is concerned, eth0 is actually a non-active interface in its own right. Br0 takes over that role, though functionally it all amounts to the same thing.
Finally, the trouble with this setup is that br0 is a physical network interface, seen and used by your physical PC. But that’s not much use to a virtual guest machine! So now we have to add a virtual interface to our physical interface -and that’s a job for a utility called tunctl. That utility probably needs to be installed to start with, so the relevant command is:
yum install tunctl
Next, issue these commands in sequence:
tunctl -t tap0 brctl addif br0 tap0
The first command creates an interface called “tap0″; the second command says it’s to be a virtual representation of the ‘br0′ physical network interface.
Once all that’s done, you can go back to virtual machines you’ve already created and add new network hardware -this time, a bridged interface will be available to you. You can remove the previous NAT one, if you like (or simply disable it within the guest OS). New guests can be created, obviously, that use the right sort of ‘let me at the world!’ interface from the get-go.
One final bit of advice as far as KVM experiments are concerned: having to start libvirtd manually before you begin is a bit of a pain. If you want to ensure libvirtd is started automatically whenever your PC reboots (and thus avoid the need to run it manually in a terminal session), just go to System->Administration->Services and click the libvirtd item, then the [Enable] button. Once it has a green check mark next to it, it’s scheduled to auto-start.
Apart from the bridged network issue, however, KVM is an absolute doddle to install, configure and run. Performance in the Windows 7 virtual machine I use is excellent -the only drawback is that the virtualized graphics hardware isn’t up to displaying the fancy, semi-transparent Aero interface. But that’s not much of a problem for me. I miss only two other things from my VMware Workstation days: movie capture and snapshots. KVM provides a menu option to take a still screen capture of your guest, which is fine. But it doesn’t have the option to capture screen motion/activity as a movie (this is something the freebie VMware Server product also lacks). There are workarounds, of course (yum install recordmydesktop puts a movie-capturing application at your disposal which will more-or-less do the job), but it would be nice to have the functionality built-in.
The lack of snapshots is a bit more of a drama, to be honest. There are snapshot capabilities that can (probably!) be used, thanks to the use of the qcow2 virtual hard disk format -and you’re supposed to be able to drop into a terminal and issue a qemu-img command that will do the necessary. But I haven’t tried it, I believe it only works for a VM that’s been shut down… and in any case, it all sounds a bit tricky at this stage. I’m really more after a ‘take snapshot’ button in the Virtual Manager window, to be honest! Meanwhile, there is a simple button to do VM cloning (though, again, the VM has to be shut down for the duration), which will do me well enough in the meantime. But this is certainly an area of VM management that it would be nice to see some development on in the next year or two!
Other than those slight niggles (oh, there’s one more: no drag-and-drop between host and guest), I think KVM is an excellent virtualization platform, and my trusty copy of VMware Workstation has remained firmly on the bookshelf for this PC’s recent rebuild.
It’s been a long-running saga, so let me first summarise:
- Bored with Ubuntu and thinking that I’d prefer something more Red Hat-ish, because that’s what we now run our Oracle databases on at work
- Therefore, install Fedora -and quite enjoy it
- Except that CD ripping is found to be broken and Stellarium refuses to display properly using the open source drivers
- Installing “proper” ATI drivers is impossible, because ATI don’t support the xorg version used by Fedora
- Because K3B is such a good CD ripper (and burner), I investigate KDE-based distros, but can’t stand any of them for long. Discover, however, that living with a mix of KDE and Gnome apps isn’t actually a bad thing but rather gives you the best of both worlds.
- CD ripping resolved, therefore, by running a Gnome distro but with some KDE apps installed (like K3B). Still leaves the Stellarium/ATI Graphics problem…
- So I install OpenSuse 11.3 -and hate every moment of it! Stellarium works and the ATI drivers install, but the distro sucks in lots of little ways (in addition to the litany mentioned last time, I should add that discovering sshd is not enabled by default was a bit of a surprise!)
- After two days with OpenSuse, I reverted to Ubuntu 10.04: Stellarium worked, K3B still does CD ripping. But I’m still bored with Ubuntu. Worse, I find some of its control changes now annoying: I want my Log Out option to be under my System menu, thanks all the same, not a little button tucked over the right-hand side of the top panel. Also, I know I can switch the windows close/minimise/maximise buttons back to the right-hand side of the window, but I don’t see why I should have to -and I know that the developers are cooking things up for 10.10 that will expect the controls to be on the left, where they put them without much consultation. All a bit Microsoft-ish if you ask me.
So, having been through just about every vaguely-plausible distro and desktop environment out there and received only disappointment for his pains, what’s a boy to do??
Buy an Nvidia graphics card is the answer!
Actually, I happened to have one sitting around in a cupboard, so I whipped out the ATI monster and slipped it into the PCI Express slot in its stead. It sounds a tad drastic but it means I’ve been able to re-install Fedora, have no graphics problems, Stellarium works, K3B works, menus are where I expect them to be, ditto windows controls …and it all behaves very like a Red Hat Enterprise distro, so I feel at home at work, if you get my drift.
Everything is thus tickety-boo …if you overlook the minor matter of having to trash more than $300-worth of ATI graphics card to get there. I have said it before, but it bears mentioning once again: ATI (i.e., AMD) should be ashamed of themselves. The quality of their Linux drivers, the convoluted installation process and the tendency of any system fitted with them to crash or otherwise have “interesting” graphical glitches happen at random moments -it all adds up to an abysmal way of doing business. I won’t say that Nvidia are completely blameless (their installation procedure isn’t exactly brilliant -on Fedora, at least, you have to manually disable the open source drivers by editing the grub.conf file before the installation will succeed, which isn’t what I’d call terribly user-friendly), but they make ATI look like a bunch of incompetent amateurs by comparison.
Funnily enough, my PC at work -which I try to keep more-or-less in synch, distro-wise, with my home PC- experienced exactly the same grief with ATI drivers, even though it was running a copy of Centos 5.5 (which is a clone of Red Hat Enterprise Linux, which ATI claims to be a fully-supported distro). I gave in there, too, and did actually go out and buy a $50 Nvidia Geforce 8400GT… which also immediately made all my graphical and stability problems melt away. So it seems to me to be a generic “feature” of ATI cards that they screw up most Linux distros!
I never had that problem with Ubuntu with the ATI card, I will admit -but then I never tried to install my own ATI drivers in that distro, either. Clicking the ‘activate proprietary drivers’ button is all it takes in Ubuntu (and is all it really ought to take anywhere else, Nvidia and ATI included), but I have no idea which drivers it actually causes to be installed. Had such an ‘automated installation’ feature been available in Fedora, I guess none of this saga would have arisen -but ATI haven’t exactly come to the party in terms of supporting Fedora nearly six months after its release, so I still say it’s more ATI’s fault than anyone else’s.
Anyway, I’m a happy Fedora man again -and discovering the joys of KVM virtualisation for the first time (very impressive, is the short version). And if anyone wants a $300 ATI graphics card, feel free to ask.