Preparing Windows EC2 instance for AMI creation

Review your EC2 settings. Start>All Programs>EC2Service Settings

Clean your disks. Start>All Programs>Accessories>System Tools>Disk clean

Clear out the bytes from deleted space. sdelete –c C:  and sdelete –c D:  (You can get sdelete from the sysinternals site).

Log off the remote desktop.

Using ElasticFox, right click the running instance and  select “Bundle into an AMI”. The S3 bucket name should be globally unique. Note the bucket name and image name carefully (you’ll need it for the registration step). Keep checking the Bundle Tasks tab of ElasticFox to see when it finishes the bundling.

After that, use the “AMI and Instances” tab to register your new AMI. The manifest file name is of the pattern “bundle/imagename.manifest”.

Using the Windows Branded Live site.

* Update - After releasing Outlook.com in 2012, Microsoft has stopped its Live service and custom domains. If you had emails setup with a custom domain, they will continue working but you cannot add new accounts or manage passwords with an admin account.

Microsoft Live has a service (formerly called Custom Domains) that allows you to create a Window’s Live hosted site. Its called the Partner Live program. (https://partners.microsoft.com/partnerprogram/welcome.aspx)
However trying to locate the free service proved difficult. So I went back to the article in CODE magazine (http://www.code-magazine.com/Article.aspx?quickid=0804112) So in short logon to admincenter.live.com to get started!
Now lets try Google’s Apps. Hmm… no way from the search page to even find it. Found the address on one BLOG page. http://www.google.com/a 
Both require that you must modify your DNS records.

Installing Moodle on Amazon EC2

To reduce the disk cost, I decided to create a separate hard disk volume that you can attach to any Windows AMI instance. Any changes that you make to the Moodle are saved to this separate volume. So when you shutdown your instance, the changes are persistent. The only disadvantage is an extra step of attaching the hard disk after the machine is running. The advantage is that you spend less. 1GB of EBS = 0.10 cents per month, vs 3GB AMI = 3x0.15 cents =0.45 cents per month. It is also easier to create snapshots of the disk in this manner. Just right click the volume and create a snapshot. The snapshot is stored on S3.

I’ve created one snapshot of the initial disk – which can be used for new Moodle installs.

Starting up Moodle.

1. Start any Windows image (I’ve tried it with ami-ec698d85 Server2003r2-i386-WinAuth-v1.02 and ami-e5698d8c Server2003r2-i386-Win-v1.02) Keep in mind that the Auth server is double the money compared to the vanilla windows.
   IMPORTANT: After you right click to launch an instance; In the dialog box, set the Availability zone =”us-east-1c” for the machine. Your volume is located in that zone. Data transfer is free if machine and volume are in the same zone.
2. Right click the instance and “Show console output”. Wait till you see a message “Message: Windows is Ready to use”. This usually takes about 10 minutes from when you start.
3. Once the instance is running, in ElasticFox go to tab “Volumes and Snapshots”. Right click and attach volume (vol-4b9d7922) as “xvdg”.
   The disk is automatically mapped to “E” if you do this after the ready message in the “Show console output”. (The volume initially contains an unzipped version of the Windows package installer from moodle.org.)
   1. If you do it before “Windows is ready to use”; you must manually change the drive letters.
      (Start>Administrative tools>Computer Management)
   2. Click Storage>Disk Management
4. Right click the instance and “Get Administrator password”.
5. [optional] If you are going to use an Elastic IP, do that now.
6. [optional] Modify DNS on your domain nameserver to point to this new instance.
7. Right click the instance and “Connect to Public DNS name”.
8. Tip (optional) – when you use your browser, you are in Internet Explorer Enhanced Security mode. To turn off Internet Explorer Enhanced Security:
9. http://www.windowsreference.com/windows-2003/how-to-disable-internet-explorer-enhanced-security-configuration-ie-esc-in-windows-server-20082003/
10. Open “E” drive
11. Edit the files E:\server\moodle\config.php (this is not needed when you haven’t configured Moodle for the first time. The file is created during Moodle configuration.)
12. Click Start>Run and enter “net stop w3svc”. This stops the IIS server.
13. Click “Start moodle.exe”
    It should start up. If there is a question about reconfiguration, press 1 and enter to proceed.
    The program will continue running in a DOS box with the message “Run STOP Moodle…..” or a blank DOS box.
    
14. Click “My public IP” to get the IP address of your instance. (If you haven’t turned off Internet Enhanced Security, you will see a number of warning dialog boxes which you can ignore.)
15. From your own PC, use a browser to open that instance. You should see the Moodle page.
    Once you are done, you can close the Remote Desktop window (do not logout, just close the window).

Stopping.

Right click the instance and “Connect to Public DNS name”.
Click “Stop Moodle.exe”.

Shutdown the server.

After the server is terminated, delete any Elastic IP addresses that you had attached.

Appendix: Editing Moodle’s config for EC2.

Once you have finished installation, the setup of moodle hard binds the name of the server in the setup files. You must change this if you want to access Moodle from outside the AMI.

Pre-startup - There are two files that should be modified.

1. D:\server\moodle\config.php
Change the line
$CFG->wwwroot = 'http://nooradel';
to
$CFG->wwwroot = 'http://<new public DNS name or elastic IP>';

2. [Optional] D:\server\apache\conf\httpd.conf
Change the line
Servername = localhost:80
to
Servername = <new public DNS name or elastic IP>:80

Helpful links

Setup instructions for Moodle on Windows

http://docs.moodle.org/en/Complete_install_packages_for_Windows

Command line for Amazon EC2.

http://docs.amazonwebservices.com/AWSEC2/2008-12-01/DeveloperGuide/

If you want to find out your own IP address inside a AMI (http://169.254.169.254/latest/meta-data/public_ipv4)

For more details see http://docs.amazonwebservices.com/AWSEC2/2008-05-05/DeveloperGuide/index.html?AESDG-chapter-instancedata.html

Setup the environment to run EC2 command line control tools. (See section on Configuring your workstation).

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1827&categoryID=100

Key solutions

The key “amazon_ec2.pem” must be used instead of “amazon_ec2.ppk” to retrieve the Windows Administrator password.

Setup in Tools:

• SSH Key template: ${home}\ec2-keys\${keyname}.ppk
• EC2 Private key template: ${home}\ec2-keys\${keyname}.pem

You can re-download the public key by going to the web page. However the private key is not stored on the server. Keep it carefully.

• Amazon_ec2.ppk (putty version of key pair amazon_ec2. )
• Amazon_ec2.pem (EC2 version of key pair amazon_ec2.)
• PK-0…..pem (Amazon account private key)
• CERT-0…pem (Amazon account public key)

Microsoft Windows Azure and Geneva and Bulldog

Over the last couple of weeks, I’ve seen a lot of information about various products from Microsoft.

Azure

A distributed operating system for the “cloud”. The idea here is that you write to the Azure API on your desktop. Run / debug it on the desktop. When you deploy it to Azure, Microsoft will run this on “server” fabric. You don’t have to worry about individual servers, scalability etc. Since your code could end up running on different servers due to load balancing, all state information must be stored centrally. Azure provides a distributed storage and a distributed SQL Server data storage to address these requirements. Azure is in CTP. You can download the SDK and play around on your own PC. If you want to deploy to Azure, you’ll need a invitation token (these were given out in PDC 2008). For the development machine, you need Windows 2008 or Vista, Visual Studio 2008.

MDM BullDog

Microsoft acquired Stratature’s MDM product (eDM). This is a robust MDM tool that is in its 4th generation. Profisee is the firm spun off by ex-Stratature employees. Ian from Profisee gave an excellent introduction to this tool. What I liked was its out of box Web UI, Workflow, Business Rules and a solid BI focus.

Geneva

This gets exciting. In the Identity space, there is a shift from applications managing your identity (uid/pwd) to claim based authentication/authorization. To use an application (RP-relying party), you need to provide a token with the appropriate claims from a trusted STS. The IP (Identity provider) STS (Secure Token Service) issues you a token with the claims, after you logon to the STS. It signs the token with a private key. The RP can decode the token and extract the claims to decide whether the user is allowed to use the application. So why is it exciting?

The IP can be Yahoo OpenID, Google, Microsoft Live or even a self issued IP. Your app can choose to trust all these STS’s; or a better way is to trust a single STS and ask the user to get a token from this STS using the token from the first STS. This provides a transparent way to federate identity. Everything is service based that makes it easy for active / passive logon. Geneva provides 3 components – Geneva server which is a Windows based STS, Geneva Cardspace which is an identity selector and Geneva framework which is an API for applications that need to use Geneva. The nice thing is that these are not bound and are based on open standards (WS-Trust, SAML, WS-Security). While searching for this, found that WCF provides a wsFederation binding that can handle claim based authentication in WCF.

Cardspace Identity selector

Enterprise case

Internet case

Using external STS

Multiple STS

Another growing trend - Virtual PCs

It is becoming easier to setup and run virtual PCs. The software from Microsoft (Virtual PC) and VMWare (player / server) are free. There are also a couple of tools to convert a physical machine to a virtual machine.

So why virtual?

For one, its easier than keeping a physical machine around. The "overhead" of running in a virtual machine is not too much - about 10-20%. So if you convert an old PC running on a 1GHZ CPU, you can speed it up by running it inside a virtual machine.

The other reason is legacy stuff. I mean all the apps that you accumulated on your last laptop. Moving to your new laptop, you really don't want to repeat all the setup stuff and tweaks you did earlier. Also most of these will probably never get used, and yet you don't want to delete them. So why not convert it to a virtual machine instead?

Disk space is cheap, so you can convert it and forget it, right? Even then, you may want to save some disk space. One nice feature in the VMware Workstation 6 version (not free) is its snapshot and linked clones. Linked clones share the same resources.
Say you start with a plain vanilla XP install (VM_XP).
Next I create a linked clone and install Firefox on it (VM_Firefox).
Then I create another linked clone and install Opera on it (VM_Opera).

The linked clone feature lets all three VMs share the same Windows files - without corrupting each other. So each will have its own linked disk to the original disk. In short this saves you a couple of gigabytes in redundant XP files.

The snapshot feature lets you take a offline snapshot of the PC and its hard disks. Useful when you are about to apply a hotpack and may want to go back :-).

Update: Fast forward to 2014:
The technology that started here, caused Amazon in 2008 to start creating massive server farms of virtual machines. It was the start of the "cloud computing" era. 

The next wave - is it Workflow?

I'm back after a long gap in posting. Mostly triggered by a migration to a new PC and followed by a lot of inertia. Now I'm looking at user centric applications designed using the principles of SOA and incontextual design. The basic idea is to create "meta-applications" or cross applications. These reside over conventional applications and work towards incorporating collaboration and convenience.
The collaboration recognizes that the process requires multiple people to provide inputs / approvals / notifications. So rather than using email, why not build it into the application?

This is where workflow across systems steps in. I've studied SAP's Guided Procedures and their CAF framework. Later this week, when looking at the Microsoft Workflow services, I was struck by how similar the two were!

Next step will be to try out a few scenarios with both technologies
1. Microsoft Workflow Foundation services - possible embedded in Microsoft Sharepoint.
2. SAP Guided Procedures in the SAP Portal.

Windows NT/XP security

An update: The emergency boot CD is an ISO that provides you with a convenient boot CD. The boot CD provides you with a boot into Linux with a simple way to reset your admin passwords. (site - ebcd.pcministry.com). This was useful to reset the passwords of NT images of defunct hardware that I was migrating to a virtual PC.

It is dangerous to allow a person to boot your PC with his floppy disk. This link discusses how you can get administrative access to a Windows XP machine by booting from a floppy disk.

The approach is rather ingenious. It edits the NT SAM database using an offline registry editor.
http://www.cgsecurity.org/nt.html. The utility chgntpw is excellent as an offline registry editing tool. A while back I was trying to find such a utility (see the BLOG on reinstalling Windows), but gave up. This seems promising...

This website also has approaches for admin access to Linux. A link from one of these pages discusses how to change Administrative passwords on Domain controllers - useful for administrators who install NT/XP half asleep ;-)

http://home.eunet.no/~pnordahl/ntpasswd/editor.html

RockXP3
While we are on security, be careful about RockXP3. One of its features is displaying passwords stored in Internet Explorer, Outlook Express and RAS. So if you save such passwords , they can be viewed with RockXP3.

Setup SMTP on a Mandrake Linux

Mandrake Linux uses Postfix rather than Sendmail.
When I try to telnet localhost smtp - it does not show any response from SMTP.

Look at /etc/syslog.conf
Find out which directory are the log files stored. In my case it was /var/log
Look at the errors in /var/log/mail/errors
Mar 6 08:29:37 linux3 postfix/smtpd[19852]: fatal: open database /var/lib/mailman/data/aliases.db: No such file or directory

. You specified two hash files in your postfix configuration and one of them doesn't exist. Try removing the /var/lib/mailman/data/aliases.db from main.cf and see if you can get basic mail delivery working first. Only after you are satisfied with how postfix is functioning, add in the mailman stuff.

After removing the mailman aliases.db, it still did not start.
The problem was that SASL parameters were turned on but SASL libraries were not installed.
Removed SASL parameters by going back to the initial version of the config files.

This page discusses setting up outbound SMTP authentication.

http://www.thecabal.org/~devin/postfix/smtp-auth.txt

Quick notes

Trying to see how Agile can be adapted into our SAP development group.

1. Installed TWIKI - very nice :-). Twiki is a WIKI site that allows users to modify the pages on the fly. It runs on Cygwin Perl and Apache. Found a very nice step-by-step install guide for Windows. Integrated this with Windows NTLM logins - again another step-by-step guide!
2. Installed a pilot site of CommunityServer. Installing CS is really easy despite the size of CommunityServer. BTW - the source code is now available. Guess I was on the bleeding edge when I needed it last week.
3. Emule - is a distributed file-sharing network. Tried it successfully today.
4. ABAP2HTML - Converts your ABAP program to a color coded HTML. You can change the VBA macro code for this program to adapt it to your needs.

Agile Programming and SAP

Martin Fowler's web articles got me rethinking about Agile Programming. He discusses various applicable methods in the article The New Methodology. The paper Is Design Dead raises a lot of questions about whether the waterfall model is really applicable for software development. He also has a nice paper on Agile with Offshore development.

But Ron Jeffries site has a compact (and quick) introduction to Extreme Programming (XP). Martin Fowler has a page full of links to XP resources on the web.

These articles are written for general software development, not really tailored for ERP software. I am interested in seeing how these ideas could be adapted to a SAP development group. Can these advantages be translated into reality with SAP development?

Basecamp and Ruby on Rails.

While surfing, I came across Rails - a framework to write web applications in Ruby.
The first application was Basecamp - a web based project management software.
I applied to Basecamp and created my project site. http://dmahajan.projectpath.com/login/

Ruby is creating a niche for itself. Rails (http://www.rubyonrails.org/) is touted as the framework to launch Ruby.

Another interesting link is Martin Fowler's site on Enterprise Software.
http://martinfowler.com/articles.html

CD Keys, offline registry editing - migrating to the new Windows XP install

The fresh install of Windows XP generated a few interesting questions.
How can you read / change the registry of another Windows instance?

1. The registry consists of 5 files in c:\winnt\system32\config. (see details here)
2. Copy these files over to your new instance. (If you have access to the drive, use the recovery console).
3. To read and manipulate these offline registry files, use regedit to load the 'hive' files. (see details here). You can even fix the offline registry and copy these files back to the original instance.

How can I recover my CD keys for Microsoft products (like Office XP, Visual Studio.NET, Office XP ....)?

There were a couple of programs that I found; RockXP , "Win KeyFinder 1.65" and "Office Keylight 1.00", "CD Key Finder: Magical JellyBean" . I liked RockXP. This thread discusses how these CD keys are stored in the registry.

Some other sites that I encountered along the way

PC Audit: http://www.belarc.com/free_download.html - shows all software and hardware in a nice local HTML page.

Windows IT Pro: http://www.jsifaq.com/

Registry tricks: http://www.mdgx.com/reg.htm

Free 60 day trial key for Office 2003: http://www.microsoft.com/office/trialkey

Creating a slipstream installer: http://www.jsifaq.com/SUBF/Tip2500/rh2529.htm

How to recover from a corrupted registry that prevents Windows XP from starting: http://support.microsoft.com/kb/307545

Finished the Nephrotic Syndrome site

It feels really nice when you finish a project and sit back to look at the results. I have just finished creating a website for parents with kids who have the Nephrotic Syndrome.
What is the Nephrotic Syndrome?

The site is a ASP.NET based website. It has a DotNetNuke portal and a CommunityServer forums application. Both applications are easy to setup and install. A bit of experience with MS SQL helps though.

To jumpstart the discussion forum, I decided to import messages from an existing discussion site @ siteboard.de. This took some time. I spent two days trying to see if someone had written a utility to import messages. Most hints on the net seemed to point to writing your own code - but I couldn't download the source code for CommunityServer from anywhere.

So I broke the problem into two parts.
1. Scanning the webpage of the old site - and building a list of threads with their messages.
2. Posting these threads and messages into CommunityServer.

Both parts were written in Perl 5.8.
The first part used regular expressions to parse the HTML files and store the appropriate data sections in a array of SB_Message objects.
The second part was simple in the end. There is a stored procedure cs_PostCreateData that creates a new post and updates all the tables. So I can create messages directly from Perl!
The challenges were
1. Learning to use the DBI module in Perl.
2. Parsing the German dates into a HTTP::Date format.

The next challenge was to take this utility and allow it to update the messages ongoing. This would let people continue posting in the old forum, but repost it in the new forum. This required some thinking - finally I decided to use the PropertyValues field to store the old message post code.

Anyway enough said.

Tracing Windows API calls

The program Traceplus (http://www.sstinc.com/products.html) lets you trace all Windows calls and see how they work!

Spyware infestation

Noticed that I have a Spyware infestation. It started with the installation of 180....
Then I noticed that this program installed a lot of other programs.
McAfee anti-virus scanner does not stop the installation of such spyware. That is SAD.

Time for help.

1. Download and install Ad-aware. It found 241 objects that it quarantined. Run a deep level scan to check the entire C drive. The definitions file was dated 16-Feb-2005. But it did not detect "Powerscan".
   
2. Go to a well known anti-spyware site. I chose the Computer Associates eTrust website to get more information about Powerscan,

I think that the best way is to restore your computer to a stable restore point that was created before the infestation.

Re-installing Windows XP

So I got a new harddisk!
I want to have a "clean" installation of Windows XP, since I've noticed that XP slows down after you install a lot of apps (typically that you never use).

A few learnings.

1. The "Recovery" XP disks that come with Dell do not have any support packs. It also does not have the drivers for the hardware. In short, it is just a plain vanilla XP installation CD. There is no "recovery" in this CD. What I had expected was that the CD would put the system back to its original factory-shipped state. No luck.
2. The following link discusses how to make your own custom recovery CDROM. It includes how you can combine SP2 with the installation files so that you don't have to do it seperately. http://www.pcmag.com/article2/0,1759,1750662,00.asp
3. The next problem is that I don't have the Microsoft CD Keys for Office and XP etc. XP was easy since the Dell had a sticker with the number. For Office - try the programs is program http://www.petri.co.il/quickly_retrieve_windows_cd_key.htm.
   Magical Jelly Bean Keyfinder: http://www.magicaljellybean.com/keyfinder.shtml
   CD Key Reader: http://www.skaro.net/cd-keyreader
4. I-Tunes - The XML database uses relative paths. So create the same directory structure in the I-Tunes music folder as in the old PC. I used sysinternals.com "Junction" utility to map folders to different drives.
5. I am still slowly installing the applications that I "really" need. Since I still need to use applications on the "Old Windows", I wanted to find out how to setup "dual-boot" in Windows XP. It is really easy. You start up with the install CD and select recovery mode. Use the commands "fixboot","fixmbr" and "bootcfg". See the installing and using Recovery console link from Microsoft. Bootcfg will regenerate boot.ini which has a list of all operating systems for dual-boot.
   One web resource: http://windows.about.com/cs/dualboot/
   Microsoft notes: Win2000 and WinXP, Multiboot with XP, Installing and using Recovery Console in XP.
6. You never know which applications are really important!

Web based forums and message boards

I was looking at how could I setup a web discussion forum.
I found that there were two easy choices

1. Use PHPBB2 (www.phpbb.com) - This is PHP based. I found a site that provides free PHPBB2 forums. www.galacnet.com. PHPBB2 looks really good and easy to setup. Within 30 minutes, I had setup a simple structure (http://phpbb.galacforums.com/Nephrotic_Syndrome/). Using the free server had the issue that half the screen was filled with advertisements. One downside was that it is not easy to install plugins and mods for Phpbb on this server.
2. Use Community Server::Forums (www.communityserver.org) - This is ASP.NET based. My hosting provider automatically installs this application. It is a nice Forums + BLOGs+ Photo Gallery application. On the plus side - it has built-in RSS feeds for all public forums. However in an anonymous post - it does not let the person type in a name. Slightly annoying because a lot of times, the anonymous user doesn't mind writing the name - just cannot register.

Final decision: Use Community Server::Forums since I can remove all advertisements on that Forum.

Migrating messages is the next challenge. The current message board is run on www.siteboard.de. So I needed a program to download all the messages, while preserving the message - response-reply relationships and dates etc. At this point I've written a short Perl program [download]that can download an entire forum from www.siteboard.de and save it as a Perl Data::Dumper file.

Setting up SSH access to a Mandrake Linux box

1. Generate your private key (use puttygen for example).
2. Save the private key in "openssh" format.
Save the private key (in putty ppk) format.
Save the public key.
3. Create a .ssh directory
cd ~
mkdir .ssh
chmod og-rw .ssh
4. ftp the private key (openssh) format to the .ssh directory
call it id_rsa if you used RSA
call it id_dsa if you used DSA.
5. Generate the authorized_keys file
cd .ssh
ssh-keygen -y -f id_rsa >> authorized_keys
============
Troubleshooting logins:
1. View /var/log/auth.log

SAP Netweaver and XI - Exchange infrastructure.

With its Netweaver application platform, SAP has focussed on a solid application development platform. It pushes SAP in two ways. It provides a solid base for their next generation applications. In addition, in itself, it provides a solid base for customers to write their enterprise extensions.

Typically, a company would have SAP and a number of other technologies. eg. Microsoft/Java for some web pages, Tibco/Microsoft/IBM for middleware, other vendors for a portal. Netweaver aims to grab this area of the market, and more vendors for data warehousing....

Netweaver is a marketing brand under which SAP is unifying its products. It is the equivalent of .NET for Microsoft. It provides an integration base for SAP's next generation technologies. By providing a platform in which you can develop web services, web pages, and .... - it is a great step ahead.

Life in General

It is easy to get caught up in day to day things and lose the big picture. The best time to get the bigger picture is after you have a nice time and are completely relaxed. Then I can ask myself, what is the point!
So what is the point of SAP, Microsoft and the others?
How does this compare with the people who started the field of computers?
Along the way, what have we forgotten in the rush to tackle day-to-day imperatives?

Can we make things simpler? Software seems to have got too complex.
Do we have an alternative for the GUI paradigm?
Can we get computers to give us the information that we want - when we want it - without us struggling to find it within the computer?
How can a computer help a disorganized person (like myself) organize themselves?
How can the computer reced to the background and just let us do our tasks?
Does a TV really make you think about it when you want to watch a program?