This is a test

Creating files for testing - part 2

2010-11-23T17:28:00.000-08:00

To test the behaviour of one of our applications, I needed to create a certain number of files of a certain file format. To do this, I wanted to copy the source file (a bitmap) multiple times and have the result go to the same directory. So the files did not overwrite each other, I needed to increment the file name (like: 0file.bmp, 1file.bmp, 2file.bmp).

With some help from one of the developers, I came up with a script that can also create a specified number of text files (file1.txt, file2.txt etc.)

The script takes two inputs:

number of files to create - mandatory
file to copy - optional

If a file to copy is not specified, the script will create as many text files as the mandatory number entered.

I've published the script on our team's repository on GitHub. You can find it here:

https://github.com/csmart/naa/blob/master/sysadmin/scripts/filecreator.sh

Creating files for testing - part 1

2010-11-23T13:51:00.000-08:00

Sometimes I need to create files of a certain size for testing purposes. Using the Linux 'dd' command, this is pretty easy.

There are lots of other posts around on how to do this. This is one of them on 'dd', including how to create the file of a certain size:

http://www.linuxprogrammingblog.com/dd-as-a-swiss-army-knife

I've noticed that some people suggest creating a file like this:

dd if=/dev/zero of=test-file bs=1M count=2

This produces a 2MB file.

and others, like this:

dd of=test-file bs=1M seek=2 count=0 (uses stdout as input)

dd if=/dev/zero of=test-file bs=1M seek=2 count=0 (uses /dev/zero as input)

Both of the above create a 2MB file.

Using 'seek' creates a sparse file.

Bulk edits of documents with Sed

2010-09-16T21:50:00.000-07:00

I recently had a need to edit a number of html documents to remove a section of html code. Doing a bit of a search, it seemed that the best way of doing this was via SED (Stream EDitor). Sed is used in Linux to filter or transform text.

The next thing I wanted to do was to find out how I could use sed to:

Find a string within an html file.
Delete the string.
Do this for more than one html.

The first thing I had to do was to create a bash script with a for loop that can cycle through the html files one by one.

The script I wrote was based on the following (I got this idea from here: http://gabeanderson.com/2008/02/01/unixlinux-find-replace-in-multiple-files/).

 for fl in *.php; do
mv $fl $fl.old
sed ’s/FINDSTRING/REPLACESTRING/g’ $fl.old > $fl
#rm -f $fl.old
done

In my case I want to delete the string and I'm dealing with .html files, so the first line of my script becomes:

for fl in *.html; do

The second line remains the same:

 #renames the .html files to .html.old

mv $fl $fl.old

The string I wanted to delete was multiple lines of html. I decided to specify the start and end of the string. Also, instead of replacing text, I just want to delete it, so I change the 'g' to a 'd'.

My third line becomes:

 #this deletes all the html between the <h4> tag and the close table cell tag  and redirects the output to .html

sed ’/<h4>/,/<\/td>/d’ $fl.old > $fl

I didn't want to keep the .old files so I uncommented the fourth line:

rm -f $fl.old #deletes the .html.old files

So, the script ends up looking like this:


#!/bin/bash

# script to replace multiple lines of text in multiple html files

for fl in *.html; do
    mv $fl $fl.old
    sed '/<h4>Are you/,/<\/td>/d' $fl.old > $fl
    rm -f $fl.old
done

To run it, I just had to make it executable and run it from within the directory containing the .html files.

Character Encoding

2010-08-26T19:26:00.000-07:00

I just learned something about character encoding. It has helped me understand some of the things I've come across when testing.

This is what I learned:

Without declaring a character encoding, your data may not be treated as you expect.
There are lots of different character encoding systems, such as ASCII, Unicode and ISO-8859.
The ASCII standard represents characters that correspond to those used by Western languages.
ASCII characters with decimal values of 32 - 127 are graphic characters. Those with values from 0 - 31 are non-printing control characters.
Unicode is a system to allocate a unique number (called a Code Point) to a specific character. The aim of Unicode is to create a single character set for the world's writing systems.
UTF-8 is a character encoding for Unicode. It is backwards compatible with the ASCII standard.
In UTF-8, code points from 0 to 127 (inclusive) are stored using 8 bits (1 byte). Code points from 0 to 127 correspond with ASCII character encoding. Code points above 127 are stored using 2 or more bytes and are used for other languages.
Some programs insert a Byte Order Mark before the first character to indicate the character encoding or endianness of the file.
Endianness specifies the ordering of multiple bytes of data in a computer system. Big endian - the most significant byte of any multibyte data field is stored at the lowest memory address, which is also the address of the larger field. Little endian - the least significant byte of any multibyte data field is stored at the lowest memory address, which is also the address of the larger field. (Brown, Christopher and Michael Barr. Introduction to Endianness)

Here's an example of endianness from http://www.allinterview.com/viewpost/162165.html

If the value is 0x0A0B0C0D then:

In case of Little Endian storage will be:
Address 1000 0D
Address 1001 0C
Address 1002 0B
Address 1003 0A

In case of Big Endian storage will be:
Address 1000 0A
Address 1001 0B
Address 1002 0C
Address 1003 0D

A couple of good articles explaining Unicode are:

http://www.joelonsoftware.com/printerFriendly/articles/Unicode.html

http://betterexplained.com/articles/unicode/

How Linux helps me become a better tester

2010-08-19T18:04:00.000-07:00

Since I've been using Linux, I have noticed how much it has improved my approach to learning and where I could make some improvements that can help me be a better tester.

I'm influenced by testers like James and Jon Bach, Michael Bolton and Pradeep Soundararajan and read a lot about testing but I occasionally find myself having trouble applying what I read about.

I just read a post by Jon Bach called "The testing moment". Some of the challenges in Jon's post reminded me about how I acted when I was trying to install Fedora recently. There were lots of things I needed to understand but I was resistant to taking the time to learn - I just wanted everything to work ASAP.

I was copying and pasting commands, jumping from one thing to another without taking the time to slow down and try to understand what I was supposed to do.

Some lessons I learned were:

Be patient - be confident that there is a solution (or at least a workaround) and in many cases, someone has probably solved it before.
Take my time - don't rush or you will overlook things.
Be methodical - not scattered. Look at the facts - what do you know, what don't you know. Know that there are things that you don't know you don't know. Don't go installing things or executing commands you don't understand - you forget you've done it or you do it out of sequence or forget something entirely.
Work at it - don't be lazy and just expect to be given the answer. You will learn more and find the next problem easier to solve.
Break down the problem - that way you won't be overwhelmed by its scale.
The whole installation process reminded me about something that James Bach keeps talking about (commonly in reference to Gerry Weinberg) - looking at things as a system. What better way to do this than by installing an operating system. That's one of the best thing about Linux - you have the power to configure it to a much greater extent than Windows. But it also means you need to understand how everything fits together and not just go blindly executing commands.

When I realised that I needed to slow down and understand more about what I was doing, I began to make some progress. I decided to apply some of the lessons I learned. One of the other challenges I had during the installation process was migrating my virtual machines from my old Ubuntu machine to my new Fedora machine (I outline the process in an earlier post). While I didn't do it without some help, it taught me some more valuable lessons.

This is what happened:

I copied the relevant virtual machines to a location on my Fedora machine.
I then mounted the virtual machines to my home directory, to replicate the setup I had in Ubuntu.
I started VirtualBox to see if it worked. There was nothing there. I realised that I needed to change the ownership of the relevant directories so they were owned by me.
I changed the ownership and started VirtualBox again. Now I could see my virtual machines but I was getting an error about my snapshots not being available. Why could I see my machines but not access the snapshots?
I Googled the error, looking for an explanation but the error message (NS_ERROR_FAILURE (0x80004005)) was generic and could have meant anything. One clue, which I ingnored in my haste, was something about not being able to find the .vdi file for the snapshot.
In my Google search I also ran across a post about checking the VirtualBox.xml file.
I then tried looking in the VirtualBox.xml file for some clues. Some time ago I had a corrupted VirtualBox.xml file, which caused me some problems, so I thought that may have been the cause of my problem now. I assumed that the UUIDs for the snapshots were somehow incorrect in the VirtualBox.xml file or the XML file for each snapshot. I started comparing the IDs between the snapshot XML and the VirtualBox.xml. I couldn't see
why there was an issue.
It wasn't until I asked for help that the problem was solved. I had overlooked the path details in the VirtualBox.xml file, pointing to each snapshot. The VirtualBox.xml file was copied from my old Ubuntu machine. In the file, it references the path to each snapshot. Under Ubuntu my user directory was /home/allan but when I installed Fedora, I decided to use /home/al. I didn't realise that the VirtualBox.xml file states the path to the virtual machine snapshots and that by using a new username, VirtualBox couldn't see any of my snapshots (it was looking for them under /home/allan/.Virtualbox/Machines, not /home/al/.Virtualbox/Machines). It was such an easy solution.

I think this is where I went wrong:

I was looking in the right place (the VirtualBox.xml file), but my assuption about a corrupt XML file mad me blind to anything else. I stuck with one cause and didn't examine others. I was being methodical but with blinkers on.
The cause of my problem was something that seemed totally unrelated. It started many steps previously - when I decided to select the username of 'al' vs. 'allan', when initially installing Fedora.
I was looking for an exact solution to my problem with a Google search but did not accept the possibility that there was a combination of factors that made my problem unique (or at least undocumented on the internet).
My brain wanted me to come to make a quick decision, so I latched on to the first one that seemed right and went with it, closing myself off from other possibilities.

It's hard to see the big picture when you are in the middle of it all, especially when you are doing something new or unfamiliar. Sometimes you need to step back from the problem and try to see it with new eyes. I think that is one of the hardest things to do. The best thing about these sort of experiences is that I can use them as opportunities to improve my ability to test.

Setting up a Network Printer in Fedora 13

2010-08-12T23:50:00.000-07:00

I just set up printing in Fedora 13.

This is how I did it:

I had to get the drivers for my network printer, so I installed foomatic:

sudo yum install foomatic

Select Administration - Printing.
Select Server - New Printer, and authenticate.
Select Network Printer - Find Network Printer.
In the Host field, enter the IP address of the printer and click Find.
Under Connections, select LPD/LPR queue 'ps' and click Forward.

The system will search for drivers.

Review install options and click Forward.
Enter a printer name and description and click Apply.

For a nicer description and screenshots, see Setting up a network printer in Fedora 13.

For info on troubleshooting printing, see Fedora Printing/Debugging.

Migrating VirtualBox from Ubuntu to Fedora

2010-08-12T17:48:00.000-07:00

One of the things I had to do when installing Fedora was to migrate my Windows virtual machines. My goal was to move the machines to a newly created LVM volume on my Fedora machine and then access the virtual machines from my /home directory.
It wasn't an easy task for me but I learned a lot of new things about Linux in the process (and testing - but that's worthy of another post).

The whole process involved the following steps:

When installing Fedora, I created an LVM (Logical Volume Manager) volume for my virtual machines, called "virtual".

I added the Fedora VirtualBox repo file to /etc/yum.repos.d/.

I use NTLMAPS, so I had to modify the virtualbox.repo file to use the NTLMAPs proxy

sudo vim /etc/yum.repos.d/virtualbox.repo

Insert the line: proxy=http://localhost:5865

I installed VirtualBox development tools on my Fedora machine.


sudo yum groupinstall "Development Tools"

I installed VirtualBox and Dynamic Kernel Module Support (DKMS).


sudo yum install VirtualBox-3.2 dkms

I mounted my old virtual machines drive and used rsync to copy the machines from my old "virtual" partition to the new LVM volume:


sudo rsync -Pa /media/virtual/VirtualBox /virtual/

I created a mount point for the virtual machines on my home directory:


mkdir /home/al/.VirtualBox

I mounted the "virtual" LVM volume to the mount point on my home directory:


sudo mount -o bind /virtual/VirtualBox/ /home/al/.VirtualBox

I edited my fstab file to to make sure virtual/VirtualBox is mounted to my mount point (/home/al/.Virtualbox) each time I reboot.


sudo vim /etc/fstab

Insert the following line:

/virtual/VirtualBox /home/al/.VirtualBox auto bind,defaults 0 0

I changed ownership of the .Virtual directory and its contents to me:

sudo chown al:al .VirtualBox -Rf

I edited the VirtualBox.xml file to ensure that I referenced the correct path to my snapshots. This is probably one of the easiest steps but it caused me the most problems as I didn't know I had to do it in the first place. The reason I had to do this is was because my user directory was /home/allan on my Ubuntu machine but on my Fedora machine, I decided to use /home/al. I didn't realise that the VirtualBox.xml file states the path to the virtual machine snapshots and that by using a new username, VirtualBox couldn't see any of my snapshots (it was looking for them under /home/allan/.Virtualbox/Machines, not /home/al/.Virtualbox/Machines).

Installing Fedora 13

2010-08-10T20:35:00.000-07:00

Having used Ubuntu for 12 months, I recently installed Fedora 13. It wasn't easy, even though I had the help of the Linux experts I work with.

This is a list of some of the things I learned about Linux while going through the installation process.

Disclaimer: Most of the Linux stuff on this blog is stuff I want to remember. To any experts out there - go easy on me - I am still drinking from the noob cup.

If you want to see the services you have running on your machine, netstat is your friend.

For TCP services:

netstat -lt

or for TCP services, including port number:

netstat -ltn

Another way to see if a service is running on your machine is to telnet to it:

telnet <server e.g. localhost> <port>

If you are behind a proxy server, you'll need to use NTLMAPS (NTLM Authorisation Proxy Server). If you want ntlmaps to automatically run on startup, add the following line to your /etc/rc.local file:

ntlmaps /etc/ntlmaps/server.cfg &

The '&' will ensure that it runs in the background and doesn't stall Fedora on boot up.

Connecting NTFS-formatted drives? Make sure you install the ntfs package.

sudo yum install ntfs-3g

Can't find where your repositories are listed? Try:

/etc/yum.repos.d

Note: if you have just updated your repository list, make sure you run the following so the operating system knows about the change:

sudo yum update

If you want to test websites on your local machine, follow these steps (I got some of this info from here):

Install Apache:

sudo yum install httpd

Depending on the nature of your website, you may need to install other packages. In my case, it was PHP and PHP-Markdown:

sudo yum install php php-markdown

Start Apache:

sudo /sbin/service httpd start

Make sure Apache starts whenever you start Fedora:

su -

/sbin/chkconfig --levels 235 httpd on

Note: 'Levels' refers to run levels. 2 - single-user, 2 - .

As root, make sure you place the website pages in the following location (Fedora is different than Ubuntu in this regard):

sudo cp <origin> /var/www/html/ -Rf

I found more Fedora basic info on Unofficial Fedora FAQ.